All Things Intelligent @allthingsintelligent - Tumblr Blog

Check out the code and start using intelligent search, recommendations, classifiers and clustering algorithms.

You can find the Yooreeka 2.0 API (Javadoc) here and you can also visit us at our Google+ home. The official blog for yooreeka is http://blog.yooreeka.com Lastly, Yooreeka 2.0 is licensed under the Apache License 2.0. The library is written 100% in the Java language.

#yooreeka #data mining #machine learning

The Sammon Mapping: Nonlinear Projections for High-Dimension Datasets

// <![CDATA[ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-42543297-1', 'tumblr.com'); ga('send', 'pageview'); // ]]>

In the context of a data mining project, or as part of constructing a machine learning algorithm, we are given data sets whose records have a lot of attributes; think of a table with many columns, if you are more familiar with modeling your data through a relational database. This situation is not problematic for a wide range of data mining and machine learning algorithms, however, at times it can be overwhelming for the purpose of analysis. In particular, these data can quickly saturate are ability to visualize information, since the human perception is limited to a small number of dimensions. Thus, herein, the words "many" and "lots" are with relation to the number three (3).

Perhaps, one may think that the simplest technique for dimensionality reduction is a straightforward linear projection, such as described in Principal Component Analysis or PCA.However, the objective of PCA is very different from the needs of the analysis that we discuss here. The objective of the PCA is the compression of the information in the dataset in such a way that the total information can be described with fewer data points (coordinates). In mathematical terms, PCA creates a sequence of dimensions such that the first dimension accounts for as much of the variability of the data in the original dataset and each subsequent dimension attempts to do the same for the remaining of the information, under the constraint that all dimensions are mutually perpendicular to each other (i.e. linearly independent).

While PCA is very useful in a number of cases, it makes no effort to preserve any geometric structures that might be present in the original data set. For certain purposes of visualization and data analysis, the preservation of the high-dimensional structures is more valuable and very revealing of the information that the data set contains.

One algorithm, that aims at achieving exactly that, is the so-called Sammon Mapping, named after John Sammon, who initially proposed it in 1969 (see "A nonlinear mapping for data structure analysis" by Sammon JW in IEEE Transactions on Computers 18, p.401–409). You can find an implementation of that algorithm in Yooreeka, just click here.

The results can be visualized by using the class ScatterGui.java.

#yooreeka

Yooreeka Roadmap

I am working on the following changes for Yooreeka:

The BeanShell scripting library has served well our project. We did investigate replacing it but we determined that it is better to continue using it. An investigation of the Spring Shell project was conducted and the benefits were not worth the effort. Another option, as a more powerful execution environment, would be to load the yooreeka library in the execution shell of an interactive (JVM) language such as Groovy or Scala. However, this is trivial if you are using Groovy or Scala, so there will be no changes to the Yooreeka project in that regard.

I am preparing a document that explicitly states the other open source projects that are used by Yooreeka, and their associated licenses. All projects that do not have a sufficiently permissive OS license will be replaced by others who do or they will be cleanly decoupled from the Yooreeka core JAR. Starting with Yooreeka 2.0, we adopted the Apache 2.0 license with the intention that you can use Yooreeka within your projects whether those are work related (proprietary) or not.

Lastly, I will introduce the Processing language for the purpose of creating powerful visualizations. If you are not familiar with Processing, I highly recommend that you take a look. Briefly, the Processing language offers the following:

Interactive programs with 2D, 3D or PDF output

OpenGL integration for accelerated 3D

Over 100 libraries that extend the core software beyond simple graphs and visualization

I think, and hope, that the above efforts will make Yooreeka more attractive and useful to researchers in Academia and industry alike.

If you have any specific requests for the Yooreeka library, please, create a ticket on the project's website or send me an email at yooreeka AT marmanis DOT com.

#yooreeka

Cloudify: A very promising (OS) DevOps project

If you are building applications on the cloud and you are looking for heavily automated, programmable, ways to provision, configure, and orchestrate these applications, you should have a look at Cloudify (http://www.cloudifysource.org/).

Cloudify is an enterprise-class (open source) PaaS stack that sits between your application and your chosen cloud. I am not aware of any other solution in this space that is as comprehensive and industrious as Cloudify -- VMware's suite of products might be better under several definitions of optimality but it is neither free nor Open Source. The company behind this project is called Gigaspaces -- Gigaspaces was a proprietary Javaspaces implementation, just in case you recall Javaspaces from the Jini days. The man behind the company is Nati Shalom, a bright and very experienced technologist who is the founder and CTO of Gigaspaces.

Moving beyond the fact that there is a commercial entity and tons of experience behind the project, the next thing to note is the clear and nearly complete documentation of the project. Quick installation guides that get you up and running within minutes, product overview pages, detailed release notes, youtube videos, and so on. There isn't a book on the platform yet but it shouldn't be long before that happens as well. There are also many blog entries, JIRA integration, public events, and community forums that support Cloudify.

In contrast to other DevOps tools that simply assist in installing your applications, Cloudify provides a mechanism to manage the entire lifecycle of your application. The breadth of integration points, the ability to mix-and-match technologies (e.g. you can use existing Chef recipes), the cloud agnosticism of the platform, and the knowledge that it is powered by a mission-critical high-performance elastic application platform GigaSpace XAP, make Cloudify a truly unique and almost irresistable choice.

I expect that Cloudify will become the de facto enterprise-class PaaS stack in the next few years. However, regardless of the probability of a ubiquitous adoption and PaaS domination, those who will adopt and use Cloudify today will have a lot to gain with minimum investment.

#cloud #devops #PaaS #gigaspaces

Big Data: Project Notes

For many years, during private conversations, I have tried to explain to people that the true value of “big data” requires much more than processing a large volume of data. Technical people are told by the marketing folks that “big data” is about “lots of data”, however, technical people should know better!

We (i.e. “techies”) should not be caught in the marketing hype du jour, unlike our marketing brethren we do not have the luxury of simply “hand waving” and “make believe”. We require a clear and thorough understanding of the things that we bring to life in the cyberspace. Computing systems do not cope well with incoherence or inconsistencies.

Many people, even established marketing and consulting firms, will not be able to pinpoint exactly what the term means. If you talk to a technical (or semi-technical) person then the term “big data” will immediately trigger a discussion around a number of technologies such as Hadoop or MongoDB, or even about cloud computing (yet another term in need of clarification!). If you talk to a business or marketing person, they will tell you how important it (”big data”) is and what the estimated market potential is by sector. They will tell you a lot of things but they will not tell you what it is and how to do it!

That is not an uncommon phenomenon, of course. It is typical during paradigm shifts where the new paradigm’s impact potential can be as profound as it is elusive. In this post I will try to define, in the simplest and briefest way that is possible, the real essence of big data.

In a typical organization, data exist in several forms, in a plethora of systems, and with a variety of semantic associations. When marketing and other business people speak of Big Data, you can think of the term as a program that brings all these data together in a way that their totality can be leveraged for analytic purposes. Hence, from a technology perspective, a big data initiative will require solutions to a number of data and analysis problems such as:

The definition of the various data sources and their categorization (e.g. internal vs. external; static vs. dynamic; and so on)

The integration of the data onto a single platform in a way that can be repeated and extended

The methodical cleansing and enrichment of the data; if you put “garbage” in you will get “garbage” out!

The construction of a system that can build domain models for (ad hoc) analysis based on the data in the background

The provision of a high-performance, scalable, and economic platform for the required computations

The availability of analytical software tools that can integrate the domain models and solve business problems in a way that is consistent, responsive, and intuitive to the end users

The first two categories of problems belong in the general area of enterprise integration, which is fairly mature in itself, but the size of the data and other (application specific) requirements may require special variations of the well known patterns or technology-specific adjustments.

The third category of problems to be solved can be described as a series of pre-processing steps that are necessary for ensuring consistency, repeatability, reliability, robustness, and maximum ROI (downstream).

The fourth category of problems relate to the ability to look at the data holistically from many different angles and in different representations, where each configuration emphasizes a specific business function or business area.

The fifth category is what we could call the “plumbing”. If you are going to process 100 Tb of data then you need to ensure that you have the proper computing infrastructure to do so. The good news here is that there is a plethora of vendors out there that are willing and able to help you build such an infrastructure. Some have technology solutions that are based solely on proprietary products, some are committed to Open Source solutions, and some use hybrid solutions based on products from both ends of the spectrum. The same can be said about the computational infrastructure, it could be entirely hosted on the “cloud” (e.g. AWS), it could be hosted entirely in your data centers, or it could be a hybrid of the two.

The sixth category is the raison d’etre of the first five! The possibilities here are limited by the imagination of your technology and business leaders. The successful solution of problems in this category depends vitally on the quality of the solutions that was achieved in the earlier stages of your big data initiative.

It is not hard to infer, from the above, that a big data architecture is needed before you even begin to entertain specific technology providers. It should also be clear that talking about Hadoop, or MongoDB or anything else for that matter, is only a small part of a very big picture. Tackling the big data challenge entails much more than data!

#big data

Big Data for Machine Learning

There is a large number of publicly available data sets on the Internet. Whether you want to learn data mining or practice with machine learning algorithms, these data are worth investigating. Some of them are real-world data both in terms of size and quality — it is actually the lack of quality that makes them “real-world” data.

In this post, I will report some data sets that I find interesting. Please, observe and respect the licenses for these data as stated in their respective websites.

The Ensembl project

The Ensembl project produces genome databases for vertebrates and other eukaryotic species, and makes this information freely available online. In particular, the human (Homo sapiens) site provides a data set based on the February 2009 Homo sapiens high coverage assembly GRCh37 from the Genome Reference Consortium. This assembly is used by UCSC to create their hg19 database. The data set consists of gene models built from the genewise alignments of the human proteome as well as from alignments of human cDNAs using the cDNA2genome model of exonerate.

You can download the data in various ways, and you can even download the entire database as a MySQL dump. You can find these data on AWS as well; if you use the AWS infrastructure that should save you some bandwidth!

The Google Books N-gram Viewer Data

If you don’t know what the Google books n-gram viewer is, you can have a look here. It basically shows you a graph that displays how your input phrases have occurred in a corpus of books over a specified period of time. There are 11 corpora of books that cover a variety of languages, including Chinese.

Each of the numbered links will directly download a fragment of the given corpus. You can obtain the 1-gram (i.e., individual words) counts, the 2-gram counts, and so on, up to the 5-gram counts. In addition, for each corpus, you can get the file total counts, which records the total number of n-grams contained in the books that make up the corpus. This file is useful to compute the relative frequencies of n-grams. Obviously, there are more fragments for 2-grams than there are for 1-grams, more for 3-grams than for 2-grams, and so on.

The corpus construction is described in the Science article written by Jean-Baptiste Michel et al. It should be noted that only the n-grams that appeared over 40 times in the whole corpus are reported. Therefore, the sum of the 1-gram occurrences, in any given corpus, will be smaller than the number given in the total counts file.

Government Sponsored Data

The United States census bureau offers a wide variety of data sets that span many decades. While you are perusing that site, make sure that you check out the DataFerret tool. You can also find US census data on AWS, for example, the data from the 2000 US Census can be found here.

You can find data from the Japan population census in the following link: http://www.stat.go.jp/english/data/kokusei/2010/summary.htm

You can obtain international data on census by using the following US census bureau website: http://www.census.gov/population/international/data/idb/informationGateway.php

The UK’s government hosts a web site (visit http://data.gov.uk/) that offers over 5,400 data sets, from all central government departments and a number of other public sector bodies and local authorities.

There is also a wealth of data offered by the World Bank’s Open Data initiative in the following website: http://data.worldbank.org/data-catalog

Freebase

Freebase is an open, graph database with more than 23 million entities; an entity is a single person, place, or thing. Full data dumps of every fact and assertion in Freebase are available in a variety of formats and are updated every week. The various data dumps can be found here: http://download.freebase.com/datadumps/

Public Data Sets on AWS

Public Data Sets on AWS provides a centralized repository of public data sets that can be seamlessly integrated into AWS cloud-based applications. AWS is hosting the public data sets at no charge for the community, and like all AWS services, users pay only for the compute and storage they use for their own applications. The following two data sets stand out:

1000 Genomes Project The 1000 Genomes Project, initiated in 2008, is an international public-private consortium that aims to build the most detailed map of human genetic variation available.Aside from the scientific importance of this data set, its sheer size (~200TB) makes it interesting for machine learning at scale. Common Crawl Corpus A corpus of web crawl data composed of 5 billion web pages. This data set is freely available on Amazon S3 and formatted in the ARC (.arc) file format.Although smaller than the 1000 Genomes project data set, these web crawl data are approximately 60TB in size.

The complete list of data that are offered on AWS can be found in the following page: http://aws.amazon.com/datasets?_encoding=UTF8

Banking Statistics

The Bank for International Settlements offers a consolidated banking statistics website: athttp://www.bis.org/statistics/consstats.htm

The mission of the Bank for International Settlements (BIS) is to serve central banks in their pursuit of monetary and financial stability, to foster international cooperation in those areas and to act as a bank for central banks. Currently, central banks in 30 countries report their aggregate national consolidated data to the BIS, which uses them as the basis for calculating and publishing global data. The data are published as part of the BIS Quarterly Review. Preliminary data, including a commentary, are released a few weeks before the publication of the Quarterly Review. Users should be aware of the limitations of the preliminary data.

#big data #data mining #aws datasets #machine learning

Google Compute Engine: An overview

(Originally posted on July 4, 2012)

Google Compute Engine (hereafter GCE) is the Infrastructure as a Service (IaaS) offering by Google, as such, it compares, and competes, directly with the IaaS offered by Amazon (AWS). Imitation is the sincerest form of flattery, as the saying goes, but I am fairly confident that comforting thoughts of adulation is not what preoccupies the minds of Amazon executives, right now.

One can only assume that Google is trying to enter a field that Amazon dominates because there is money in it. In order to take a fair share of the market, they must offer a better product than AWS (or the other incumbents) and do so with serious incentives. So, can they do that?

We do not have the answer to that question, of course. What is publicly available today is fairly limited for use in production. However, given the engineering potency of Google, it is worth having an early and careful look at GCE.

The GCE offering brakes down into the following broad categories:

Computing

Network

Storage

Tools

The first three are the fundamental blocks of any data center. The last is crucial for gaining adoption because even if computing, networking, and storage are less expensive or work better on GCE than, say, on AWS, the ability to setup an environment, operate it, and integrate it with some in-house infrastructure is crucial. Moreover, just offering these capabilities will not be enough for Google, the tools must be user friendly and integrate well with existing development environments as well as data center environments (e.g. vSphere from VMware).

It remains to be seen whether the speed, scale, and global footprint that Google has achieved for its own operations can scale when other customers jump on the same data centers. For example, right now, it seems that GCE offers only two “zones” (us-central1-a and us-east1-a) and only 8 types of instances; the former limitation renders GCE more “local” than “global” and the latter limitation indicates fewer options for setting up exactly the cloud infrastructure that you need. Compare that situation with Amazon’s eight regions (each of which has two or more availability zones): US East (Northern Virginia), US West (Oregon), US West (Northern California), EU (Ireland), Asia Pacific (Singapore), Asia Pacific (Tokyo), and South America (Sao Paulo). Amazon has proved that they can do that and do it well, Google might be ranked first based on traffic but most of that traffic is on the search site, which is fairly simplistic compared to, say, the Amazon retail site. Lastly, it should be noted that, so far, the overall Google infrastructure has been running only Google products.

The equivalent of the AWS web console is the Google APIs Console, which is the “one-stop shop” for creating and managing multiple API projects — a project is the concept used by Google for grouping GCE resources, defining team membership to these resources, defining group ownerships, and billing for the use of these resources.

There is also a command line utility (the gcutil tool) that can run on Linux distros and Mac OS X, which have Python 2.6.x or 2.7.x installed — currently, the gcutil tool does not support Python 3.x. For MS Windows, the gcutil tool is supported if you install Cygwin.

Authentication and authorization for GCE is based on the OAuth 2.0 protocol. The SSH connection to a running instance seems to be established by the following command:

ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -i /Users/<someuser>/.ssh/google_compute_engine -A -p 22 <someuser>@<IP>

At this time, it is not clear neither how the SSH key pairs are created and maintained nor what are the accepted formats or key lengths — for example, AWS does not accept DSA keys and the allowed key lengths are 1024, 2048, and 4096.

The default instances seem to be Ubuntu based, and the name of the host is set by the name of the instance via the Dynamic Host Configuration Protocol (DHCP). The API is JSON over HTTP and it supports three main channels, the gcutil tool (and its siblings, I suppose), the UI console, and code (through open source libraries and third parties).

From a hardware perspective, the infrastructure seems to be built on Intel Sandy Bridge. The CPU utilization is based on the concept of a GCEU (Google Compute Engine Unit), which is a unit of CPU capacity that describes the compute power of GCE instance types. The minimum power of one logical core (a hardware hyper-thread) on the GCE Sandy Bridge platform is considered equivalent to 2.75 GCEU. The number of virtual cores can be 1, 2, 4, or 8 depending on the machine (instance) type. The number of GCEUs is proportional (x 2.75) to these numbers, of course. You can have ephemeral as well as persistent storage; the latter ranges between 128 GB and 1024 GB. The amount of memory (RAM) per virtual core is 3.75 GB; a table with all the details about the various GCE machine types can be found here.

Google’s virtualization technology is based on Kernel Virtual Machines (KVM) — a full virtualization solution for Linux on x86 hardware with virtualization extensions (Intel VT or AMD-V) — and Linux cgroups, a Linux kernel feature to limit, account and isolate resource usage (CPU, memory, disk I/O, etc.) of process groups.

Each project is assigned its own Private Virtual Network (PVN — a concept that probably corresponds to the AWS VPC) — and that virtual network seamlessly extends across geographic locations (regions). From a security perspective, it is important to note that the traffic within your virtual network never goes through the public Internet infrastructure — that makes sense since packets with a private destination address are ignored by all public routers. The private IPv4 space is based on RFC 1918, so the range of IPs ought to be 172.16.0.0/12. In terms of external networking capabilities, GCE offers external IPs (types supported are “reserved”, “ephemeral”, “none”) with dynamic attachment to an instance, 1-to-1 NAT, and (of course) built-in firewalls. Some limitations include the blocking of outgoing SMTP traffic (presumably to avoid abuse) and the restriction to UDP, TCP, and ICMP for network traffic between your instances and the (public) Internet.

The above is a summary of the information that I have obtained from publicly available sites (primarily Google IO 2012 content). I have not covered storage sufficiently here but I have requested access to the service for a fair and balanced review. If and when I get it, I intend to update this post.

#cloud #Google #amazon

Trending Blogs

Recently Viewed Blogs

All Things Intelligent