DataWeave

DataWeave provides actionable data to businesses by creating curated datasets on the Web and letting them access this data through data APIs, dashboards, visualizations, reports, and alerts.

[dataweave]

Mining Twitter: Understanding and Analyzing Social Reactions to Products and Brands

[This post was written by Dipanjan. Dipanjan works in the Engineering Team with Mandar, addressing some of the problems related to Data Semantics. He loves watching English Sitcoms in his spare time. This blog post is cross posted on the PriceWeave blog as well.] This is the second post in our series of blog posts which we shall be presenting regarding social media analysis. We have already talked about Twitter Mining in depth earlier and also how to analyze social trends in general and gather insights from YouTube. If you are more interested in developing a quick sentiment analysis app, you can check our short tutorial on that as well. Our flagship product, PriceWeave, is all about delivering real time actionable insights at scale. PriceWeave helps Retailers and Brands take decisions on product pricing, promotions, and assortments on a day to day basis. One of the areas we focus on is “Social Intelligence”, where we measure our customers' social presence in terms of their reach and engagement on different social channels. Social Intelligence also helps in discovering brands and products trending on social media. Today, I will be talking about how we can get data from Twitter in real-time and perform some interesting analytics on top of that to understand social reactions to trending brands and products. In our last post, we had used Twitter’s Search API for getting a selective set of tweets and performed some analytics on that. But today, we will be using Twitter’s Streaming API, to access data feeds in real time. A couple of differences with regards to the two APIs are as follows. The Search API is primarily a REST API which can be used to query for “historical data”. However, the Streaming API gives us access to Twitter’s global stream of tweets data. Moreover, it lets you acquire much larger volumes of data with keyword filters in real-time compared to normal search.

Installing Dependencies

I will be using Python for my analysis as usual, so you can install it if you don’t have it already. You can use another language of your choice, but remember to use the relevant libraries of that language. To get started, install the following packages, if you dont have them already. We use simplejson for JSON data processing at DataWeave, but you are most welcome to use the stock json library.

[root@dip]# pip install twitter [root@dip]# pip install simplejson [root@dip]# pip install prettytable [root@dip]# pip install matplotlib [root@dip]# pip install nltk

Acquiring Data

We will use the Twitter Streaming API and the equivalent python wrapper to get the required tweets. Since we will be looking to get a large number of tweets in real time, there is the question of where should we store the data and what data model should be used. In general, when building a robust API or application over Twitter data, MongoDB being a schemaless document-oriented database, is a good choice. It also supports expressive queries with indexing, filtering and aggregations. However, since we are going to analyze a relatively small sample of data using pandas, we shall be storing them in flat files. Note: Should you prefer to sink the data to MongoDB, the mongoexport command line tool can be used to export it to a newline delimited format that is exactly the same as what we will be writing to a file. The following code snippet shows you how to create a connection to Twitter’s Streaming API and filter for tweets containing a specific keyword. For simplicity, each tweet is saved in a newline delimited file as a JSON document. Since we will be dealing with products and brands, I have queried on two trending products and brands respectively. They are, ‘Sony’ and ‘Microsoft’ with regards to brands and ‘iPhone 6’ and ‘Galaxy S5’ with regards to products. You can write the code snippet as a function for ease of use and call it for specific queries to do a comparative study.

import io import simplejson as json import twitter # Go to https://apps.twitter.com/ to create an app and get values for these credentials CONSUMER_KEY = '' CONSUMER_SECRET = '' OAUTH_TOKEN = '' OAUTH_TOKEN_SECRET = '' # Authenticate with OAuth auth = twitter.oauth.OAuth(OAUTH_TOKEN, OAUTH_TOKEN_SECRET, CONSUMER_KEY, CONSUMER_SECRET) # Create a connection to the Twitter Streaming API twitter_stream = twitter.TwitterStream(auth=auth) QUERY = 'microsoft' OUT_FILE = 'tweets_'+QUERY+'.json' print 'Filtering the public timeline for "{0}"'.format(QUERY) stream = twitter_stream.statuses.filter(track=QUERY) # Write one tweet per line as a JSON document. with io.open(OUT_FILE, 'a', encoding='utf-8',buffering=1) as f: for tweet in stream: f.write(unicode(u'{0}\n'.format(json.dumps(tweet, ensure_ascii=False)))) print tweet['text']

Let the data stream for a significant period of time so that you can capture a sizeable sample of tweets.

Analyses and Visualizations

Now that you have amassed a collection of tweets from the API in a newline delimited format, let's start with the analyses. One of the easiest ways to load the data into pandas is to build a valid JSON array of the tweets. This can be accomplished using the following code segment.

import pandas as pd DATA_FILES = ['tweets_microsoft.json', 'tweets_sony.json', 'tweets_galaxys5.json', 'tweets_iphone6.json'] data_frames = dict() for data_file in DATA_FILES: data = "[{0}]".format(",".join([line for line in open(data_file).readlines()])) data_frames[data_file.split('_')[1].split('.')[0]] = pd.read_json(data, orient='records') # All the values should be of data frame type print {k:type(v) for k,v in data_frames.items()} # to see an individual sample data frame print data_frames['sony']

Note: With pandas, you will need to have an amount of working memory proportional to the amount of data that you’re analyzing. Once you run this, you should get a dictionary containing 4 data frames. The output I obtained is shown in the snapshot below.

Note: Per the Streaming API guidelines, Twitter will only provide up to 1% of the total volume of real time tweets, and anything beyond that is filtered out with each “limit notice”. The next snippet shows how to remove the “limit notice” column if you encounter it.

# Capture the limit notices by indexing into the data frame for non-null field containing "limit" # df is a data frame here limit_notices = df[pd.notnull(df.limit)] # Remove the limit notice column from the DataFrame entirely df = df[pd.notnull(df['id'])] print "Number of total tweets that were rate-limited", sum([ln['track'] for ln in limit_notices.limit]) print "Total number of limit notices", len(limit_notices)

Time-based Analysis

Each tweet we captured had a specific time when it was created. To analyze the time period when we captured these tweets, let’s create a time-based index on the created_at field of each tweet so that we can perform a time-based analysis to see at what times do people post most frequently about our query terms.

from prettytable import PrettyTable pt = PrettyTable() pt = PrettyTable(['Brand \ Product', 'First tweet timestamp (UTC)', 'Last tweet timestamp (UTC)']) for key in sorted(data_frames.keys()): pt.add_row([key, data_frames[key]['created_at'][0], data_frames[key]['created_at'][-1]]) print pt

The output I obtained is shown in the snapshot below.

I had started capturing the Twitter stream at around 7 pm on the 6th of December and stopped it at around 11:45 am on the 7th of December. So the results seem consistent based on that. With a time-based index now in place, we can trivially do some useful things like calculate the boundaries, compute histograms and so on. Operations such as grouping by a time unit are also easy to accomplish and seem a logical next step. The following code snippet illustrates how to group by the “hour” of our data frame, which is exposed as a datetime.datetime timestamp since we now have a time-based index in place. We print an hourly distribution of tweets also just to see which brand/product was most talked about on Twitter during that time period.

brands = ['sony', 'microsoft'] products = ['iphone6', 'galaxys5'] brands_grouped_time = [data_frames[key].groupby(lambda x: x.hour) for key in brands] products_grouped_time = {key:data_frames[key].groupby(lambda x: x.hour) for key in products} # dividing by 1000 in the distribution because people talked more about brands during this time period for brand, brand_grouped_time in brands_grouped_time.items(): print "\nNumber of relevant tweets by the hour (UTC) for", brand pt = PrettyTable(['Hour', 'Total Tweets', 'Tweet Distribution']) pt.align["Tweet Distribution"] = "l" for hour, group in brand_grouped_time: pt.add_row([hour, len(group), '*'*(len(group) / 1000)]) print pt # dividing by 100 in the distribution because people talked less about products during this time period for product, product_grouped_time in products_grouped_time.items(): print "\nNumber of relevant tweets by the hour (UTC) for", product pt = PrettyTable(['Hour', 'Total Tweets', 'Tweet Distribution']) pt.align["Tweet Distribution"] = "l" for hour, group in product_grouped_time: pt.add_row([hour, len(group), '*'*(len(group) / 100)]) print pt

The outputs I obtained are depicted in the snapshot below.

The “Hour” field here follows a 24 hour format. What is interesting here is that, people have been talking more about Sony than Microsoft in Brands. In Products, iPhone 6 seems to be trending more than Samsung’s Galaxy S5. Also the trend shows some interesting insights that people tend to talk more on Twitter in the morning and late evenings.

Time-based Visualizations

It could be helpful to further subdivide the time ranges into smaller intervals so as to increase the resolution of the extremes. Therefore, let’s group into a custom interval by dividing the hour into 15-minute segments. The code is pretty much the same as before except that you call a custom function to perform the grouping. This time, we will be visualizing the distributions using matplotlib.

import matplotlib.pyplot as plt def group_by_15_min_intervals(x): if 0 <= x.minute <= 15: return (x.hour, "0-15") elif 15 < x.minute <= 30: return (x.hour, "16-30") elif 30 < x.minute <= 45: return (x.hour, "31-45") else: return (x.hour, "46-00") brands_grouped_time = {key:data_frames[key].groupby(lambda x: group_by_15_min_intervals(x)) for key in brands} products_grouped_time = {key:data_frames[key].groupby(lambda x: group_by_15_min_intervals(x)) for key in products} # Plot for brands plt.ylabel("Tweet Volume") plt.xlabel("Time") plt.title("Brands Social Trend") plt.plot([float(str(hour[0])+'.'+hour[1].split('-')[0]) for hour, group in brands_grouped_time['sony']][1:-1], [len(group)for hour, group in brands_grouped_time['sony']][1:-1],'r', label='Sony') plt.plot([float(str(hour[0])+'.'+hour[1].split('-')[0]) for hour, group in brands_grouped_time['microsoft']][1:-1], [len(group)for hour, group in brands_grouped_time['microsoft']][1:-1],'b', label='Microsoft') plt.legend() # Plot for products plt.ylabel("Tweet Volume") plt.xlabel("Time") plt.title("Products Social Trend") plt.plot([float(str(hour[0])+'.'+hour[1].split('-')[0]) for hour, group in products_grouped_time['iphone6']][1:-1], [len(group)for hour, group in products_grouped_time['iphone6']][1:-1],'r', label='iPhone 6') plt.plot([float(str(hour[0])+'.'+hour[1].split('-')[0]) for hour, group in products_grouped_time['galaxys5']][1:-1], [len(group)for hour, group in products_grouped_time['galaxys5']][1:-1],'b', label='Galaxy S5') plt.legend()

The two visualizations are depicted below. Ofcourse don’t forget to ignore the section of the plots from after 11:30 am to around 7 pm because during this time no tweets were collected by me. This is indicated by a steep rise in the curve and is insignificant. The real regions of significance are from hour 7 to 11:30 and hour 19 to 22. Considering brands, the visualization for Microsoft vs. Sony is depicted below. Sony is the clear winner here.

Considering products, the visualization for iPhone 6 vs. Galaxy S5 is depicted below. The clear winner here is definitely iPhone 6.

Tweeting Frequency Analysis

In addition to time-based analysis, we can do other types of analysis as well. The most popular analysis in this case would be frequency based analysis of the users authoring the tweets. The following code snippet will compute the Twitter accounts that authored the most tweets and compare it to the total number of unique accounts that appeared for each of our query terms.

# Just to jog your memory, this was already initialized earlier brands = ['sony', 'microsoft'] products = ['iphone6', 'galaxys5'] # For brands brands_user_coll = {key:df.pop('user').apply(pd.Series) for key, df in [(key, data_frames[key]) for key in brands]} brands_authors = {key:brands_user_coll[key].screen_name for key in brands_user_coll.keys()} brands_authors_counter = {key:Counter(brands_authors[key].values) for key in brands_authors.keys()} # Display the results in a neat tabulated form for brand in brands_authors_counter.keys(): print "\nMost frequent (top 10) authors of tweets for", brand pt = PrettyTable(['Author', 'Tweet Count']) [pt.add_row([a, f]) for a, f in brands_authors_counter[brand].most_common(10)] print pt num_unique_authors = len(set(brands_authors[brand].values)) print "There are {0} unique authors out of {1} tweets".format(num_unique_authors, len(data_frames[brand])) # For products products_user_coll = {key:df.pop('user').apply(pd.Series) for key, df in [(key, data_frames[key]) for key in products]} products_authors = {key:products_user_coll[key].screen_name for key in products_user_coll.keys()} products_authors_counter = {key:Counter(products_authors[key].values) for key in products_authors.keys()} # Display the results in a neat tabulated form for product in products_authors_counter.keys(): print "\nMost frequent (top 10) authors of tweets for", product pt = PrettyTable(['Author', 'Tweet Count']) [pt.add_row([a, f]) for a, f in products_authors_counter[product].most_common(10)] print pt num_unique_authors = len(set(products_authors[product].values)) print "There are {0} unique authors out of {1} tweets".format(num_unique_authors, len(data_frames[product]))

The results which I obtained are depicted below.

What we do notice is that a lot of these tweets are also made by bots, advertisers and SEO technicians. Some examples are Galaxy_Sleeves and iphone6_sleeves which are obviously selling covers and cases for the devices.

Tweeting Frequency Visualizations

After frequency analysis, we can plot these frequency values to get better intuition about the underlying distribution, so let’s take a quick look at it using histograms. The following code snippet created these visualizations for both brands and products using subplots.

# Brands Tweets Visualizations fig, axes = plt.subplots(2, sharex=True) axes[0].hist(sorted(brands_authors_counter['sony'].values()), bins=20, alpha=0.7, label='Sony', log=True, color='r') axes[0].set_title('Sony') axes[1].hist(sorted(brands_authors_counter['microsoft'].values()), bins=20, alpha=0.7, label='Microsoft', log=True, color='b') axes[1].set_title('Microsoft') for ax in axes: ax.set_xlabel('Number of Tweets') ax.set_ylabel('Number of Authors') # Products Tweets Visualizations fig, axes = plt.subplots(2, sharex=True) axes[0].hist(sorted(products_authors_counter['iphone6'].values()), bins=20, alpha=0.7, label='iPhone 6', log=True, color='r') axes[0].set_title('iPhone 6') axes[1].hist(sorted(products_authors_counter['galaxys5'].values()), bins=20, alpha=0.7, label='Galaxy S5', log=True, color='b') axes[1].set_title('Galaxy S5') for ax in axes: ax.set_xlabel('Number of Tweets') ax.set_ylabel('Number of Authors')

The visualizations I obtained are depicted below.

The distributions follow the “Pareto Principle” as expected where we see that a selective number of users make a large number of tweets and the majority of users create a small number of tweets. Besides that, we see that based on the tweet distributions, Sony and iPhone 6 are more trending than their counterparts.

Locale Analysis

Another important insight would be to see where your target audience is located and their frequency. The following code snippet achieves the same.

# Top ten locales for Brands for brand in brands: print 'Top 10 locales for', brand pt = PrettyTable(['Language', 'Tweets']) top_ten = dict(data_frames[brand].lang.value_counts()[:10]) top_ten = [[key, top_ten[key]] for key in sorted(top_ten.keys())] top_ten.sort(key = lambda row: row[1], reverse=True) [pt.add_row(row) for row in top_ten] print pt # Top ten locales for Products for product in products: print 'Top 10 locales for', product pt = PrettyTable(['Language', 'Tweets']) top_ten = dict(data_frames[product].lang.value_counts()[:10]) top_ten = [[key, top_ten[key]] for key in sorted(top_ten.keys())] top_ten.sort(key = lambda row: row[1], reverse=True) [pt.add_row(row) for row in top_ten] print pt

The outputs which I obtained are depicted in the following snapshot. Remember that Twitter follows the ISO 639–1 language code convention.

The trend we see is that most of the tweets are from English speaking countries as expected. Surprisingly, most of the Tweets regarding iPhone 6 are from Japan!

Analysis of Trending Topics

In this section, we will see some of the topics which are associated with the terms we used for querying Twitter. For this, we will be running our analysis on the English language tweets. We will be using the nltk library here to take care of a couple of things like removing stopwords which have little significance. Now I will be doing the analysis here for brands only, but you are most welcome to try it out with products too because, the following code snippet can be used to accomplish both the computations.

from collections import Counter import nltk import copy # This is just a rough compilation, it can be done in a better way ignore_terms = [] ignore_terms.extend(nltk.corpus.stopwords.words('english')) ignore_terms.extend(['-', '#', '', 'rt']) # ignoring tokens like retweets and symbols # Analysis for brands print '\n\nAnalysis for Sony' sony_df = copy.copy(data_frames['sony']) sony_en_text = sony_df[sony_df['lang'] == 'en'].pop('text') sony_tokens = [] [sony_tokens.extend([t.lower().strip(":,.") for t in txt.split()]) for txt in sony_en_text.values] sony_tokens_counter = Counter(sony_tokens) [sony_tokens_counter.pop(t, None) for t in ignore_terms] print '\nMost common terms:' print sony_tokens_counter.most_common(20) print '\nMost common phrases:' nltk.Text([token.encode('utf-8').strip() for token in sony_tokens]).collocations() print '\n\nAnalysis for Microsoft' microsoft_df = copy.copy(data_frames['microsoft']) microsoft_en_text = microsoft_df[microsoft_df['lang'] == 'en'].pop('text') microsoft_tokens = [] [microsoft_tokens.extend([t.lower().strip(":,.") for t in txt.split()]) for txt in microsoft_en_text.values] microsoft_tokens_counter = Counter(microsoft_tokens) [microsoft_tokens_counter.pop(t, None) for t in ignore_terms] print '\nMost common terms:' print microsoft_tokens_counter.most_common(20) print '\nMost common phrases:' nltk.Text([token.encode('utf-8').strip() for token in microsoft_tokens]).collocations()

What the above code does is that, it takes each tweet, tokenizes it and then computes a term frequency and outputs the 20 most common terms for each brand. Ofcourse an n-gram analysis can give a deeper insight into trending topics but the same can also be accomplished with ntlk’s collocations function which takes in the tokens and outputs the context in which they were mentioned. The outputs I obtained are depicted in the snapshot below.

Some interesting insights we see from the above outputs are as follows.

Sony was hacked recently and it was rumored that North Korea was responsible for that, however they have denied that. We can see that is trending on Twitter in context of Sony. You can read about it here.

Sony has recently introduced Project Sony Skylight which lets you customize your PS4.

 There are rumors of Lumia 1030, Microsoft’s first flagship phone. People are also talking a lot about Windows 10, the next OS which is going to be released by Microsoft pretty soon.

Interestingly, “ebay price” comes up for both the brands, this might be an indication that eBay is offering discounts for products from both these brands.

To get a detailed view on the tweets matching some of these trending terms, we can use nltk’s concordance function as follows.

print 'Tweets for Sony talking about hack' nltk.Text([token.encode('utf-8').strip() for token in sony_tokens]).concordance('hack') print 'Tweets for Microsoft talking about Lumia 1030' nltk.Text([token.encode('utf-8').strip() for token in microsoft_tokens]).concordance('1030')

The outputs I obtained are as follows. We can clearly see the tweets which contain the token we searched for. In case you are unable to view the text clearly, click on the image to zoom.

Thus, you can see that the Twitter Streaming API is a really good source to track social reaction to any particular entity whether it is a brand or a product. On top of that, if you are armed with an arsenal of Python’s powerful analysis tools and libraries, you can get the best insights from the unending stream of tweets. That’s all for now folks! Before I sign off, I would like to thank Matthew A. Russell and his excellent book Mining the Social Web once again, without which this post would not have been possible.

[dataweave]

Mining Twitter in Depth to Understand and Analyze Product Trends

[This post was written by Dipanjan. Dipanjan works in the Engineering Team with Mandar, addressing some of the problems related to Data Semantics. He loves watching English Sitcoms in his spare time. This blog post is cross posted on the PriceWeave blog as well.] Social media can be defined as virtual communities and networks, where social interaction takes place among people and a wide variety of content is shared including ideas, opinions, information, pictures, videos and much more. Due to the massive growth of social media in the last decade, it has become a rage among data enthusiasts to tap into the vast pool of social data and gather interesting insights like trending items, reception of newly released products by society, popularity measures to name a few. As you are aware, we are constantly evolving PriceWeave, which has the most extensive set of offerings when it comes to providing actionable insights to retail stores and brands. As part of the product development, we look at social data from a variety of channels to mine things like: trending products/brands; social engagement of stores/brands; what content "works" and what doesn't on social media, and so forth. We do a number of experiments with Twitter data, and this series of blog posts is one of the outputs from those efforts. In some of our recent blog posts, we have seen how to look at current trends and gather insights from YouTube the popular video sharing website. We have also talked about how to create a quick bare-bones web application to perform sentiment analysis of tweets from Twitter. Today I will be talking about mining data from Twitter and doing much more with it than just sentiment analysis. We will be analyzing Twitter data in depth and then we will try to get some interesting insights from it. To get data from twitter, first we need to create a new Twitter application to get OAuth credentials and access to their APIs. For doing this, head over to the Twitter Application Management page and sign in with your Twitter credentials. Once you are logged in, click on the Create New App button as you can see in the snapshot below. Once you create the application, you will be able to view it in your dashboard just like the application I created, named DataScienceApp1_DS shows up in my dashboard depicted below. On clicking the application, it will take you to your application management dashboard. Here, you will find the necessary keys you need in the Keys and Access Tokens section. The main tokens you need are highlighted in the snapshot below. I will be doing most of my analysis using the Python programming language. To be more specific, I will be using the IPython shell, but you are most welcome to use the language of your choice, provided you get the relevant API wrappers and necessary libraries. Installing necessary packages After obtaining the necessary tokens, we will be installing some necessary libraries and packages, namely twitter, prettytable and matplotlib. Fire up your terminal or command prompt and use the following commands to install the libraries if you don't have them already.

[root@dip]# pip install twitter [root@dip]# pip install prettytable [root@dip]# pip install matplotlib

Creating a Twitter API Connection Once the packages are installed, you can start writing some code. For this, open up the IDE or text editor of your choice and use the following code segment to create an authenticated connection to Twitter's API. The way the following code snippet works, is by using your OAuth credentials to create an object called auth that represents your OAuth authorization. This is then passed to a class called Twitter belonging to the twitter library and we create a resource object named twitter_api that is capable of issuing queries to Twitter's API.

import twitter CONSUMER_KEY = 'REPLACE WITH YOUR KEY' CONSUMER_SECRET = 'REPLACE WITH YOUR SECRET' OAUTH_TOKEN = 'REPLACE WITH YOUR TOKEN' OAUTH_TOKEN_SECRET = 'REPLACE WITH YOUR TOKEN SECRET' auth = twitter.oauth.OAuth(OAUTH_TOKEN, OAUTH_TOKEN_SECRET, CONSUMER_KEY, CONSUMER_SECRET) twitter_api = twitter.Twitter(auth=auth) print twitter_api

If you do a print twitter_api and all your tokens are corrent, you should be getting something similar to the snapshot below. This indicates that we've successfully used OAuth credentials to gain authorization to query Twitter's API. Exploring Trending Topics Now that we have a working Twitter resource object, we can start issuing requests to Twitter. Here, we will be looking at the topics which are currently trending worldwide using some specific API calls. The API can also be parameterized to constrain the topics to more specific locales and regions. Each query uses a unique identifier which follows the Yahoo! GeoPlanet’s Where On Earth (WOE) ID system, which is an API itself that aims to provide a way to map a unique identifier to any named place on Earth. The following code segment retrieves trending topics in the world, the US and in India.

import json WORLD_WOE_ID = 1 US_WOE_ID = 23424977 IND_WOE_ID = 23424848 world_trends = twitter_api.trends.place(_id=WORLD_WOE_ID) us_trends = twitter_api.trends.place(_id=US_WOE_ID) india_trends = twitter_api.trends.place(_id=IND_WOE_ID) print world_trends print us_trends print india_trends

Once you print the responses, you will see a bunch of outputs which look like JSON data. To view the output in a pretty format, use the following commands and you will get the output as a pretty printed JSON shown in the snapshot below. To view all the trending topics in a convenient way, we will be using list comprehensions to slice the data we need and print it using prettytable as shown below.

from prettytable import PrettyTable world_trends = [trend['name'] for trend in world_trends[0]['trends']] us_trends = [trend['name'] for trend in us_trends[0]['trends']] india_trends = [trend['name'] for trend in india_trends[0]['trends']] pt = PrettyTable(field_names=['World Trends', 'US Trends', 'India Trends']) for world_trend, us_trend, india_trend in zip(world_trends, us_trends, india_trends): pt.add_row([world_trend, us_trend, india_trend]) print pt

On printing the result, you will get a neatly tabulated list of current trends which keep changing with time. Now, we will try to analyze and see if some of these trends are common. For that we use Python's set data structure and compute intersections to get common trends as shown in the snapshot below. Interestingly, some of the trending topics at this moment in the US are common with some of the trending topics in the world. The same holds good for US and India. Mining for Tweets In this section, we will be looking at ways to mine Twitter for retrieving tweets based on specific queries and extracting useful information from the query results. For this we will be using Twitter API's GET search/tweets resource. Since the Google Nexus 6 phone was launched recently, I will be using that as my query string. You can use the following code segment to make a robust API request to Twitter to get a size-able number of tweets.

query = 'Nexus6' count = 100 search_results = twitter_api.search.tweets(q=query, count=count) statuses = search_results['statuses'] # Iterate through 5 more batches of results by following the cursor for _ in range(5): print "Length of status list", len(statuses) try: next_results = search_results['search_metadata']['next_results'] except KeyError, e: break # create a dictionary of parameters to be passed to the search method kwargs = dict([kv.split('=') for kv in next_results[1:].split('&')]) search_results = twitter_api.search.tweets(**kwargs) statuses += search_results['statuses'] # Print one sample tweet by slicing the list print json.dumps(statuses[0], indent=2)

The code snippet above, makes repeated requests to the Twitter Search API. Search results contain a special search_metadata node that embeds a next_results field with a query string that provides the basis of making a subsequent query. If we weren't using a library like twitter to make the HTTP requests for us, this preconstructed query string would just be appended to the Search API URL, and we'd update it with additional parameters for handling OAuth. However, since we are not making our HTTP requests directly, we must parse the query string into its constituent key/value pairs and provide them as keyword arguments to the search/tweets API endpoint. I have provided a snapshot below, showing how this dictionary of key/value pairs are constructed which are passed as kwargs to the Twitter.search.tweets(..) method. Analyzing the structure of a Tweet In this section we will see what are the main features of a tweet and what insights can be obtained from them. For this we will be taking a sample tweet from our list of tweets and examining it closely. To get a detailed overview of tweets, you can refer to this excellent resource created by Twitter. I have extracted a sample tweet into the variable sample_tweet for ease of use. sample_tweet.keys() returns the top-level fields for the tweet. Typically, a tweet has some of the following data points which are of great interest.

The identifier of the tweet can be accessed through sample_tweet['id']

The human-readable text of a tweet is available through sample_tweet['text']

The entities in the text of a tweet are conveniently processed and available through sample_tweet['entities']

The "interestingness" of a tweet is available through sample_tweet['favorite_count'] and sample_tweet['retweet_count'], which return the number of times it's been bookmarked or retweeted, respectively

An important thing to note, is that, the retweet_count reflects the total number of times the original tweet has been retweeted and should reflect the same value in both the original tweet and all subsequent retweets. In other words, retweets aren't retweeted

The user details can be accessed through sample_tweet['user'] which contains details like screen_name, friends_count, followers_count, name, location and so on

Some of the above datapoints are depicted in the snapshot below for the sample_tweet. Note, that the names have been changed to protect the identity of the entity that created the status. Before we move on to the next section, my advice is that you should play around with the sample tweet and consult the documentation to clarify all your doubts. A good working knowledge of a tweet's anatomy is critical to effectively mining Twitter data. Extracting Tweet Entities In this section, we will be filtering out the text statuses of tweets and different entities of tweets like hashtags. For this, we will be using list comprehensions which are faster than normal looping constructs and yield substantial perfomance gains. Use the following code snippet to extract the texts, screen names and hashtags from the tweets. I have also displayed the first five samples from each list just for clarity.

status_texts = [ status['text'] for status in statuses ] screen_names = [ user_mention['screen_name'] for status in statuses for user_mention in status['entities']['user_mentions'] ] hashtags = [ hashtag['text'] for status in statuses for hashtag in status['entities']['hashtags'] ] # get samples of first five entities texts = status_texts[0:5] scr_names = list(set(screen_names))[0:5] hash_tags = hashtags[0:5] tweet_words = words[0:5] # display the results as a table pt = PrettyTable() pt.add_column('Tweets', texts) pt.add_column('Screen Names', scr_names) pt.add_column('HashTags', hash_tags) pt.add_column('Words', tweet_words)

Once you print the table, you should be getting a table of the sample data which should look something like the table below but with different content ofcourse! Frequency Analysis of Tweet and Tweet Entities Once we have all the required data in relevant data structures, we will do some analysis on it. The most common analysis would be a frequency analysis where we find out the most common terms occurring in different entities of the tweets. For this we will be making use of the collection module. The following code snippet ranks the top ten most occurring tweet entities and prints them as a table.

from collections import Counter # get top ten entities top_words = [item[0] for item in Counter(words).most_common()[:10]] top_words_freq = [item[1] for item in Counter(words).most_common()[:10]] top_screen_names = [item[0] for item in Counter(screen_names).most_common()[:10]] top_screen_names_freq = [item[1] for item in Counter(screen_names).most_common()[:10]] top_hashtags = [item[0] for item in Counter(hashtags).most_common()[:10]] top_hashtags_freq = [item[1] for item in Counter(hashtags).most_common()[:10]] # print the results as a table pt = PrettyTable() pt.add_column('Words',top_words) pt.add_column('Frequency',top_words_freq) pt.add_column('Screen Names',top_screen_names) pt.add_column('Frequency',top_screen_names_freq) pt.add_column('Hashtags',top_hashtags) pt.add_column('Frequency',top_hashtags_freq) print pt

The output I obtained is shown in the snapshot below. As you can see, there is a lot of noise in the tweets because of which several meaningless terms and symbols have crept into the top ten list. For this, we can use some pre-processing and data cleaning techniques. Analyzing the Lexical Diversity of Tweets A slightly more advanced measurement that involves calculating simple frequencies and can be applied to unstructured text is a metric called lexical diversity. Mathematically, lexical diversity can be defined as an expression of the number of unique tokens in the text divided by the total number of tokens in the text. Let us take an example to understand this better. Suppose you are listening to someone who repeatedly says "and stuff" to broadly generalize information as opposed to providing specific examples to reinforce points with more detail or clarity. Now, contrast that speaker to someone else who seldom uses the word "stuff" to generalize and instead reinforces points with concrete examples. The speaker who repeatedly says "and stuff" would have a lower lexical diversity than the speaker who uses a more diverse vocabulary. The following code snippet, computes the lexical diversity for status texts, screen names, and hashtags for our data set. We also measure the average number of words per tweet.

# A function for computing lexical diversity def lexical_diversity(tokens): return 1.0*len(set(tokens))/len(tokens) # A function for computing the average number of words per tweet def average_words(statuses): total_words = sum([ len(s.split()) for s in statuses ]) return 1.0*total_words/len(statuses) print lexical_diversity(words) print lexical_diversity(screen_names) print lexical_diversity(hashtags) print average_words(status_texts)

The output which I obtained is depicted in the snapshot below. Now, I am sure you must be thinking, what on earth do the above numbers indicate? We can analyze the above results as follows.

The lexical diversity of the words in the text of the tweets is around 0.097. This can be interpreted as, each status update carries around 9.7% unique information. The reason for this is because, most of the tweets would contain terms like Android, Nexus 6, Google

The lexical diversity of the screen names, however, is even higher, with a value of 0.59 or 59%, which means that about 29 out of 49 screen names mentioned are unique. This is obviously higher because in the data set, different people will be posting about Nexus 6

The lexical diversity of the hashtags is extremely low at a value of around 0.029 or 2.9%, implying that very few values other than the #Nexus6 hashtag appear multiple times in the results. This is relevant because tweets about Nexus 6 should contain this hashtag

The average number of words per tweet is around 18 words

This gives us some interesting insights like people mostly talk about Nexus 6 when queried for that search keyword. Also, if we look at the top hashtags, we see that Nexus 5 co-occurs a lot with Nexus 6. This might be an indication that people are comparing these phones when they are tweeting. Examining Patterns in Retweets In this section, we will analyze our data to determine if there were any particular tweets that were highly retweeted. The approach we'll take to find the most popular retweets, is to simply iterate over each status update and store out the retweet count, the originator of the retweet, and status text of the retweet, if the status update is a retweet. We will be using a list comprehension and sort by the retweet count to display the top few results in the following code snippet.

retweets = [ # Store out a tuple of following three values (status['retweet_count'], status['retweeted_status']['user']['screen_name'], status['text']) # for each status for status in statuses # as long as the status has been retweeted if status.has_key('retweeted_status') ] # Display the top 5 retweets with necessary fields pt = PrettyTable(field_names=['Count', 'Screen Name', 'Text']) [ pt.add_row(row) for row in sorted(retweets, reverse=True)[:5] ] pt.max_width['Text'] = 50 print pt

The output I obtained is depicted in the following snapshot. From the results, we see that the top most retweet is from the official googlenexus channel on Twitter and the tweet speaks about the phone being used non-stop for 6 hours on only a 15 minute charge. Thus, you can see that this has definitely been received positively by the users based on its retweet count. You can detect similar interesting patterns in retweets based on the topics of your choice. Visualizing Frequency Data In this section, we will be creating some interesting visualizations from our data set. For plotting we will be using matplotlib, a popular Python plotting library which comes inbuilt with IPython. If you don't have matplotlib loaded by default use the command import matplotlib.pyplot as plt in your code. Visualizing word frequencies In our first plot, we will be displayings the results from the words variable which contains different words from the tweet status texts. Using Counter from the collections package, we generate a sorted list of tuples, where each tuple is a (word, frequency) pair. The x-axis value will correspond to the index of the tuple, and the y-axis will correspond to the frequency for the word in that tuple. We transform both axes into a logarithmic scale because of the vast number of data points. Visualizing words, screen names, and hashtags A line chart of frequency values is decent enough. But what if we want to find out the number of words having a frequency between 1-5, 5-10, 10-15... and so on. For this purpose we will be using a histogram to depict the frequencies. The following code snippet achieves the same.

for label, data in (('Words', words), ('Screen Names', screen_names), ('Hashtags', hashtags)): # Build a frequency map for each set of data and plot the values c = Counter(data) plt.hist(c.values()) # Add a title labels plt.title(label) plt.ylabel("Number of items in a bin") plt.xlabel("Bins (number of times an item appeared)") # Display as a new figure plt.figure()

What this essentially does is, it takes all the frequencies and groups them together and creates bins or ranges and plots the number of entities which fall in that bin or range. The plots I obtained are shown below. From the above plots, we can observe that, all the three plots follow the "Pareto Principle" i.e, almost 80% of the words, screen names and hashtags have a frequency of only 20% in the whole data set and only 20% of the words, screen names and hashtags have a frequency of more than 80% in the data set. In short, if we consider hashtags, a lot of hashtags occur maybe only once or twice in the whole data set and very few hashtags like #Nexus6 occur in almost all the tweets in the data set leading to its high frequency value. Visualizing retweets In this visualization, we will be using a histogram to visualize retweet counts using the following code snippet.

# Using underscores while unpacking values in a tuple is idiomatic for discarding them counts = [count for count, _, _ in retweets] plt.hist(counts) plt.title("Retweets") plt.xlabel('Bins (number of times retweeted)') plt.ylabel('Number of tweets in bin') print counts

The plot which I obtained is shown below. Looking at the frequency counts, it is clear that very few retweets have a large count. I hope you have seen by now, how powerful Twitter APIs are and using simple Python libraries and modules, it is really easy to generate very powerful and interesting insights. That's all for now folks! I will be talking more about Twitter Mining in another post sometime in the future. A ton of thanks goes out to Matthew A. Russell and his excellent book Mining the Social Web, without which this post would never have been possible.

[dataweave]

Analyzing Social Trends Data from Google Trends and YouTube

[This post was written by Dipanjan. Dipanjan works in the Engineering Team with Mandar, addressing some of the problems related to Data Semantics. He loves watching English Sitcoms in his spare time.] In today's world dominated by technology and gadgets, we often wonder how well a particular product or technology is being perceived by society and if it is truly going to leave a long lasting impression. We regularly see fan wars breaking out between Apple and Android fanatics (no offence to Windows mobile lovers!) on social media where each claim that they are better than the rest. Another thing we notice quite often is that whenever a new product is launched or in the process of being launched, it starts trending on different social media websites. This made me wonder if there was a way to see some of these trends and the impact it is causing on social media. There are different social media channels where people post a wide variety of content ranging from personal opinions to videos and pictures. A few of the popular ones are listed below.

Facebook

Twitter

YouTube

Instagram

Pinterest

Today, I will discuss two such ways we can do this, namely Google Trends and YouTube. If you want to know how we can perform data mining using Twitter, refer to my earlier post "Building a Twitter Sentiment Analysis App using R" which deals with getting data from Twitter and analyzing it. Now, we will be looking at how easy it is to visualize trending topics on Google Trends without writing a single line of code. For this, you need to go to the Google Trends website. On opening it, you will be greeted by an interactive dashboard, showing the current trending topics summarized briefly just like the snapshot shown below. You can also click on any particular panel to explore it in detail. This is not all that Google Trends has to offer. We can also customize visualizations to see how specific topics are trending across the internet by specifying then in the interface and the results are shown in the form of a beautiful visualization. Google gets the data based on the number of times people have searched for it online. A typical comparison of people's interest in different mobile operating systems over time is shown below. Interestingly, from the above visualization, we see that 'Windows Mobile' was quite popular from 2007 till mid 2009 when the popularity of 'Android' just skyrocketed. Apple's 'iOS' gained popularity sometime around 2010. One must remember however that this data is purely based on data tracked by Google searches. Coming to YouTube, it is perhaps the most popular video sharing website and I am sure all of you have at least watched a video on YouTube. Interestingly, we can also get a lot of interesting statistics from these videos besides just watching them, thanks to some great APIs provided by Google. In the next part, I will discuss how to get interesting statistics from YouTube based on a search keyword and do some basic analysis. I won't be delving into the depths of data analytics here but I will provide you enough information to get started with data mining from YouTube. We will be using Google's YouTube Data API, some Python wrappers for the same and the pandas framework to analyze the data. First, we would need to go to the Google Developers Console and create a new project just like the snapshot shown below. Once the project is created, you will be automatically re-directed to the dashboard for the project: There you can choose to enable the APIs you want for your application. Go to the APIs section on the left and enable the YouTube Data API v3 just like it is depicted in the snapshot below (click it if you are unable to make out the text in the image). Now, we will create a new API key for public API access. For this, go to the Credentials section and click on Create new key and choose the Server key option and create a new API key which is shown in the snapshot below (click to zoom the image). We will be using some Python libraries so open up your terminal or command prompt and install the following necessary libraries if you don't have them.

[root@dip]# pip install google-api-python-client [root@dip]# pip install pandas

Now that the initial setup is complete, we can start writing some code! Head over to your favorite Python IDE or console and use the following code segment to build a YouTube resource object.

from apiclient.discovery import build from apiclient.errors import HttpError import pandas as pd DEVELOPER_KEY = "REPLACE WITH YOUR KEY" YOUTUBE_API_SERVICE_NAME = "youtube" YOUTUBE_API_VERSION = "v3" youtube = build(YOUTUBE_API_SERVICE_NAME, YOUTUBE_API_VERSION, developerKey=DEVELOPER_KEY)

Once this is complete, we will be using this YouTube resource object to search for videos with the android keyword. For this we will be using the search method to query YouTube and after getting back a list of results, we will be storing each result to its appropriate list, and then display the lists of matching videos, channels, and playlists using the following code segment.

search_response = youtube.search().list( q="android", part="id,snippet", maxResults=50 ).execute() videos = [] channels = [] playlists = [] for search_result in search_response.get("items", []): if search_result["id"]["kind"] == "youtube#video": videos.append("%s (%s)" % (search_result["snippet"]["title"], search_result["id"]["videoId"])) elif search_result["id"]["kind"] == "youtube#channel": channels.append("%s (%s)" % (search_result["snippet"]["title"], search_result["id"]["channelId"])) elif search_result["id"]["kind"] == "youtube#playlist": playlists.append("%s (%s)" % (search_result["snippet"]["title"], search_result["id"]["playlistId"]))

Based on the query I ran, most of the results seemed to be videos. The output I obtained is shown in the snapshot below. Since the playlists and channels we obtained are very less in number, I decided to go ahead and analyze the videos obtained. For that, we create a dict of video identifiers and video names. Then we pass a query to the YouTube API's videos method, to get the relevant statistics for each video.

videos = {} for search_result in search_response.get("items", []): if search_result["id"]["kind"] == "youtube#video": videos[search_result["id"]["videoId"]] = search_result["snippet"]["title"] video_ids_list = ','.join(videos.keys()) video_list_stats = youtube.videos().list( id=video_ids_list, part='id,statistics' ).execute()

I know you must be interested by now to see what kind of data is present in video_list_stats. So for that, I will show you the relevant statistics obtained for a video from the API in the following snapshot. Now we will be using pandas to analyze this data. For that, the following code segment is used, to get this data into a pandas data frame.

df = [] for item in videos_list_stats['items']: video_dict = dict(video_id = item['id'], video_title = videos[item['id']]) video_dict.update(item['statistics']) df.append(video_dict) df = pd.DataFrame.from_dict(df)

Now, we can view the contents of this data frame. I will be showing the output of the first few rows with the relevant columns in the snapshot below. I have considered only the important data points which include viewCount, likeCount, dislikeCount, commentCount indicating the number of views, likes, dislikes and comments on the videos respectively. Once we have this table of clean and formatted data, we can do all sorts of analytics on it, like getting the mean and median for number of views, seeing which are really popular videos and so on. Some examples with required code segments are depicted below. I have used the IPython shell for analyzing the data. Mean and Median of different counts Top ten most viewed videos Top ten most liked videos Line chart showing counts of likes, views and comments Thus you can see by now that a lot of interesting analysis and visualizations can be built on top of this data. For more details on how to customize and use the Youtube API with Python, you can refer to this page for sample code segments.

[dataweave]

Building a Twitter Sentiment Analysis App using R

[This post was written by Dipanjan. Dipanjan is a Data Engineer at DataWeave who works with Mandar addressing some of the semantics related problems like product clustering and data normalization.] Twitter, as we know, is a highly popular social networking and micro-blogging service used by millions worldwide. Each status or tweet as we call it is a 140 character text message. Registered users can read and post tweets, but unregistered users can only view them. Text mining and sentiment analysis are some of the hottest topics in the analytics domain these days. Analysts are always looking to crunch thousands of tweets to gain insights on different topics, be it popular sporting events such as the FIFA World Cup or to know when the next product is going to be launched by Apple. Today, we are going to see how we can build a web app for doing sentiment analysis of tweets using R, the most popular statistical language. For building the front end, we are going to be using the 'Shiny' package to make our life easier and we will be running R code in the backend for getting tweets from twitter and analyzing their sentiment. The first step would be to establish an authorized connection with Twitter for getting tweets based on different search parameters. For doing that, you can follow the steps mentioned in this document which includes the R code necessary to achieve that: http://rpubs.com/dipanzan/twitterAccess After obtaining a connection, the next step would be to use the 'shiny' package to develop our app. This is a web framework for R, developed by RStudio. Each app contains a server file ( server.R ) for the backend computation and a user interface file ( ui.R ) for the frontend user interface. You can get the code for the app from my github repository here which is fairly well documented but I will explain the main features anyway. The first step would be to develop the UI of the application, you can take a look at the ui.R file, we have a left sidebar, where we take input from the user in two text fields for either twitter hashtags or handles for comparing the sentiment. We also create a slider for selecting the number of tweets we want to retrieve from twitter. The right panel consists of four tabs, here we display the sentiment plots, word clouds and raw tweets for both the entities in respective tabs as shown below. Coming to the backend, remember to also copy the two dictionary files, 'negative_words.txt' and 'positive_words.txt' from the repository because we will be using them for analyzing and scoring terms from tweets. On taking a close look at the server.R file, you can notice the following operations taking place. - The 'TweetFrame' function sends the request query to Twitter, retrieves the tweets and aggregates it into a data frame. - The 'CleanTweets' function runs a series of regexes to clean tweets and extract proper words from them. - The 'numoftweets' function calculates the number of tweets. - The 'wordcloudentity' function creates the word clouds from the tweets. - The 'sentimentalanalysis' and 'score.sentiment' functions performs the sentiment analysis for the tweets. These functions are called in reactive code segments to enable the app to react instantly to change in user input. The functions are documented extensively but I'll explain the underlying concept for sentiment analysis and word clouds which are generated. For word clouds, we get the text from all the tweets, remove punctuation and stop words and then form a term document frequency matrix and sort it in decreasing order to get the terms which occur the most frequently in all the tweets and then form a word cloud figure based on those tweets. An example obtained from the app is shown below for hashtags '#thrilled' and '#frustrated'. For sentiment analysis, we use Jeffrey Breen's sentiment analysis algorithm cited here, where we clean the tweets, split tweets into terms and compare them with our positive and negative dictionaries and determine the overall score of the tweet from the different terms. A positive score denoted positive sentiment, a score of 0 denotes neutral sentiment and a negative score denotes negative sentiment. A more extensive and advanced n-gram analysis can also be done but that story is for another day. An example obtained from the app is shown below for hashtags '#thrilled' and '#frustrated'. After getting the server and UI code, the next step is to deploy it in the server, we will be using shinyapps.io (https://www.shinyapps.io/) server which allows you to host your R web apps free of charge. If you already have the code loaded up in RStudio, you can deploy it from there using the 'deployApp()' command. For more details on app building and hosting check out the official tutorial here. You can check out a live demo of the app here: https://dipanjan.shinyapps.io/twitter-analysis/ It's still under development so suggestions are always welcome.

[dataweave]

A Peek into GNU Parallel

[This post was written by Jyotiska. Jyotiska is a Data Engineer at DataWeave with keen interest in Python and experimentation. He works with Sanket in the Products and Strategy team.] GNU Parallel is a tool that can be deployed from a shell to parallelize job execution. A job can be anything from simple shell scripts to complex interdependent Python/Ruby/Perl scripts. The simplicity of ‘Parallel’ tool lies in it usage. A modern day computer with multicore processors should be enough to run your jobs in parallel. A single core computer can also run the tool, but the user won’t be able to see any difference as the jobs will be context switched by the underlying OS. At DataWeave, we use Parallel for automating and parallelizing a number of resource extensive processes ranging from crawling to data extraction. All our servers have 8 cores with capability of executing 4 threads in each. So, we experienced huge performance gain after deploying Parallel. Our in-house image processing algorithms used to take more than a day to process 200,000 high resolution images. After using Parallel, we have brought the time down to a little over 40 minutes! GNU Parallel can be installed on any Linux box and does not require sudo access. The following command will install the tool:

(wget -O - pi.dk/3 || curl pi.dk/3/) | bash

GNU Parallel can read inputs from a number of sources -- a file or command line or stdin. The following simple example takes the input from the command line and executes parallely:

parallel echo ::: A B C

The following takes the input from a file:

parallel -a somefile.txt echo

Or STDIN:

cat somefile.txt | parallel echo

The inputs can be from multiple files too:

parallel -a somefile.txt -a anotherfile.txt echo

The number of simultaneous jobs can be controlled using the --jobs or -j switch. The following command will run 5 jobs at once:

parallel --jobs 5 echo ::: A B C D E F G H I J

By default, the number of jobs will be equal to the number of CPU cores. However, this can be overridden using percentages. The following will run 2 jobs per CPU core:

parallel --jobs 200% echo ::: A B C D

If you do not want to set any limit, then the following will use all the available CPU cores in the machine. However, this is NOT recommended in production environment as other jobs running on the machine will be vastly slowed down.

parallel --jobs 0 echo ::: A B C

Enough with the toy examples. The following will show you how to bulk insert JSON documents in parallel in a MongoDB cluster. Almost always we need to insert millions of document quickly in our MongoDB cluster and inserting documents serially doesn't cut it. Moreover, MongoDB can handle parallel inserts. The following is a snippet of a file with JSON document. Let’s assume that there are a million similar records in the file with one JSON document per line.

{“name”: “John”, “city”: “Boston”, “age”: 23} {“name”: “Alice”, “city”: “Seattle”, “age”: 31} {“name”: “Patrick”, “city”: “LA”, “age”: 27} ... ...

The following Python script will get each JSON document and insert into “people” collection under “dw” database.

import json import pymongo import sys document = json.loads(sys.argv[1]) client = pymongo.MongoClient(“mongodb://somehost:someport”) db = client[“dw”] collection = db[“people”] try: collection.insert(document) except Exception as e: print “Could not insert document in db”, repr(e)

Now to run this parallely, the following command should do the magic:

cat people.json | parallel ‘python insertDB.py {}’

That's it! There are many switches and options available for advanced processing. They can be accessed by doing a man parallel on the shell. Also the following page has a set of tutorials: GNU Parallel Tutorials.

[dataweave]

JSON vs simpleJSON vs ultraJSON

[This post was written by Jyotiska. Jyotiska is a Data Engineer at DataWeave with keen interest in Python and experimentation. He works with Sanket in the Products and Strategy team.]

Without argument, the most used file format used in DataWeave is JSON. We use JSON everywhere -- managing product lists, storing data extracted from crawl dumps, generating assortment reports, and almost everywhere else. So it is very important to make sure that the libraries and packages we are using are fast enough to handle large data sets. There are two popular packages used for handling json -- first is the stock json package that comes with default installation of Python, the other one is simplejson which is an optimized and maintained json package for Python. The goal of this blog post is to introduce ultrajson or Ultra JSON, a JSON library written mostly in C and built to be extremely fast.

We have done the benchmark on three popular operations -- load, loads and dumps. We have a dictionary with 3 keys - id, name and address. We will dump this dictionary using json.dumps() and store it in a file. Then we will use json.loads() and json.load() separately to load the dictionaries from the file. We have performed this experiment on 10000, 50000, 100000, 200000, 1000000 dictionaries and observed how much time it takes to perform the operation by each library.

dumps operation line by line

Here is the result we received using the json.dumps() operations. We have dumped the content dictionary by dictionary.

Number of dicts json (in ms) simplejson (in ms) ujson (in ms) 10000 84.106 115.876 25.891 50000 395.163 579.576 122.626 100000 820.280 1147.962 246.721 200000 1620.239 2277.786 487.402 500000 3998.736 5682.641 1218.653 1000000 7847.999 11501.038 2530.791

We notice that json performs better than simplejson but ultrajson wins the game with almost 400% speedup than stock json.

dumps operation (all dictionaries at once)

In this experiment, we have stored all the dictionaries in a list and dumped the list using json.dumps().

Number of dicts json (in ms) simplejson (in ms) ujson (in ms) 10000 21.674 21.941 14.484 50000 102.739 118.851 67.215 100000 199.454 240.849 138.830 200000 401.376 476.392 270.667 500000 1210.664 1376.511 834.013 1000000 2729.538 2983.563 1830.707

simplejson is almost as good as stock json, but again ultrajson outperforms them by 150% speedup. Now lets see how they perform for load and loads operation.

load operation on a list of dictionaries

Now we do the load operation on a list of dictionaries and compare the results.

Number of dicts json (in ms) simplejson (in ms) ujson (in ms) 10000 47.040 8.932 10.706 50000 165.877 44.065 45.629 100000 356.405 97.277 105.948 200000 718.873 185.917 205.120 500000 1699.623 461.605 503.203 1000000 3441.075 949.905 1055.966

Surprisingly, simplejson beats other two, with ultrajson being almost close to simplejson. Here, we observe that simplejson is almost 400% faster than stock json, same with ultrajson.

loads operation on dictionaries

In this experiment, we load dictionaries from the file one by one and pass them to the json.loads() function.

Number of dicts json (in ms) simplejson (in ms) ujson (in ms) 10000 99.501 69.517 15.917 50000 407.873 246.794 76.369 100000 893.989 526.257 157.272 200000 1746.961 1025.824 318.306 500000 8446.053 2497.081 791.374 1000000 8281.018 4939.498 1642.416

Again ultrajson steals the show, being almost 600% faster than stock json and 400% faster than simplejson.

That is all the benchmarks we have here. The verdict is pretty clear. Use simplejson instead of stock json in any case, since simplejson is well maintained repository. If you really want something extremely fast, then go for ultrajson. In that case, keep in mind that ultrajson only works with well defined collections and will not work for un-serializable collections. But if you are dealing with texts, this should not be a problem.

[dataweave]

James Gleick's The Information -- The Beginnings

[The author of this post is Sanket. Sanket heads Products and Strategy at DataWeave. When he manages to find time he plays with his daughter or reads two more pages of a book he's been trying to finish since several months.]

James Gleick’s 2011 book The Information (ISBN: 0375423729, Pages: 527, Publisher: Knopf Doubleday Publishing Group) is an important book. It is an essential reading of our time---the fabled Information Age. The subtitle of the book is very apt: A History, a Theory, a Flood. As it suggests, the book traces the quest of human being towards understanding the nature of “information”, representing it (in various forms spoken word, phoneme, written symbols, electric pulses, etc.), and communicating it.

The rest of this blog post is not a review of the book (just go read the book instead of this!), but a series of thoughts and ideas formed upon reading the book. Just like the book, I am going to write them in three parts.

Talking Drums

While there are no distinct sub parts to the book, we can deem the initial chapters as The History. Gleick starts with a fascinating account of the “talking drums of Africa”. Drums speaking the numerous African tongues and dialects developed by illiterate primitives were used to communicate information over hundreds of miles within minutes, a feat unachievable by the fastest runner or the best horse in Europe.

Europeans used smoke signals that typically communicated binary (or at best a very limited number of) status messages. Whereas the drums communicated complex messages not just warnings and quick messages, but “songs, jokes, etc”. A drummer added sufficient contextual redundancy to ensure error correction (while such concepts as we know of them today did not exist at that time!). African languages being tonal, and the drums limited by conveying just the tonal element, the redundancy was essential to resolve context. For instance, if a drummer wanted to say something about a chicken (koko), he would drum: "koko, the one that does the sound kiokio".

The Written Word

While the oral culture was wondrous in itself (Homer’s epics possibly evolved from oral transfer across generations), the invention of writing presented possibilities and difficulties, hitherto unheard of. Firstly, it had its detractors among which were influential people such as Plato. "Writing diminishes the faculty of memory among men. […]" Compare this to the qualm of our age: “Is Google making us stupid?”

Gleick argues on the basis of dialectics of earlier historians that writing was essential for the possibility of conscious thought. Plato and other Greek philosophers, while being skeptical about writing, developed logic through the faculty of writing. The oral culture lacked the comprehension of syllogisms as shown by sociological experiments.

Babylonians developed advanced mathematics for their time as indicated by their cuneiform tablets. Regardless of the initial resistance to writing, it marched on to produce the sciences, arts, and philosophy of the highest quality. The notion of “history”, leave aside “history of information”, was inconceivable without first inventing writing.

The Telegraph

Information also presented the problem of communication. Beacons, smoke signals, and the rest were all but limited by their range, both of vocabulary and distance. Then came along telegraph. Of course, the first telegraph systems were mechanical, wherein an intricate system of wooden arms was used to relay messages from one station to the next. The positions of the arms of the telegraph implied different messages. The number of states in which the telegraph could exist increased quite a bit. However, the efficacy of a mechanical system was quite limited.

When Morse developed his electrical telegraph system, he put a lot of thought into one of the primary problems of communication---that of “encoding”. That is to say, representing the spoken words into written symbols at one level; then representing the written symbol into a completely new form, that of electric pulses, the so called “dots and dashes”.

Morse and his student Veil wanted to improve the efficiency of the encoding. Veil visited the local newspaper office and found that, “there were 15000 E’s, followed by 12000 T’s, and only 200 Z’s”. He had indeed discovered a version of Zipf’s law. It is shown today that the Morse code is within 15% of the optimal representation of the English language.

It took a while to get the right intuition to develop the encoding. People thought about as many wires as there were letters in the alphabet. Then came the idea of using needles deflected by electromagnetism--again one per letter. The number of needles were reduced by Baron Schilling in Russia, first to five and then to just one. He used combinations of left and right signals to denote letters. Gauss and Weber came up with a binary code using the direction and the number of needle deflections.

Alfred Vail devised a simple spring-loaded lever which had the minimum functionality of closing and opening of a circuit. He called it a key. With the key an operator could send signals at the rate of hundreds per minute. Once the relay was invented (which occurred almost simultaneously), long distance electric telegraphy became a reality.

"A net-work of nerves of iron wire"

Even the electric telegraph was not beyond the scorn of some of the thinkers of the day. Apparently, someone assigned to assess the technology--he was a physician and scientist--was not impressed: "What can we expect of a few wretched wires?"

Regardless, newspapers immediately saw the benefits of the new technology. Numerous newspapers across the globe called themselves The Telegraph or a variant of it. News items were described as "Communicated by Electric Telegraph" (a la "Breaking News" or "Exclusive Story"). Very soon fire brigades and police stations started linking their communications. Shopkeepers started advertising their ability to take orders by telegraphs. That was perhaps the beginning of online shopping!

People, especially writers need an analogue, a way of describing new things in comparison to what they see in nature. The telegraph reminded them of spider webs, labyrinths and mazes. But perhaps for the first time the word network was used, as something that connects and encompasses, and can be used as a means of communication. "The whole net-work of wires, all quivering from end to end with signals of human intelligence."

[dataweave]

Extract dominant colors from an image with ColorWeave

[This post was written by Jyotiska with contributions from Sanket. Jyotiska is a Data Engineer at DataWeave with keen interest in Python and experimentation. He works with Sanket in the Products and Strategy team.]

We have taken a special interest in colors in recent times. Some of us can even identify and name a couple of dozen different colors! The genesis for this project was PriceWeave's Color Analytics offering. With Color Analytics, we provide detailed analysis in colors and other attributes related to retailers and brands in Apparel and Lifestyle products space.

The Idea

The initial idea was to simply extract the dominating colors from an image and generate a color palette. Fashion blogs and Pinterest pages are updated regularly by popular fashion brands and often feature their latest offerings for the current season and their newly released products. So, we thought if we can crawl these blogs periodically after every few days/weeks, we can plot the trends in graphs using the extracted colors. This timeline is very helpful for any online/offline merchant to visualize the current trend in the market and plan out their own product offerings.

We expanded this to include Apparel and Lifestyle products from eCommerce websites like Jabong, Myntra, Flipkart, and Yebhi, and stores of popular brands like Nike, Puma, and Reebok. We also used their Pinterest pages.

Color Extraction

The core of this work was to build a robust color extraction algorithm. We developed a couple of algorithms by extending some well known techniques. One approach we followed was to use standard unsupervised machine learning techniques. We ran k-means clustering against our images data. Here k refers to the number of colors we are trying to extract from the image.

In another algorithm, we extracted all the possible color points from the image and used heuristics to come up with a final set of colors as a palette.

Another of our algorithms was built on top of the Python Image Library (PIL) and the Colorific package to extract and produce the color palette from the image.

Regardless of the approach we used, we soon found out that both speed and accuracy were a problem. Our k-means implementation produced decent results but it took 3-4 seconds to process an entire image! This might not seem much for a small set of images, but the script took 2 days to process 40,000 products from Myntra.

Post this, we did a lot of tweaking in our algorithms and came up with a faster and more accurate model which we are using currently.

ColorWeave API

We have open sourced an early version of our implementation. It is available of github here. You can also download the Python package from the Python Package Index here. Find below examples to understand its usage.

Retrieve dominant colors from an image URL

from colorweave import palette print palette(url="image_url")

Retrive n dominant colors from a local image and print as json:

print palette(url="image_url", n=6, output="json")

Print a dictionary with each dominant color mapped to its CSS3 color name

print palette(url="image_url", n=6, format="css3")

Print the list of dominant colors using k-means clustering algorithm

print palette(url="image_url", n=6, mode="kmeans")

Data Storage

The next challenge was to come up with an ideal data model to store the data which will also let us query on it. Initially, all the processed data was indexed by Solr and we used its REST API for all our querying. Soon we realized that we have to come up with better data model to store, index and query the data.

We looked at a few NoSQL databases, especially column oriented stores like Cassandra and HBase and document stores like MongoDB. Since the details of a single product can be represented as a JSON object, and key-value storage can prove to be quite useful in querying, we settled on MongoDB. We imported our entire data (~ 160,000 product details) to MongoDB, where each product represents a single document.

Color Mapping

We still had one major problem we needed to resolve. Our color extraction algorithm produces the color palette in hexadecimal format. But in order to build a useful query interface, we had to translate the hexcodes to human readable color names. We had two options. Either we could use a CSS 2.0 web color names consisting on 16 basic colors (White, Silver, Gray, Black, Red, Maroon, Yellow, Olive, Lime, Green, Aqua, Teal, Blue, Navy, Fuchsia, Purple) or we could use CSS 3.0 web color names consisting of 140 colors. We used both to map colors and stored those colors along with each image.

Color Hierarchy

We mapped the hexcodes to CSS 3.1 which has every possible shades for the basic colors. Then we assigned a parent basic color for every shades and stored them separately. Also, we created two fields - one for the primary colors and the other one for the extended colors which will help us in indexing and querying. At the end, each product had 24 properties associated with it! MongoDB made it easier to query on the data using the aggregation framework.

What next?

A few things. An advanced version of color extraction (with a number of other exciting features) is being integrated into PriceWeave. We are also working on building a small consumer facing product where users will be able to query and find products based on color and other attributes. There are many other possibilities some of which we will discuss when the time is ripe. Signing off for now!

[dataweave]

Implementing DataWeave's Social API for Social Data Analysis

[The author of this post is Apoorv. Apoorv did his internship at DataWeave during January and February, 2014. Here he shares his experiences with Twitter API, MongoDB, and implementing REST APIs.]

In today's world, the analysis of any social media stream can reap invaluable information about, well, pretty much everything. If you are a business catering to a large number of consumers, it is a very important tool for understanding and analyzing the market's perception about you, and how your audience reacts to whatever you present before them.

At DataWeave, we sat down to create a setup that would do this for some e-commerce stores and retail brands. And the first social network we decided to track was the micro-blogging giant, Twitter. Twitter is a great medium for engaging with your audience. It's also a very efficient marketing channel to reach out to a large number of people.

Data Collection

The very first issue that needs to be tackled is collecting the data itself. Now quite understandably, Twitter protects its data vigorously. However, it does have a pretty solid REST API for data distribution purposes too. The API is simple, nothing too complex, and returns data in the easy to use JSON format. Take a look at the timeline API, for example. That's quite straightforward and has a lot of detailed information.

The issue with the Twitter API however, is that it is seriously rate limited. Every function can be called in a range of 15-180 times in a 15-minute window. While this is good enough for small projects not needing much data, for any real-world application however, these rate limits can be really frustrating. To avoid this, we used the Streaming API, which creates a long-lived HTTP GET request that continuously streams tweets from the public timeline.

Also, Twitter seems to suddenly return null values in the middle of the stream, which can make the streamer crash if we don't take care. As for us, we simply threw away all null data before it reached the analysis phase, and as an added precaution, designed a simple e-mail alert for when the streamer crashed.

Data Storage

Next is data storage. Data is traditionally stored in tables, using RDBMS. But for this, we decided to use MongoDB, as a document store seemed quite suitable for our needs. While I didn't have much clue about MongoDB or what purpose it's going to serve at first, I realized that is a seriously good alternative to MySQL, PostgreSQL and other relational schema-based data stores for a lot of applications.

Some of its advantages that I very soon found out were: documents-based data model that are very easy to handle analogous to Python dictionaries, and support for expressive queries. I recommend using this for some of your DB projects. You can play about with it here.

Data Processing

Next comes data processing. While data processing in MongoDB is simple, it can also be a hard thing to learn, especially for someone like me, who had no experience anywhere outside SQL. But MongoDB queries are simple to learn once the basics are clear.

For example, in a DB DWSocial with a collection tweets, the syntax for getting all tweets would be something like this in a Python environment:

rt = list(db.tweets.find())

The list type-cast here is necessary, because without it, the output is simply a MongoDB reference, with no value. Now, to find all tweets where user_id is 1234, we have

rt = list(db.retweets.find({ 'user_id': 1234 })

Apart from this, we used regexes to detect specific types of tweets, if they were, for example, "offers", "discounts", and "deals". For this, we used the Python re library, that deals with regexes. Suffice is to say, my reaction to regexes for the first two days was much like

Once again, its just initial stumbles. After some (okay, quite some) help from Thothadri, Murthy and Jyotiska, I finally managed a basic parser that could detect which tweets were offers, discounts and deals. A small code snippet is here for this purpose.

def deal(id): re_offers = re.compile(r''' \b (?: deals? | offers? | discount | promotion | sale | rs? | rs\? | inr\s*([\d\.,])+ | ([\d\.,])+\s*inr ) \b | \b\d+% | \$\d+\b ''', re.I|re.X) x = list(tweets.find({'user_id' : id,'created_at': { '$gte': fourteen_days_ago }})) mylist = [] newlist = [] for a in x: b = re_offers.findall(a.get('text')) if b: print a.get('id') mylist.append(a.get('id')) w = list(db.retweets.find( { 'id' : a.get('id') } )) if w: mydict = {'id' : a.get('id'), 'rt_count' : w[0].get('rt_count'), 'text' : a.get('text'), 'terms' : b} else: mydict = {'id' : a.get('id'), 'rt_count' : 0, 'text' : a.get('text'), 'terms' : b} track.insert(mydict)

This is much less complicated than it seems. And it also brings us to our final step--integrating all our queries into a REST-ful API.

Data Serving

For this, mulitple web-frameworks are available. The ones we did consider were Flask, Django and Bottle.

Weighing the pros and cons of every framework can be tedious. I did find this awesome presentation on slideshare though, that succinctly summarizes each framework. You can go through it here.

We finally settled on Bottle as our choice of framework. The reasons are simple. Bottle is monolithic, i.e., it uses the one-file approach. For small applications, this makes for code that is easier to read and maintainable.

Some sample web address routes are shown here:

#show all tracked accounts id_legend = {57947109 : 'Flipkart', 183093247: 'HomeShop18', 89443197: 'Myntra', 431336956: 'Jabong'} @route('/ids') def get_ids(): result = json.dumps(id_legend) return result

#show all user mentions for a particular account @route('/user_mentions') def user_mention(): m = request.query.id ac_id = int(m) t = list(tweets.find({'created_at': { '$gte': fourteen_days_ago }, 'retweeted': 'no', 'user_id': { '$ne': ac_id} })) a = len(t) mylist = [] for i in t: mylist.append({i.get('user_id'): i.get('id')}) x = { 'num_of_mentions': a, 'mentions_details': mylist } result = json.dumps(x) return result

This is how the DataWeave Social API came into being. I had a great time doing this, with special credits to Sanket, Mandar and Murthy for all the help that they gave me for this. That's all for now, folks!

[dataweave]

Web Scraping at Scale: Python vs Go

[The author of this post is Jyotiska. Jyotiska is a Data Engineer at DataWeave. He is a Python charmer and claims to have measured the size of big data, with a mere ruler. He posted this post originally on his personal blog. He has reposted it here with minor edits.]

Scaling is a common challenge when you have to deal with a lot of data everyday. This is one thing we face at DataWeave a lot and have managed to tackle it fairly successfully. Often there is no one best solution. You have to keep looking and testing to find out what suits you the best. One thing we do a lot in DW, is crawl and extract data from HTML pages to get meaningful information out of them. And by a lot, I mean in millions. Speed becomes an important factor here. For instance, if your script is taking one tenth of a second to process a single HTML page, it is not good enough as this will turn out to be a huge bottleneck when you are trying to scale your system.

We use Python to take care of all our day to day operations, and things are going smooth. But as I mentioned, there is no best solution, and we need to keep exploring. Thanks to Murthy, I became curious in Go. I knew about Go for a long time, probably since it came out but I never paid much attention to it. Last weekend, I started playing around with Go and liked it immediately. It seemed like a perfect marriage between C and Python (probably C++ too). Then I decided to do a small experiment. I wanted to see how Go performs against Python for scraping webpages at scale. The Regular Expression package of Go is as good as Python, so I didn't face much problem building a basic parser. The task of the parser is: I will feed it an apparel product webpage from an Indian online shopping website and it will give me the 'Title', 'MRP' (price), and 'Fabric' information of the apparel and the product 'Thumbnail URL'.The parser function looked something like this:

func parse(filename string) { bs, err := ioutil.ReadFile(filename) if err != nil { fmt.Println(err) return } data := string(bs) var title = regexp.MustCompile(`(?i)(?s)<div class="[^<]*?prd-brand-detail">(.*?)<div class="mid-row mb5 full-width"`) var mrp = regexp.MustCompile(`(?i)(?s)\{"simple_price":(.*?),"simple_special_price":(.*?)\}`) var thumbnail = regexp.MustCompile(`(?i)(?s)<meta property="og:image" content="(.*?)"`) var fabric = regexp.MustCompile(`(?i)(?s)<td>Fabric</td>\s*<td[^>]*?>(.*?)</td>`) titleString := HTML(title.FindStringSubmatch(data)[1]) fabricString := fabric.FindStringSubmatch(data)[1] mrpString := mrp.FindStringSubmatch(data)[1] thumbnailString := thumbnail.FindStringSubmatch(data)[1] fmt.Println(filename, titleString, fabricString, mrpString, thumbnailString) } def parse(filename):

htmlfile = open(filename).read() titleregex = re.compile('''<div class="[^<]*?prd-brand-detail">(.*?)<div class="mid-row mb5 full-width"''', re.S|re.I) fabricregex = re.compile('''\{"simple_price":(.*?),"simple_special_price":(.*?)\}''', re.S|re.I) mrpregex = re.compile('''<meta property="og:image" content="(.*?)"''', re.S|re.I) thumbnailregex = re.compile('''Fabric</td>\s*<td[^>]*?>(.*?)</td>''', re.S|re.I) result = re.search(titleregex, htmlfile) title = result.group(1) result = re.search(fabricregex, htmlfile) fabric = result.group(1) result = re.search(mrpregex, htmlfile) mrp = result.group(1) result = re.search(thumbnailregex, htmlfile) thumbnail = result.group(1) print filename, title, fabric, mrp, thumbnail

Nothing fancy, just 4 simple regexes. My testbed is a Mid-2012 MacBook Pro with 2.5 GHz i5 processor and 4 gigs of RAM. Now, I had two options. First, run the script with 100 HTML files sequentially and monitor the running time. Or, I could create 100 goroutines and give each of them a single HTML file and let them handle its own parsing using Go’s built in concurrency feature. I decided to do both. Also, I made a Python script which does a similar thing, to compare its performance against the Go script. Each experiment was performed 3-4 times and the best running time was taken into account. Following is the result I received:

Number of HTML Files Go Concurrent (in sec) Go Sequential (in sec) Python (in sec) 100 0.39 0.40 0.95 200 0.83 0.80 1.93 500 2.22 2.18 4.78 1000 4.68 10.46 14.58 2000 10.62 20.87 30.51 5000 21.47 43 71.55 10000 42 79 157.78 20000 84 174 313

It is obvious that using Go beats Python in all the experiments and more than 2x speed in every cases. What is interesting, using concurrency results in much faster performance than processing the HTML files sequentially. Possibly, deploying 500-1000 goroutines together actually speeds up the execution process as each goroutine can work without coming in each other's way. However, I was able to deploy maximum 5000 goroutines at once. My machine could not handle more than that. I believe using powerful a CPU will let you process the files faster than what I have done. This begs for more experiments and benchmarking, something that I absolutely love to do! The following is a graphical view of the same experiment: If the differences are not visible properly, perhaps this one will be helpful:  

If you are not satisfied yet, keep reading! I was able to get Murthy intrigued into this. He started playing with the Python code and kept optimizing it. He felt that compiling regexes every time the loop is executed adds overheads and takes unnecessary time. Globally compiled and initialized regexes will be a better option. He tried out two experiments, one with plain CPython and the other with PyPy which has JIT compiler for Python. Following is the result he observed. He tested this on a CentOS server with Python 2.6 running, 8 cores and 16 gigs of RAM.

Test Case (on 1000 HTML files) Execution Time (in secs) CPython (without function calls, regexes compiled globally) 16.091 CPython (with function calls, regexes compiled globally) 20.483 CPython (with function calls, globally compiled regexes passed as function parameters) 16.898 CPython (with function calls, regexes compiled locally inside function) 17.582 PyPy (with function calls, globally compiled regexes passed as function parameters) 4.510 PyPy (without function calls) 3.567 CPython + re2 (with function calls, globally compiled regexes passed as function parameters) 1.020 CPython + re2 (without function calls) 0.946

Not bad at all! PyPy does it under 4 seconds. Although the CPython result is same as mine. Go provides a runtime parameter GOMAXPROCS which can be passed as a parameter while running the go code. The variable sets and limits the number of OS threads that can execute user-level Go code simultaneously. I believe that by default the limit is set as 1. But to parallelize the process using multiple threads I set it as 4. This is the result I got.

Number of HTML Files Go Concurrent Single Thread Go Concurrent 4 Threads 100 0.397 0.196 200 0.827 0.396 500 2.22 0.988 1000 4.68 1.98 2000 8.27 4.03

So, 1000 HTML files under 2 seconds and 2000 files in around 4 seconds. As you can observe from the results, using multiple threads gives you 2x more gain than the previous result, more than 4x gain over sequential processing result and more than 7x gain from CPython result. However, I was not able to run 5000 goroutines concurrently as my OS did not permit. Still, if you can run your job in batches, the advantages will be good enough. Google if you are reading this, can you invite me over to your Mountain View office so that I can run this on your 16000 core machine?

[dataweave]

India's Air Quality in numbers and how it affects us all

India has the most number of deaths by chronic respiratory diseases per 100,000. Yes, you heard that right. Worse than Zambia, Uganda and Pakistan, Mozambique, Ethiopia or Somalia.

A common measure of air pollution is the mean annual concentration of fine suspended particles of less than 10 microns in diameters in the air. 10 microns is about 100 times thinner than a human hair. The size of such particles in the air is directly linked to their potential for causing health problems. Particles less than 10 micrometers in diameter, pose the greatest problems because they can get deep into our lungs, and may even get into our bloodstream.

The world average of PM10 (Particulate Matter up to 10 micrometers in size) hovers around 71 µg/m3 but the quality might range from 21 to 142 µg/m3. Here is an image from an 8 year study of air quality in 1100 urban areas released by the WHO.

The contrast of air quality between the developing and the developed world is visibly stark. People living in the urban areas in and surrounding India, China, Mexico and large parts of the middle east, Africa and few south american countries are considerably economically backward than their counterparts in the US and Europe, who enjoy cleaner air.

According to WHO numbers from 2008, the number of premature deaths attributable to urban outdoor air pollution is estimated to 1.34 million worldwide. It’s no surprise where those deaths are coming from.

Source:http://gamapserver.who.int/gho/interactive_charts/ncd/mortality/chronic_respiratory_diseases/atlas.html

India is on top of this list with 175 (Male) and 125 (Female) deaths per 100,000 (age standardised). Other regions include large parts of China, Africa and south east asia.

The WHO claims that of the 1.34 million deaths worldwide 1.09 million deaths could be avoided if the mean annual Air Quality Guideline values of PM10=20μg/m3 along with (the smaller particulate matter 2.5 micrometers) PM2.5=10 μg/m3 are implemented.

The DPCC (Delhi Pollution Control Committee) has near real time Air Pollution Level data on their site. The clear break up of the data along with the prescribed limit helps us get a clear sense of the pollution in the air. It also lets us monitor this data by localities.

Numbers might lack the intimacy and urgency of images but they help us see things clearly and take measurable actions. The availability of data on the DPCC site is a step in the right direction. This can help focus on making positive changes to improve the air quality in India.

[dataweave]

10 Free Data Visualization Tools

“Visualizations act as a campfire around which we gather to tell stories.” Al Shalloway (2011)

Data is most effective when visualized. Why should we suffer the dull monotony of a sheet of numbers when the same data can be visualized in colorful graphs brimming with vitality? Over the past few years a number of useful and free visualization tools have made it more and more easy for us to consume data in meaningful and insightful ways. Here is a list of 10 such free data visualization tools with examples:

1. D3

D3.js is a JavaScript library for visualizing simple and complex data. D3 helps visualize data using HTML, SVG and CSS. D3’s strength is its ability to utilize the full capabilities of modern browsers without tying itself to a proprietary framework. D3 combines powerful visualization components and a data-driven approach to DOM manipulation. We’re huge fans of D3 at DataWeave and rely on the extensive list of resources on its website: http://d3js.org/

[ A dashboard on PriceWeave made using D3]

2. Tableau Public

Tableau aims to make databases and spreadsheets understandable to ordinary people. It has hundreds of visualization types, such as maps, bar and line charts, lists, heat maps and more. Two important features of Tableau are its built-in mapping which automatically geocodes down to the city and zip code level and its built-in date functionality, which lets readers filter to a time period or drill down from months to days to hours. You can even combine views into a dashboard to show different sides of the same story.

Some features on Tableau Public are reserved for paid users. More about Tableau Public at:  https://www.tableausoftware.com/products/public

3. Many Eyes:

Many Eyes is probably the simplest tool to start using. It combines graphical analysis with community, encouraging users to upload, share and discuss information. It is very well documented and includes valuable suggestions on when to use what kind of visual data representation.

Many Eyes includes more than a dozen output options -- from charts, graphics and word clouds to treemaps, plots, network diagrams and some limited geographic maps. The only seeming drawback of Many Eyes is that both visualizations and data sets are public on the Many Eyes site and can be easily downloaded, shared, reposted and commented upon by others. Access Many Eyes here. 

  4. Processing:

Processing is a stand out interactive visualizations tool. It enables you to write much simpler code which is in turn compiled into Java. There is also a Processing.js project to make it easier for websites to use Processing without Java applets, plus a port to Objective-C so you can use it on iOS. It is a desktop application, but can be run on all platforms.

Ex: http://www.openprocessing.org/sketch/77118

5. Google Chart Tools:

Google Chart Tools offers an API for creating Web graphics from data. Google offers a comparison of data size, page load, skills needed and other factors to help decide which option to use.The visualization API includes various types of charts, maps, tables and other options and helps pull  data in from a Google spreadsheet.

6. Weave

Weave is a web-based visualization platform designed to enable visualization of any available data by anyone for any purpose. It is an application development platform supporting multiple levels of users – novice to advanced – as well as the ability to integrate, analyze and visualize data at “nested” levels of geography, and to disseminate the results in a web page. Learn more about Weave at http://oicweave.org/

Here is a visualization on trees in Boston made using Weave: http://demo.oicweave.org/weave.html?file=demo_Boston_Trees_meters_2011.weave

7. Datawrapper:

DataWrapper is an open source tool that covers the entire cycle of cleaning, visualizing and publishing data.  Visualizations using Datawrapper have appeared in high profile publications such as Washington Post, Der Spiegel and The Guardian. You can find more details on the DataWrapper site: http://datawrapper.de/

8. Chartbuilder

Chartbuilder began as an inhouse project of the design loving team of Quartz who were frustrated with the sub standard charts they found online. Chartbuilder works best when you have to turn out beautiful and aesthetically pleasing graphs in a short time. It is open source and the full code can be found here: https://github.com/Quartz/Chartbuilder/

9. Jolicharts

Jolicharts helps users build and share dashboards and export data. You can upload your data in JSON, CSV or XLS to Jolicharts and your data will be automatically prepared for analysis. It is free to use for up to 50 MB of calculation power and 5 datasources per dashboard. Learn more about it here: https://jolicharts.com/

10. Plotly

Plotly is a collaborative data analysis and graphing tool. It provides online graphing, analytics, a Python command line, and stats tools for individuals and collaboration, as well as scientific graphing libraries for Python, R, MATLAB, Perl, Julia, Arduino, and REST. Plotly claims to streamline one’s workflow, all in one place and comes very close to doing it. Take a look at the plotly gallery.

Source: Plotly Visual

What visualization tools do you use on your projects? How has your experience been with them? Let us know in the comments below. 

[dataweave]

2013- A year in review

Over the last year Data & Analytics has continued to grow in prominence and become an important part of businesses worldwide. Being part of this data revolution we at DataWeave have had a fiercely intense year. In this post we give you a glimpse into the year that was.

Stats:

Since Data is at the core of our business we can’t help but share our most important metrics. In 2013 we served a total of 27,081,937. That’s a whopping 500+ % increase over last year’s 4,464,310 datapoints.

This means that we served 74197 datapoints a day in 2013! That’s about one datapoint for every citizen of Liechtenstein twice a day all year. *wink*  

PriceWeave:

PriceWeave which started out as a Price Intelligence product has grown steadily into an all around Competitive Intelligence Suite. Along with upgrades to core features, PriceWeave also received important and powerful new features that make pricing, merchandising, and assortment planning decisions easier and faster than ever.  

The new PriceWeave now has Price Intelligence, Assortment Intelligence, and Gap Analysis among other indispensable competitive intelligence features. We are currently working on a bunch of exciting features for PriceWeave that we will roll out within the next few months. More on PriceWeave on the PriceWeave Blog.

Wins:

New Geographies: In 2013, we expanded our offerings to multiple geographies. We now have clients across south-east Asia and are actively receiving requests from around the world.

Multi-Language Support: We cracked the language barrier to make ourselves available for non-english speaking countries. We now serve PriceWeave in multiple languages across SEA.

Multi-country deals: With business growing at a fast pace many of our associations with our clients went international. We now have multi-country deals with some major retailers and brands.

In the News:

1. We were declared one of the 100 most promising start ups in India by NextBigWhat and featured in the top 100 start ups in India on Sutra HR.

2. Mint did a feature on us in their Indian Start ups in big data write up.

3. Economic times mentioned us in an article on big companies seeking innovation with smaller players.

4. Addicted To Success listed us as one of the top 10 exciting start ups in India

Blogs:

This year our blogs were more popular than ever. Apart from blogs about our systems and technology we branched out to investigative and exploratory blogs. Some of our most popular blogs were:

1. An onions-to-onions analysis: an investigation into the rising Onion prices in India http://blog.dataweave.in/post/61933646874/an-onions-to-onions-analysis

2. Of broken pumpkins and wasted lemonade http://blog.dataweave.in/post/65327903530/of-broken-pumpkins-and-wasted-lemonade

3.  Tweeting Bad: An analysis of the TV show Breaking Bad on twitter http://blog.dataweave.in/post/65600030714/tweeting-bad

4. Our blog post on Amazon's entry into India was republished by nextbigwhat. http://blog.dataweave.in/post/53376567061/amazon-forays-into-india-with-aggressive-discounts-on

Hackathon:

We won a prize in the Business of API Hackathon organized by Bessemer Ventures conducted at Microsoft. We built a quick prototype of 'AlertGram', an universal alerting system.

Events:

We have been actively participating in popular technology events in Bangalore and around India. We contributed and learned from community events such as TechSparks, MobiSparks, StartUp Saturday, Design Day and Droidcon, among others.

We were 'data sponsors' at Startup Weekend in February where several teams used our data and API to build their apps. We also provided our APIs for APIGee hackathon in March.

Sanket gave a talk about the DataWeave platform in BlrDroid's March meetup and spoke about the DataWeave platform's technical architecture at the Yahoo! Hack Day in Hyderabad.

Karthik was a panelist on the Big Data panel at the 'Startup Festival' and on the Omni Tech Summit on eTailing in India.

Mandar moderated a panel discussion on Big Data Analytics at the Siemens Big Data Conference in September 2013 and was a member of the program committee for two conferences in the area of big data: ICADL 2013 and BDA 2013

Between Sanket and Mandar two research papers were published at ICADL 2013.

Getting Bigger Faster!

From a six member team last year we grew to a 20 member strong team this year.

With the addition of Sravan, Abhishek, Mukesh, Mithun & Rajesh our tech team is stronger than ever!

Other additions to the team were in Q&A (Sukruth, Amar, Shiva, Lakshmi, Naveen, Harish & Aarthi) and Content & Marketing (Supreeth).

We also have firecracker Interns in Jyotiska, Navneet and Apoorv.

Thank you!

DataWeave couldn’t have grown this big, this fast, in the right direction without the help and support of awesome folks who are there to guide and  support us unconditionally. We thank Rajan Anandan, Blume Ventures, 5ideas, Times Internet, and other angels associated with us as investors and advisors.

Also invaluable to us are our users, readers of our blogs, our friends and family who drive us to build DataWeave and get us closer to our goal of making the world a tiny bit better using data.

Here’s wishing you a very Happy New Year! We look forward to a productive and data driven 2014 with you!

[dataweave]

Data Projects & Advocacy

Data is a powerful tool to ensure transparency and influence change. When visualized right, data can be a great advocacy tool. In this post we’d like to share a couple of projects & initiatives that are doing interesting things with data advocacy & visualization.  

1. The Ballot

The Ballot is a visual compendium of information about Indian democracy. Starting from August 15th, they have been posting on topics as varied as criminal records of current Lok Sabha members to how an ordinary bill becomes an act.

They visualize and comment on important facts and figures we ought to know as India prepares for the 2014 General Elections. The richly illustrated graphs and charts on The Ballot make it easy to consume relevant nuggets of information and improve our overall understanding of the Indian Democracy and its state today.

[The above image is from a post on women in all the Lok Sabhas.]

2. Visualizing Advocacy

Visualising Information for Advocacy is a book by the Tactical Tech team about how advocates and activists use visual elements in their campaigns. This guide features over 60 case studies from around the world that provide an introduction to understanding visual information and a framework for using images for influence. It has been developed from their experience on working to help campaigners and activists around the world to use information, visual representation and digital technologies in their work, over the past ten years.

Tactical Tech, the team behind Visualizing Advocacy, is an organisation dedicated to the use of information in activism. They make toolkits, films and guides, and organize trainings and events that support rights advocates and helps activists with tools and techniques to bring about positive social change.

These are but two examples from the fertile & fast evolving landscape of data advocacy. Apart from being effective tools to help us understand the world better, they are bold steps towards making the civil society more transparent and corruption free.

At DataWeave, we continue to enable users to access data for Advocacy. Two such APIs we’ve released for insights into the Indian Democracy are

1. Lok sabha and Assembly Elections Results Data (Candidate wise)  

Candidates wise result list of all contestants for Lok sabha and Assembly elections. Data can be accessed by State, House & Year .

2. Member Of Parliament Local Area Development Scheme .

Under The Member of Parliament Local Area Development Scheme (MPLADS), each MP has a choice to suggest works upto Rs. 5 Crores per annum to be taken up in his/her constituency. This data specifies the details of the allotted amount to each MP, its usage and unspent balance since 15th Lok Sabha 2009-2010.

Check out our full list of APIs here.

If you manage or know of other data advocacy projects feel free to write to us at [email protected] & we promise to do our bit and spread the word about your project. Also get in touch with us if you are looking for a particular data set that aids you in your Data Advocacy project.

About DataWeave:

DataWeave helps businesses make data-driven decisions by providing relevant actionable data. The company aggregates and organizes data from the web, such that businesses can access millions of data points through APIs, dashboards, and visualizations. To use our platform for your data needs write into us at [email protected].

[dataweave]

Tweeting Bad

There is little contest to the fact that we are in a golden age of narrative television. With shows like Madmen, The Walking dead and Deadwood counted among all time greats like The Sopranos and The Wire and the recent hits of Netflix produced House of Cards & Orange is the new black, it’s a veritable Oak forest. None stands as tall though as AMC’s Breaking Bad.

The build up to the finale was arguably the most anticipated single episode of a TV series of all time. In this blog we visualize the twitter activity running up to the finale and the days that followed. All we have left to give you is this. 

This analysis is based on a sample of 300,000 tweets narrowed in from the 1.2 million tweets that we collected by tracking Breaking Bad related keywords using the Dataweave platform.

1. Tweets/ Day:

The trendline speaks for itself. Twitter activity around Breaking Bad was most right after the finale. The run up to the finale looks quite gradual too.

2. Language

Apart from English, obviously the most tweeted-in language, a sizeable portion of the tweets came from Spanish, French, Portugese and Dutch.  The non obvious languages that showed up were Tagalog ( an Austronesian language spoken widely in the Philippines) & Estonian.

3. Common terms:

This tag cloud represents the frequency of the terms used in tweets related to breaking bad. Not surprisingly breaking bad & goodbyebreakingbad turn up as widely used keywords as well as BrBaQuotes that though less used is a strong topical hashtag. Mike seems to be the most popular tweeted about character followed by Jesse, Marie, Hank & Lydia. Skyler, Saul, Todd & Gomez follow. Also noticeable is the spoiler keyword, probably calls to fellow fans to tread lightly, lest they run into spoilers before they watched the finale.

As ardent fans of the show here at Dataweave, we conceived this post as a small way of saying goodbye to our favourite show. All we can say is we did it for us and we liked it.

About Dataweave (Say our name!):

DataWeave helps businesses make data-driven decisions by providing relevant actionable data. The company aggregates and organizes data from the web, such that businesses can access millions of data points through APIs, dashboards, and visualizations. To use our platform for your data needs write into us at [email protected]

[dataweave]

Of broken pumpkins and wasted lemonade

[The idea for this post was given by Mahesh B. L., who is a friend of DataWeave. We are going to have some fun with data below.]

Ayudha Puje (ಆಯುಧ ಪೂಜೆ) or Ayudha Puja marks the end of Navaratri. It is the day before Dasara. Ayudha Puje is essentially the worship of implements (or tools/weapons) that we use. While this is practiced in all southern states, since Dasara is Karnataka's nADa habba (ನಾಡ ಹಬ್ಬ) or "the festival of the land" it is done so with added fervour in Karnataka. At least  it does seem so if you see the roads for the next few days!

As part of Ayudha Puje, everybody washes and decorates their vehicles (well, they are our major weapons in more than one sense) and worships them by squashing a suitable amount of nice and juicy lemons with great vengeance. Ash gourds (Winter Melons) are also shattered with furious anger.

[Image Courtesy: Wikipedia] See above a small sample of unsuspecting victims. Alas, they will soon be squashed beyond recognition!

Just how many lemons got squashed on the last Ayudha Puje in and around Bangalore? We dug up some data from a few sources (Praja, Bangalore City Traffic Police, RTO, a Hindu article), and came with a quick and dirty estimate on the number of vehicles in Bangalore. We estimate that there are about 55 lac vehicles in and around Bangalore. You can download the data we used and our approximations here.

So, assuming one lemon per wheel, upwards of 13 million lemons got squashed on October 12, 2013 that could otherwise have been put to some good use such as preventing cauliflowers from turning brown, or serving Gin Fizz to an entire country. (Of course, we know that not everyone performs Ayudha Puje. But let's not be insensitive to the plight of our victims by digressing.)

We don't have an estimate on how many Ash Gourds were broken, but we are sure the quantity would have been at least enough to serve tasty Halwa to every person in Karnataka.

Do you want to do some serious analysis on Lemons and Pumpkins yourself? Take a look at our Commodity Prices API. Register to get an API key and start using it. Like this:

http://api.dataweave.in/v1/commodities/findByCommodity/?api_key=b20a79e582ee4953ceccf41ac28aa08d&commodity=Lemon&start_date=20131009&end_date=20131015&page=1&per_page=10

DataWeave helps businesses make data-driven decisions by providing relevant actionable data. The company aggregates and organizes data from the web, such that businesses can access millions of data points through APIs, dashboards, and visualizations.

[dataweave]

DataWeave's Telecom Recharge Plans API

Several months ago we released our Telecom recharge plans API. It soon turned out to be one of our more popular APIs, with some of the leading online recharge portals using it extensively. (So, the next time you recharge your phone, remember us :))

In this post, we’ll talk in detail about the genesis of this API and the problem it is solving.

Before that---and since we are into the business of building data products--some data points.

As you can see, most mobile phones in India are prepaid. That is to say, there is a huge prepaid mobile recharge market. Just how big is this market?

The above infographic is based on a recent report by Avendus [pdf]. Let's focus on the online prepaid recharge market. Some facts:

There are around 11 companies that provide an online prepaid recharge service. Here’s the list: mobikwik, rechargeitnow, paytm, freecharge, justrechargeit, easymobilerecharge, indiamobilerecharge, rechargeguru, onestoprecharge, ezrecharge, anytimerecharge

RechargeItNow seems to be the biggest player. As of August 2013, they claimed an annual transactions worth INR 6 billion, with over 100000 recharges per day pan India.

PayTM, Freecharge, and Mobikwik seem to be the other big players. Freecharge claimed recharge volumes of 40000/day in June 2012 (~ INR 2 billion worth of transactions), and they have been growing steadily.

Telcos offer a commission of approximately 3% to third party recharge portals. So, it means there is an opportunity worth about 4 bn as of today.

Despite the Internet penetration in India being around 11%, only about 1% of mobile prepaid recharges happen online. This goes to show the huge opportunity that lies untapped! 

It also goes to show why there are so many players entering this space. It’s only going to get crowded more.

What does all this have to do with DataWeave? Let's talk about the scale of the “data problem” that we are dealing with here. Some numbers that give an estimate on this.

There are 13 cellular service providers in India. Here’s the list: Aircel Cellular Ltd, Aircel Limited, Bharti Airtel, BSNL, Dishnet Wireless, IDEA (operates as Idea ABTL & Spice in different states), Loop Mobile, MTNL, Reliable Internet, Reliance Telecom, Uninor, Videocon, and Vodafone. There are 22 circles in India. (Not every service provider has operations in every circle.)

Find below the number of recharge plans we have in our database for various operators.

In fact, you can see that between the last week and today, we have added about 300 new plans (including plans for a new operator).

The number of plans varies across operators. Vodafone, for instance, gives its users a huge number of options.

The plans vary based on factors such as: denomination, recharge value, recharge talktime, recharge validity, plan type (voice/data), and of course, circle as well as the operator.

For a third party recharge service provider, the below are a daily pain point:

plans become invalid on a regular basis

new plans are added on a regular basis

the features associated with a plan change (e.g, a ‘xx mins free talk time’ plan becomes ‘unlimited validity’ or something else)

We see that 10s of plans become invalid (and new ones introduced) every day. All third party recharge portals lose significant amount of money on a daily basis because: they might not have information about all the plans and they might be displaying invalid plans.

DataWeave’s Telecom Recharge Plans API solves this problem. This is how you use the API.

Sample API Request

http://api.dataweave.in/v1/telecom_data/listByCircle/?api_key=b20a79e582ee4953ceccf41ac28aa08d&operator=Airtel&circle=Karnataka&page=1&per_page=10

Sample API Output

We aggregate plans from the various cellular service providers across all circles in India on a daily basis. One of our customers once mentioned that earlier they used to aggregate this data manually, and it used to take them about a month to do this. With our API, we have reduced the refresh cycle to one day. 

In addition, now that this is process is automated, they can be confident that the data they present to their customers is almost always complete as well as accurate.

Want to try it out for your business? Talk to us! If you are a developer who wants to use this or any other APIs, we let you use them for free. Just sign up and get your API key.

DataWeave helps businesses make data-driven decisions by providing relevant actionable data. The company aggregates and organizes data from the web, such that businesses can access millions of data points through APIs, dashboards, and visualizations.