#twimpact

[twimpact]

Our TWIMPACT real-time ad optimization pitch at DataDays 2012.

[mikiobraun]

My talk "TWIMPACT: On Real-time Twitter Analysis" given at the Apache Hadoop Get Together in Berlin on April 18, 2012.

[twimpact]

Here is a nice demo Leo put together. You see a timelapse video of places people are talking about on Twitter during March 2011. Shown is the average activity over the last hour. On March 11, there was that huge earthquake in Japan which dwarves all other locations for quite some time.

For this demo, we've extracted place names from about 16000 cities from open street map data (about 500k variations all in all) and then matched these names in the tweets (i.e. we're not using the geolocation but get the locations from the tweet texts themselves). The resulting stream is run through our analysis database to compute the location trends online.

We're currently putting together a real-time version of this for our website.

[twimpact]

Not the real TWIMPACT...

You might have stumbled upon the paper "Can Tweets Predict Citations? Metrics of Social Impact Based on Twitter and Correlation with Traditional Metrics of Scientific Impact" by Gunther Eysenbach in which a measure called "twimpact" is proposed. Unfortunately, this work has nothing to do with us, and as it seems, [his paper also contains some methodological flaws](http://scholarlykitchen.sspnet.org/2012/01/04/tweets-and-our-obsession-with-alt-metrics/). One can only wonder how he wasn't aware of the "other" twimpact, a small search on twitter would have been enough to reveal that the name already exists... .

[twimpact]

Preview of our NIPS Poster. Come to our demo on Wednesday night at #NIPS2011!

[twimpact]

That's what I call stable memory use ;)

[twimpact]

Real-time seems to be the next big thing in big data. Map-Reduced has shown how to perform big analyses on huge data sets in parallel, and the next challenge seems to be to find a similar kind of approach to real-time.

When you look around the web, there are two major approaches out there which try to building something which can scale to deal with Twitter-firehose-scale amounts of data. One is starting with a MapReduce framework like Hadoop and somehow finagle real-time or at least streaming capabilities on it. The other approach starts with some event-driven “streaming” computing architecture and makes it scale on cluster.

These are interesting and very cool projects, however from our own experience with retweet analysis at TWIMPACT, I get the feeling that both approaches fall short of providing a definitive answer.

In short: One does not simply scale into real-time.

Read the whole post on Mikio's blog

[twimpact]

Google suspends real-time search

A lot of things happened that last week. Google opened their new service Google+ to a small set of people (don't bother getting an invite now, they seem to have closed down the set of users for now), and also changed their layout for the search page and their calendar.

As a "side effect", Google real-time search is apparently gone. Mashable confirms this in an article and gets some insight. Apparently, the agreement between Twitter and Google has ended, and Google did not extend it for now, with Google planning to focus on their own service Google+. Google states that it's still crawling the publicly accessible pages from Twitter, which is of course hardly the same as having access to the Twitter stream.

I'm actually not that surprised as Google's real-time search always felt like a half-hearted attempt at real-time search. I think the challenges are somewhat different between real-time search and web search.

Web pages are relatively static, both in content and relevancy. In real-time search, however, the information is changing very rapidly and has a high probablity of becoming obsolete quickly. This leads to quite different technological challenges, so that it's probably hard to fit real-time search to Google's existing infrastructure easily. The amount of cacheable information is also very limited.

Real-time search requires real-time relevancy measures. It doesn't really make sense to just show the most recent messages matching your query. For popular events, there might be thousands of hits, swamping any really relevant hit after a few minutes or even seconds.

Displaying a list of all hits doesn't make sense. Often, you have many near identical hits, and some form of aggregation would really be useful. Google is doing something like this for news, but news lives on a much slower timescale than real-time search.

Naturally, these are also topics we're very much interested in at Twimpact. For example, our retweet based trending and user Twimpact score is a good starting point to get a better estimate of relevancy (This is currently demoed at our Japanese trending site). Currently, we're also moving to an infrastructure which does most of the analysis in memory to deal with the real-time requirements. This allows us to process literally thousand of messages in real-time with relatively modes hardware requirements. You can get a glimpse on beta.twimpact.com.

As far as I remember Google had some plans of incorporating, for example, the social graph of a user to refine the search results, but I don't know how far that went. Let's see whether they take the time to create something better. In the interim, a site like Topsy gives you a more comprehensive real-time search feature than Twitter's own search.

^MB

Some more links on the later state of Google's real-time search:

http://mashable.com/2010/08/26/google-improves-real-time-search/

http://searchenginewatch.com/article/2066677/How-Twitter-Data-Can-Improve-Real-Time-Search