TWIMPACT Dev Blog

Development blog for TWIMPACT beta.twimpact.com Members Mikio Braun Leo Jugel

[twimpact]

Video for Mikio's talk given at this year's Berlin Buzzword conference. The title of the talk is "Beyond scaling: real-time event analysis with stream mining".

[twimpact]

$YHOO dominates $AAPL for once over tumblr acquisition. Live at http://play.streamdrill.com/vis/

[mikiobraun]

Ben Lorica: Scalable streaming analytics using a single-server

During our Bay Area trip two weeks ago, we had the chance to chat with Ben Lorica, Chief Data Scientist of O’Reilly Media at the Ritual Roaster Coffee shop in the Mission in San Francisco. It turns out, what we did very much resonated with Ben who had recently become interested in alternatives to scaling, and single-server systems. He was kind enough to write this great blog post about streamdrill.

[mikiobraun]

I review six seminal Google Big Data papers and discuss what hints they give to go beyond MapReduce.

[mikiobraun]

[mikiobraun]

[mikiobraun]

In this post, I explain how streamdrill helps you solve the top-k problem for event streams in real-time.

[twimpact]

Leo has been busy packaging the streamdrill demo into an exectuable jar. Download your version now!

[twimpact]

The last few weeks we've been working on extracting the real-time analysis engine behind TWIMPACT's social media demos. The result is streamdrill which we've just launched in a beta version.

Streamdrill is a real-time event processing engine which solves the top-k problem. You can pipe in up to several 10k events per second and instantaneously query the most active entries over the past minute, hour, day, or week.

If you're interested, we'll spin up a small instance for you to play with.

We already have clients in Python and Scala available here.

[twimpact]

Our TWIMPACT real-time ad optimization pitch at DataDays 2012.

[nosql]

Mikio L. Braun:

Many of the tools like Hadoop or NoSQL data bases are quite new and are still exploring concepts and ways to describe operations well. It’s not like the interface has been honed and polished for years to converge to a sweet spot. For example, secondary indices have been missing from Cassandra for quite some time. Likewise, whether features are added or not is more driven by whether it’s technically feasible than whether it’d make sense or not. But this often means that you are forced to model your problems in ways which might be inflexible and not suited to the problem at hand. (Of course, this is not special to Big Data. Implementing neural networks on a SQL database might feasible, but is probably also not the most practical way to do it.)

While an interesting read I’m not sure I really got it—my understanding is that the author’s advise is that disregarding your backend storage or Big Data architecture, you should always think of your data and processing tools in terms of higher concepts as data structures, operations on data structures, and processing algorithms.

Original title and link: Levels of Abstractions in Big Data (NoSQL database©myNoSQL)

[twimpact]

At TWIMPACT, we're a big fan of stream mining algorithms to do real-time event processing. One of their interesting features is that they let you trade exactness for computation time. However, people often ask us why that won't be a problem. In this post, I collect 4 reasons why you don't want your real-time big data analytics to be exact.

[twimpact]

We've launched a new demo based on our retweet analysis of 2011. The interface is similar to Google Trends and lets you search and compare keyword terms. Click on the above picture to go to trends.twimpact.com. The data is based on the 300000 most active retweets for each day based on the public Twitter feed, which is about 4.3 million tweets per day. For more information, have a look at this blog post.

[mikiobraun]

Some impressions from the Big Data and Social Media meeting I attended a month ago in Munich.

[mikiobraun]

Slides for my talk “Scalability Challenges in Big Data Science” held at BerlinBuzzwords 2012. I give an overview of different scalability challenges when you try to scale data science and machine learning methods. I talk about MapReduce, large scale SVM training, stream mining, real-time and what we did for Twitter.

[mikiobraun]

My talk “TWIMPACT: On Real-time Twitter Analysis” given at the Apache Hadoop Get Together in Berlin on April 18, 2012.

[mikiobraun]

Talk given at the Apache Hadoop Get Together, Berlin, on April 18, 2012.