Hopper Storm Hackathon
On Saturday, I attended the Hopper Storm Hackathon at Notman House. My friend Victor from work asked me to go and hack for a few hours to learn a bit more and find out some more about what Hopper was and maybe win a prize.
Hopper (whose website I cannot find) offers a product called Storm that follows a map/reduce sort of process. Instead of running in batch like hadoop, Storm runs from a feed- most notably a social media feed like twitters firehose or any other massive data feed. For this hackathon we had a few feeds available to us, Victor and I picked up a third guy (George) and together we decided to pick something we knew we could finish in a day. We decided to visualize geotagged tweets ontop of google maps with the option of following tweets with certain hashtags. Not exactly a difficult project, but we knew we could get something running end to end in the timeframe we had.
The code provided by the Hopper team was all java examples (the Hopper team were all writing Scala) so we did our processing on their cluster then saved our results to a riak database provided by hopper (to which I must add, wow riak is easy...incredibly simple key/value store to connect to and query). Our frontend was some javascript to make an AJAX call to a python REST endpoint we were running that queried the riak database and just dumps the results back to the frontend. You can check the code on my github (its a real mess, but it was a hackathon- cut me some slack), but the final result was pretty cool.
We were getting down to the end and trying to think of something to add in the last few minutes. There was a sales rep there from a company called Semantria that does sentiment analysis and natural language processing for big data. We thought a cool idea would have been to add sentiment analysis to tweets showing up in the same area and try to do some more classification and grouping to provide more info on the frontend than just tweets in the same location. Unfortunately we just didnt have time, but that would have been pretty damn cool.
Unfortunately we didn't win the hackathon (not that I'm surprised, we didn't exactly innovate). The winners did a pretty cool analysis of the twitter feed using finangle and redis to find youtube videos linked in tweets to find videos that are being shared by users and deciding whether they are trending. They managed to pick out some seemingly popular youtube videos (several popular k-pop songs) that had been linked a bunch. It would have been pretty cool as well to see some time analysis too- since anything linked 12 hours ahead or behind us will be big in Korea or Southeast Asia.
Etiher way it was a fun day! Definitely inspirational.