Andrew .R. Patterson

Just a blog about ML, IR, and all related projects!

[andrewrpatterson]

What have I been doing?

Since graduating from university roughly 6 months ago, I’ve been splitting myself between my own projects, job hunting, and flipping burgers!

Like many projects before this one, my current project circles around social media analytics and the tools we need for the task.

The online reputation of ‘Still Game’

One of the first projects was a simple 1 hour long project to discover the online popularity of the new Still Game season after its 9-year hiatus. I only tracked the shows mentions over the first and second episode and, from first inspection, discovered that roughly 30% of all tweets in Scotland mentioned still game when the show was aired. Incredible!

Distributed processing pipeline

Another project, and one still in development, is a distributed processing chain for classifying and clustering tweets. The most novel aspect of this project for me is using Apache Kafka as a message broker between processing nodes, which I have discovered is particularly useful for splitting up development tasks into more atomic components.

A better crawler

In my previous projects I have written a twitter crawler specifically for a task and in some cases built a hybrid crawler that uses both the streaming and search endpoints. At some point I discovered that doing it this way took up a great deal of my time; I must spend hours, even days, on building crawler functionality for almost every project!

So I started building a different crawler, one that, given valid credentials, can connect to and disconnect from the stream, run ad-hoc queries, and give the user statistics about the application, such as number of tweets per second.

While this application is still in development, I believe this will be the game-changing application for me, especially when run as a service.

[andrewrpatterson]

Shannon Entropy for a cluster of tweets

Introduction

The LSH-based event-detection approach that I was using recently became too slow to output events (where we take an event as as having an entropy>=3.5) as the entropy calculation was done in PostgreSQL.

I’m not knocking Postgres by the way, actually I find Postgres to be perhaps the best freely available database out there, but an entropy calculation in SQL, unfortunately involved lots of cartesian join, which at the scale of of Twitter, is frankly unfeasible.

So to remove this abhorrent bottleneck, I’ve moved this portion of the process to a Java application. My problem however is that there isn’t any readily available code for Shannon entropy in java, for working with clusters of documents, or anywhere that tells you if you have to change the calculation.

Background

In this context, Shannon entropy is a measure of diversity within a set of documents; the higher the entropy, the more diverse a set of documents is.

Pretty simple!

The solution

If working with clusters, simply add every token, of every document to a list then pass it in.

[andrewrpatterson]

Trends on Twitter

Overnight, from about 1am until 10am this morning, I let my trend detection application to run uninterrupted and discovered a few things which I’m quite surprised it did, especially for how simple the application actually is. To detect a trend, I must define what a trend is first and I chose the simplest definition: a trend occurs when it is mentioned by many.

The Application

Firstly, the application is an apache storm application I created which draws its data from the Twitter Streaming API, filtered by location(glasgow) and language (english). The tweet text is then passed along to an EntityExtractor bolt which uses a rudimentary stop word list to remove function words and emits each token to the TrendCounter bolt.

The trendCounter bolt keeps a hash map of all tokens to their respective occurrence counts. if an occurrence count rises above a threshold it them emits them to the next bolt, which just outputs the token as a trending token.

Results

Of course the results show an exceptional amount of stop words such as “it” and “I’m” and also shows a lots of word variance such as “it” and “it’s”, but it has detected some trends too. Last night in Glasgow there was thunder storm going on and the application managed to detect the terms “thunder”, “lightening”, and “rain”.

Conclusion

Even though the system used is extremely simple and naive  in its assumptions, the underlying principle still holds true. That being said, it was clear that some of the trends discovered were related and probably occurred in the same tweets together. 

The next steps with this approach is to take a more vigorous stop word/stemming approach to clear up trends and to remove the user defined threshold value for what determines a trend. Instead, the application should take a classification based approach where a terms are determined to be trending or non-trending based on previous trend patterns.

**if any one wants the code just drop me an ask!*

[andrewrpatterson]

Jenkins...

So thats the trouble with Jenkins fixed, for now at least. Turns out the ssh for GitHub I set up shouldn’t have had a passphrase associated with it. 

[andrewrpatterson]

A quick update

So since my dad setup my Fedora server for me (because Im far, far to lazy) I’ve been spending most of my time streamlining the development process.

Why?

Well the first few things I set up was the YUM installations of CouchDB/Postgres and allowing remote connections to the server, including SSH. This allowed me to start crawling tweets but quickly I realised tat copying a jar file using scp was quite a laborious task.

So I went and setup Jenkins. Now I remember I used Jenkins in uni and of course everyone complained about it, but that was mostly because of the way the school’s IT was setup; slow, and the need to create multiple ssh tunnels smh.

The jenkins on my machine is much easier to use however. The only downside is trying to get it to work with github! But we’ll see if I can fix it tonight. 

The main objective of this task is for me to be able to allow continuous delivery of my code to the fedora machine by simply pushing to github. If this works well  then I will have the sleekest build pipeline every *laughs through nose*.

In other news, I finally convnced Virgin Media to give me the SuperHub2 to replace my ageing SuperHub. I’m hoping that it will imrove the connection quality to my TV and to my other devices which I’ll find out on Friday. 

Strangely though, the tech support said there was no difference between the two versions but sales said there was; Obviously I know enough about this stuff to agree with the sales team but i find it shocking nonetheless. #sellouts

[andrewrpatterson]

A quick review of IntelliJ IDEA

I’ve been taken more of an interest in using this IDE since my project for my financial engineering systems course(run by Glasgow University and JPMorgan) was to be completed using IntelliJ.

What I initially liked about it was the test coverage feature. Having never used this kind of feature before I think its preettty cool. It nicely quantifies my testing into a simple measurement which is really useful for TDD.

That being said, I actually found it fairly difficult to get access to that feature as most of my projects are built using Maven/Ant/Gradle and I really expected to click a single button on my tests and it would like auto-detect it. That’s software I guess...

Anyways, It also has built in support for github’s repositories and even Gist. So I would definitely recommend it to anyone thinking about using it.

[andrewrpatterson]

K-Means Clustering code

Since I finished uni seevral weeks ago, I have been filling my free time with completing the areas of my dissertation which I couldn’t complete (this was because I either ran out of time or the frameworks I was using were not up to scratch).

I developed a clustering package as part of a new framework I’m developing which would be better suited to the original application. As part of the package I have implemented a K-Means clustering algorithm to be used.

Why didn’t I just copy code?

Most frameworks that are available for this purpose do not work in an online setting as well. For example, Weka’s implementation does not allow the creation of clusters after training or to decide some threshold by which an instance can be a member of a cluster. Thats why the approach I had taken uses threshold distances and for K to be a number used in training.

The other issue with many existing implementations is that they require more space than required to run and doesn’t take full advantage of the object oriented paradigm of JAVA.

Use Case

I intend to use this for the online clustering approach I had taken in my dissertation.

My dissertation approach trained an LDA topic model and used topic inference to convert all unseen tweets into topic vectors and then clustered the topic vectors using cosine similarity.

This approach will instead train model using K-Means and then also have the facility of adding unseen tweets to the model.

The code

You may notice in the code that it shares some of the functionality provided by Weka, namely the Instance interface and Instances class. At first, during experimentation, I elected to use double[] arrays as vectors but that approach still requires me, the lazy programmer, to develop the supporting components. So instead I chose to use Weka's functionality and speed up development. The approach means that effectively I have created my own implementation side-by-side with Weka's

The code is still being tested and will release in ~48hrs.

Future Considerations

For good software engineering practice I should consider extending Weka's implementation. Another reason for taking this approach is that I would only need to store the cluster centroids instead of the Cluster abstraction I chose

[andrewrpatterson]

First exam over!

I had my first of 6 exams today which I’m glad to say is finally over.

It was the Distributed Algorithms & Systems course which, although I didn't really turn up to the lectures, I think went kinda well.

I certainly don’t have full marks or even a B, but all the material I learned was from reading the course book so I think I done better than I should have!

[andrewrpatterson]

Prototype Application

It's been a while!

I've been given two weeks to create an application for "Good Tweet, Bad Tweet" and I've been really stuck on my data.

Like what structure should my data take?

Because I've missed the obvious i've wasted so much time on it. The obvious solution is usually the simplest, and the simplest is usually the best.

So I believe i've figured out what I'm going to do:

Use MongoDB as a backend data store with the data structure like {tweet_id, tweet_json, entity, classification, date_added}

Use Django as my front-end client application framework

Sandwiched in between 1 and 2 is Tornado; to provide the REST API.

Now why did I not just decide to use SQL or use MongoDB inside Django? Well My supervisor wants a NoSQL database (what the boss wants the boss gets;) ) and I need to reduce the coupling between the client application and the data as much as possible.

Last year, this decoupling wasn't done in a Django application and it proved difficult in the end. That project had two applications: one for the interface and one providing url mappings to the data(API). Bit overcomplicated I must admit but the difficulty lay in improving clients access to the data i.e. streaming.

I personally had the task of implementing the streaming service and it was an absolute pain having to import Django models into a Tornado application just so that accessing the data was consistent.

Thats why I'm doing this project differently. This project is being built from the data up and not from opposite ends meeting in a malformed middle.

I digress...

Though my system architecture seems pretty small and basic right now it will obviously need improved. some ideas I've thought about are:

Redis for caching entity:tweet Lists

A streaming service for live classification viewing

Reading the above sounds a like I've been thinking about scalability a lot! caching needs no explaining but the streaming service should reduce server load as connection churning should be kept to a minimum in those instances.

[andrewrpatterson]

Project Ideas

I posted previously about a few ideas I have for my project.

The Application

I feel that when a user clicks on an entity to view it's reputation a "summary view" should be presented to the user. This summary view would ideally be  a large print text display that tells the user the percentage of sentiment compared to all classified tweets.

The following view would be based on graphs. The primary graph - the one that is largest first - I think should be a line graph showing changes in reputation over time*. along the bottom of the graph would be smaller graphs that when clicked become the primary. These graphs could be:

barchart showing the size difference between positive, negative, and indifference

colour coded map showing reputation per country.

The Classifier

My current thinking is that I have server applications/modules called FeedManager, FeedClassifier, and FeedEnrichment, UserService.

FeedManager

This module will look up in the shared database for the entities that are currently being watched. For each entity create a new stream for downloading corresponding tweets.

For each tweet that comes through, tag it's entity, and place in a temporaral resource, such as a queue.

Feed Classifier

This module is multithreaded. Each thread listens on the FeedManager Queue for tweets and removes a tweet, classifies it as positive, negative,  or indifferent, then places in the shared database along with the entity in question.

This will probably be an extended version of an already existing framework.

FeedEnrichment

This module watches the database for any new classified tweets. if a tweet is added, the module adds it to the worklist.

Anything can go on here. We could use Klout to add a more accepted measure of influence to a Tweet in regards to there author.

Regardless of how its done, a tweet influence factor should be added.

UserService

The userservice will be our webserver/database so that we can treat our data and information as a web service. This should mean that any number of apps could be created.

Ideally, both a REST service and websockets service would be available.

[andrewrpatterson]

Back to Uni!

Yesterday I started back at Glasgow uni. Though I resent the 9am start I have to put up with I’m glad to be back as I get to do an individual project this year :) My project is called “good tweet, bad tweet”. It focuses on finding an entity in Twitter and analysing that entity for sentiment. Though my dry musings may not seem enthusiastic I really am. Tomorrow I’ll be having my first discussion with the project and we’ll be agreeing on a set of objectives and outcomes. Can not wait!

[andrewrpatterson]

AWS charged me $280 when all instances and volumes were stopped?

[andrewrpatterson]

Twitter API Libraries

After using the Twython Library I reckon I’ll make my own in Python.

I’ve used Twython for some time now and have noticed its limitations, especially when using the Streaming API for only retrieving filtered tweets.

If I make any progress on the project I’ll post updates.

For now though, I’ll attempt to create a python wrapper for the streaming sample endpoint along with twitter credentials stored in a configuration file for ease of use.