#nationalmood

[st3v3nhunt]

Mood of the nation - week one

Mood of the nation - week one

Stuff found out

There are several tools available for exploring the Twitter API. I used a combination of the web search, the apigee console and twurl.

Twitter's search API only contains tweets from the last 6-9 days.

Topsy API is available for historical data but it is pretty expensive. The trial could be used but has a limit of 1000 tweets in total so there is little point in using that.

More research about the available data might have been a good idea before pitching the idea.

Clearly, historical sentiment analysis is something that is in demand. At least enough for a company to exist that charges pretty large sums of money in order to access the data. And Twitter are not going to do it for free.

The term 'NHS' had over 250K tweets in the past 30 days according to Topsy

Incoming changes

Now I am aware that historical sentiment analysis is not feasible due to financial restrictions (I am not going to stump up any cash until I figure out something that could provide a return on the investment!). I need to pivot. I see there are two choices:

Start collecting relevant data on a daily/weekly basis so I will have some historical data to use at a later date.

Look at 'real-time' analysis. Rather than work from a historical base - use the data that twitter makes available through the search API i.e. the last 6-9 days. This could then be tied to into the streaming API.

At this moment in time I am leaning towards using what is available i.e. the data in the search API

Future work

The focus for next week will be on understanding how to do sentiment analysis on the tweets. Options include doing this myself or using an existing service like Sentiment140. Once that is done it will be about tying that into other data such as geographic region of the tweet and other interesting data sets.

[st3v3nhunt]

The Idea

The Idea

How does the nation feel about the NHS?

That is my idea in one line.

In more than one line what is it?

I want to see what the nation thinks or maybe how the nation feels about the NHS and how that differs over time. I want to see if there is a correlation between the area in which a citizen lives and the quality of the health services provided in that area. I want to understand if there are correlations between other types of human activity and how citizens feel about the NHS. Do areas with higher rates of crime or higher unemployment think the NHS is better or worse than those areas with lower levels of crime and higher employment? Is this cause or effect? How does the coverage of the NHS within the media effect how the nation feels about the NHS? What other external factors are there that impact the way the NHS is seen and thought of by the citizens it serves? Does having this knowledge, whatever that might end up being, lead to anything useful? I certainly hope so.

Sounds good but...doesn't this exist already?

It is almost certainly already possible to know how the nation feels about the NHS so why did I want to do this?

Mainly because I think there will be several interesting technical challenges to overcome that will provide opportunity for creativity and innovation. Maybe even some room to have a bit of fun along the way. But also, it might provide some useful insight that is not already provided elsewhere.

More details please

I have yet to think in too much detail about the technical makeup of the solution but it will need to be relatively quick to develop (relative to what though, who knows?). That probably means the data gathering will be done with Ruby or Node.js and spat into a mongoDB database. The main driver is rapid application development; I need to be able to test my hypotheses as quickly as possible.

Ideally I want to get the data sorted within a couple of weeks so I can spend the rest of the time on the analysis and visualisation of the data. That is going to be the really interesting bit. And probably the most fun bit. Playing around with JavaScript data visualisation libraries is going to be a hoot. Do I go for D3.js or CanvasJS or how about Raphaël.

Coming soon to these very pages

I will be looking to blog about the progress on a weekly basis, the first 'proper' week is this coming Friday - 04/10/2013.

[st3v3nhunt]

The Dragon's Den

The Dragon's Den

The den of impending doom

Walking into the den was just like walking into the meeting room I walk into everyday at 10:00 for the daily stand-up. Why the stand-up is in a meeting room and not around the Kanban board is another post entirely.

Back to the den...as I walked in the 3 dragons each let out a lick of flame as they raised their gnarled snouts and glared at me. Their teeth, aged by millennia of consuming insignificant peasants such as myself were glinting in the dull 60W light provided by the strip bulbs in the overhead fittings. There was a dull thud of water dripping from the roof of the cavern in the background due to the air conditioning's leaky overflow. The height of the cavern was such that it stretched far out of sight; even it dwarfed the size of the dragons. As The Three paced back and forth waiting for my introduction I felt an overwhelming fear my voice would not be heard in this hall of infinite emptiness I stood within. As I opened my mouth to begin the pitch a heat so intense the sun would have felt dim in comparison blazed across the void, my skin ignited and darkness fell upon me. For that reason I was out...and back to the real world.

Everybody took it reasonably seriously, bribes were given, masks (of the real dragons) were worn, Post-it notes were used in place of piles of cash, they sat down, and I stood up. It was good fun and my pitch was successful. I went for the full 12 weeks and got it on the provisos I would be able to show significant progress within 3. I am confident this is possible.

Check back for detail on the pitch...

[st3v3nhunt]

10 percentage points of time (is awesome)

10 percentage points of time (is awesome)

You've got to ask yourself one question. Do I feel lucky? Well, do ya, punk?

From Friday 4th October I can say without any hint of irony I will become one of the luckiest workers I know. Why is that, I here you ask. From this date I, along with a bunch of my co-workers are able to spend Friday afternoons working on what we want (with a few caveats).

We do not have carte blanche but the general premise is awesome. We should work on something that will benefit the organisation in some way. And that way might be as simple as making the individuals more involved and better at what they do in the other 90% of their time at work.

How does it work?

An idea needs to be thought of and pitched to the 'dragons' (3 peers). The pitch is for a maximum of 12 weeks investment. The idea can be worked on individually or a team can be formed. There needs to be regular reviews of the progress among peers at which point additional people may join the team or the idea may be scrapped.

Caviar Cat

There is always some small print. For us it is pretty sensible stuff like:

The employer has the rights to what is made but it will be open source anyway so no biggy.

No money will be provided by the org (time is what is being provided).

Anything really, really important will trump the time available to do your own thing but this is really the last resort.