words typed by hand

Stuff written and liked by me. Possibly real, possibly not. Probably nonsense. My own opinion and nobody else's.

[st3v3nhunt]

Mood of the nation - week one

Mood of the nation - week one

Stuff found out

There are several tools available for exploring the Twitter API. I used a combination of the web search, the apigee console and twurl.

Twitter's search API only contains tweets from the last 6-9 days.

Topsy API is available for historical data but it is pretty expensive. The trial could be used but has a limit of 1000 tweets in total so there is little point in using that.

More research about the available data might have been a good idea before pitching the idea.

Clearly, historical sentiment analysis is something that is in demand. At least enough for a company to exist that charges pretty large sums of money in order to access the data. And Twitter are not going to do it for free.

The term 'NHS' had over 250K tweets in the past 30 days according to Topsy

Incoming changes

Now I am aware that historical sentiment analysis is not feasible due to financial restrictions (I am not going to stump up any cash until I figure out something that could provide a return on the investment!). I need to pivot. I see there are two choices:

Start collecting relevant data on a daily/weekly basis so I will have some historical data to use at a later date.

Look at 'real-time' analysis. Rather than work from a historical base - use the data that twitter makes available through the search API i.e. the last 6-9 days. This could then be tied to into the streaming API.

At this moment in time I am leaning towards using what is available i.e. the data in the search API

Future work

The focus for next week will be on understanding how to do sentiment analysis on the tweets. Options include doing this myself or using an existing service like Sentiment140. Once that is done it will be about tying that into other data such as geographic region of the tweet and other interesting data sets.

[st3v3nhunt]

The Idea

The Idea

How does the nation feel about the NHS?

That is my idea in one line.

In more than one line what is it?

I want to see what the nation thinks or maybe how the nation feels about the NHS and how that differs over time. I want to see if there is a correlation between the area in which a citizen lives and the quality of the health services provided in that area. I want to understand if there are correlations between other types of human activity and how citizens feel about the NHS. Do areas with higher rates of crime or higher unemployment think the NHS is better or worse than those areas with lower levels of crime and higher employment? Is this cause or effect? How does the coverage of the NHS within the media effect how the nation feels about the NHS? What other external factors are there that impact the way the NHS is seen and thought of by the citizens it serves? Does having this knowledge, whatever that might end up being, lead to anything useful? I certainly hope so.

Sounds good but...doesn't this exist already?

It is almost certainly already possible to know how the nation feels about the NHS so why did I want to do this?

Mainly because I think there will be several interesting technical challenges to overcome that will provide opportunity for creativity and innovation. Maybe even some room to have a bit of fun along the way. But also, it might provide some useful insight that is not already provided elsewhere.

More details please

I have yet to think in too much detail about the technical makeup of the solution but it will need to be relatively quick to develop (relative to what though, who knows?). That probably means the data gathering will be done with Ruby or Node.js and spat into a mongoDB database. The main driver is rapid application development; I need to be able to test my hypotheses as quickly as possible.

Ideally I want to get the data sorted within a couple of weeks so I can spend the rest of the time on the analysis and visualisation of the data. That is going to be the really interesting bit. And probably the most fun bit. Playing around with JavaScript data visualisation libraries is going to be a hoot. Do I go for D3.js or CanvasJS or how about Raphaël.

Coming soon to these very pages

I will be looking to blog about the progress on a weekly basis, the first 'proper' week is this coming Friday - 04/10/2013.

[st3v3nhunt]

The Dragon's Den

The Dragon's Den

The den of impending doom

Walking into the den was just like walking into the meeting room I walk into everyday at 10:00 for the daily stand-up. Why the stand-up is in a meeting room and not around the Kanban board is another post entirely.

Back to the den...as I walked in the 3 dragons each let out a lick of flame as they raised their gnarled snouts and glared at me. Their teeth, aged by millennia of consuming insignificant peasants such as myself were glinting in the dull 60W light provided by the strip bulbs in the overhead fittings. There was a dull thud of water dripping from the roof of the cavern in the background due to the air conditioning's leaky overflow. The height of the cavern was such that it stretched far out of sight; even it dwarfed the size of the dragons. As The Three paced back and forth waiting for my introduction I felt an overwhelming fear my voice would not be heard in this hall of infinite emptiness I stood within. As I opened my mouth to begin the pitch a heat so intense the sun would have felt dim in comparison blazed across the void, my skin ignited and darkness fell upon me. For that reason I was out...and back to the real world.

Everybody took it reasonably seriously, bribes were given, masks (of the real dragons) were worn, Post-it notes were used in place of piles of cash, they sat down, and I stood up. It was good fun and my pitch was successful. I went for the full 12 weeks and got it on the provisos I would be able to show significant progress within 3. I am confident this is possible.

Check back for detail on the pitch...

[st3v3nhunt]

10 percentage points of time (is awesome)

10 percentage points of time (is awesome)

You've got to ask yourself one question. Do I feel lucky? Well, do ya, punk?

From Friday 4th October I can say without any hint of irony I will become one of the luckiest workers I know. Why is that, I here you ask. From this date I, along with a bunch of my co-workers are able to spend Friday afternoons working on what we want (with a few caveats).

We do not have carte blanche but the general premise is awesome. We should work on something that will benefit the organisation in some way. And that way might be as simple as making the individuals more involved and better at what they do in the other 90% of their time at work.

How does it work?

An idea needs to be thought of and pitched to the 'dragons' (3 peers). The pitch is for a maximum of 12 weeks investment. The idea can be worked on individually or a team can be formed. There needs to be regular reviews of the progress among peers at which point additional people may join the team or the idea may be scrapped.

Caviar Cat

There is always some small print. For us it is pretty sensible stuff like:

The employer has the rights to what is made but it will be open source anyway so no biggy.

No money will be provided by the org (time is what is being provided).

Anything really, really important will trump the time available to do your own thing but this is really the last resort.

[st3v3nhunt]

Cumulative Flow Diagrams (CFDs)

Cumulative Flow Diagrams (CFDs)

What is a CFD?

A CFD is a graph showing the state of the work over a period of time based on data acquired from Kanban.

More in-depth explanations available at Wikipedia, Google and David J Anderson.

Graphs and colours…I want one

To create a CFD is pretty straight forward. The number of stories in each state represented on the board need to be recorded on a regular basis. Daily is typical. With the data a chart can be generated, plotting the accumulated data against time.

Colours are great but why do I need one?

There is a host of reasons why using CFDs are a good idea. Personally, I like the fact that from a relatively small amount of easily obtainable data a huge amount of information can be discovered. The information can provide some very revealing insights into the flow of the work.

Insights sound interesting, like what…

It is probably best to use a real world example. The commentary and insights below were gained after studying this CFD. Available as a larger image here.

The CFD is from a project where the time and cost were fixed with the scope being variable. Effectively a time box where as much as possible was to be delivered.

Backlog (ideas hangout)

Although there are stories added throughout the duration of the project there are a couple of big bangs (19/08 & 31/08). Ideally all stories should be added on a regular, free flowing basis.

Approximately 40% of the stories created were not done - how much time was spent on creating the stories that were effectively wasted (not much)?

Refinement (ideas thought about)

There were always some stories being 'refined'. One might assume that is a good thing as stories were always being thought about and made ready for development.

UX & Design (interface)

The state was only used within the first couple of weeks. This might have been due to that being the time when the front end was being designed and refined before it was developed.

Ready to go (blast off)

A large number of stories were ready to go in week 2, for the full week. Not one story moved from this state during the week. Not a good thing, the board will have looked pretty stagnant at this time.

A couple of stories ended the time box as being refined - this means time and effort was spent on doing the work to get them into this state and that was wasted.

In progress (rolling)

Week 3 saw many stories in progress, perhaps too many. How many people were in the team and how many of the stories were actually being worked on? Were there blockers stopping the progress of the stories?

Ready to verify (awaiting testing)

Stories were not sat in the queue for very long. Actively being tested as soon as available.

Bit of queue building up towards the end of the time box, was this caused by the knowledge the time was closing?

Verfying (is it ok)

Some stories were stuck in verifying for a long time. Possibly there was no environment available to complete the verification?

Verified (done the thing right)

Some of the stories are in verified for a small period of time, suggesting they were accepted almost immediately.

The last week or so saw a few stories stack up in verified. This was due to environment problems. Once that was sorted the stories quickly moved into accepted.

Accepted (done the right thing)

It was 2 weeks before anymore than 2 stories were accepted.

It looks like the stories were accepted in chunks rather than when they were ready * to be accepted. Is this due to the person accepting them not being available / engaged? Or not having an environment in which the stories could be accepted?

Waste (…)

The stories added around 19/08 - were they wasted 27/08? If so, why were they created and perhaps this should not happen in the future?

What about the lead and cycle times?

TBC…

[st3v3nhunt]

Cumulative Flow Diagram

[st3v3nhunt]

What work has work where working work wrangles waiting work while wandering work wistfully wishes waiting work would wander, wearily towards work?

What work has work where working work wrangles waiting work while wandering work wistfully wishes waiting work would wander, wearily towards work?

What…when…how…wow…WHAT?

This is a little bit about how work at where I work works.

Visualisation of work

Kanban. Is the system used to visual the work. It includes who is involved, what they are involved in, what state work is in and whether there are things getting in the way. The currency of the work is a story. The more stories completed, the richer everybody is. Flow is everything.

GBP, USD and…stories

The vessel of work is a story. Stories have a size, small (S), medium (M) or large (L). The size of a story is dictated by the complexity of the work involved. This is often based on the number of systems that will need to be changed to complete the work.

What does a story look like?

There is no set definition of what a story is (perhaps there never will be although it would probably be a good idea). The level at which a story is created varies. Some of the experiences differing levels of story have provided are:

One project had stories focused on the technical activities required to get the work done. They were very much like tasks. Stories at this level moved through the board quickly, this built momentum. It worked well for technical people. It did not work well for non-technical people. The board was flooded with stories that made little sense to business people. However good it might have been for the technical team, not having the business understand the state of the project is not a good position to be in. Especially when a goal of the project is to increase the level of participation of the business in the development of products.

Another project had stories written at a much higher level, one where they would provide some real value to the Product Owner (PO). There were far fewer stories and each story was much clearer about what was being delivered. This approach worked much better from the point of view of looking at the board and understanding the state of the project. Business people were much more bought in. This is a very good thing. However, it was not all roses and sunshine. The major loss was the flow of stories through the board. The board stagnated. How to improve on this is an on-going concern. Probable solutions revolve around another board, a 'technical' board, however, this has its' own challenges.

Free flowing

One of the goals of Kanban is to increase the rate at which stories flow through the board. This is referred to in two ways - cycle time and lead time.

Lead time

This is how long it takes a story to get across the entirety of the board. From initial state to end state. It is useful to see how long an idea takes to go from inception to realisation. Often used by the business.

Cycle time

This is how long is takes a story to get to done i.e. from the time it leaves the backlog until it reaches the end state. Provides information on the time it takes to do the work. Used by the team doing the work.

Why should anybody care about lead and cycle time?

There are many reasons to care about lead and cycle times. My number one reason is, the smaller the time it takes to get something to the end state (which should really be the production environment) the quicker the product can be iterated.

The quicker the product can be iterated, the quicker the wants of the users of the product can be met, the happier those users can be made, the more users there will be. Ultimately ending in the beating of competitors whilst providing the best product possible, built based on the needs of the users. Win, win.