Bayesian Project Management
I am currently working on a particularly complex project involving many moving parts, at least three different companies all with their own internal requirements, and which is a mixture of IT development and process migration and improvement.
In short, there is a lot that can go wrong and pretty much every day we have a new surprise to deal with.
So the other evening, while I was re-reading a chapter of Nate Silver’s excellent The Signal And The Noise, it occurred to me that there are many aspects of project management that should really be treated more explicitly as a formal prediction or forecast.
Take timeline estimation for example. Whereas informally we often try and make a judgment about how a new or changed requirement could impact the go-live date, I think our planning would benefit significantly by being more formal in our treatment of time (or cost) estimates.
In practice this means two things:
Thinking and talking in terms of probabilities and ranges
Constantly updating our estimates based upon new information
Living with Uncertainty
For the first point, current practices are very focused on using fixed estimates, often leading to conflict and recrimination when the actual figure turns out to be different from the initial estimate.
Indeed when you look at a traditional Gant chart or other visualization of a project plan, all of the activities are typically set out in a very linear and causal fashion, all building up to a fixed go-live date.
Reality, however, is very different. Even for smaller individual tasks, when someone says that it will take 5 days to complete activity X, in practice it could take 5 days, however it could also take 4 or maybe 6 or more, depending on the activity. People try and round to whole numbers but there is often an implicit probability distribution built into their estimates.
Each of these individual estimates + error then compound at the work-stream and project level, so a fixed implementation timeline of two months, would be far more usefully expressed as something like:
10% probability of 7 weeks (if everything goes extraordinarily well)
70% probability of 8 weeks
10% probability of 9 weeks
10% probability of 10 weeks or more
I know that many people, particularly project owners / sponsors, would not be very comfortable with these kinds of estimates as in general people are not comfortable working with probabilities and uncertainty. Often people mistake expressions of uncertainty for lack of competence and will try and push for a “definite answer”.
Unfortunately the real world is full of uncertainty, and ignoring it or pretending it isn’t there doesn’t make it disappear. In fact, pretending that we can forecast the future with absolute certainty leads to overconfidence and leaves us less prepared to adapt to changing circumstances.
Institutional Statistics
What does this mean in practice? The probability estimates are not just numbers based upon the gut feel of whomever is responsible for that activity, but should be based upon real historical data.
This means that if you were to say that the time estimates for completing activity Z are:
4 days or less, 10% probability
5 days, 80% probability
6 days or more, 10% probability
then looking at all instances of where the company or team has performed activity Z,
In 10% of cases, it should have taken 4 days or less
In 80% of cases, it should have taken 5 days
In 10% of cases, it should have taken more than 5 days
One way to address this challenge is at the Project Management Office level. Rather than asking individual areas or teams to forecast their timelines (which we know are often wildly inaccurate due to the Planning Fallacy), the PMO should begin to construct and set of statistics at both the activity and project level.
Then, when constructing a project plan, they would be able to look first at actual historical data on the distribution of how long similar projects took to implement, giving them greater insight into the potential range of total time required for their particular project.
However perhaps more usefully, they would also be able to look at distributions for how long specific activities have taken too. Whereas it can often be hard to find a comparable project, most projects are made up of very similar activities, such as recruitment, training, setting up computers, systems changes, contract signing etc.
So for example, rather than building your project plan based on the HR estimate of hiring X positions in 4 weeks and then being surprised when it takes longer, the inherent uncertainty would already be built in based upon actual data on historical recruiting activities.
Bayesian Updating
The other key element of maintaining more realistic project estimates is through constant updating as new information appears.
I have seen so many projects managed by assuming that all requirements will be finalized by X date, only to see conflict arise when the scope needs to be modified in some way.
Maybe it is possible to implement some projects this way, however in my experience, it is far better to think of project scope and requirements as fluid:
External factors can lead to changing requirements in the middle of the project (i.e., competitor products, regulatory needs etc.)
Testing with real-world examples can lead to the realization that the initial requirements were not quite correct
Plain old human error can mean that somebody made a mistake or forgot to include something initially
Methodologies like Agile are certainly one way of addressing and incorporating this inherent uncertainty in project management, however whatever methodology you use, I believe it is key to, almost on a daily basis, take new evidence and use it to update the project time and cost estimates.
Returning to our recruiting example from above. Suppose we had initially estimated the hiring of 4 positions to take 4 weeks with 80% probability.
Then our HR contact comes back to us and says that having reviewed the specific profiles and salary ranges, they realized that these positions are more complicated to fill. We would then look again at our historical data and see how this could impact our estimates, and we may revise our timeline to be 6 weeks with probability 80%, but with a 20% probability of 8 weeks.
Even if we don’t have the historical data available, we should still go through a formal updating process, looking at our original timelines, and figuring out how this new information could affect the timings and probabilities.
Business Deadlines
Perhaps the biggest challenge to this type of project management comes in the form of imposed and hard-business deadlines.
Even though the Project Manager might estimate the project will require 8 weeks with 80% probability or up to 12 weeks with 20% probability, the reality could be that for whatever reason the business needs to launch in exactly 8 weeks.
I would argue that a probabilistic approach is still useful for two reasons:
By sensitizing the project sponsor to uncertainty early on, later conversations regarding time or cost risks should be at least slightly easier than if you had used fixed and 100% confident estimates from the start
By being aware of inherent uncertainties, both initially and also as they change with new evidence, it will be easier to prepare contingency plans to try and reduce the uncertainty and bring the project back on track when needed