Estimation? It’s Complicated…
We are a web software shop. We frequently work for startups, including those who are just starting. Inevitably we get asked to estimate batches of work.
I never have a quick and easy answer when it comes to estimations.
On one hand I perfectly understand the rationale behind requesting estimates. If I were starting my own project or even a company, I would like to know how much I need to bet in order to turn it into a business or at least validate the hypothesis that it can be turned into one in the first place. In fact, the scale of that bet would influence my decision.
Even if the information in terms of the expected cost wouldn’t make me turn away from the idea altogether it may influence decisions about who will build it for me and how I will structure the development process. It’s not that I think the expected cost should be the determining factor in terms of making the ultimate decision when it comes to building software. Pretty much the opposite. Though, it has to be taken into account.
In either case, an estimate provides some information which is often pretty important in deciding on a course of action.
What’s the problem with estimates then?
Estimates are most often interpreted as commitments, which they are not. Not only does it influence how people act in a situation when it turns out they were not accurate, it also changes the whole discussion about estimation. A thing I would frequently hear is that since a project is well defined providing a precise estimate in such a case should be easy.
It’s not for a number of reasons.
Precise Estimates
One thing is that the human brain is pretty much incapable of estimating in abstract metrics such as space and time. Not only that. It also doesn’t get better with simply more experience at estimation. Research conducted by Roger Buehler et al showed that even for tasks we are familiar with, and even when asked for an estimate that we’re 99% sure of, we’d get that estimate wrong more often than not.
Let me rephrase, even when we play it super-safe the quality of our estimates is mediocre at best.
In software development, in the majority of cases, we simply can’t play it super-safe. A new project or a product is always new so familiarity with the subject matter is always limited and clients very, very rarely are willing to pay for confidence levels as high as 80%, let alone 99%.
The bottom line is that even for a fairly well known domain we can’t expect precise estimates.
Scope of Work
That’s not all though. Another argument in this discussion is what gets estimated. First we estimate a batch of work and then we never, never ever, in the end build exactly that. I’m yet to see a project where the scope doesn’t change over the course of building it.
There’s a number of good reasons for that.
- As the project progresses and a client sees the parts that are already working they have more insight on what should and what should not be there.
- As time passes clients learn more about their business environment and that results in changes in how they think about the product.
- We are only human and simply forget stuff.
- We have to take care of all the rework that happens once a decision maker changes their mind after something has already been built.
The worst thing that could happen is when a death cycle starts: we keep adding new features to a product, which makes it late and this means we have to add even more features to catch up with the market. This in turn makes the product even later, which forces us to add even more stuff… Now, go figure out what happens with reliability of estimates in this scenario.
The tricky part is that it’s much easier for us to add new stuff to the plate than to remove existing things. Most commonly we end up building more stuff than what was part of the initial scope. How does that work for estimates?
Knowledge Discovery
Another challenge is related to the learning process that is an inherent part of every project. We start a project with a bunch of assumptions. These assumptions will frequently get invalidated which, more often than not, means more work than expected.
A common answer to that is to add more details to specifications. That doesn’t really work. It would if we were talking about the known unknowns. In other words, we would know exactly what questions we should ask. The big risks are in the unknown unknowns – ones that we can’t predict or ask about up front. We will become aware of them eventually and they would affect the scope of work but the trigger is having parts of the app built and getting feedback from a client.
There’s another reason why going into more detail with specification is most often a bad idea. It doesn’t come for free. It means spending time and money on scoping out all the details and delays the start of actual work. The latter is frequently much more costly because of the Cost of Delay.
In fact, Douglas Hubbard argues that adding more and more details makes estimators more confident while the quality of their estimates gets worse.
When you look at a project from the perspective of a knowledge discovery process, the moment where you naturally know the least is before it commences. It is at this exact point where we are expected to prepare estimates.
Collaboration
Finally, there’s something that is almost never taken into account, despite the fact that this factor itself may be responsible for increasing the effort needed to build the same batch of work by a factor of two or more.
This magic factor is the quality of collaboration. It affects a team working on a project on many levels.
The first thing is the availability of the client. If we get all the information we need in a timely manner we go in with much fewer non-validated assumptions when we build specific features. This highly reduces the amount of tweaks needed to satisfy the needs of a client.
Then we have feedback. The faster we get feedback about the stuff we built the faster we can improve it and thus we shorten the cycle time of finalizing features and limit the number of tasks that are in progress. This results in less multitasking and a much better efficiency of work.
If that wasn’t enough we also have more psychological factors. If collaboration on an interpersonal level works well people tend to be much happier. Happy people also results in more efficient work. Not to mention that it also means that a team is much more likely to go the extra mile.
It all stacks up to a point where the quality of collaboration is typically the biggest leverage one can use to limit effort and the cost of building a project. At the same time we never know how this part will go until we start working together. On one hand it is a major factor that influences any estimate, on the other it can hardly be known at the time when estimates are prepared.
The Bottom Line
Let’s go back to what I started with. Estimates often have a crucial role to play in the decision making process. We can’t get them right though. Are we doomed then?
Not really. There are two basic lessons to learn here. One is that the reasons of commonly low quality estimates are not specific for us – these are general observations. That means that when someone provides an estimation with precision, they’re either unaware of all the uncertainty involved or they’re one of these “let’s throw an appealing number at a client and then we’ll figure something out” guys.
If the latter is true I’d challenge the idea that it’s wise to work with such a partner. It’s not going to turn into a trust relationship.
If the former is the case, there’s at least some hope. With some dose of luck the outcome of such a project can be OK and a reasonable estimate can make a decision about kicking the project off easier. There’s a downside too. Such an approach is reckless. It means a lack of awareness of risks and as a result no risk management. In the case where a budget is not sufficient to cover every single thing, which is a common situation, tradeoffs have to be made when we have the least options available.
I don’t think I need to mention that it is much better to actively manage risks, including those related to the scope of work, from the very beginning, as this universally yields better outcomes. For that a level of awareness of the challenges facing estimation is a hard prerequisite.
Fixed Budget
After sharing all that, one could get an impression that we wouldn’t consider working under fixed budget constraints. After all, we say that the estimates we provide have a high level of uncertainty which translates to a lack of hard commitment that we would finish a defined batch of work within a specified budget.
Such an impression would be wrong. We have nothing against working under a fixed budget. One thing that both parties need to be aware of is that in this scenario we don’t treat the scope as fixed.
There are consequences of that fact for both parties. One thing is that we need to work very closely with a client to define what the crucial parts of the app are and in what order features should be developed.
It doesn’t mean that we need to have all the decisions up front. In fact, my common advice is to define a set of features that the application must include in every single, even the most crazy scenario that can happen. I call it a Minimal Indispensable Feature Set. It definitely doesn’t mean that such a set of features would constitute an MVP. It shouldn’t. It shouldn’t be viable for release. At the same time it would be a pretty safe choice as these features are assumed to be ultimately part of the app in all possible cases.
Further decisions can be delayed till we get feedback from building this set of features, which provides a lot of value:
- We will be further down the knowledge discovery process so we will know more about the work in general.
- We will get some feedback from a client about the early releases.
- We would uncover some of our assumptions – unknown unknowns – and move them to a more predictable domain, the one where we at least know what questions we should ask.
- Based on the data points from the early stage of the project we will be able to tell much more and with much better precision what our pace of development is and thus vastly improve our estimates.
From that point on we can continue making decisions about the scope in a similar manner, deciding on fairly small chunks of work that we know we will need, and continuously iterate on that while learning more about the work we do. It also means that our understanding of what would fit the fixed budget will be improving continuously and thus we will inform the tradeoff decisions. We will suck the most out of the available budget.
By the way, we can use exactly the same approach in the case of where we have continuous funding for a project. Although, I rarely see enough determination and discipline on the side of the client in such scenarios to go for that.
Meaning of Estimates
My message here is twofold. The less important part is explaining why we do what we do with the estimates and why I would always start answering a question about estimates with “It’s complicated…”
The important part is that in order to have a meaningful discussion about the estimates we need to understand all the caveats behind the process. Otherwise, we simply go with the answer that is most comforting for us initially, which is likely also the one that would introduce a lot of troubles down the line.
It is very rarely, if ever, a sensible choice.