On Estimations

Alice needs to make a decision that depends on the time that Bob needs to complete a task. Hi Bob! How long will you need to complete this task?.

Bob scratches his head and answers, About 5 days... yeah, a working week.

Alice notes down the estimate. But when she gets back to it, she realises that it is rather ambiguous, leaving room for various interpretations, including:

That a week is the expected value. Bob may be thinking that it will take approximately a week, give or take "a few days". Alice recognises that she doesn't know how confident Bob is in his estimate, and thus she cannot gauge the meaning of "some days".
That it is "unlikely" to take less than a week. Alice not knowing about Bob's certainty precludes the precise meaning of "unlikely".
That it is unlikely to take more than a week.

Estimates are a tool to address uncertainty. Bob doesn't know how long he needs to complete the task, but he has an intuition about what time frame is realistic. With his estimate, Bob is trying to communicate the likelihood of different outcomes, which he can do more precisely borrowing from the abstractions that probability theory uses to model uncertainty.

Bob and Alice agree to describe estimates as confidence intervals (x, y), where:

The probability of completing the task in less than x is 15%.
The probability of completing the task in more than y is 15%.

Bob can now say I estimate that this task will take between 2 and 10 working days, and be confident that Alice understands, precisely, that the probability of completing the task in 1 or 2 days is 15% and the probability of finishing within 10 days is 85%. She can also infer that there is a 70% chance to complete the task within 2 to 10 days.

While confidence intervals don't fully describe the probability distribution of the estimate, they provide sufficient information for practical decision making and rigorous assessment of the estimation process quality. For instance, assuming that Bob's estimates are accurate, Alice can infer that:

It is risky to assume that the task will take less than one or two days.
It may be safe to assume that the task won't take longer than 2 weeks, although there is still some chance of exceeding that time frame.
Bob's confidence in the estimate isn't particularly strong, suggesting that it would be wise to follow up a few days after he starts working on the task and update the estimate.

Unfortunately, there is no reason to believe that Bob's estimates are magically accurate. After all, humans are famously bad at estimating. However, Alice and Bob can evaluate the quality of the estimates and understand how to calibrate the estimation process to rectify the inaccuracies. They start collecting estimates and the actual duration of the tasks once they are completed. Analysing the samples, they can determine how often the task duration aligns with the predicted interval, surpasses the pessimistic end, or falls short of the optimistic end. Bob can then adjust his estimation process until he provides reliable estimates:

If the actual duration falls within the predicted interval only 50% of the time, Bob needs to provide wider intervals than he initially believes.
If the actual duration of the tasks falls within the predicted interval around 70% of the time, but only 5% of the tasks were shorter than predicted and 25% longer, Bob needs to bias towards being longer predictions than he initially believes.
If the actual duration of the tasks falls into the predicted interval 90% of the time, Bob needs to narrow his intervals.

Through this iterative process, Bob can refine his estimation process to consistently provide accurate estimates.

Confidence intervals are a relatively straightforward and effective way to describe estimates using probability theory. Nonetheless, Alice and Bob have numerous other possibilities to explore. What matters is that:

Estimates are precisely defined, so that their quality can be assessed using historical data.
Estimators can produce accurate estimates, possibly using historical data to calibrate their process.

Without these properties, estimates are unreliable and lack the necessary feedback loop to improve them with practice. Unsurprisingly, people find these estimation processes frustrating. Estimators feel that they are expected to provide wild guesses that are so uncertain that everyone feels compelled to artificially extend all timelines significantly to have any chance of meeting them. In contrast, when estimates defined precisely using probability theory, can be collected and compared to actual outcomes. This validation process enables estimators to improve their skills and allows stakeholders to make better decisions, only padding timelines affected by low confidence estimates.