I simply do believe that the central reason is that there is a lot of (also psychological) pressure to give lower time estimates than what is realistic.
> Hofstadter’s Law: Everything takes longer than you think it will, even when Hofstadter’s Law is taken into account.
> Mayfer’s Law: The miscalculation of time required for achievement enables the undertaking of endeavors far larger than we would knowingly attempt to achieve.
---
James T. Kirk: How much refit time before we can take her out again?
Montgomery Scott: Eight weeks, Sir - but ya don't have eight weeks, so I'll do it for ya in two.
James T. Kirk: Mr.Scott. Have you always multiplied your repair estimates by a factor of four?
Montgomery Scott: Certainly, Sir. How else can I keep my reputation as a miracle worker?
I personally tend to use (jokingly) "Latin" times (useful for countries like Italy, Spain, Greece, Portugal, but also large parts of France and Romania), basically you take initial Swiss (proverbially exact) estimated time, double it and pass to the next bigger unit of measure, i.e. 2 weeks=4 months, 2 days=4 weeks, etc.
The author's thesis is that underestimation is a good thing, because it makes more ambitious projects get approved.
The unspoken assumption here is that decision-makers are too conservative in what they approve. That is, some projects get refused due to too-high costs, even though the benefits outweigh their costs (and the costs are affordable to the individual or organization).
Underestimating the cost is a way for ambitious projects to get approved despite the conservatism. I.e., "It's easier to get forgiveness than permission." This is true even if the underestimating is due to unconscious cognitive biases (as opposed to, say, project proposers consciously, rationally and deliberately low-balling cost estimates).
I wonder if the conservatism and underestimation co-evolve into some equilibrium, either at the organizational level, at the cultural level, or the average cognitive biases of all humanity?
That is, it seems the author might say the counterfactual world where all estimates are accurate is a dreary one where few ambitious projects are attempted. But I wonder if, in such a world, a similar number of ambitious projects would be attempted. This could be the case if decision makers in a world of perfect estimates would be less conservative about accepting ambitious projects, knowing that the time estimates would be reliable.
In other words, the reason an underestimation cognitive bias is beneficial is because decision makers are conservative. And the reason conservative decision making is beneficial is because the cognitive bias exists. It's a circular dependency, some sort of feedback loop: Conservatism increases lowballing, and lowballing increases conservatism.
Project proposers (perhaps unconsciously) divide their estimates by some number k before making their proposal, so project deciders multiply the estimates from proposers by k before making their decision.
The value of k is some social construct, or perhaps it's based on the strength of the cognitive bias, which might have some biological basis. It would be interesting to study cost overruns and empirically measure k in different sizes / types of organizations in different cultures to see if k is some universal constant that applies to all humanity, or varies in different sociological environments.
Now that I think about it, I'm realizing story points in agile are a system that tries to measure a specific team's k, allowing measuring and correction for cognitive biases within that one team.
People wildly underestimate how long certain kinds of tasks will take. This is usually regarded as a Bad Thing.
Imagine how this plays out if you're a decision maker, deciding whether to do highly ambitious project X. If you knew ahead of time how long X would take, you would never do X. If you don't start on X, there's a zero percent chance you ever finish it. But because you have an unrealistically low estimate of how long X will take, you decide to do it. So now there's a nonzero percent chance you finish X. Once you've started X, the project has a nonzero chance to stay alive, even though it ends up using massively more time/cost than planned. As a decision maker, you don't shut down project X when it overruns its budget for a couple reasons: Partly due to sunk-cost fallacy, and partly due to continued underestimation of how long the remaining parts of X will take.
The author's point is that this underestimation is actually a Good Thing: If we could accurately estimate time costs up-front, fewer highly ambitious projects would be attempted (and therefore, fewer would be completed).
