The real problem is not with estimation but with the management denial.
I was asked to estimate a project I owned as a “green” lad. I offered 6 months with a real detailed plan and up to a year with some extras. Based on past rates we had etc. I even did some statistical modeling. They disagreed. They decided to alter the approach, that the principal engineer had suggested and “it would take 2-3 months.”
Two years later I learned they were still working to finish it.
Another example: they wanted a Byzantine Fault Tolerance rewrite along with the networking layer. I said we need 6 months. They said we know how it should be done in 1-2 months. I finished my stuff and moved on — they picked couple of people to work on the rest that had said 1-2 months. It took them a year to reach a point they could test it.
Now why did I say 6 above? Because that is a project I have done multiple times, I can do everything in my sleep and the delay beyond all other fundamentals is random interruptions. Did that matter? No.
When top head works with wishful thinking estimates are ways to find scapegoats; nothing more.
Your first mistake was misunderstanding the relationship with management.
Time estimations are bargaining, you ask for something and then get a counter offer from management, they don't care that the counter offer from them is delusional. The bargaining is the point, the less time you bargain for the more stressful the development can become. Don't estimate what it should take, estimate what it could take and bargain from there.
I think it mostly correlates with how technical PMs are.
People that were fairly technical prior to deciding to move to PM roles, those you can actually explain the context and reason with.
PMs that are more biased towards Business/Marketing are big on C Level networking etc., those are the ones that feel they are doing a good job if they try to bleed a stone with deadlines. They are adversarial like GP mentions and not someone you can reason with.
The other part is that management/PMs usually care about deadlines, not work effort (duration in man-hours, story points, or whatever).
Allow me to indulge in a fictional narrative:
You are handed "high level" (a.k.a rooted in the fuzzy non-technical alternate reality that is product management) requirements resulting in known unknowns and unknown unknowns. You're junior and naive and you put real tech-borne effort numbers in front of each task; or you're experienced and you put O(t) numbers on the first ones and random but high numbers on the second ones.
Anyway, the estimates end up landing past the desired deadline, management mutters TLAs such as "OKR" and whatnot, you'll have to justify why it doesn't fit, so you communicate work effort.
You are summoned into a retroplanning meeting to cull out stuff based on work effort so that it fits management's expected deadline, and try to pretend you're doing agile when it's all waterfall Gantt in disguise.
Project starts. Implicitly man-hours become wall-clock-hours, irrespective of other tasks such as CI/tooling/dependency breakage, code hygiene, or customer support escalations; your effort estimates became t{project start}+effort == deadline. You are now held accountable for the latter based on you communicating the former.
Time is now the scarcest of resources, you do your best, conjure magic, work extra hours, take shortcuts, and make it to the deadline, sacrificing body and soul. In a cloudy spreadsheet checkboxes have been checked, management is happy. Deployment is smooth but soon corner case bugs and escalations start trickling in. Management is either oblivious or not so happy after all.
Time for a new quarter to start. You know issues borne out of past quarter will rear their ugly head, and there's much tech debt to repay before interests quickly accrue towards tech bankruptcy. You're junior and swear to yourself to not make the same mistake, and you start inflating numbers; you're experienced and you realise the whole number dance is a sham as they don't matter anyway, but you still are required to do the dance because that's how the game is played, so you turn Scotty and just throw outrageously big numbers up in the air and just make sure the deadline is not too ludicrous, still hanging to the hope hoping that someone, somewhere, has seen the light of reason and built a company around it, and maybe you could go to these greener pastures; you're very experienced and you realise the only winning move is not to play, so you quit (not the job, the _industry_) and move to a remote plot of land with literal green pastures and a small cabin mountainside to raise sheep, or cut wood, or whatever, where the smallest time unit is the measurement error of eyeballing the zenith-relative angular position of a giant fireball 8.3 light-minutes away.
The tell is that as soon as they don’t like how things are going they flip from calling them estimates to commitments.
Also if someone’s estimates aren’t perfect, that’s something you can use to justify withholding salary. The game is fixed. It’s just a good deal less fixed for us than for the average person.
Well your responsibilities warrant a 20% raise but since your estimates are bad and your <bullshit other reason> was not 5 out of five, we’re giving you a 6% raise. Better luck next year right?
>>>. The real problem is not with estimation but with the management denial.
^^ This is the truth. All estimation processes in any company are just ceremony. Management already has a deadline fixed. They will make you agree to their deadlines, even though whatever process stipulated by them says otherwise.
The real problem is saying "yes we can do it all in that timeline" instead of saying "That's fine, but we won't get to everything. We will work with you to prioritise the most important things to get done in that time."
I know it's painful, but just hold that line. It's not lying or negotiating or padding; in fact it may be the only true statement you can say.
If a civil engineer were building a bridge, and couldn't do it safely in the budget and timeline, they'd say so, and the project manager would nod and diligently write that down. We have a more subservient relationship with project managers in software because we choose to.
You have to get them to take that perspective. What happens is "we want all this stuff how long is it going to take" and then they fight that number down while not answering any clarifying questions. So you have to sumo wrestle a bunch of assholes twice your weight into even framing the context correctly before the conversation you recommend can even take place.
> They will make you agree to their deadlines, even though whatever process stipulated by them says otherwise.
I'm not sure how. I mean, if I know it takes X time to do something, what would make me to commit to Y<<X time when I know it will fail and will be made my fault? If they want to record it as a deadline, so be it, but if it's an impossible deadline I'd make every effort to be on record that I said it from the start that it's impossible.
Not EVERY company. Just the ones that don't understand how estimation works... and unfortunately in software, that's still a very very common case. In more established kinds of engineering it works differently... which gives me hope that eventually our industry will understand too.
Management denial is not universal. It can't be - companies that have realistic management should out-compete companies where management is politics and unreality.
The problem is that management unreality can take a decade or several to destroy a place. But when you see that kind of management, know that that's the road they're on. That's your cue to start looking for somewhere with management that at least tries to live in reality.
I hope you are right. I have never been in or seen a workplace where management is not playing the 'get the order at (nearly) all costs - let others handle the problems' game. Again, I hope I am wrong. But these type of companies seem to outcompete all others. The only unicorn in a decade in my geographical area, is a company that plays this game very very well in the IT sector. As far as I have seen, they play a variation called 'the customer takes the problems (and the full financial impact). Due to building a good set of core competencies, that the competition mismanaged over years, they can get away with this strategy.
You’re forgetting that in companies with the management properties you say are bad, those properties may cause actual value to be delivered faster, even if it’s at long term cost. That value can accrue, gain interest, and that company can be more successful than the one with the management properties you say are bad.
I think you’re conflating success of a company with the pleasure of its employees. These 2 factors are usually (not always) opposing not correlated.
I don't disagree with you but I think people also need to understand that management often pick the team that provides lower estimates because team members will have to work harder to deliver it.
Management might not be so dumb. If someone said 2 months and you say 6 months, they might go with the person who said 2 months but will accept 5 months secretly.
The 2 month version experiences scaling issues at a 20% rollout, or takes 20s for a database query before returning a page.
In 6 months you need a better scaling version while the current version is creating cascading database failures; it's got terabytes of denormalized data that is out of sync and a 2nd team can't migrate the data in 12 months.
In 12 months, you find 5 other teams copied the 2 month solution.
In 2 years, you have 0 devs left that can solve the problem and a 1000% grown dev team that is great at the 2 month solution.
You ask the teams why they don't use the scalable solution and they say, "do we have any examples of that, the unscalable way is our convention"
I was asked to estimate a project I owned as a “green” lad. I offered 6 months with a real detailed plan and up to a year with some extras. Based on past rates we had etc. I even did some statistical modeling. They disagreed. They decided to alter the approach, that the principal engineer had suggested and “it would take 2-3 months.”
Two years later I learned they were still working to finish it.
Another example: they wanted a Byzantine Fault Tolerance rewrite along with the networking layer. I said we need 6 months. They said we know how it should be done in 1-2 months. I finished my stuff and moved on — they picked couple of people to work on the rest that had said 1-2 months. It took them a year to reach a point they could test it.
Now why did I say 6 above? Because that is a project I have done multiple times, I can do everything in my sleep and the delay beyond all other fundamentals is random interruptions. Did that matter? No.
When top head works with wishful thinking estimates are ways to find scapegoats; nothing more.