
Project delays: why good software estimates are impossible (2015) - theinnerspirit
http://chrismm.com/blog/project-delays-why-software-estimates/?
======
dang
[https://news.ycombinator.com/item?id=12149515](https://news.ycombinator.com/item?id=12149515)

------
jacques_chester
I feel like this has been posted before, not too long ago.

And even though it's "impossible", I and hundreds of my peers at work do it
every week with a workable degree of accuracy, relative to company-wide work
in progress.

In any case, the usual problems are:

1\. We software developers think we're the first to face uncertainty,
including massive uncertainty. We are not. We are in fact lightweights in this
department. Ask a civil, structural or plant engineer for war stories. Or read
_Industrial Megaprojects_ to see how skipping a single water-temperature test
cost billions of dollars.

2\. We tend to embrace the nirvana fallacy. "Perfect or very accurate
estimation is worthless; therefore, _all_ estimation is worthless".

3\. "Some projects are pure research". Yes. But very few. And even they
contain bog-ordinary implementation details. Writing a fantastical new
language? Well guess what: parsers have been written before, and someone with
data or relevant experience could break down the task into subtasks and
estimate them.

I think the real reason that we so dread estimates is:

1\. They aren't treated as estimates. People turn them into deadlines. Or try
to negotiate with the estimate. Which means they aren't estimates any more.

2\. We get them wildly wrong the first few times and throw our hands in the
air. Improving estimates is hard. If it were easy, we wouldn't need to
consciously do it.

~~~
lotyrin
It's because most software is under the umbrella of office people playing
grown-up and not something where reality matters or thinking happens.
Engineering disciplines, when faced with uncertainty do some math and decide
that if the deviation of possible quantities is too close to the tolerable
limits of those quantities, you have to adjust one or the other.

High school level math and problem solving plus having ever worked with
software clearly state that if you have a software team, you could at the very
least collect a single statistic of estimation error (actual effort over
expected effort) and get a long-tail (it's going to be more more than it'll be
less) distribution. Now you can spec out new projects, get values for expected
effort and factor in the error distribution to get a distribution of possible
actual effort for the project. If some high percentile of that distribution is
more time/money than you can tolerate you have to reduce risks by cutting
features or increase the budget (or have a significant fraction of projects
fail).

Of course, there's lots more sophistication that could (should) go into this
but even this is beyond any part of software industry I've been allowed to
participate in, because it gives answers that people with authority find
inconvenient.

~~~
UK-AL
Yes, that is part of the problem. People asking for estimates arn't looking
for probability distribution of potential completion dates.

They are looking for estimates to hold you to account, because that's how you
run a good business right?

If you gave your answer to someone in management. It would just look like your
trying obfuscate the answer.

~~~
jacques_chester
> _They are looking for estimates to hold you to account, because that 's how
> you run a good business right?_

There's a difference between estimates and their misuse.

~~~
UK-AL
I don't think most business leaders are looking for estimates, where the
developer admits it's probably way off, but it's his best guess.

Or huge range of possible release dates(because there is huge variation
between the estimate and actuals), as with other estimation methods.

------
didgeoridoo
OP means this in the sense of "good estimates for software are impossible",
but I submit that the alternate reading also obtains: that estimates for GOOD
software are impossible. This is because, pace Brooks, the process of creation
is also the process of eliciting the requirements that demand that creation in
the first place. "Good" software, software that solves a real problem in a
feasible way, is rarely so well-defined at the outset that your only risk
factors are difficulties in finding the right library or algorithm. Frequently
it is only possible for requirements to be understood once an early version of
a solution actually exists.

~~~
wvenable
Perfectly well-defined software is completed software. The act of defining the
requirements completely and absolutely is writing the code.

This is what distinguishes software development from other disciplines. If you
want to build a bridge, you create the plans and derive your estimate from the
plan. But software development is the creation of a plan. Do engineers know
exactly how long that planning process takes? Probably not. It's insignificant
compared to actual construction. But for software development, that is
everything.

~~~
learc83
Exactly. I think we really should be comparing software development creating
processes instead of creating objects.

Look at something like a consultancy that develops safety procedures for
organizations or a company that handles building logistics operations.

Software doesn't have the same physical constraints that bridge building does.
Anytime you have fewer constraints, you have more possible solutions which
take more time to sort through.

------
vmarshall23
I've always thought, if you can, with reasonable accuracy, predict how long it
will take to build a piece of software, then it's probably already done, and
you should just apt-get it or whatever.

However, if it's something truly novel, then kind of by definition, it's
impossible to predict how long it will take, because it's never been done
before.

Of course, there's a million shades of gray in between, but it's not like
bridge building. Bridges are pretty unchanged in 1000's of years. Most
software you write, should be brand new, otherwise, why do it?

~~~
BurningFrog
> _it 's probably already done, and you should just apt-get it or whatever._

This leads to the next quagmire:

How do you reliably estimate how long it takes to find the already written
software that does X?

~~~
vmarshall23
Heh. How long does apt-cache search take. Kidding, and I do recall that being
a real problem for most things at one point. You'd go to FreshMeat and just
_hope_ that the highest rated one was good. But, re: apt, if it gets in the
main repo, then I'm hard-pressed to think I'm smarter than all those people.

That said, for some classes of software, Drupal plugins and Minecraft mods
come to mind, are all varying degrees of abysmal. Those take a lot longer to
sort through.

Things like "evidence based scheduling" and your permutations around
"velocity" and burndown charts seem to _help_ , particularly over time, but
I've found no magic bullet.

------
gh1
I think it boils down to this.

1\. If you are doing something that people have done before, then it is
possible to come up with a reasonable estimate. How? Just talk with someone
who has done something similar before and they will help you understand the
complexity involved. Example : Rubik's cube.

2\. If you are doing something that no one has done before, then it is more
difficult to come up with an estimate. Example: Unified theory of forces. In
the 1970, people assumed that we are very close. It's been 45 years since then
and we aren't any closer.

Of course, I have often made the mistake of underestimating tasks of type 1
because I didn't talk to anyone with enough experience. Maybe this has to do
with self confidence. I am very confident of my abilities so I usually tend to
underestimate things rather than overestimate them.

~~~
the_duke
Even 1.) is very hard to estimate, and info from people who did something
similar is of quite limited value.

Skills, technical environment (system, programming language, libraries,
available tooling) and team efficiency vary A LOT.

\-- Regarding your comment of overestimation: I think this is a problem for
almost everyone. When you get a task described, you often think "hey, this
will be easy", without understanding all the requirements in detail, the
complexity it will entail or the broader effects in a system. That's where
only experience can help (your own or someone elses input)

~~~
gh1
On a different note, I have heard it being said that it is better to
overestimate than underestimate. This is apparently the best way to manage
other people's expectations and maintain credibility.

~~~
matwood
Over is probably the wrong word. I call it realistic estimation. Every
stakeholder/business person wants it done yesterday. Their job is to push and
prod and squeeze out as much efficiency as possible. It's the estimators job
to give a realistic estimate, which often means pushing back to a lot of
pressure, and that can be very hard. It's important to realize the most
important thing for the business is an estimate as close to reality as
possible though. This point is often missed when there is pressure to estimate
it shorter, when in fact it is not the estimate the business person wants
shorter, but the complete task.

And yes, it does take time to build that trust. Often times when I'm at a new
company, the business people are used to short estimates and overruns. I have
to train them to understand that while my estimates are longer, they are
typically closer to correct. It only takes once or twice of having something
done when expected to win them over.

~~~
MaulingMonkey
> Over is probably the wrong word. I call it realistic estimation.

I generate realistic estimations by attempting to overestimate and failing to
do so sufficiently.

Watching coworkers try to estimate how long it'll take to deliver software
really makes you wonder if pessimists actually exist, or if they're rather
some kind of myth or mirage.

------
booleandilemma
James T. Kirk: Scotty, progress report?

Montgomery Scott: Almost done, sir! You'll be fully automated by the time we
dock.

James T. Kirk: Your timing is excellent, Mr. Scott. You've fixed the barn door
after the horse has come home. How much refit time before we can take her out
again?

Montgomery Scott: Eight weeks, sir -- [Kirk opens his mouth] -- but ye don't
have eight weeks, so I'll do it for ye in two.

James T. Kirk: Mr. Scott. Have you always multiplied your repair estimates by
a factor of four?

Montgomery Scott: Certainly, sir. How else can I keep my reputation as a
miracle worker?

James T. Kirk: [over the intercom] Your reputation is secure, Scotty.

[https://m.youtube.com/watch?v=t9SVhg6ZENw](https://m.youtube.com/watch?v=t9SVhg6ZENw)

~~~
cbanek
This is good advice and I've actually followed it for estimating software
projects. If you say 4x of what you think, and your boss/client doesn't balk,
you probably have it right.

~~~
haalcion3
I'm roughly between 3x and 10x off, depending on the type of work. For
greenfield work on small well-understood tasks and with application and test
infrastructure setup, I'm ~3x off. Constantly changing complex applications
without tests, I'm ~10x off. However even these multipliers have a 1-20x
margin of error.

------
matchagaucho
It puzzles me why some insist on being called "Software Engineers", yet don't
invest in automated, repeatable practices (i.e. true _Engineering_ ).

By all means, take risks and explore Software. But do it under the premise of
Software _Development_ if Engineering practices will not be applied.

In agile environments, push for more "spikes" if estimates cannot reliably be
provided for a story.

~~~
jerf
"invest in automated, repeatable practices (i.e. true Engineering)."

We do. We call those "libraries". We have more of them than most disciplines
could dream of, and they work better than most disciplines could dream of.

So where most disciplines are taking the closest things to libraries they have
and still applying a lot of drudge work to put them into the current context,
with "unexpected work" forming small single digit percentages of the work, we
get to yank all the easy drudge work off the shelf and the resulting
"unexpected work" makes up the majority of what we're doing.

I'm not going to pretend everything is perfect in the land of software
engineering, but the answer to "why are we so stupid that we don't do the
obvious thing that people have been complaining about for 50 years now?" is
that we aren't that stupid. We are very good at shrinking the obvious, easy
parts of the problem down, so all that's left is the hard part. Which, since
everything isn't hunky dory in the world of software engineering, we also
often add to with self-inflicted injuries. Still, the amount of off-the-shelf
crap that I can just push a button and have staggers the mind.

If "real engineering" worked like computer science, we'd be down to a guy
pointing and five minutes of fiddling with his phone to lay a bridge down
across the average river, four minutes of which would be rebuilding the bridge
in various different aesthetic styles to see which suited the location best.
"Do I like the fake aged wood look, or do I go with modernistic concrete?
Let's see the wooden one again...."

Your homework is to write "hello world" in your favorite language and step
through the resulting program in gdb or equivalent, starting from the moment
the OS turns control over to the program, one assembler instruction at a time.

I apologize if this comes off a touch crabby, though upon re-reading I'm not
sure exactly what to tone down about it. I'm getting really tired of the
inferiority complex programmers have about "real engineering". Yes, we do real
engineering. If other engineers had the advantages we had, they'd be much more
"agile", and they'd make the same messes we do. They work the way they do
because of the constraints they have on them, for exactly the same reason the
space programmers work the way they do and for exactly the same reasons it
would be wildly inappropriate for all of us to work the same way NASA does all
the time.

~~~
jcrites
Great points! One other remark I'll share about software engineering is that
the _amount_ of engineering rigor required or warranted depends a lot upon the
task and problem.

If I'm building a back-office website for a few colleagues, I may not invest
the same rigor as if I'm building a website that will be accessed by hundreds
of millions of people. Many software tasks do not require great rigor: you add
rigor until the incremental benefit is less than the incremental cost.

As a result, I would guess that the average software engineering task is less
rigorous than the average real-world engineering task, but that's only because
real-world engineering frequently risks or protect lives. Even the tire rope
swing, the hammock, and the treehouse put lives at risk if built poorly. The
back-office website on average does not. Business might be disrupted or
individuals might be inconvenienced, but typically no one can die.

Yet, many types of software involve just as much rigor if not more as other
engineering disciplines. For an example, read "They Write the Right Stuff", a
story of NASA avionics software engineers [0]. I consider software engineering
to be an engineering discipline because of the fact that rigor is required,
and because a good engineer must know how much of it is appropriate, and be
able to apply it effectively. A software engineer produces software to
specified reliability and tolerances, according to requirements.

[0] [https://www.fastcompany.com/28121/they-write-right-
stuff](https://www.fastcompany.com/28121/they-write-right-stuff)

------
segmondy
This is hogwash.

As a team lead who lives in this world.=, good software estimates are not
impossible. If you are doing the kind of work that you have done before, then
it's very possible. If you are doing R&D, then sure it's damn near impossible.
But if the work and requirements are clear, then there are plenty of ways to
make "Good software estimates".

I fight estimates when I don't have a good business requirement, but tell me
what your problem is and the solution you desire and I will give you a good
enough estimate. Estimate is never just a date, it also includes the unit of
work to be done, and the resources.

~~~
UK-AL
If your doing similar work over and over again as a programmer, your not doing
a good job.

~~~
jacques_chester
Now suppose I substituted a different profession here:

> _If you 're doing similar work over and over again as a civil engineer,
> you're not doing a good job._

Hmm. Not sure I feel comfortable with that.

> _If you 're doing similar work over and over again as a small business
> accountant, you're not doing a good job._

Actually, I'd rather prefer you to stick to the basics, please, no need for
discussions of complex derivatives of treasury stock.

> _If you 're doing similar work over and over again as an orthopaedic
> surgeon, you're not doing a good job._

Thanks, but I think I'll get a second opinion.

~~~
UK-AL
Civil engineering and software engineering are fundamentally different because
software is digital.

You can't download the building package. You actually have to build the
building again. If it's the same building I would expect them reuse the same
blueprints. So that holds true.

Where as in software it's quite easy and desirable to reuse software. Which is
why you should not be rewriting the same thing over and over.

As a result, the problems you face are unique. Your not just stamping out the
same building again and again.

~~~
jacques_chester
> _You actually have to build the building again._

You also have to factor engineering time into the plan. If you read
_Industrial Megaprojects_ , the author shows that projects which _don 't_ do a
bit of upfront estimating and research tend to go off the rails on a scale
that software struggles to match.

~~~
learc83
Some upfront planning is required but, there is an overall problem with
comparing software to building construction.

If you make a blueprint and give it to 2 construction companies to build the
building, you'll end up with 2 very similar buildings.

If you give a software spec to 2 separate software companies you'll get 2
completely different programs.

You can't create a blueprint/spec that is sufficiently detailed so that 2
different companies/teams will build essential the same software.

A blueprint/spec that detailed would be the code itself.

The closest analog to software would be designing processes, or maybe
designing a factory or assembly line. And the analog to the code in that
situation would be the factory or assembly line itself.

~~~
timdev2
Software shouldn't be compared to building construction, more appropriately
compared to building design. Building construction is analogous to software
deployment: a largely solved problem that is predictable.

------
siliconc0w
One thing i'm a big fan of is the 'eta till the eta'. So many
developers/managers often shoot from the hip with estimates - usually with the
rationale that if it's all made up anyway might as well seem smart.

Yet if you spend just like 20 minutes to try and break down the tasks and do
some research you can usually reduce the margin of error significantly. Our
brains are good at ignoring complexity and our egos are good at telling
ourselves we understand things we don't. Break it down and do you research.
Stuff like poking through github repos of libraries you might be adopting.
Lots of forks? lacking documentation/tests? What kinds of issues? Who is using
it and why? You still can certainly still run into unexpected complexity but
you get good at recognizing where it's likely to pop up. It's hard, and you
won't get it right always, but it doesn't mean that shit is 'impossible'.

------
dsjoerg
the reason good software estimates are impossible is because the
career/political cost of your project being late is less than the
career/political cost of making a longer (and more correct) estimate and
sticking to it.

~~~
medina
too sad, too true :-(

------
ktRolster
The trick is not to say, "It will be done at exactly time X," but rather to
say, "There's a 95% chance it will be done before time X."

For people having trouble with this, Jim Shore does a good job giving tips:
[http://www.jamesshore.com/Agile-
Book/estimating.html](http://www.jamesshore.com/Agile-Book/estimating.html)

~~~
bottled_poe
While this may be true, most clients/managers won't accept that sort of
answer.

~~~
ktRolster
They all accept that kind of answer......it's a matter of phrasing it
correctly.

------
zenogais
Another one of these?

Bad estimation techniques - eg. guesses without a basis in historical data
(most developers don't collect historical data) - are usually wrong. No one
would argue this point.

If you do iterative development with intelligent planning and collect some
historical data to use in estimates the game changes. First, most of the
examples of "unexpected complexity" you gave can now be identified early and
factored into your estimates. That leaves the 1% of truly unexpected stuff
that you can then deal with as it comes up and factor into your future
thinking in a post-mortem. Second, since you now identify risks earlier and
can factor them into estimates, most of your work becomes estimable, and with
historical data your estimates will improve.

There's even multiple process products around this idea already, but they're
mostly used in the "safety critical" industry, see [1] [2].

[1]: [http://resources.sei.cmu.edu/library/asset-
view.cfm?assetid=...](http://resources.sei.cmu.edu/library/asset-
view.cfm?assetid=5283) [2]:
[http://www.sei.cmu.edu/tsp/](http://www.sei.cmu.edu/tsp/)

~~~
jayvanguard
The SEI work has been around a long time and it has had very limited success
outside of academic or very controlled settings. It had its heyday in the late
90s but never really caught on broadly because it is impractical to collect
the level of detail necessary, with consistent teams, languages, environments,
and practices in a fast moving computing environment.

------
mslate
If you want a job as a software engineer, then you have to estimate.

------
amelius
> Say that you are an experienced programmer, and you only run into
> unexpectedly complexity about 5% of the time. If you are starting a new
> project and split it up into 10 tasks [...] there’s a 40% chance that you
> will run into a complex task, and blow your estimations for this project.

Solution: divide the project into fewer tasks :)

~~~
snarf21
Right, it is like asking a builder "how long to build my new house?" Way too
many variables and things that are independent variables.

Prioritized (short) iterative development is the only path to success.

~~~
ewjordan
Except that if I'm hiring someone to build my house, if they tell me "well, I
can guarantee you that we'll prioritize iteratively and get it done as fast as
possible" I'd tell them to piss off, I can't throw good money at that. You at
least need a baseline estimate, and some data as to how often things go over
or under.

Apparently software engineers as a group seem to think "fuck you, it's done
when we feel like it" is a valid answer?

I disagree, and have always offered estimates, probabilities and contingencies
when talking to product owners when I was a lead. I tracked data, and adjusted
accordingly, and ended up giving estimates that were statistically sound.

I don't necessarily expect individual contributors to do the same, but anyone
that's a lead should be an adult and account for their team's actual
productive capacity. This is absolutely expected of a building company, even
if you don't expect a single dude with a hammer to be able to estimate how
long the project will take.

~~~
learc83
Given the number of software estimates that fail and the number of software
projects that collapse, the vast majority of developers are incapable of doing
so.

If you are giving regularly accurate long-term estimates, I'd suggest that you
create a consultancy that performs this service or teaches how to do it. Most
companies and teams just aren't capable of doing this.

------
eb0la
Quoting Charles Simonyi (1986):

"SIMONYI: We have great difficulty in predicting the time it will take to
write a program. There are valid reasons why this is so. It doesn’t mean we
shouldn’t try our best to predict, because there are reasons why a prediction
might be useful, just like when a weather prediction has economic as well as
other benefits.

Really good programs will live forever and take forever to write, at least as
long as the hardware exists, and maybe even longer."

Source: [https://programmersatwork.wordpress.com/programmers-at-
work-...](https://programmersatwork.wordpress.com/programmers-at-work-charles-
simonyi/) (great book, btw!).

------
partycoder
If you have done it before it's easy to estimate how much it's going to take.
Some things are actually really hard to estimate, like estimating how much
root causing and fixing a bug will take.

Now, if you want to estimate properly... ask for a min time estimate, max time
estimate, and confidence level.

To say "I am not sure", is actually fine. That can be the start of a
conversation where you can have the opportunity to improve that estimate.
Whereas a closed answer like "4 hours" won't.

------
batoure
This is an interesting read, mostly because I remember working in the world it
was murky and depressing. The unexpected happens, this is an undeniable truth
but blowing estimates because of the unexpected should be an edge case and not
the norm. There are 3 key thing the team I work with now does to manage the
unexpected.

1\. Top level commitments should be vague

-the statement "we will build you a website" says nothing about what will be on it

2\. Scope the top level commitment, don't over scope, commit only to
achievable scope

\- If you don't know if something can be done don't commit to it, commit to
identifying if it is achievable. Green engineers I have worked with tended to
miss deadlines and when we dug into it the problem was often that they took on
tasks we didn't even want completed. Commit to delivering something you are
certain you can do.

3\. Write the scope down, and stick to it.

\- A good scoping document is like a good legal contract, it keeps everyone
honest while limiting liability. It gives someone a commitment they can hold
you to, and it gives you an exit clause if they try to rewrite the rules. If
someone tries to change your agreed scope after the fact, adding work means
timelines must be adjusted and reducing timelines means work must be removed.

4\. If you work somewhere where you can't seem to make 1-3 happen find a new
job its just not worth it.

~~~
UK-AL
The problem with scoping a project is that no one really knows what's they
want at first.

Developing software also involves discovering the real user issues by
experimentation.

If you scope it, you may deliver the project, but you probsbly haven't
delivered what they actually needed.

The best software is developed with feedback loops.

~~~
bbcbasic
Yes it's no good producing a turd, even though it's an on budget and on time
turd.

------
DonaldFisk
Large Limits to Software Estimation:
[http://scribblethink.org/Work/kcsest.pdf](http://scribblethink.org/Work/kcsest.pdf)

Mathematical Limits to Software Estimation: Supplementary Material:
[http://scribblethink.org/Work/Softestim/softestim.html](http://scribblethink.org/Work/Softestim/softestim.html)

------
sharpshadow
It's even worse than that. Knowledge decay let you run into complex tasks
earlier. Your acquired experience gets less useful over time or as stated in
this article: Half of what a programmer knows will be useless in 10 years.

The way to access a Rubik's cube stays the same whereas accomplishing a coding
task is enviromental dependend. Software changes over time making your
experience how to solve a task redundant at some point.

------
pcai
This is wrong. If the variables that affect estimates can be predicted, then
they can be accounted for.

Source: i used to work at an organization you've heard of that could reliably
predict, down to the day, how long it would take ~4000 engineers to build and
release the next version of its product. Three years in advance.

~~~
wvenable
I almost think it's far easier to estimate the output of 4000 engineers than
the output of 10. It's also far easier to cut things to fit a deadline 3 years
out.

------
kol
The classic in the topic: Evidence Based Scheduling, by Joel Spolsky
[http://www.joelonsoftware.com/items/2007/10/26.html](http://www.joelonsoftware.com/items/2007/10/26.html)

------
ww520
Good software estimates are impossible because business stake holders don't
like the true amount of time to get software done and always push to shorten
the estimates.

------
hakanito
Someone once said, when estimating software projects you should multiply the
time you think it will take with Pi. It has been fairly accurate so far

------
z3t4
u can build a house with just walls and a roof. Its important to understand
the requirements. Then 80% of the project time will be spent polishing, so it
helps to know the amount of polishing expected.

------
Tiquor
Software development of novel ideas is art, not manufacturing.

------
known
Software Requirements != Customer Expectations

------
jomamaxx
It depends largely on what you are building and the granularity (and validity)
of the requirements.

