
Software Estimation Is a Losing Game (2014) - tustlem
https://rclayton.silvrback.com/software-estimation-is-a-losing-game
======
kzhahou
I can estimate some tasks very accurately. I know that adding this new flag to
my system will take a day.

Other tasks I can accurately say I don't know how long they'll take. This new
feature is... a couple of weeks? This new application is... a few months? I
can roughly guess (estimate!) but there's a wide certainty margin.

That uncertainty is never built into the mathematical model used by project
managers, which is invariably a simple addition of scalar estimates. The only
common way of dealing with uncertainty is to _pad_ the estimates (multiply by
two, etc), but that's still not a statistical boundary -- it's just a scalar.

Furthermore, when adding uncertain values, they wind up not being independent,
for a number of reasons -- the lessons from one feature are carried on to the
next, or the next task is easier because of new existing code, or because as
you approach the original deadline you invariably throw out feature
requirements. So you can't necessarily just add all the uncertainty values.

So a better approach to estimation would be to formulate a ( _simple_ ) model
which encodes each estimate as {guess, uncertainty, independence} and then
provides a function for compounding the guesses and uncertainty and
independence factors in a realistic way.

~~~
wai1234
So, small tasks you CAN estimate, large ones not so much. Perhaps the
technique to use is right in front of you: decomposition.

~~~
ddingus
On a small task, everything is small. The percent error can be totally gross
(not precise at all) and not have a meaningful impact overall. A large error
on small is just a multiplier of small. It's all small.

When you decompose a larger task into smaller ones, that's all still true for
each of the smaller ones, right?

But you must remember to add the errors! And that is precisely why overall
confidence may not likely improve with decomposition vs. an aggregate, or
coarse, high level estimate of the whole thing.

And this is true for more than software. Why?

New tasks can be broken down into two coarse buckets: done before and novel.

Novel things really are novel! It's important to recognize this case and
qualify it. Progress might look like:

week 1 - 10 percent done, week 2 - 10 percent done, week 3 - 15 percent done,
week 4 - 50 percent done! Yes!, week 5 - 5 percent done... everybody take a
drink, week 6 - 40 percent done,

etc...

Because it's novel, there are new dynamics in play, and even the best planning
may not account for, and actually usually does not account for, things that
just come up and that must be dealt with.

For the done before case, it's better as a lot is known, but there are still
many factors out of the team control. Updates, changes, regressions, unfound
bugs, all contribute to variance. The more the organization completely owns,
the less this may be true, but it's always somewhat true.

The pragmatic person might classify everything as having some percentage of
novel, and break it down into known novel and unknown, "it just hit us!" type
novel and go from there and likely be better off when having to manage
expectations.

Another way to think of this, depending on it being software or something
else, is production vs "R&D" and the research and development part just isn't
cog / production like. Time and materials estimates, if they are even given,
must be considered accordingly.

If they are not, expectations do not match realities, and now there is a
management and potentially a funding problem in addition to the primary
problem needing a solution.

------
thoman23
One of the comments on the original article nails it. The real problem is
conflating "estimates" with "deadlines" or "commitments". You see it with the
casual replacement of "story estimating" with "story sizing" in our language.
"Sizing" implies a degree of certainty that may not be there.

Estimating is most definitely useful. But you have to fight every step of the
way to keep an estimate from becoming a "promise". Promises that can't be kept
are not good for any of the parties involved and lead to things like developer
burnout, poor quality, technical debt, surreptitiously reduced scope, lack of
trust...it goes on and on.

------
roel_v
Pardon my French, but it's a load of bollocks. Sure, I hate estimating and I'm
bad at it, but I've worked with many others who were better than me at it and
_did_ consistently deliver what promised, when they promised it. And yes I'd
much rather just sit and contemplate and re-re-rework 'the design' and be all-
important about 'it's done when it's done' \- but I fully well understand that
any customer will just fire me and take someone who just does what he says he
will.

Building construction is also fraught with unexpected events and project
management issues etc., but I'd laugh any building who would propose that I
just keep paying them until it's done, or who will start building and go home
when whatever budget I give him is used, out of the room. Why would my
customers be any different?

(articles like this poison the minds of young impressionable developers who
think it's actually reasonable to have ideas like this, btw, which is why we
shouldn't just shrug them off)

~~~
Nyetan
Actually, that's an excellent point -- how do buildings/bridges/etc. actually
get built? I mean, laugh if you will, but can you not think of any examples of
construction projects that straight up ran out of money and had to halt? Or
that became enormous money and time sinks, inching their way but never-quite-
making it to completion?

As I understand it (still pretty new to the industry), we're supposed to give
realistic estimates for a product's completion, considering that 1.
requirements are always incomplete and 2. unexpected complications will always
arise. 'Done' is a moving target, and part of its estimation is ensuring the
product is deliverable at every stage (with varying degrees of completeness),
regardless of whether how many times everything goes pear shaped.

~~~
tikhonj
Sure. Developments get abandoned all the time, either because they run out of
time and money or because economic conditions change. One I was reading about
recent is the Fontainbleu Resort in Las Vegas[1]—the second-tallest building
on the Strip and unfinished.

And even more developments end up hilariously past their budget on both time
and money. "Megaprojects" like the Big Dig[2] are common examples of this.

These are the failures big enough to be notable. I'm sure there are far more
smaller, less interesting but more indicative failed projects as well.

[1]:
[https://en.wikipedia.org/wiki/Fontainebleau_Resort_Las_Vegas](https://en.wikipedia.org/wiki/Fontainebleau_Resort_Las_Vegas)

[2]:
[https://en.wikipedia.org/wiki/Big_Dig](https://en.wikipedia.org/wiki/Big_Dig)

~~~
Nyetan
It was more of a rhetorical question, with the remaining literalness directed
at the OP :) but yeah, those are fantastic examples! The Berlin Brandenburg
[1] was what came to mind immediately, but you're absolutely right that
history is like to be littered with other dead developments.

[1]: [http://www.bloomberg.com/news/features/2015-07-23/how-
berlin...](http://www.bloomberg.com/news/features/2015-07-23/how-berlin-s-
futuristic-airport-became-a-6-billion-embarrassment)

------
sbuttgereit
Software time/cost estimates are essential. If I run a business, I need to
have some understanding if an investment will have a return. If I have no way
to gauge that at all, then there are many more predictable endeavours where I
can spend the money and realize a return. When you invest, do you simply say
the future is unknowable and bet on raw emotion? I should hope not. That's not
to say you may err in your judgment of available facts, you may still lose,
but at least you understand some of the parameters about what constitutes
success.

Software time/cost estimates are exactly that: estimates not guarantees. We
tend to think of such things in terms of project management and contract
enforcement. If anything, that is where we go wrong. What we should do is not
try to understand the whole project in terms of an unalterable roadmap, but
rather as a guideline for managing expectation and for informing decisions
that are made during the course of a project. I use my estimates to sell the
project, yes, but I also tell the client that as we go along we will be making
many decisions that will move us away from our initial plan as we better
understand both the issues and opportunities a project affords. Sometimes
spending more time than planned will make sense, sometimes it won't and we cut
the feature. Sometimes we find that we have failed to identify important
features and sometimes we identified a feature early on that isn't as
important as we thought. The overall project plan and estimate does set a
benchmark to ensure the economics are still correct, after all if we lose that
we lose the point of the project entirely, but we don't turn that plan into
dogma. We don't lose sight of the reason that caused the project to exist.

So, yes, we need estimates no matter how hard they are to get accurate. What
we need to stop doing is equating that estimate as an oath to be upheld
regardless of what new knowledge comes along.

------
jcadam
If I'm ever asked for an estimate, it's after the project schedule has already
been set in stone.

On my current project, I am the sole full-time developer. The requirements
were gathered from the client without my involvement. The schedule was laid
out without my input. Then the whole thing was dropped on me.

~~~
moron4hire
I really wish more articles on software engineering practices would at least
_acknowledge_ that this is the reality for a huge number of developers in the
real world. I just can't take them seriously without it.

If your new-fangled-to-you development ideology starts by assuming a
fundamental level of congenial cooperation between different departments of
your organization, then you've designed your system for a fairy-tale world. I
don't suspect most people are able to change organizations easily. I suspect
most people are stuck dealing with the hand they've been dealt. The lies they
were told during their interview. The other employees that lied during their
interviews. It's a really messy world out there.

~~~
mikekchar
I think the problem is that software process engineering does not really
include the skills necessary for transforming organizations. To be honest, I
don't know of any really good software process person who thinks that a
process will do anything on its own. The processes and practices are only half
of the battle.

It gets even more complicated because, at least in my experience, software
processes are really subtle and it is extremely easy to _appear_ to be "doing
the right thing", while at the same time failing miserably at it. Throw in the
power politics, the bad actors who don't really care if something is
successful or not (as long as they can extract money from it), and naive
developers who are _absolutely sure_ they know what to do but really don't,
and it makes it very difficult.

It is difficult to be the one instigating process change. Such a role is
fraught with danger. You will be attacked by the power politics people. You
will be attacked by the money extracting people. You will be attacked by the
naive developers who think they know better than you. You almost certainly
will not be thanked by anyone (except very far down the road if you somehow
manage to succeed... and even then probably nobody will remember it was you
who started the ball rolling). And finally, because it is incredibly hard and
the necessary skills are very difficult to acquire, you will almost certainly
fail many, many times before you ever succeed.

Even worse, it's not something you can write down and say "Do X,Y, and Z and
everyone will be a happy family working hard to ensure success". It is
something that requires dedication to understanding every single individual
who is involved in the development. You have to sit and listen to them,
understand their goals and help them meet those goals (even if they aren't
"Make a successful software product" and even if you personally think the
person is unbearably awful). You have to devote yourself to finding common
ground and bringing everyone together.

Don't get me wrong. The processes and practices are crucial to a successful
organizational change. These practices, if well chosen, will support you as
you influence, cajole, flatter, criticize and motivate the people around you.
It is next to an impossible job and without these tools, you will almost
certainly fail.

It is unfortunate that people dramatically underestimate the difficulty of
cultural change in an organization. They write about the easy bits (the
processes that support the change) and sweep the hard bits under the carpet.
To be fair, I'm not sure what you would say that wouldn't sound like "business
BS" to most engineers, though.

Anyway, most organizations do not have many people conscientious (possibly
read "stupid") enough to attempt real, lasting cultural change. The vast
majority that do try are often incredibly naive and lack all of the basic
skills necessary to be successful. The end result is unfortunately all too
predictable.

To anyone who is really serious about building an amazing team, I encourage
you to suffer the slings and arrows of outrageous management and to work on
the people skills that will eventually bring you what you want. It sucks for a
very long time, but as you get a better understanding, it can get better.

------
lifeisstillgood
A more appropriate question is not, does software estimation help improve
project management, but is project management a suitable model for software?

I am a proponent of the idea software is a new form of literacy. We generally
do not project manage the next paragrpH or the next chapter. writers are often
faced with deadlines, but rarely have to justify themselves on a daily basis.

Improving estimation will just result in slightly better inputs to a process
that at best hinders us and sucks creativity and joy.

~~~
knieveltech
Creativity and joy are fantastic qualities that are difficult to quantify on
an accountant's spreadsheet. Clients have to be able to budget sufficiently
for project work, and the only way they can do that is with an estimate. This
is how business is done. "Uh..we'll send you a bill when we're done
for...like...whatever it costs. I guess?" Isn't going to go over well when
responding to a potential clients RFP.

~~~
oldmanjay
Of course, given the historical fact that traditional project management
doesn't work and the estimates are a farce, why should we happily continue
doing this cargo cult uselessness?

~~~
AnimalMuppet
They're not useless. Except sometimes they are.

I have to give estimates all the time. They're estimates, but they're in the
ballpark.

But recently, I had to give an estimate for porting some software to a new
environment. I knew it was going to present me with a bunch of obstacles, and
I didn't know what the obstacles were, or how long each one would take. _I
didn 't even know how many obstacles there would be._

I didn't want to give an estimate for that. They made me give them one anyway.
I did, _but I knew that no reliance whatsoever could be placed on the
estimate._ Whereas other tasks, estimates were close to "bet the company on
it" solid.

~~~
knieveltech
When we run into stuff like that at work we budget a set number of hours to
breaking the problem up into chunks, researching and prototyping the sticky
parts, and then using that information to put together a realistic estimate.
We call it a "planning project".

~~~
lifeisstillgood
That's sensible - and is a way to lose bad clients (not itself a problem).

I am not saying project mgmt does not work, I am saying the plan the task, do
the task approach is not appropriate for software development, at a given
level of granularity.

If you ever watch Grand Designs, they start with a design, a plan, a schedule,
and it all goes out the window ten minutes later. This is not that being on
top of the process will not give you control, but that relying on estimates
and plans as a means of control is worthless - and that's for something people
have been doing for hundreds of years

------
wai1234
I have to laugh when I see these posts. So, software development is a special
activity that should not be constrained by budgets or deadlines (or
estimates)? Where do I sign up? There are so many strawmen here I can't
possibly address them all. If you can't provide a credible estimate for a
piece of work, you either a) aren't very good, b) don't properly understand
the work to be done, or c) perceive some aspect of the job that is unknowable
to you (through lack of experience or other novelty). Fixing each of these is
up to you. 'a' is obvious (you either work hard to get better or find
something else to do). 'b' means you need to ask more questions that allow you
to sufficiently complete your understanding of the problem so an estimate is
possible. 'c' means you need to isolate the novelty and explain that this part
of the problem is new to you and you are not qualified to provide an estimate.
Perhaps someone else can provide that insight or you can define an initial
activity designed to help you better understand the novel problem until you
can estimate the remaining work. NO area of engineering is so cut and dried
that estimation is easy.

What IS common is a combination of poorly defined projects and a profound lack
of self-awareness. By all means, push back against poorly defined projects ("I
can't estimate the work until you tell me what it is."). Lack of self-
awareness is squarely in YOUR lap ("I don't understand why it always takes
longer than I thought it would..." Well, duh, maybe you should consider that
reality for what it is and account for it next time).

Estimating work and committing to those estimates is what a professional does.
Deal with it. If estimates are imposed on you that you do not agree with, that
has nothing to do with the virtue of estimation. That's a social problem, not
a technical one.

~~~
igammarays
"So, software development is a special activity that should not be constrained
by budgets or deadlines (or estimates)?"

Of course it is constrained, but how can you estimate it? Because "software
development" is the continuous process of converting a real world idea into
instructions that are so tiny and specific that a computer can understand
them. Obviously you can't know in advance what those instructions are, or
you've already written the program. When do you stop "pushing back against
poorly defined projects"? You can literally continue asking for more specifics
until the project is complete. This is different from other areas of
engineering, because almost NOTHING demands preciseness like a computer does.

"Estimating work and committing to those estimates is what a professional
does. Deal with it."

The reality is, we have been trying to deal with, for decades. And failing
badly. The author is suggesting (probably rhetorically) that perhaps we need
to abandon the idea entirely, because it is not working (for very large
projects).

~~~
sheepmullet
We can estimate better but it's often not worth it.

It makes sense when there is a large cost to not meeting a deadline. Or when
it's difficult to change scope.

------
DonaldFisk
There are Mathematical Limits to Software Estimation:
[http://scribblethink.org/Work/kcsest.pdf](http://scribblethink.org/Work/kcsest.pdf),
[http://scribblethink.org/Work/Softestim/softestim.html](http://scribblethink.org/Work/Softestim/softestim.html)

In general, unless you've done something very similar before, estimation can't
be done accurately. Your manager should know and accept this.

The best you can do is probably this: Repeatedly split the project into tasks,
estimate the time it will take for each, until the next task takes several
hours, and then do the next task. Keep track of estimates and actual times
spent. Your manager should ask you to do this and keep it updated. Joel
Spolsky wrote something similar here:
[http://www.joelonsoftware.com/articles/fog0000000245.html](http://www.joelonsoftware.com/articles/fog0000000245.html)
(I'm sure I read similar advice elsewhere, but can no longer find it.)

It should go without saying that you should include unit testing, code review,
debugging, etc. in your estimates. And, of course, making and updating the
time estimates.

The project, when "complete", will still usually require maintenance, so work
on the project only stops when the software is no longer used.

You should never be given deadlines, as you won't know in advance what
problems you might encounter, and the quality of your work (and probably your
health) will suffer. This will result in a build up of technical debt,
increasing maintenance costs.

It's well worth reading The Mythical Man-Month, especially if you're managing
a project, or working with others on the same project.

~~~
xacaxulu
The Mythical Man-Month is REQUIRED reading for anyone who might ever come
close to estimating a software project in any setting. +++++1

------
j2bax
I've found that using past data is very helpful. Look at your past projects,
compare your initial estimates to the actual outcome when the project was
finished. After a dozen or so projects, you should be able to formulate a
percentage buffer to help you get closer on future estimates. Continually make
this assessment and eventually you will find that you are getting much closer
to accurate budgets/timelines.

~~~
moron4hire
If I have any past projects that are good indications of my current work, it
means I'm stuck doing the same things over and over again.

~~~
j2bax
I don't think building on your past successes and mistakes and using the raw
data to improve your future endeavors should be considered stuck doing the
same thing over and over again. Surely there will be some overlap of knowledge
with each project you take on... Or do you only go for projects with
completely foreign problems to anything you've ever worked on before?

~~~
moron4hire
I've never worked on two projects that did the same things other than "let
people login", which I usually have settled in the project before anyone even
_starts_ talking about estimates. I've never worked on two projects with even
the same reporting frameworks. Hell, only a few of them have even been in the
same _application_ framework.

I've never worked on two projects with the same team. There's as much learning
about how to work with new, particular individuals as there is to learn about
what the client needs.

Insomuch that anything has remained constant, it was pretty much just the
underlying programming language, but even that has changed dramatically over
the years, as well as opinions on the best way in which one should write it.
Certainly my understanding of it has drastically improved.

And I'm not exactly sure what you're suggesting by "raw data". What sort of
data? Collected and stored how? Reported on in what ways?

------
triggercut
_Open article_ _Ctrl+f "Risk Management"_ "Zero results" _Ctrl+f "Risk"_ "Zero
results"

Now, I don't work in software engineering, I work in traditional engineering,
but I find the lack of any formal Risk Assessment, Management or Lessons
Learned in any Project Management methodology/framework a little scary.

------
Nadya
I use my best judgement to estimate the time the task will take. I then double
that number and report that as my estimate. For times great than one month -
multiple by a time and a half instead.

One month? No, six weeks. Two hours? No, four hours.

The reality is typically somewhere between the two numbers. So the estimate
remains fairly accurate and most things are finished ahead of schedule.

There are too many variables that cannot be accounted for when giving an
estimate. How often am I going to get interrupted? Is there going to be legacy
code that causes a bug to surface and has to be refactored? Etc.

------
jacques_chester
Aside from starting with some nirvana fallacy at the beginning, the conclusion
is basically an accidental advertisement for XP.

