
Poor developer estimates, a.k.a. guesstimation - renderrobot
http://www.alexefish.com/post/5234c52057d0aa0015000002
======
rfergie
Also be aware of the selection bias when it comes to estimations.

The companies A, B and C all put in estimates for a product based on similar
premises.

Company C gets selected because their estimate is the lowest.

This could be because company C is the best and can do the work quicker but it
is also possible that company C has just made a low estimate pretty much by
chance.

Estimates that are wrong (lower than the actual cost) are more likely to be
selected so it always looks like estimates are bad even if all the estimates
submitted actually do cluster around the true value

~~~
jacques_chester
I think you're mixing bids with estimates.

~~~
InclinedPlane
Is there some Universe where developer time is not synonymous with cash money
dollars?

~~~
jacques_chester
No, I mean, the process of inviting tenders or bids is distinct from
estimating.

Bidders will inevitably perform _some_ sort of estimate, even if it's only a
wild guess. But even the most rigorous internal estimation process may then be
passed to management or sales staff who will use it only as a starting point
for the bid.

The _estimate_ didn't change; instead it was an input into the _bid_.

And in as the grandparent comment demonstrates, buyers often pick the lowest
bidder.

------
iancw
I recently read Software Estimation (Steve McConnell, author of Code
Complete). My main takeaway was to distrust developer intuition and instead
build estimates from data. One example is to look at the duration of past
projects of "similar size" (still requires some qualitative analysis). Another
example is to count the number of distinct "features" or "components," and
build an estimate from those. Both of these still require some qualitative
analysis. That will be the weak point of any estimate. The goal is to minimize
inputs from developer intuition and use as much quantitative analysis as
possible when estimating.

~~~
sateesh
My main takeaway from the McConnel's book has been not to provide an estimate
as a single point number (say something like it will be done in 5 weeks) but
rather as a range (say something like 5-8 weeks), as providing a single point
number would give the impression that it is the target date (while it is not).
But unfortunately most of the times all I have got is blank stares when I give
estimates as a range and a question why don't you give me the time it takes to
finish :(. Wish more number of people who do project management read at least
1st two parts of McConell's book on software estimation.

~~~
DougWebb
I ran into the same problem, and eventually the only workable solution was to
use ranges within the software development group, but to only communicate the
high end of the range to the project management group and executive team. Most
of the time our projects came in "under budget" this way, which seems self-
serving, and sometimes projects we thought we could complete on-time were
shelved as being too-expensive, but whenever we tried to give those groups
better information (eg: the range) they always went straight for the low-end
and gave that out to marketing and other groups as a committed deadline date.

------
onion2k
This is actually a solved problem in some industries using a technique called
"3 point analysis". I've no idea why it hasn't really caught on among
developers; it's very useful. Essentially it goes like this:

1\. Estimate the very best possible outcome. How long would something take if
everything went to plan?

2\. Estimate the very worst possible outcome. If everything that can go wrong
does, how long will the task take?

3\. Estimate the most likely outcome. Knowing what you do about the client,
the task, the tech, etc, how long is it likely to take.

(Point 3 is the normal way of estimating.)

The advantage of 3 point analysis is that you can use the difference between
the times to estimate risk. If the best and worst outcomes are similar, then
there's little risk of things going horribly wrong. If the best and the likely
times are similar but the worst is much longer then there's something that can
completely screw up - identify what that is and defend against it. If the
likely and worst times are similar but the best time is much shorter, identify
why that's the case and try to make sure it happens. And so on.

Better yet, by taking the aggregate best times for a set of requirements and
comparing it against the worst, you can get a good view of the overall risk of
a project.

Very useful indeed.

~~~
jacques_chester
You've gotten to the first step, which is to provide 3 estimates. The next
step is to break estimates into smaller parts, estimate them individually,
then roll it back up to build a pseudo-statistical profile of probable
outcomes.

This is the PERT 3-point estimation technique. It's taught in every project
management course in the world.

It so happens that almost nobody uses it in practice, because it is a pain in
the rear end to do in Excel. It also so happens that I'm working on a tool to
make it easy[1].

All I have right now is a landing page, proof-of-concept code and some
articles. Oh, and a _pile_ of research papers and books. It's turned out to be
a deeper rabbit hole than I expected.

[1] [http://confidest.com/](http://confidest.com/)

edit: how come "professional" is the only word that was spelled wrong on the
landing page? Because Murphy is _hilarious_ , that's why.

~~~
ams6110
_It 's turned out to be a deeper rabbit hole than I expected._

That's quite ironic, given the subject of this thread....

~~~
jacques_chester
I keep a list of subjects for articles I want to write for the website.

One of them is "Why Research Is Almost Impossible To Estimate".

------
msluyter
I've always found this article -- "Evidence Based Scheduling" \-- by Joel
Spolsky to be rather compelling:

[http://www.joelonsoftware.com/items/2007/10/26.html](http://www.joelonsoftware.com/items/2007/10/26.html)

I'm curious if anyone else has tried it?

~~~
jacques_chester
As part of my research on estimations I've been working through Edward
Merrow's book _Industrial Megaprojects_. It's based in part on a large
database of megaprojects that his firm consulted on.

He considers Monte Carlo-based methods to at best pointless, at worst harmful
to outcomes (p 324):

    
    
        The average megaproject cost estimate when Monte Carlo simulation 
        was used overran by 21 percent, with a standard deviation of 26
        percent and with a sharp right skew. When Monte Carlo simulation 
        was not used, teams were actually more sensitive to basic risks as
        they set contingency.
    
        ... The use of Monte Carlo simulation has no relationship to success
        of megaprojects or any of our other five figures of merit of projects:
        cost growth, cost competitiveness, schedule slippage, schedule
        competitiveness or production attainment.

------
PaulHoule
Many developers don't believe that management is going to listen to their
estimates so they either avoid giving an estimate or they tell management what
they want to here.

A friend of mine worked at a place that was 'trying' agile (at the behest of
top management) and had them entering estimates for tickets and recording time
for them.

My friend heard from his boss that his "estimates were too high" and when he
found at that the lead developer wasn't making estimates OR putting time in,
he stopped, and started looking for his next job.

~~~
greenyoda
Exactly. The conversation frequently goes something like this:

Boss: Give me an estimate on how long it will take to do Project X.

Developer: Six months.

Boss: That's not acceptable! We promised it to the customer in one month.

Developer: OK, one month.

------
dbg31415
A lot of the poor estimates come from not taking time to break out the
features and subtasks of the project. Planning. Because clients never want to
pay for planning.

We spend about 35% of the budget on planning and estimates. And they're
usually fairly close, for projects where we've done similar things in the
past.

When I worked at a smaller dev / consulting shop, we'd be lucky if we got
5-10% of the budget to spend on planning. Short sighted to say the least.
Projects succeed or fail based on planning.

------
knodi
Lets look at the definition of the word estimate.

estimate the cost: calculate roughly, approximate, guess; evaluate, judge,
gauge, reckon, rate, determine; informal guesstimate, ballpark.

Of cause the developer is estimating because there are requirements missing.
Hanse the word "estimate". Its more so the client doesn't to understand the
word estimate.

------
rparet
It was mentioned before in the thread, but I'll just go ahead and leave this
here again:

[http://en.wikipedia.org/wiki/Planning_fallacy](http://en.wikipedia.org/wiki/Planning_fallacy)

Trying to improve developer estimates is just another case of doing the wrong
thing, righter.

~~~
jacques_chester
Every time there's a discussion on estimation the Planning Fallacy is brought
up as a kind of trump card.

But it's not. First, psychologists have done research on how to ameliorate it.
See the "unpacking effect" paper I linked elsewhere. Kahneman and others have
also pushed using "reference classes". In the estimation literature this is
called "estimation by analogy" and it is a well-established technique.

Which reveals the next thing about the Planning Fallacy; it comes from the
psychological literature. There are at least two other bodies of literature
which need to be dealt with before we give up entirely. The first is ordinary
project estimation literature; most of the advanced stuff is by people from
the Operations Research field. The second is work done by Statisticians
working on what they call forecasting which is essentially parametric
estimation of timeseries data.

These 3 bodies of literature seem to have evolved largely in isolation. I
haven't see OR papers talking about the psych research, I haven't seen the
psychologists citing the _International Journal of Forecasting_ and so on.

------
workmanw
This one is my favorite: [http://www.quora.com/Engineering-Management/Why-are-
software...](http://www.quora.com/Engineering-Management/Why-are-software-
development-task-estimations-regularly-off-by-a-factor-of-2-3)

~~~
jacques_chester
You can make the argument that this story reveals important features about
taking estimation _seriously_.

1\. The author gave a point estimate, instead of a range.

2\. The author didn't decompose the task. They took the highest level view and
performed a back-of-the-envelope calculation. A closer reading of a
differently-scaled map would have revealed many of these delays, giving at
least one or two orders-of-magnitude changes in the initial estimate.

3\. The author did not seek out historical data or models for this kind of
project. Hiking and bushwalking are well-known. There are tables of travel
time which can be used to establish a better idea of the outcome. Experienced
walkers could have been sought out for their expert judgement of the initial
estimate.

It is not as good a parable as people make it out to be. The problem was not
an unknown or unknowable problem domain. It was due to ignorance of the basics
of estimating.

~~~
jkubicek
Your issues are exactly why that story is a great parable for poor project
planning.

~~~
jacques_chester
Absolutely. But it's often trotted out as a parable for why project planning
is pointless, or impossible.

Which is a very different conclusion to draw.

~~~
JoeAltmaier
I don't know, in software projects the country is not just unknown but
potentially unknowable; no-one may have walked that kind of terrain before.
THEN there are no experts, its useless to predict. And diving deeper into the
problem may involve actually solving it.

That's my issue with software planning. I can often DO the damn project in the
time it takes to estimate doing it. What responsible course of action can I
take then? Investigate but don't save the code, give the estimate, then go
back and type it in? Blow smoke, give an estimate for the deep dive but call
it the project?

~~~
jacques_chester
All estimates are wrong, what matters is improving their business value by
making them more accurate.

That some estimates will be less accurate due to _uncertainty_ is not by
itself a reason to totally abandon estimating.

The generalised case of your argument is that estimation is imperfect in all
cases, therefore, we must abandon estimation. This is known as the Nirvana
Fallacy: "we have a partial, imperfect solution. Because it is not complete
and perfect, it is worthless".

Even modest improvements in estimate accuracy can have enormous value.

By the way, estimating is not planning. An estimate is an estimate. A plan is
a plan. They are different things.

~~~
JoeAltmaier
Wasn't talking about uncertainty at all, so I'm not sure why that comment. And
no, its not really cool to pretend I said something else so you can debunk
that.

I'll repeat myself then: the task of diving into a problem to estimate its
time is, for many problems, approximately equal to the time to solve the
problem. And I honestly have a hard time figuring out a responsible approach
in those cases.

~~~
jacques_chester
> _Wasn 't talking about uncertainty at all, so I'm not sure why that
> comment._

Because you wrote:

> _in software projects the country is not just unknown but potentially
> unknowable_

Which is uncertainty.

> _I 'll repeat myself then: the task of diving into a problem to estimate its
> time is, for many problems, approximately equal to the time to solve the
> problem._

Non points-scoring questions: does that happen to you often? Can you give an
example? How did you know that the time to estimate would be equivalent to
time to implement?

I agree that estimates themselves have a cost/benefit ratio and that sometimes
it is going to be negative.

~~~
JoeAltmaier
I write in C++. Estimating can require prototyping classes and methods; this
IS development, often all that is needed to deploy.

E.g. I'm being asked to estimate the time for an audio-monitoring feature to
play a tone when someone speaks yet their voice is muted in the space (Sococo
Teamspace has spaces where people work together). The voice-level feature is
already in place; its being used by the mic-selection dialog. So the whole
estimate involves showing the GUI engineer the API he's already using.

I have to fill out forms online, define the 'feature', time-box development
and keep this record up-to-date as the work progresses.

OR I just doorbell Tom and say "Tom, use the same API as the mic-selection
dialog". In fact, its taken longer to type this message than the fake work I
have to do for this.

So that's a degenerate case. Other cases involve changing timers for idle
connection probing (done before the request was finished being uttered);
investigating silent-participant overhead (which I did during the sprint
planning meeting using netmon on the idle participants in that meeting);
aggregate audio packets via our media node to reduce router overhead for
bursts of UDP.

That last one is illustrative. To make the estimate I reviewed the media send
path for the right place to put in the code (half an hour). Then decided the
transport layer was ideal place to aggregate packets using a Nagle timer. I
identified 5 cases (idle; aggregate packet under construction; oversize
packet; normal packet to aggregate; normal packet that blows the aggregate
limit). The constructors for packets need changes to allow header extensions
to identify the aggregation boundary for unpacking.

That took a couple of hours. Plus the time to enter the tickets and put in the
estimates.

The work will take a few minutes, since I have identified everything that will
need to be done. The 'estimation' process has dominated the project time. I
can be done with the project before the project manager even notices the
tickets I've entered into the database!

SO the whole estimation/recording process is some silly circle-jerk to make
management feel involved. IT wasted my time, delayed the project and kept me
from doing more useful tasks that our customers could really benefit from.

Btw sorry for my snarky tone above; I was in the middle of this sorry process
when I resorted to reading HN/commenting to let off steam.

~~~
jacques_chester
Wow. I don't take it personally. I gather that your internal process requires
all changes to have estimates, regardless of task.

Estimates can be performed multiple times. Conceptually what you've done here
is performed 3/4s of the total work (study the change, identify change points,
perform basic design) before proceeding to estimate.

You've come across the Cone of Uncertainty in miniature: all that extra work
substantially reduced uncertainty and so the estimate range was tiny.

But because of the size of the task, you have a clearly pathological case
where estimation had a net negative value. This is a good case of where
proceeding directly to the work would have been more effective.

In your position I'd be pissed off too.

------
chrisbennet
"Could everyone in the room who is good at estimating how long a software
project will take raise your hands?"

"OK, I see a couple of hands"

"Of the people who raised their hands, how many write software all day?"

(No hands are up).

------
mbrock
I think one thing we don't do enough of is prototyping. When we've built
something before, we can usually estimate how long it's going to take this
time if the differences are not huge and qualitative. But if we're building
something new, or doing something that we've never done before, just thinking
and guessing is a terrible way to estimate. We should learn to take the time
to try things to see how difficult they are.

~~~
krisgee
That's because in most cases if you build a prototype that works your
boss/client doesn't want you to waste time making a better version, they just
want the one that works now and doesn't require a hundred more (billable)
hours from you.

------
phigcch
I find that the most precise approach is usually to make an estimate based on
experience, and then multiply it with either e or Pi.

------
pbreit
I usually get pretty close by doubling the estimate and then cutting scope in
half (or quadrupling estimate).

~~~
ams6110
The classic rule of thumb is to double the estimate and bump up the unit of
measure. So an estimate of 1 hour becomes 2 days; 1 day becomes 2 weeks, etc.

------
DanielBMarkham
I've ran across this topic on 3 occasions in the last week in various forms:
online, with a team, and on a LinkedIn board. Wow, must be something in the
air.

So here's the short version:

1) Agile relationships with customers boil down to Time and Materials. You're
there working sprint-by-sprint doing stuff. Each sprint the customer and the
team agrees on what it can do. The team decides, but the customer can always
fire them. This isn't estimation, this is just how an Agile engagement is
supposed to work.

2) Most Agile teams split estimating into two parts: how difficult it is to
deliver, and when it can be done. It's important to understand that by
separating these concepts, you're not pressured to make time commitments when
spitballing story difficulty. That's a good thing. Decouple those concepts and
leave them decoupled.

3) As a trailing indicator, you should use past team performance as an
estimate of how the big picture is going to play out. [insert long discussion
here about the various ways to do that]

4) None of this is business or contract management. That's another can of
worms. Yes, it all gets mixed up sometimes, but there are ways to keep
visibility at the right level to let all the players work effectively.

5) None of this will make-up for working in a crappy environment. Sucky
situations still suck.

Estimating can be a pain in the ass, but it doesn't have to be. I _think_
we've reached the point where most of the pain is gone if you set the
environment up the correct way. (fingers crossed)

Shameless plug: I've got a no-nonsense Agile Tune-Up email series in case
anybody's interested. It covers estimation (along with many other Agile team
topics) [http://bit.ly/15sz0Pl](http://bit.ly/15sz0Pl)

ADD: As you can tell from the many comments, this is also a topic that has
been done to death. There are more methods out there than you can shake a
stick at. (Hence my reluctance to dive in with "This is the perfect way to
estimate!"). But the key takeaway is this: _lots of simple, quick estimates
that converge over time beat any kind of complex model done only once_. If you
only get one thing, that's the thing to understand. We keep trying to "fix"
this by creating models, when the real answer is that over time the team
learns to estimate each project as a separate entity. The more engaged it is
with the problem and the farther along it is, the better the estimates get.
It's a funnel. Don't spend a lot of effort trying to polish up the first part
of the funnel: rather, make a commitment to using empirical knowledge and re-
estimating to provide better and better estimates as you go along. There's a
lot more value added to everybody's job that way.

Hell, I don't care if you use KLOC, some kind of FPA, phases of the moon, or
your lucky astrology mood watch to estimate, as long as you continue to re-
estimate as you go along and your estimates quickly converge on the actual
result.

~~~
nradov
That technique tends to break down when you have multiple agile teams working
through a long, complex dependency chain to deliver a single project. It's
also problematic when your customer has to make fixed schedule commitments to
_their_ customers based on your work.

~~~
DanielBMarkham
No, it doesn't.

Complex programs and dependencies are handled the way complex projects and
dependencies are handled: visiblility, simplification, and just-in-time
architectural support. Works at the team level, works at the program level.

Fixed calendar dates are also not an issue. Remember, the goal is to separate
delivery from scheduling. There's no magic fix to make you automatically hit
dates, but you can generate real numbers on what it takes to meet arbitrary
calendar dates if that's your situation. You should know, for instance, that 2
more teams need to be spun up within the next week if your group is going to
hit the date.

It's a myth that Agile team techniques don't scale out. This was true several
years ago but a lot has changed since then. A better way of describing the
situation is that there are multiple models for scaling out Agile teams and
estimation, many of which have years of real-world traction. We don't have the
same number of data points as we do with, say, how well stand-ups map to the
average team, but we have data and we have demonstrable traction on the
problem. All of the normal project and program management tools are available
to Agile teams and programs.

For a discussion of how simple systems can scale out into complex projects,
check out my 15-minute video. (Sorry for the additional plug, but it is
relevant here) [https://vimeo.com/57146799](https://vimeo.com/57146799) Also
for a review of the various Agile Program management systems out there,
[https://vimeo.com/64452664](https://vimeo.com/64452664)

