
What do you mean ‘we need more time’? Project schedule estimation in software - hepha1979
https://blogs.dropbox.com/tech/2015/10/what-do-you-mean-you-need-more-time/
======
13hours
Almost at the end : "Yes, all this sounds like a lot of work". Exactly. And it
will most probably still turn out to be wrong, because you will forget to ask
some questions, and there will be things that you simply cannot foresee that
will influence the time.

Turns out it's pointless to even try to do this. So why waste valuable time to
do all this, if you know it's not the truth anyway? The answer is to accept
that you will learn things as you go along, and that things take as long as
they take, and to rather deal with that. Marketing wants a product by
Christmas. Fine, we can do that. They just cannot say what needs to be in that
product by Christmas. They can have whatever the last production quality
release is at that stage. Instead of taking many weeks or months to try and
answer all these estimation questions, we start and tackle the high risk items
first. We learn from them, readjust, and as we get closer to the deadline, we
keep communicating with all the other stakeholders what we think will be
delivered on that date. It's a narrowing cone of uncertainty. One sprint
before the deadline we'll be pretty accurate in predicting what it will be. 2
or 3 sprints before, accurate enough to be able to do marketing. At the start
: the best we can predict is simply whether we feel we can have something
useful at the deadline.

~~~
soft_dev_person
Except some of us make a living selling development of projects and solutions
to customers that require pretty accurate (cost) estimates prior to even
getting the contract.

Crazy world, I know...

~~~
eterm
Absolutely this, clients need to know their costs upfront.

Meanwhile, I've been on the other side, working with a shop that insisted they
were agile, and so refused to give an upfront cost.

That project ended up delivering a substandard project where key parts did not
work well. Their answer was to tell us they would of course be happy to be
paid for another 2 week sprint to fix those bugs...

As a customer it wasn't very satisfying.

~~~
13hours
I don't think working fixed price quote would have given you a different
outcome though? The team produced bad quality product, fixed cost or agile
won't change that.

Selecting the team to build the product on price is often a bad idea, and
that's all fixed cost gives you above agile (the ability to select on price).

~~~
eterm
A fixed cost makes it easier to turn and around not pay (in whole or part) if
the product is really not up to the standard specified. It's harder to do that
if the cost is not fixed but overrunning and the response is "well we just
need more time".

Also fwiw, the team wasn't selected on price.

~~~
jsprogrammer
Not pay? Heh. For work performed? OK...

That is why my contracts require payment up front.

~~~
SapphireSun
Sounds like they should have negotiated for a base rate with bonus pay out on
meeting QC targets. For heavens sake people, align your incentives!

~~~
codeisawesome
Everyone, on both sides of this river, they need to grow up.

------
cateye
Analogies like "painting a room" are pretty useless because they are very
misleading. Especially when it is an innovative product, a better analogy
would be: "painting a room on Mars".

With software development, there are a lot more "unknown attributes" that
determine the main part of the effort estimation unless it is a repetitive
task.

So, a critical/skeptical engineer would always drive the customer nuts by
asking questions nobody can know and the conclusion will end somewhere like it
is not possible to give a upfront estimation and stop the project. Or the
effort to give the estimation approaches the effort to develop.

~~~
erikb
Before NASA sent their first drone to Mars they made a lot of predictions, a
lot of tests, guessed a lot of things that may get wrong, etc. Do some of
their missions still fail? Sure. But the amount of success is much much larger
because they didn't just sent a drone there and watched what happened.

So even if all you say is correct it's still worth improving one's estimation
skills.

~~~
lmm
> Before NASA sent their first drone to Mars they made a lot of predictions, a
> lot of tests, guessed a lot of things that may get wrong, etc. Do some of
> their missions still fail? Sure. But the amount of success is much much
> larger because they didn't just sent a drone there and watched what
> happened.

How much effort went into estimating Opportunity's lifetime at 90 days? How
useful has that estimate proven to be?

~~~
TheLogothete
That is cherry picking. Did you try to comprehend the post you are replying
to? Take into account all of the estimations, not just one failed estimation.

~~~
lmm
Sure, take into account all the estimations. How much value have they
delivered? Show me how NASA has got more science done than if they had "just
sent a drone there and watched what happened". Because I don't believe they
have.

~~~
erikb
The funny thing is despite having the opposite opinion I understand so well
how you see it. I wish I could show you my point of view better.

~~~
TheLogothete
That was exactly how I felt. It is a matter of pay grade, I think.

------
mzahir
[http://qr.ae/ROlwZp](http://qr.ae/ROlwZp) \- I've found this answer on quora
to be one of the best explanations as to why software estimates are so off.

~~~
mannykannot
Digging down into the details is one of the best responses to absurdly
unrealistic schedules.

------
geebee
Estimates are a tricky thing. Ultimately, I don't think they work in software.
We should be looking for a different solution.

Think of it this way - a physician can prescribe a medication and tell you how
long you'll be on it, but can she estimate how long it will take before your
blood pressure falls within normal parameters? She can tell you what has
happened in the past, but suppose you take the medication and your blood
pressure remains stubbornly high. Has she failed to meet her targets?

A lawyer may be able to tell you, roughly, how long litigation tends to take,
but can your lawyer estimate how long a case will take, what it will cost,
what the outcome will be, and the date it will all be wrapped up? If that
doesn't happen, did the lawyer fail to meet targets?

Well, in both cases, sort of, yeah. You didn't get the outcome you wanted. But
these professions, run by the people who staff them, understand what can be
predicted and what can't.

This blog entry uses the task of estimating painting a room as an example, and
focuses on breaking the task into subsections. Ultimately, I think the entire
approach is deeply flawed, and it all starts by thinking that software
development is similar to something like painting a room, just a more
complicated version of it.

In my opinion, it's very different. Of course, it's very different from
prescribing medication or litigating as well - in fact, I'd say that it's
generally considerably less predictable than even these things. I'd say it's
closer to writing a novel, creating a film, or recording an album, except that
software needs to _work_ , and can be much more mundane at times as well.
Ultimately, I don't think the analogies are useful, we need to realize that
software is its own thing, and steer clear of too much reasoning by analogy or
example.

Unfortunately, the anxiety caused by making estimates under conditions of
uncertainty (and possibly duress) is a source of stress and unhappiness for
developers, and I do think it's one of the things that drives people out of
the field. I actually think this may also be one of the reasons people love
creating apps as a hobby but hate doing it for money.

------
Theodores
The thing that annoys me most about project estimation is when the micro-
managing types don't let you even look at the problem space first, an instant
answer is expected. Of course the figure plucked out the air could be 'x days'
with 'x' being greater than the time one expects internally so backside is
covered, yet that is not really a true answer.

At the moment I am putting together a 'simple' feature on a website, where I
have been able to explore the problem space first. For the $content there
actually needs to be half a dozen date fields that were not in the original
brief, I doubt I would have discovered this had I not rolled a little bit of
code first and explored 'prior art' in the area.

I need to put together a semi-working prototype to discover such things -
particularly when UX is involved. Yet, far too often, the opportunity to do
actual preliminary dev work is denied by those that 'know best', preferring a
spec. in a Word document (or something else equally 1990's) instead of a rough
working prototype in code (with back-of-envelope sketch).

My current feature-being-added will need to be re-factored at some stage, once
users have started using it and problems/features overlooked in the original
brief catered for. It is not as if the users of my new feature will know that
'actually they wanted it to work this way' until they have something to see.
So why we have these silly cast-in-stone processes I do not know.

I also must say that I find the metaphor of painting a room is quite a poor
one, I would not need to paint 'an alcove first' to get a handle on how long
the whole room would take.

~~~
LoSboccacc
> don't let you even look at the problem space first

this is the core of the issue. when one has the time to break down all
unknowns and discover them, estimate are quite spot on. but who want to pay
for the estimate?

so we are left trying to guess what time a particular unknown is going to take
to solve.

the good metaphor here is asking how much time it will take to solve a puzzle
magazine, from cover to cover by looking at the title.

~~~
GFischer
The point you're making is exemplified in this article which makes an analogy
to a hike from San Francisco to Los Angeles:

[http://www.michaelrwolfe.com/2013/10/19/50/](http://www.michaelrwolfe.com/2013/10/19/50/)

from

[https://www.quora.com/Why-are-software-development-task-
esti...](https://www.quora.com/Why-are-software-development-task-estimations-
regularly-off-by-a-factor-of-2-3)

------
work-on-828
As a new grad who has worked a couple different jobs now, some things I don't
get about this:

\- What do you do when you don't have a spec and when you ask the senior
engineer you are working with questions, they don't respond or push back
against having a spec or a plan longer than the very next feature?

\- The author says that estimation is easier to do if you are doing something
you have done before. What do you do if you are always working with a new
framework or language?

\- How do you break down the steps if you don't know what a project entails
until you are finished?

\- What do you do if asked to estimate how long something will take to debug?
What if you just started on a project and the error message you are trying to
debug doesn't make sense? My approach there is to try to understand the system
I am working with. If the system doesn't have documentation and the existing
people on the project don't have time to give a walk through of things, I
figure that means source diving. However, I've gotten the feedback that I
should avoid trying to "understand the universe" when debugging. What can I do
to produce accurate estimates of how long it takes to find a bug in a system
without spending the time to gain a mental model of the system?

\- This excercise takes a lot of time. What do you do when asked for an
estimate in a meeting rather than over email? What

\- What do you do if the thing you are building relies on getting an external
API to work and that API is either undocumented or is documented in a foreign
language? Assume that the google translation is not making sense and this is
your first project and you don't have a budget to hire a professional
translator? This sounds like something you would need to get working before
you could actually give an estimate.

Why do experienced engineers ask for estimates anyway? Every time I give one I
feel like I am lying and I try to warn people "I really don't know how to come
up with estimates but I am guessing 3 hours". How do I deal with it when they
are upset about it taking a week and a half?

~~~
DougWebb
For a lot of these, rather than trying to estimate the thing you're asked to
estimate, you need to either estimate how long it'll take you to do the
analysis you need in order to come up with an estimate, or you need to provide
a time-limit on the task after which you'll provide a status update and have a
discussion about the next step.

Managers don't really care about time, they care about budgets and risks. In
your debugging example, the existence of the bug is a big unknown risk, and
your manager needs to manage that. You obviously can't tell how long it'll
take to fix the bug, but you can say "Give me an hour to debug it, and after
that I should have a better idea of what the cause is and how long it'll take
to fix. I might even have it fixed already by that point." That gives your
manager a budget (one hour) and risk control (more information in a fixed
time), and the option after a small investment to bring someone else in to
assist.

~~~
work-on-828
It sounds like you are saying that when I take an hour to work on debugging
something, I should spend 55 minutes taking notes on the system and where the
problem might be and 5 minutes turning those notes into a coherent email. Is
that a reasonable approach?

I certainly prefer to think of debugging as a process of learning about a
system rather that trying to try things at random without making a mental map
of where I am.

~~~
DougWebb
I'm saying that when asked for a debugging estimate, you should commit to
spending no more than an hour (or other amount, depending on the problem)
before stopping to report on your progress. How you spend that hour depends on
the circumstances; if you could predict that then you probably already know
enough to give a somewhat accurate estimate.

When you stop to report on your progress, you don't need to deliver detailed
notes. It could be as simple as "yeah, fixed it" or "figured out the problem,
I need three hours to refactor the code to resolve it" or "based on what I'm
seeing, Bob would be better at fixing this; I can spend a few minutes to show
him how to reproduce it."

"try things at random" is not debugging, so you're right about the process.

~~~
work-on-828
usually when I debug things my method is sort of a mix of linear and binary
search where I start at one place, see if the a value is what I expect it to
be, then move along to a different part of the code to see if something is
wrong there, reducing the search space and repeating as necessary.

Sometimes my method is instead to look at data output at a certain level of
abstraction like system calls (which is usually not helpful except for
detecting a network hang or a path config whose directory doesn't exist) or
database queries or something. That is usually more helpful if I either
already suspect I know where the problem is and need to know what model name
or file name to `ag` for.

If (Hephaestus be praised) there is actually an automated test framework in
place, I'll start by writing a failing test or copy-pasting an existing test
and modifying it to fail, then (if it isn't JavaScript), inserting a debugger
into a function the test calls.

All of this tends to be pretty successful.

But I've occasionally found myself on a project where I just have a sense that
I don't know what is going on or how pieces fit together and I feel a strong
temptation to stop and fix that sense of bewilderment. When I resist this urge
is when it feels like I am trying things at random.

------
gjkood
A pet peeve of mine:

Scenario: Finished requirements, analysis, design; completed development and
testing; in UAT; Production deployment: 1 week away.

Customer: Hey this field name does not exactly represent what we want to
convey. Can we change its name quickly before we go live?

Eager Novice (who is unfortunately in front of the customer at that moment):
Oh yeah, that's just a small change in the Customer model. I think we an get
this ready and unit tested in 2 hours.

Customer: Great. Lets git er done!

Project Manager (wary of customer and afraid to rock the boat on an already
shaky project): Sure we can do that.

Grizzled Veteran/Grey Beard (who for some reason wasn't consulted on the
change request): Hmmm....have we done an impact analysis of the change? What
are the downstream impacts on code that references this field? What do we need
to do to propagate the change to the db? What about the seven or so Web
Services that are impacted by this field change? How long does it take to do a
regression test? etc...

Net effect: 2 Hrs initial estimate given to customer; 4 Days worth of work in
actuality; Production deployment moved down a week..

~~~
edc117
Agree 100%. Golden rule: Never give off the cuff estimates (even if you're
sure).

------
kbouck
Regardless of project/planning methodology, having a ever-growing checklist of
less-obvious estimation items is a simple way to avoid forgetting to consider
them. The book "Software Estimation: Demystifying the Black Art" [1] by Steve
McConnell contains a number of these.

Some examples:

    
    
      - Team or company meetings
      - Team member sick days, holidays, vacations
      - Bugs in third party software
      - Maintenance of previous systems
      - Maintenance of build automation
      - Demos
      - Interviews
      - ...
    

... or your own list of whatever other items you have historically forgotten,
or tend to forget when estimating.

In the painting example, the moving of the furniture was something I
overlooked, despite having made the effort to carefully think of all tasks. A
estimation checklist built up from previous painting projects could have
triggered me to consider this task.

[1] [http://www.amazon.com/Software-Estimation-Demystifying-
Devel...](http://www.amazon.com/Software-Estimation-Demystifying-Developer-
Practices/dp/0735605351)

------
Gratsby
I'm about to being a rant here because this is a larger problem than anybody
likes to admit.

Developers are good at development. By and large, they are not good at task
management, project management, and certainly not estimation. One of the
primary reasons for this is that their plates are always overflowing and they
work more efficiently than many people in other professions. They shift
priorities on the fly when work in one area cannot be progressed. Development
time is social currency. You can make your boss happy by progressing his
priority, or you can make someone else happy by progressing their priorities.
Not many developers truly answer to a singular master. And few allocate time
for learning although we all spend a significant amount of time doing it - how
else do we maintain relevant skill sets?

Can a developer really estimate a project and tell his manager that 3 weeks in
the timeline are for 2 unknown and unforeseen problems that require
interaction with vendor support? No, because the manager will fight back and
say that those 3 weeks are really just zipping up logs and making three phone
calls and "if it takes anything more than that, hand it over to me!" Most
can't tell their manager that it will take 2 weeks to parse a CSV because the
source data is shit and they have too many other things they are working on.
So they stay an extra 6 hours in the office and get called a rock-star until
the day that they opt to go to their child's school play instead.

The solution to this problem is simple. Take task management and project
estimation out of the hands of developers. It's time for the return of the
secretary to the workplace. A single $50-80k administrative assistant could
easily support, manage the priorities of, and improve the outward facing
communication for five developers. With experience doing so, that admin could
provide better time estimates for all of them and would be better at
negotiating realistic timelines than any of them individually.

People have been coming up with methodologies and software to solve this
problem for decades. Developers don't have brains built for project
management. Yes, they _can_ do it, but they don't get paid to work in MS
Project. They get paid because they know the difference between a bubble sort
and a quick sort and when to use either. Conversely, I don't know a soccer mom
who doesn't successfully manage a more challenging schedule and competing set
of priorities better than every nose-down developer I've ever met. Give
appropriate work to appropriate people. Developers have their talents.
Scheduling, estimating, and task management are not among them.

~~~
squeaky-clean
Going to join you in this rant if you don't mind.

> Most can't tell their manager that it will take 2 weeks to parse a CSV
> because the source data is shit and they have too many other things they are
> working on. So they stay an extra 6 hours in the office and get called a
> rock-star until the day that they opt to go to their child's school play
> instead.

Ugh! For real! I had this happen to me recently. I have trouble telling people
"No", and pushy PMs who also happen to be my friends means that I often bite
off more than I can chew, and have to either "be a rockstar" (which you start
to realize isn't such a compliment), or deliver late / fail to deliver.

So I just started being a dick about it, and it works for me. If I say no too
politely, it's really easy to convince me to change to a yes. My turning point
was when discussing a project that was much more complex than management
assumed it would be (I'm compelled to link the relevant xkcd [0]), and I
already felt overwhelmed on my current projects.

The discussion was something like "Ok, how about by this date? What about by
this date? What if we dropped these features?", I thought for a moment and
responded "Ok, look. I can promise you any date you want, and we can shake
hands and leave this meeting. But I'm telling you the reality is I can't do
it, and it won't be done. If you'd like to take some of my other projects and
move them into the _will not be done_ category, we can talk. I'm only saying
this because I don't want to agree to a deadline and then be constantly
thinking 'Oh well, that's the date I'm getting fired.'"

I'm very lucky that everyone here is really nice, and realized how stressed I
must have been to react like that. So we've reprioritized some things and are
beginning to outsource some of my more menial tasks that took up a lot of
time.

Didn't really intend for that story to have a point, just venting something my
non-dev friends don't relate to. I guess the point could be that the problem
isn't just caused by bad managers, or bad dev estimates, or bad process, it's
a little bit of everything, and the ratios are different for every company or
person.

[0] [http://xkcd.com/1425/](http://xkcd.com/1425/)

~~~
ThomaszKrueger
Personal experience here - everything got better when I really started being a
dick. I flat out tell them that they can't change reality and things take
time; and I will not work extra hours or during the weekend to keep promises I
didn't make. I had the same feeling of "when are they going to fire me" but
actually it looks I am getting much more respect.

~~~
squeaky-clean
Exactly the same here. I think it's not really "being a dick", but just a
dissonance between how the different types of people operate. I'm usually so
passive that any sort "No, I want my free time after work" feels rude, and I'm
probably not very good at being assertive and nice at the same time yet, haha.

The other developers always seemed to get the message that when I said
"Maybe... I'm not sure if I have the time for it... I can see...", I really
meant "No, but please don't make me say no.", while anyone else hears it as
"Cool, he said he would see if he has the time for it" and then follow up with
me a few days later "Did you make time for that project?"

I'm making an effort to flat out say "No" more often, but it's nice that other
people are also beginning to realize that I only mean "Yes" when I actually
use the word "yes".

------
bluesign
I think also one of the biggest mistakes is making over estimates, or playing
too safe. Especially when working with big teams and multiple projects (with a
lot of dependencies).

~~~
pjc50
Why - has that led to a spate of projects completed early?

~~~
Ensorceled
It leads to what I call "do it right syndrome".

Since I have time, I'll do it right, with a factory class, and extendible
configuration package (that I'll write myself so it will be just perfect) and,
for optimal speed, I'll use Red-Black trees that I'll have to write my own
implementation of since there isn't a standard library ...

~~~
pestaa
What's wrong with separation of concerns, extendability, optimal speed, etc.?

More often than not, I observed doing it right the first time is actually a
huge time saver compared to hacking some stupid prototype nobody can
understand later.

~~~
sombremesa
Agreed, except in cases where the code will never be maintained again (e.g.
game releases on N64). In my experience an "over-estimate" that leads to the
so called "do it right" syndrome happens if _someone else_ estimates for a
developer. When the developer makes the estimate him/herself and doubles it,
you can be damn sure they are going to need all that time and then some to
even get it in the ballpark of long-term acceptable.

(That being said, the company needs to have standards for code acceptability
in the first place, which doesn't always happen)

------
beat
The tendency to estimate poorly - particularly to underestimate - is one of
the key reasons to use a very tight, short-iteration agile process.

If I estimate something will take two days, and I'm off by 50%, it'll take
three days. If I estimate it'll take two years, and I'm off by 50%, it'll take
three years. And because I can get a lot more practice at two day schedules
than two year schedules, I'll improve my accuracy over time, through steady
practice.

This takes us back to the triangle of constraints that Kent Beck talked about
in the first Extreme Programming book. There are three constraints on projects
- schedule, scope, and resources. You can control _two_ of them. Generally,
resources are inflexible. So in practice, projects are either schedule-bound
(we must hit this date) or scope-bound (we must have these features). A
project that tries to do both, without a significant resource surplus, is
probably doomed.

Since estimation improves over time with short iterations, it seems better to
me to be scope-constrained than schedule-constrained. With steady progress,
you can achieve the goals eventually. Good agile processes are much better at
saying where you are and where you're going than when you'll get there.

~~~
p4wnc6
I've never experienced estimation improving over time in an Agile process. In
fact, since short estimation cycles are just as susceptible to political games
as longer cycles, and this dominates the whole process in any organization,
the length of the estimation cycle really has nothing to do with the
estimation issue.

But, what shorter estimation cycles are responsible for is a decoupling of the
overall big picture progress from the immediate tasks being prioritized. This
often leads to sprints that are well scoped, and everything in the sprint gets
completed close to on time (at least as close as in longer work cycles, but
not usually any closer), but, _crucially_ all of the work has to be scrapped
because the whole sprint, _conceptually_ , was not right. In my experience,
this happens maybe 1/5 of the time in Agile exactly in ways that would be
prevented with longer-term planning.

It's very similar to the classical fractal effects of measuring the coast of
Britain. By refining your ruler, you merely _think_ you're being more
accurate, and a lot of people are doing a lot of performative managerial crap
for the sake of the Scrum performance, like a ritual. It _feels_ like you are
estimating better, but really because the whole meta-level _concept_ you are
working on isn't guided by longer-term thinking, you end up having so much
round off error, added up over more and more cycles, that the overall waste is
colossally greater than systems that involve some more significant longer-term
pre-planning and which work on variable cycle lengths instead of a cookie-
cutter, one-size-fits-all approach (which is the core philosophy of Agile,
though people try to obfuscate this fact by No True Scotsman-ing it with the
Agile Principles).

I also can't miss the opportunity to point out this great article on the
political games we play during estimation: <
[http://research.cs.queensu.ca/home/ahmed/home/teaching/CISC3...](http://research.cs.queensu.ca/home/ahmed/home/teaching/CISC322/F09/files/EstimatingGames.pdf)
>

~~~
beat
All excellent points. But having done big waterfall, bad corporate agile, and
good agile, the problem of work failing conceptually is common to all of them.
The question is, how do you recover from it? I think good agile recovers more
gracefully from conceptual failures than either waterfall (which tends to
throw good money after bad, because admitting failure is not an option), or
corporate agile (because corporate agile is usually just buzzword-compliant
waterfall).

~~~
p4wnc6
I'm not saying waterfall is a good alternative. Waterfall is just as shitty as
Agile, but one benefit is that wastes less of everyone's time with meetings
and busywork. But waterfall is equally guilty of a one-size-fits-all
mentality, from the opposite view of Agile, and this cookie-cutter property is
what makes both methods attractive to bureaucracies.

I'm a fan of common sense. If it's clear in a given team and project context
that two week sprints, story points, and frequent meetings are really going to
help, then just do it. And when that stops working, just switch and do
something else. When architects or researchers are telling you a given project
direction needs to slow down for some more intensive piloting research, do it
and don't hesitate to violate sacred Scrum principles.

This stuff should be figured out organically, based on the given project and
given personnel. Never with a fixed mandate to a single prescribed method or
time frame.

~~~
beat
Continuous improvement of the process itself is part of any good agile
process. Corporate buzzword compliance agile is often guilty of not doing that
- the "cookie-cutter property".

But importantly, I've found that a refined agile process saves more work than
it costs, by figuring out what doesn't need done before doing it, and
prioritizing immediate value over future value. It's very important to be able
to say "This is good enough for now, but we know it's not good enough for the
future". It keeps the perfect from being the enemy of the good. Likewise, it's
important to be able to say "Well, I guess that didn't work", and have it be
an integral part of the process. Without process, you have no guidance over
what wrong turns you took and what those wrong turns cost.

~~~
p4wnc6
I dunno, what you say just reads like a bunch of buzzwords that Agile, even
supposedly "refined" or "good" versions of Agile, never prioritizes or
delivers. More often, when quality work is produced it's produced _in spite
of_ Agile, not because of it, which is a shame because then people falsely
attribute that success back to Agile.

I admit though that it's hard to know exactly what you mean just exchanging
textual comments like this, so I'm happy to leave it at that.

------
orky56
One of the fundamental issues is that upper management sees estimates as bell
curves as though 50% of the time a project may finish early. In reality, very
few projects on time, let alone early. No one wants to hear about reducing
scope but at the same time an MVP that solves the customer problem without
feature creep should be evangelized within organizations. In reality, schedule
follow more of an exponential function where slippage time goes from bad to
worse very quickly as resources/scope/schedule snowball very quickly. Things
get scrapped before we see this in practice.

------
jahaja
I may be cynical at this point but overeager project scheduling/estimation
processes and top-down software processes in general are really where software
development as a creative and experimental occupation dies.

~~~
crpatino
If you can do "creative software development" and get paid for it, count your
blessings. And better yet, give a very big "Thank you!" to your manager.

The reality of this trade, regardless of you doing in-house work or a product
for the market, is that everything is required for yesterday. There is an
"economy of pressure" where every stakeholder pushes everyone else as much as
they can in the expectation that they will be pushed as well. If you do not
get to experience this, it means someone in your chain of command is very good
at pushing others, and that some other part of the overall system is
shouldering most of that pressure all by themselves.

~~~
jahaja
So if I understand you correctly you're saying that "creative software
development" is inherently incompatible with the hierarchical realities of the
corporation?

If so, I think you're probably right.

"The reality of this trade [..] is that everything is required for yesterday"

But it just isn't. This is an artificially created pressure for selfish
reasons, and should not just be accepted in passing.

~~~
bigmanwalter
The game is rigged. Only way to win is not to play.

~~~
sombremesa
Or according to the parents of this reply, to play under the wing of an expert
player.

------
dreamfactory2
article assumes specs are known. they might be for some kind of brochureware
but for any serious work they never are, and in business reqs change day to
day. 60% of software features are never used - so that's 60% overspend right
there. there's no reason not to do fixed cost agile (personally wouldn't do it
any other way). trying to fix scope otoh is just bad business from the get go
- a) you might have target outcomes but you don't know exactly what you're
going to need to get there b) useful software tends to be ongoing in terms of
reqs

------
mikeskim
You could just use data, machine learning and optimal control to make better
estimates. It has aleady worked for fighter jets, self driving cars, and
automated factories. With enough data it will work for anything.

~~~
chillacy
What kind of features would we use? Which prior projects will help predict how
long it takes a team of 3 developers (one of them senior with a history of
underestimating time, two junior with little track history) to produce a novel
photo sharing app where the requirements might or might not change every day?

The problem I see is that there isn't enough data, and the problem space (the
number of possible projects and teams) is too large to draw meaningfully on
past data.

Unless we're looking to predict specific CRUD apps where the requirements are
known up front and a team cranks these out consistently.

Humans have already been flying fighter jets, driving cars, and doing work in
factories, so we know it's a learnable skill. We have not yet learned how to
estimate arbitrary software production.

------
a3n
As long as sales, or its local equivalent, commits to as yet non-existing
products and features, it will ever be thus.

------
anon4
> The fact that I technically only asked how long the fix would take is
> something only an engineer would bother pointing out. -_-

Well excuse me for not fitting your prejudiced profile for a neurotypical
extroverted 20-something white male. Yes, I might have needed the
clarification between "how long will it take you to paint the room" vs "how
long until the room is back the way it was with the walls in a different
colour", but you help noone when you talk down to the painter like that. You
asked a question and I misunderstood, because we use slightly different
language and I have a ton of other things to think about all the time.

Can we stop this "oh let's treat engineers like they're children with mental
development issues" already? I'm not some god in an ivory tower, I don't want
you to kneel before me, I just want the usual everyday respect you afford to
your peers and to engage with me as one professional with another.

This kind of treatment is not ok.

~~~
Ensorceled
Agreed. As an engineer, I can also claim that if I say "we can deliver that
March 15th", then I get a argument about why can't it go out with the Feb 28th
release and why is it going to take 30 days to do something simple and why ...
all because marketing people can't understand delivery cycles.

So, let's just not have that type of conversation at all.

~~~
ams6110
Ever talk to a contractor about a home renovation, or even a painter as in
this story? If you start questioning why something will take so long, in my
experience the usual answer is something like "if you think you know better
than me, do it yourself." Especially if they are good at what they do. Most
good tradespeople have more work than they can do, and will just walk if they
think a customer is going to be a pain in the ass. How often will developers
take that approach?

~~~
crpatino
Well tradespeople usually work for multiple clients on smallish projects. For
_good_ tradespeople, a significant number of those projects come from repeat
customers, which means their optimal strategy is not to serve all customers
equally, but to prioritize those that are more likely to bring repeat business
down the road. A difficult customer is usually not someone you want to do
repeat business with, so they get "managed out" pretty reliably.

Most software engineers, on the other hand, work for one big customer at a
time (your employer). And though I agree that a bad employer at the end of day
is detrimental to your career, you cannot afford to switch jobs on trivial
matters, on risk of getting labeled as a grasshopper. That's why you do not
see developers "walking".

------
davidgrenier
> These are the skills that elevate someone from a good programmer to a great
> software engineer

I stopped reading at engineer

~~~
ddingus
You really shouldn't.

While the term engineer is loaded with the licensing and professional aspects,
and understandably so, the people working with software very often do actually
have to perform the task of engineering solutions.

Dismissing a valid dialog on this basis does no one any real good.

Despite the issue you raise, the query and discussion remain valid and high
value.

------
alistairSH
_In tech, we spend little time talking about the softer skills like
communication, project management, and prioritization._

Well, that's not a good start. Maybe my career has been abnormal, but I've
spent a lot of time thinking and talking about project management skills,
particularly estimation. It's something my employer expects all non-junior
employees to do - more than half my team are certified in scrum or have taken
courses in other PM methodologies.

For most of what we do, the hard part is project management. Sure, the techie
code stuff is usually more fun, but that's not even half of my daily workload
(and hasn't been since well before my title changed from "product architect"
to "project manager").

