

Friction: why we can't build software as fast as we think we can - dools
http://iaindooley.com/post/51974072812/friction-why-you-cant-build-software-as-fast-as-you

======
swalsh
Here's the thing, estimating software development time is essentially
equivalent to estimating time to think + overhead. Overhead includes actually
typing the code, booting the computer, dealing SVN issues, etc. The overhead
is relatively consistent, and can be minimized and predicted. Time to think is
highly variable. Previously I used to work for a consulting company. After
about a year of working there the company started to settle in a particular
niche. We built libraries to do things, then we just started to churn out
work. Predicting development time became an exact science, because all we had
to do was estimate overhead. There was very little new things, so time to
think was minimized to nearly nothing. I hated working in a code factory, so I
left for greener pastures. The new place has a lot of new things, trying to
fit it together is a real challenge, and requires thinking. My estimates are
no where near where they were at my old company.

My main point is this, estimating software time is hard because you don't know
what you don't know. If you can solve that problem, you solved one of
humanity's greatest challenges.

------
wpietri
This is a good start, but I wish the author had taken it deeper. E.g.:

* Can a team discover patterns in the things they underestimate? * Can a team do better by not estimating? * Can a business do better by not estimating? * What are the axes along which scope can be cut? * What emotional needs keep stakeholders from cutting scope?

I also have found a lot of value in trying to stop using terms like "launch";
I think they confuse things more than they help.

~~~
A1kmm
> Can a team discover patterns in the things they > underestimate?

If you do almost the same project twice, definitely. If you are doing
different projects of a similar nature all the time, you can at least get an
idea of the risk to the schedule and of systemic biases in estimation (i.e.
the statistical distribution of time taken to complete a task / work package
as a function of the initial estimate).

> Can a team do better by not estimating? > Can a business do better by not
> estimating?

One of the goals of project management is to ensure that organisations don't
start projects that do not have a valid business case (and cancels them as
soon as it becomes clear that they don't); to work out if a project has a
business case, it is necessary to estimate the benefits they project will
bring to the organisation, and the costs and risks of the project. Businesses
that can't do at least some basic form of cost-benefit analysis will likely
end up going out of business under a series of projects that don't make
financial sense.

The biggest win-win is to take an agile approach; instead of taking on a
massive project which is hard to estimate, planning it up front, and deploying
it in a big bang (the waterfall method), break it up into smaller iterations,
each of which immediately delivers benefit to the organisation. Smaller
projects are much easier to estimate, and if the organisation doesn't have the
cash flow to continue after one iteration, they still get the benefit of the
first iteration. The second iteration is only planned near the end of the
first iteration, and so knowledge from the first iteration goes into the
planning of the second, again making estimates more accurate.

> What are the axes along which scope can be cut?

When looking at a project, you have to look at all six variables at the same
time: time, scope, cost, risk, benefit, and quality. If you want to cut the
time / cost / risk, you might cut the scope, but that is likely to have
implications for the quality and benefits. To assess whether a reduced scope
makes sense, you need to look at the estimated effect that has on the benefits
for the organisation - if cutting the scope leaves the project with no
meaningful benefits for the organisation, because the products are unusable,
but it still has some cost, then cutting the scope is not worthwhile. If
cutting the scope reduces the cost and risk significantly, but still delivers
most of the benefits, then it is probably worth it.

Delivering a project in iterations does have some overhead costs (and
sometimes risks due to the rapid pace of change), so it ultimately comes down
to a trade-off between iterating fast enough to manage the estimation risk but
not so fast that it causes new problems - and that trade off is going to
depend on external factors like the culture of the organsiation.

> What emotional needs keep stakeholders from cutting scope?

Often, stakeholders don't know much about software project management, and I
think the application of the waterfall method, creating projects with huge
scope, is applied because senior management wrongly believe that it is the
best way to manage risks.

Up-front planning of large projects works very well on other types of projects
with lower levels of uncertainty (for example, when constructing a large and
complex building that could take hundreds of human years, up front planning
probably works much better than agile approaches), and so managers apply what
they know and insist that software projects be planned out from start to
finish with full scope.

~~~
artsrc
> Businesses that can't do at least some basic form of cost-benefit analysis
> will likely end up going out of business under a series of projects that
> don't make financial sense.

You can't execute an estimate and make money with it, so they are waste in a
lean sense of waste.

There are usually cheaper* ways of dealing with the problem than producing
estimates.

Where I work now I don't think I have ever provided a written estimate (and no
one else has) for my work.

Who does estimates for the time to produce all these estimates?

* Estimates have subtle costs and benefits apart from the time to produce them.

~~~
wpietri
Yes to all of this.

In the Lean sense, estimates are necessary waste if you use them to make
productive decisions. That is, if by wasting time on estimates you can avoid
an even larger waste, you win.

But otherwise, they're pure waste. And that's something I've seen quite a bit
of. E.g., when estimates are used to give managers a sense of control. Which,
in this case, means an opportunity to yell at people who fail to meet the
estimates. Curiously, they almost always conclude the problem is a failure of
execution, rather than over-optimistic estimates. And by "curiously", I mean
"predictably."

------
nickthemagicman
I think the problem with programming is fricking programmers.

We write so much obtuse code with no documentation, adhering to no standards,
and with no tutorials that you can literally get side tracked for days on some
esoteric dependency that has you bashing your head against the wall. (getting
mcrypt working with php I'm looking at you)

You should keep other developers in mind when writing software.

~~~
dools
A couple of people in this thread have mentioned "tools" and I think this is
mostly because some friendly admin at YC decided to editorialise the title of
my post so that it differs from the article (and subtly changes the meaning of
it), so they're talking about tools that help us build software as fast as we
can _think_... which is different from the original title of the article which
says "as fast as we think we can". Anyway.

I think that where tools can help is by making a wider variety of different
projects _more predictable_ because domain experts create easy to use tools
that abstract away alot of the friction for us.

One notable example is jQuery: it has allowed a huge variety of non-javascript
experts to very reliably produce work that, in the old days, would have been
wildly unpredictable due to skull crushing differences in browser behaviour
and the intricacies of JavaScript as a language.

------
jconley
Reminds me of a great Steve McConnell book, Software Estimation: Demystifying
the Black Art.
[http://www.amazon.com/gp/aw/d/0735605351/ref=redir_mdp_mobil...](http://www.amazon.com/gp/aw/d/0735605351/ref=redir_mdp_mobile)

~~~
jacques_chester
I agree with this recommendation. McConnell has done his usual trick of
distilling lots of material into a concise, readable account.

------
mckoss
There are two different questions here:

\- Why can't we build software as fast as we think?

\- Why can't we build software as fast as we think we can?

The author probably meant the latter (which is more associated with estimation
and prediction).

But I find the former question MUCH more interesting. We've all had the
experience of envisioning software solutions that would be useful to us - but
then the actual task of developing that software seems much more difficult
than the task SHOULD be.

I attribute much of this to IMPEDANCE rather than FRICTION. We are often not
using the best tools for the job, and hence suffer great amount of superfluous
work in solving any particular problem.

General purpose programming languages may always have this problem; being able
to describe any solution means that they are not particularly well suiting to
expressing solutions to particular classes of problems).

I've always experienced myself being much more efficient when I have domain-
specific languages specifically tailored to the class of problems I want to
solve. I think we are moving (slowly) in the direction of developing more
targeted languages (e.g. Angular or Dart from Google for client-side web
development).

~~~
dools
Actually the title of the article is "Friction: why you can't build software
as fast as you think you _can_ " (emphasis here only).

Some admin was obviously in an editorial mood and has updated it to confuse
everyone.

------
thruflo
The article moves away from its interesting proposition to just restate
something bland about keeping scopes tight.

We can't build software as fast as we can think because we don't have the
tools. What those tools should be is an interesting and important thought
experiment.

~~~
danmaz74
>We can't build software as fast as we can think because we don't have the
tools

Actually, the real "problem" is that, as soon as we have the tools to easily
solve something that before was difficult, we move the target a step farther.
But that's also why programming is so interesting, and programmers are in such
high demand, so I wouldn't really call it a "problem".

~~~
lttlrck
The right tools move the goalposts. If this wasn't the case things would
stagnate pretty fast.

------
tudorconstantin
The only estimation I give on tasks is "as fast as possible"

------
macca321
AKA You Ain't Gonna Need It, sensible advice indeed.

To me though, Friction in software development is something slightly
different:

<https://www.google.co.uk/search?q=%22zero+friction%22+ayende>

~~~
dools
YAGNI to me is something different from scope creep.

YAGNI, to me, is more about programmers tendency to want to "solve each
problem only once" which leans to premature abstraction and generalisation,
more often than not before they truly understand the scope of the problem.

Being ruthless about what features you include is more about being able to
say, at the end of the project when it's overdue and everyone's hassled out of
their brain "well, at least we know we got it done as fast as we possibly
could have".

I'm always reminded of the HHGTTG bit (can't remember which book it's in)
about the ship waiting until a new society arises that is capable of
manufacturing lemon scented napkins so they can take off[1].

I think we spend a _lot_ of time in software building lemon scented napkins :)

[1][http://thelostcityofcarcosa.com/2012/11/12/lemon-soaked-
napk...](http://thelostcityofcarcosa.com/2012/11/12/lemon-soaked-napkins/)

------
jacques_chester
In their doctrinal manual, _Warfighting_ , the US Marine Corps use the same
term and with a similar meaning. The original analogy comes from von
Clausewitz.

    
    
        Friction is the force that resists all action and saps energy. It
        makes the simple difficult and the difficult seemingly impossible.
    
        Friction may be mental, as in indecision over a course of action.
        It may be physical, as in effective enemy fire or a terrain
        obstacle that must be overcome. Friction may be external, imposed
        by enemy action, the terrain, weather, or mere chance. Friction
        may be self-induced, caused by such factors as lack of a clearly
        defined goal, lack of coordination, unclear or complicated plans,
        complex task organizations or command relationships, or
        complicated technologies. Whatever form it takes, because war is a
        human enterprise, friction will always have a psychological as
        well as a physical impact.

