

The strange case of the inverted chart - gjm11
http://lesswrong.com/lw/9sv/diseased_disciplines_the_strange_case_of_the/

======
Maxious
Somehow I think this post would have been more valuable if it hadn't played
stupid games about imagination.

Sorry to spoil it but the punchline is that the study that everyone claims
proves "[software bugs] detected during system testing cost 10 times more to
fix than in the construction phase and 10-25 times more to fix when detected
post-release" actually didn't prove that and software engineering academics
have been parroting it because they're citing a powerpoint presentation that
doesn't mention the caveats of the study.

Would it have been so hard to lead with that fact?

~~~
hammerdr
It isn't entirely false:

If one assumes that bugs are introduced at construction (the writing of the
software) then you can use that as a fixed point to determine the cost of bugs
based on when they are found. Which, using the original study, would mean that
in testing the bugs would be more costly and that in production the bugs would
be even more costly.

What _is_ interesting is that it is a function of time and not phase. So, for
example, if you were continuously deploying code and a QA found a bug the same
day it would effectively be the same cost (assuming everything else is equal)
as a code review done by a developer. It would also be significantly less than
a bug found several months down the pipeline once your code was finally
integrated into a QA environment in a non CD example.

~~~
barefoot
Agreed: When I work on a block of code I am quickest at making changes to it
within the hour (as I still have it's design and structure loaded into my
brain).

I'm slower to make the same changes after a few days or a week, and even more
so after six months.

My experience is that I'm not alone. Well structured code, unit tests, and
documentation help - but will never eliminate the work of loading the system
into your brain again.

------
eps
By the way, regarding _inverted charts_ per se.

There is an anecdote of one physicist showing an X-Y chart to another. Latter
elegantly explains the phenomena, but then the first guy says "oh, you are
looking at it upside-down", to which the other guy says - "ah, now it makes
even more sense."

~~~
gjm11
That's a failure mode that's had quite a bit of attention at Less Wrong; see
e.g. <http://lesswrong.com/lw/ia/focus_your_uncertainty/> and
[http://lesswrong.com/lw/ii/conservation_of_expected_evidence...](http://lesswrong.com/lw/ii/conservation_of_expected_evidence/)
.

------
SoftwareMaven
""defects detected in the operations phase (once software is in the field)
cost more to fix the earlier they were introduced"""

I've never heard it phrased that way "in industry" (as opposed to academia,
where I have no experience). Instead, what I've heard is:

""defects detected in the operations phase (once software is in the field)
cost more to fix than those detected and fixed earlier"""

While perhaps not backed up with published results (again, I don't know), just
by 1) taking the actual development cost of fixing a bug as constant (which is
_extremely_ generous for comparing a bug in dev versus production) and 2) then
counting the number of other people involved in identifying and resolving the
bug shows it to be true.

------
Drbble
Has anyone ever made the claim that LW is lampooning? I have only heard that
testing early is important because it is expensive to fix mistakes after
development, and more expensive after release.

I guess the straw man argument is that design bugs are worse than
implementation bugs in a project, but that's also true. In both cases, the
issue is that time cures all wounds, where cure is used in the sense of made
permanent, not fixed, or fixed is used on the sense of locked, not restored to
working order.

------
ThomPete
Feynman have a great little story about checking the facts behind the facts.
Anyone got a link to that?

------
JanezStupar
The good news is that most people reading and using the data will be too
illiterate to know the difference.

