
An apology to readers of Test-Driven iOS Development - Morendil
http://blog.securemacprogramming.com/2012/09/an-apology-to-readers-of-test-driven-ios-development/
======
jamieb
"The second problem, and the one that in my opinion I’ve let down my readers
the most by not catching, is that the table is completely false."

There is a big difference between "unsupported by hard data" and "completely
false", especially when the author later admits to not being willing to pay to
see the first paper cited in support.

It also seems, (from this post:
[http://lesswrong.com/lw/9sv/diseased_disciplines_the_strange...](http://lesswrong.com/lw/9sv/diseased_disciplines_the_strange_case_of_the/))
that in fact the first line is supported by data, if only in aggregate. The
author of that blog posts makes much of the phrase "this study didn't
accurately record" while glossing over the next part of the sentence that
states "so we will use average rates summarized from several other studies."
It is common practice to aggregate data, and provided it is done properly, the
aggregation can reduce error, not increase it.

The claim that "a bug introduced in Requirements, costs more to fix the later
it is discovered" is claimed to be supported by data. Well done for demanding
to see that data. But epic fail for creating controversy by claiming that the
opposite is true.

------
Morendil
Amusing exchange over at Reddit:

Redditor 1: "I bet not a single f$%k was given about that table among the
readers of the book :)"

Redditor 2: "That table, or something like it, features strongly in every
software engineering textbook that I've ever seen. The numbers differ, but the
increasing order of magnitude differences are roughly the same. It's the
essential justification for nearly everything in software engineering. "A
single f$%k" doesn't begin to describe the importance of that table."

And yes, "the essential justification for nearly everything in software
engineering" is broadly correct. Anytime you see an argument that "it's
important to get the requirements right" for instance, it's based on studies
that purport to show these numbers.

~~~
ajross
Indeed, with the semantic cavet that "software engineering" is used in the
sense of "software development management process nattering" and not (always)
the actual engineering of working software.

The basic falseness of that chart is pretty well known among the hacker set.
Or maybe it's a failure of interpretation among the natterers: the "high cost"
of mistakes in the past is made mostly of work already spent. The _immediate_
cost of correcting a past mistake is often (of course not always!) vastly
lower than people think. And the farther you are from the actual code involved
(i.e. if your role is "architect"), the higher your overestimate of that value
is going to be.

So this is a feedback loop: the "high level" people at the start of the
process have a built-in bias against iterative solutions (which they perceive
as "expensive") and thus a built-in predilection to overdesign as they try to
get everything right the first time.

~~~
mcguire
I'm glad someone else sees the difference between the engineering of working
software and the "process nattering" that seems to make up official "software
engineering" as far as I can tell.

You do miss a couple of steps in the feedback loop:

1\. " _The 'high level' people at the start of the process have a built-in
bias against iterative solutions._ "

2\. " _A built-in predilection to overdesign as they try to get everything
right the first time._ "

3\. The wild-ass guesses that make up the overdesign prove to be completely
wrong.

4\. The project fails miserably or succeeds, miserably.

5\. The "high level" people perceive software as "very expensive", reinforcing
their biases.

------
grayprog
I know Graham. And he's both a great guy and a serious developer. I've also
read his book and though I've not yet finished it, I've seen this table, as
it's in the beginning.

Having read this blog post, I now respect him even more. Both for the
intellectual honesty and for his efforts to try and reconcile the data in the
table, with all its consequences.

I wish more people (including me) were this dedicated and willing to admit
mistakes.

Maybe that's what makes him a specialist in software security, of all things.

------
Shish2k
> I therefore no longer consider Table 1-1 in Test-Driven iOS Development to
> be representative of defect fixing in software engineering

Surely this makes it _more_ representative, in a literal sense -- if he'd
checked his data at the start, he wouldn't have to go back to check it + issue
an apology + reprint the books now :P

~~~
marshray
The general idea ("bugs found later tend to cost more to correct") is
uncontroversial.

It's also accepted that this is a very difficult thing to study. There are so
many subjective factors involved that's it's really hard to quantify the data
or the results. But it seems to me that if Software Engineering as a wants to
make progress methodically, it needs to throw out the old unsubstantiated
assumptions in order to make room for new conclusions with a solid basis.

------
robomartin
The time and cost to fix bugs can have a huge variance which is sometimes
dependent on the nature and field of the software being written. There are
bugs that take seconds to fix and others that take months.

I have personally experienced hunting down a bug for six months nearly full
time (10 to 12 hours per day) until it was finally found. This was a real-time
hardware system and the code was that of an FPGA. The culprit was a
coefficient in one of many polyphase finite impulse response filters. The
calculations used to generate this coefficient were done in a huge Excel
spreadsheet. At one point, in the hundreds of equations in the spreadsheet,
the author had used the ROUND() function when the ROUNDUP() function should
have been used. This was enough cause a buffer issue in the FIFO feeding the
polyphase filter. These are tricky problems to track down when you are
literally looking for events that take somewhere in the order of single-digit
nanoseconds to occur.

On the other hand, there are those bugs where you know exactly what is going
on and where in the code it is happening the instant you see the behavior.
We've all experienced that during the course of software development.

Fixing bugs for a dating website has vastly different requirements than, say,
fixing bugs in a flight control system.

One argument is for more up-front planning in order to avoid some bugs. At one
point this can quickly become counterproductive. Sometimes it's better to just
start writing code and fix issues as they come up.

Now, if we are talking about fixing bugs after the fact, that's a different
matter. One example here might be if you inherit the code to a complex website
or an engine control system and, without any familiarity with the code, are
required to fix bugs. This can easily take cubic time as it requires to learn
the code-base (and sometimes the subject matter) while also trying to hunt
down bugs.

This is why I tend to take such tables or studies with great skepticism. I
haven't really paid much attention to these studies, but I remember looking at
one or two of them and thinking that they tended to focus on narrow cases such
as fixing bugs in-house, with a stable programmer team and great familiarity
with the code base.

~~~
lttlrck
Skepticism is deserved. It's a very fine balance, and bearing in mind not much
software exist that is entirely bug-free.

What is the cost of a bug that is never discovered? Is it negative? What is
the cost of fixing a bug that would never have caused a problem (at each stage
of development)?

It's way more complex than bugs == bad

~~~
thenonsequitur
Agreed. Indeed, your argument naturally follows from a "bug fixes are
features" type mindset.

------
thebooktocome
Previously on LessWrong:
[http://lesswrong.com/lw/9sv/diseased_disciplines_the_strange...](http://lesswrong.com/lw/9sv/diseased_disciplines_the_strange_case_of_the/)

~~~
cousin_it
Note that the author of that post is Laurent Bossavit, the author of the
"Leprechauns" book that was referenced in the OP.

------
diego_moita
Call me cynical but I am getting very skeptical against a lot of well
established "truths" in Software Engineering: the cone of uncertainty, the
orders of magnitude difference in programmers productivity, the efficiency of
TDD, ...

Most of these well established claims simply don't have enough empirical data
to sustain them. They're all bloggers' hand waving and empty claims.

~~~
itmag
_the orders of magnitude difference in programmers productivity_

I am skeptical of this too. It makes more sense to have huge swings in
_ability_ , not _productivity_. Ie a poor programmer won't take 10x as long to
code a given feature, he will just hit a ceiling of ability and not be able to
do it at all.

~~~
robomartin
>Ie a poor programmer won't take 10x as long to code a given feature, he will
just a ceiling of ability and not be able to do it at all.

10x longer is not a poor programmer, that's an incompetent programmer. If
someone needs ten days to code something that can be done in one day something
is very seriously wrong.

~~~
shawabawa3
> 10x longer is not a poor programmer, that's an incompetent programmer. If
> someone needs ten days to code something that can be done in one day
> something is very seriously wrong.

That's just not true, it's all relative. Linus Torvalds supposedly coded git
to the point where it was self-hosted in 1 day. Even a very good programmer
could take more than 10 days to do that, an average (but not incompetent)
programmer could take months.

~~~
jkubicek
Linus has guessed it took him about two weeks to get git to the point it was
self-hosted.

<http://www.spinics.net/lists/git/msg24132.html>

It's still impressive, though.

Edit: Should have read further. It looks like Linus got git self-hosting in
two days.

<https://lkml.org/lkml/2005/4/8/9>

------
consultutah
Hopefully this will end in a discussion in which the actual data underlying
the table can be had and matched up.

Anecdotally, it seems to make sense that it is more expensive to find and fix
a defect later in the process. If a dev finds a defect while implementing a
feature, only his costs are involved. However, if a defect is found later in
the our process, at least 2 qa analysts are involved: one to find the defect
and another to confirm it. After that a project manager schedules out time and
assigns the defect out to a developer, possibly not the one that introduced
the defect gets the assignment to fix it. The developer fixes the defect, a
build is made by the build person. The original tester retests the defect and
marks the fix as being verified.

That seems complex, but there are possibly even more steps than that. Unit
tests may need to be written, customer test cases may need to be updated.

I don't know if the costs end up being exponentially greater, but they would
seem to be greater.

At any rate, it would be good to have someone independently validate that
data.

~~~
philwelch
It's not written in stone that you need 2 QA analysts, a project manager, a
developer, and a build master to fix a bug. That only proves that bureaucracy
is expensive, not that fixing bugs "late" in the process is expensive!

I'm not sure what exactly the second QA person adds to the process. As a
developer, I need to be able to reproduce a bug myself in order to have any
chance of fixing it, so there's your confirmation step right there.

Project manager? No, just have your QA person file the ticket as a bug (with
the correct priority) and have your developers pull tickets off your bug
tracker themselves. At worst, a slight email/verbal nudge from the usual boss
should do.

Build person? No, have a continuous, automated build system. The job of the
build person should be to maintain that system, not to run individual builds.

Now you're down to one QA person to file the ticket, one developer to fix the
bug, and the same QA person again to close the ticket. Add in a few minutes of
another developer's time to code review the first developer (you don't even do
_that_ despite having all that process?) and it's still cheaper.

Unit tests? Developers write their own unit tests. Have a failing unit test
that reproduces the root cause of the bug _before_ you fix the bug--that's a
best practice anyway. For testing above and beyond that, it's not unreasonable
to have SDETs maintain that stuff, though you already have the root cause of
the bug captured in a unit test so your main concern should be whether any
existing tests actually rely upon the broken behavior.

~~~
marshray
_It's not written in stone that you need 2 QA analysts, a project manager, a
developer, and a build master to fix a bug._

Well - that depends. For plenty of large software engineering projects (think
aerospace) it's a given that you'll have that organization many times over.

On the other hand, a web startup may produce a high-quality shipping product
with just two or three devops. The number of "lines of code" may even end up
being similar between the two deliverables.

The code that gets burned into an ASIC requires a completely different set of
development considerations than a non-critical "make deploy_production" web
service.

But how do you quantify all these confounding factors?

~~~
philwelch
The hope is that adding all those people adds value somewhere, it doesn't just
add cost.

~~~
marshray
Right - the aerospace people aren't stupid - far from it. They simply have
vastly different goals and constraints, and thus a different organization and
a different development process.

------
sunraa
Lots of props to the blog author for going through the hoops that I suppose we
should all be going through. I've always placed the CC books up there in the
pantheon of great software engineering books. This is a chink in the armor and
I hope Mr McConnell takes the time to provide a response although I'm not
holding my breath. Another object lesson in not accepting things at face
value.

------
jakejake
Based on my own subjective experience I've found the numbers regarding testing
to be suspicious simply because "bugs" are so varied that any specific number
would be subjective.

I'd always assumed that these numbers referred to an architectural type of bug
where, once made, more code is built upon the bug and so a cascading effect
occurs. The more code that relies on the bug, the worse it is to fix because
you have to fix all of it's dependent code. In some cases you may need to
repair data, or entirely refactor sections of an application.

But there's other bugs that are more "typo" level bugs where obviously the
time to fix is exactly the same no matter what phase of development.

I had just always assumed those numbers were a worse-case average to encourage
developers to better plan their architecture. In part because the context of
code complete is more about planning and estimation.

Though there may not be support for the exact figures, I think the point of
them is to not let things pile up and to try to put thought into your design
especially at the architectural levels.

~~~
MattRogish
This seems to have been my experience and interpretation as well. A bug is not
a mistake in coding, but a mistake in selecting core architecture, 3rd party
libraries, etc.

Making a mistake picking the wrong storage engine can cause major problems if
you find it can't scale to your load after 12 months of development. That's a
ton of code to change if you're switching from MySQL to Redis (for a crazy
example).

Make a mistake in copy or layout, and that's usually much easier to change.

------
unreal37
It seems he wasn't able to see some of the sources. One cost money to see, one
is out of print and not available...

It may be true and noble that since he isn't able to verify the data himself,
he shouldn't have used it. But if you can't see the sources, you also can't
state that the data is incorrect. One of those books he couldn't find might
contain the data that directly corroborates this.

I suspect the cost of fixing defects has more to do with your internal
process, and less to do with how much work it takes to find and fix the actual
defect. I have a client I work for now for which I need about 10 days lead
time to get code from development to production. So it might take 1 hour to
fix a bug that is discovered, and 8 hours to do the paperwork and go through
the formal process of moving code through testing, staging and finally to
production.

------
aptwebapps
I wonder if he tried asking McConnell just how he compiled the table. It does
seem a bit odd to construct such a simple table from eight different sources.

~~~
mcguire
...none of which seem to have the actual data to support the table.

------
smoyer
I seem to remember first seeing a table like the one the author describes in
"The Mythical Man Month" but my copy is currently at home. The data underlying
that book was gathered from projects at IBM in the '60s and '70s and I don't
really doubt that the underlying data was fairly represented.

The bigger question is whether improvements in processes and tools have
obsoleted this data. TDD and automated regression testing would be one place
where inter-phase defects could become less costly. On the other hand,
projects that have uncorrected architectural errors can be completely
functional and yet their maintenance costs never decrease.

If someone has a copy handy, please validate my memory otherwise I'll check
when I get home this evening.

~~~
alttag
I've just gone through it page-by-page, and didn't see a table like that.
Using the 20th Anniversary Edition, I also looked through the summarized list
of claims in each chapter, and the 20-year retrospective, and did not see the
table or a section that might have been a textual version of the same data.

Perhaps the closest bit I found was in the chapter "Plan One to Throw Away":
"The total cost of maintaining a widely used program is typically 40 percent
or more of the cost of developing it" (p 121). Similarly, another section
quotes Capers Jones, "Focus on _quality_ , and productivity will follow" (p
217, emphasis original).

That, I think, is about as close as MMM gets to these claims.

~~~
smoyer
I couldn't find it either but in addition to "Code Complete" this data is also
shown in a chart in "The Software Project Survival Guide", also by Steve
McConnell (p29).

The attribution is "Researchers have found" but Boehm and Papaccio are listed
in the references.

------
zwdr
I wish more authors would check their references like that. Not that I am
interested in iOS-development, but still.

------
habitue
It's great to see such intellectual honesty. It looks like the real
perpetrator, however, is McConnell.

~~~
Maxious
"The fact that no studies have produced findings that contradict the 10x claim
provides even more confidence in the 10x claim. When I consider the number of
studies that have been done, in aggregate I find the research to be not only
suggestive, but conclusive—which is rare in software engineering research."
Steve McConnell
[http://forums.construx.com/blogs/stevemcc/archive/2011/01/09...](http://forums.construx.com/blogs/stevemcc/archive/2011/01/09/origins-
of-10x-how-valid-is-the-underlying-research.aspx)
<http://news.ycombinator.com/item?id=4117417>

~~~
nateberkopec
Yikes, that doesn't strike me as very intellectually sound.

"The fact that no studies have produced findings that contradict the existence
of a teapot orbiting Jupiter..."

~~~
gruseom
I think if you read the article you'll find that that's unfair. His argument
is that there are many studies that support it _and_ none that contradict it,
and that such consistency is unusual in the literature. That may be wrong but
it's not trivially stupid as you make it sound.

------
ludflu
Much respect for the frankness and forthrightness with which he addressed
this. I've always taken the exponential increase in the cost of bug fixes with
time for granted. I won't do so in the future. We need more empirical studies
of software development!

------
Tashtego
Anecdotally, I have certainly found that although the cost in time may not
vary as much as this table would indicate, the cost in stress ramps up even
faster. Fixing a bug in production is usually a highly stressful endeavor for
all involved. I would love to see a similar table phrased in terms of stress
comparing different development methodologies currently in vogue (test and
throw it over the wall, CI, automated pushes vs. manual pushes, etc.)

------
emmapersky
A google scholar search for the title of the paper behind the paywall reveals
numerous copies freely available through university websites.

------
f4stjack
So yeah, long story short from what I gather is: I couldn't find the
referenced sources to the table I am referencing so it must be false!

What a leap of logic, seriously...

~~~
ludflu
He didn't say it was false. He did say it was unsupported.

~~~
gruseom
"The second problem, and the one that in my opinion I’ve let down my readers
the most by not catching, is that _the table is completely false_."

~~~
ludflu
"it" meaning the conclusion about the exponential increase in the cost of
fixing bugs over time.

------
MaxGabriel
Would people who've read the book recommend it? It sounds interesting to read
because I'm not experienced with TDD, and iOS tends to shy away from it.

