
The Outrageous Cost of Skipping TDD and Code Review - ericelliott
https://medium.com/javascript-scene/the-outrageous-cost-of-skipping-tdd-code-reviews-57887064c412
======
NumberSix
The article appears to assume what it is asserting about TDD (Test Driven
Development):

Of course, this is a fictional example making a lot of assumptions. Your
mileage will vary based on lots of factors, including your team’s experience
with TDD, the presence or absence of other quality measures (such as code
review), etc…

Here are the assumptions made:

    
    
        Production bugs without TDD: 100
        Production bugs with TDD: 40 (60% fewer, middle ground between 40% — 80%)
        Average time to fix bug at implementation (TDD) phase: 1 hour (this number is only used to derive the production bug fix cost)
        Average time to fix a production bug: ~15 hours
    

The author does try to justify this by saying:

The relative results of this fictional example ring true to me based on my
experience with real production projects, both with and without TDD.

BUT...

Where is the specific data and how representative is the author's experience
of all or most software development?

Software development is enormously varied. Kent Beck initially hawked TDD
based on a project to rewrite Chrysler's payroll system that was cancelled.

[https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compens...](https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compensation_System)

Also see:

[https://stackoverflow.com/questions/10149798/how-tdd-is-
rela...](https://stackoverflow.com/questions/10149798/how-tdd-is-related-to-
extreme-programming)

The Chrysler C3 project is not the most convincing argument for TDD or other
of Beck's ideas. But even assuming the C3 project was a roaring success -- a
highly debatable proposition, one should ask how applicable the results are to
many other quite different types of software projects.

Similarly, what is the author of this article's actual experience and how
applicable is it to other types of software projects?

I don't question that automated tests can be helpful and indeed critical in
some projects but I do question rigid orthodoxies like TDD or requiring unit
tests for every function or class after implementation (many people promote
unit tests everywhere while claiming they are not doing TDD).

Tests take additional time to design and implement and thus cost more. They
also have maintenance costs. Architectural and other changes in the software
as it is developed can and do result in revising or rewriting the tests as
well.

Many successful software projects were designed and implemented in the 1970's,
1980's, 1990's, and even today without TDD and without unit tests -- or code
reviews although I am focusing here on TDD and unit tests.

~~~
ericelliott
While the example is fictional, the numbers used for it come from a meta-study
of TDD research, and for the relative cost of fixing a bug in production vs
implementation time, "Integrating Software Assurance into the Software
Development Life Cycle (SDLC)" from the IBM System Sciences Institute.
Original sources and raw data are available if you click through the links.

None of the examples or figures presented rely on the Chrysler C3 project, or
anything published by Kent Beck. I'm unsure why you'd mention them except to
build a straw man argument. Straw men aside, this is all independent research
with lots of 3rd party citations.

Yes, tests do take additional time and do incur maintenance costs as well, and
that additional time is accounted for in the numbers presented. TDD still
easily wins.

Yes, you can produce successful projects without TDD. You can also run a
marathon with only 1 leg -- but maybe it would be easier with 2.

As for questioning my credentials, I don't blame you. It's certainly a lot
easier to try to cast doubt on somebody's credentials than it is to come up
with a valid argument or think critically about your own beliefs and bias. If
you have any sincere doubts, a sampling of projects I've worked on are listed
at the bottom of the article.

~~~
NumberSix
The linked article TDD: the Art of Fearless Programming by Ron Jeffries and
Greg Melnik, IEEE Spectrum May/June 2007
[https://www.computer.org/csdl/mags/so/2007/03/s3024.pdf](https://www.computer.org/csdl/mags/so/2007/03/s3024.pdf)

cites Kent Beck's books Extreme Programming Explained (1999) and Test Driven
Development: By Example (2002)

Kent Beck is Mr. TDD It is reasonable to question his claims about C3 as many
have done.

The IBM article Integrating Software Assurance into the Software Development
Life Cycle (SDLC)
[https://www.researchgate.net/publication/255965523_Integrati...](https://www.researchgate.net/publication/255965523_Integrating_Software_Assurance_into_the_Software_Development_Life_Cycle_SDLC)

has the following section:

PROCESS TO SECURE CODE

In the event of a vulnerability finding, the software code may require
redesign and implementation. This iterative cycle is costly in time and
resources. To truly understand security threats to a system, security must be
addressed beginning with the initiation phase of the development process. For
an organization this means they must allow the IA controls and requirements to
drive design and influence the software requirements. Therefore, any
identified security threats found during the requirements and analysis phase
will drive design requirements and implementation. Security defects discovered
can then be addressed at a component level before implementation. The cost of
discovery and mitigation can be absorbed within the review, analysis and
quality check performed during the design, and implementation of our SDLC. The
resultant product is one with security built in rather than security
retrofitted. A study was performed by the IBM System Science Institute in
order determine the relative cost in order to fix defects within the SDLC.
Figure 2 displays their findings.

followed by a figure 3 (not 2) which is a simple graphic.

There is no identifiable reference to the underlying data which is presumably
some sort of study for the US Department of Defense for security software
issues presumably for military related software projects. This does not appear
to be a general study, nor is it at all clear what was studied or how.

I am not questioning credentials but extrapolation from in Kent Beck's case a
payroll system to general software development or in the IBM case from some
sort of DoD software security projects to general software development.

DoD and aerospace projects often have unusually high costs for bugs in the
production systems. A bug in a flight avionics system can cause a literal
crash with loss of millions or even billions of dollars (Space Shuttle for
example) and lives.

~~~
ericelliott
The costs discussed are counted in development time, not the monetary value of
rockets. That point is clearly evident with a cursory glance at the math. Just
look at the numbers. The values all fall on the same curve. The cost increase
from design phase to implementation phase is the same relative increase from
implementation phase to testing phase, which is the same relative increase
from the testing phase to the production phase.

If it were counting the cost of rockets in the production phase, it would
clearly jump that value off that cost curve in a dramatic and clearly notable
way.

The mathematical evidence of these facts is clearly visible in this chart:
[https://cdn-images-1.medium.com/max/2000/1*jOS7CZX-gWfOoKGPU...](https://cdn-
images-1.medium.com/max/2000/1*jOS7CZX-gWfOoKGPUU29kg.png)

Which is a reproduction of the graph that appears in the original paper using
identical data.

Of course, the data I cited was not the only data available on the economic
impact of software quality. There is a whole chapter on the topic of the
economic benefits of reducing bugs in the book "The Economics of Software
Quality" by Capers Jones, Olivier Bonsignour Addison-Wesley, Jun 3, 2011.

The book is packed with valuable references if you're looking for more data on
why it's worthwhile to reduce production bug density using techniques like TDD
and code review. I strongly suggest you have a look at it if you have any
doubts about the value of reducing production bug density.

