
Problems with TDD - fogus
http://www.dalkescientific.com/writings/diary/archive/2009/12/29/problems_with_tdd.html
======
ShabbyDoo
I find TDD impossible for all but the most trivial of coding tasks. As the
article asks, what should the resulting API look like? The act of solving a
problem gives me insight into the actual requirements, and the API results
from that knowledge.

Another issue I have with unit testing is that the cases which, for me at
least, identify the most bugs pre-production are more integration-ish. Does
the timeout setting for the HTTP API work as expected? Does this transaction
really roll-back against an Oracle DB with the "thin" driver on JBoss? These
are cases which are really hard to catch with traditional, use-the-UI testing,
either manual or automated. I should read more about what others are doing for
these semi-integration cases.

A couple of years ago, I committed a test case called TestHowDatabasesWork
because of a team argument about how Oracle handled concurrency and locking.
It got a good laugh, but it also documented a few behaviors specific to the
problem at hand.

~~~
CWuestefeld
Agree. TDD in the middle tiers of the app is easy, but that's not where the
difficulties lie.

Testing the UI layer is quite difficult (at least in the absence of something
like MVC, which too many developers can't handle and only want VB-like or
WebForms-like coding). And in any paradigm I've ever heard of, real testing of
the bottom layers of DB interaction is darned near impossible.

As the OP asserts, imposing a well-defined testing regimen is where most of
the bang for the buck derives.

~~~
steveklabnik
> And in any paradigm I've ever heard of, real testing of the bottom layers of
> DB interaction is darned near impossible.

Is it that because the code is written in an untestable way? A pretty standard
mocking approach should be able to test this.

~~~
mbrubeck
In my experience, problems "at the bottom" usually occur because underlying
third-party components like the database (or the filesystem or the network
stack or the remote server) don't behave the way you thought they did. They're
buggy or undocumented, or you just misunderstood their specs. When you mock
them away for testing, the mock objects act like _you_ expect and don't have
the same quirks as their real-life counterparts. If you don't mock them away,
then you have hard-to-run or hard-to-reproduce tests, or tests that give
different results with different instances of the external dependencies.

~~~
steveklabnik
Ah. That makes sense. I guess I've had the luxury of only working with MySQL,
Postres, and sqlite, and have never run into bugs with them.

~~~
ShabbyDoo
I think the issue is that a mock of a DB doesn't imitate all the nuanced
behaviors of the real thing. Transactionality, concurrency, isolation, error
generation, etc. So, while you can prove functional correctness with a
DBUnit/etc., you can't prove that the difficult stuff works.

~~~
steveklabnik
Isn't that the job of the DB package, and not of you, the user?

~~~
ShabbyDoo
It's the job of the DB package to perform as specified. The problem is that
one can write code that works correctly under the wrong assumptions of DB
behavior. Here's an example I've experienced:

We were building (to be purposefully vague) an auction system where an auction
could end at any point in time. Bids flowed in via a queue. It didn't really
matter within reason which order the bids were processed, but we had to make
sure that a bid which caused an auction to end was processed in isolation from
other bids that depended on the auction still being open. Our code relied upon
the DB's locking behaviors (SELECT FOR UPDATE) to ensure that the code
processing a particular bid laid claim to the state (auction status, etc) used
for its work. All fine and good except that we didn't quite understand how
Oracle implemented locking at various isolation levels. So, some code which
would have worked against, say, DBUnit, failed under Oracle. This wasn't a
failure of the DB, but we wouldn't have caught this problem unless we ran an
integration test.

~~~
steveklabnik
Ah. Interesting.

Databases have always been one of my weaker points, and lately, ORMs have
allowed me to be pretty lazy on that front. Thanks for walking me through
that.

------
michael_dorfman
I think that TDD is like Wittgenstein's ladder, which can be discarded once
you climb through the window, or the Buddha's raft, which can be discarded
once you cross the river.

TDD offers novice (or stuck) programmers two powerful nudges: 1) to break
tasks down to trivial units, and 2) to think about what you're trying to
accomplish before writing the code. Once you get the hang of doing this, TDD
is just dogma to cling to.

~~~
BrianHV
I have to admit I've always been puzzled by the opposition to TDD. I don't see
it as a novice thing, or a dogmatic thing. Yet many in this community see it
as such, so I'm wondering what they're seeing that I'm not.

To me, the fundamental benefit of TDD is that it proves that the code is in
the state I think it's in before I make a change, and it proves that the
change I made did what I expect. I'm not aware of any other technique that
does that. And I find that to be compelling enough to use the technique
routinely.

It seems to me the article author's basic point is that TDD alone isn't
sufficient to prove all the requirements are met. I agree with that, but I
also don't think that's sufficient to dismiss the practice.

~~~
pvg
_To me, the fundamental benefit of TDD is that it proves that the code is in
the state I think it's in before I make a change, and it proves that the
change I made did what I expect._

I think this is the sort of statement that makes some of us foam at the mouth
and gesticulate wildly at well-intentioned TDD proponents. TDD doesn't prove
anything of the sort. It proves that the code passed your tests before you
made the change and passed them after - which is a very, very different thing
from proving the change you made did what you expect.

 _I'm not aware of any other technique that does that._

Because there isn't one, in the non-trivial case but there are many that try
to help - from static typing to design-by-contract and of course a zillion
other testing methodologies. In fact, chances are very good that what you do
is not strictly unit testing. And that's fine and good. TDD didn't invent
testing.

[http://blogs.msdn.com/cashto/archive/2009/03/31/it-s-ok-
not-...](http://blogs.msdn.com/cashto/archive/2009/03/31/it-s-ok-not-to-write-
unit-tests.aspx)

Another rather irritating thing is the insistence that TDD is a design
methodology. If you haven't yet, read the informative case of an XP adherent
trying to write a sudoku solver with TDD. Here's a blog post that links to all
the relevant articles - the XP'ers attempt, Peter Norvig's solution and a
reference to Coders at Work where Norvig talks about it a bit.

[http://ravimohan.blogspot.com/2007/04/learning-from-
sudoku-s...](http://ravimohan.blogspot.com/2007/04/learning-from-sudoku-
solvers.html)

Long story short, if you don't know how to solve a problem and reason about a
problem TDD won't help you. Neither will Design Patterns, UML or any
particular fashionable technique, of course.

~~~
BrianHV
I think perhaps in my quest for concision I failed to communicate. I certainly
didn't mean to imply that TDD could provide certainty for the entire system.

But I have to differ with you in one key point. TDD specifically proves that
the code does _not_ pass your test before you made the change. The red bar is,
in my mind, the most important part of the practice.

Given that you have a failing test, you make a change, and then you have a
passing test, what more do you feel is required for proof that the change did
what you expect? It's possible that "proof" is too strong a word, but I find
that process gives me sufficient confidence.

(Side note: I'm honestly trying to figure out the best way to write code. I
don't think TDD is the silver bullet, but I think it's a useful tool in the
mechanical part of development.)

~~~
pvg
We're using 'pass' in slightly different senses, what I meant is (and this
applies to any test methodology) - in the strictest sense, getting the
expected outputs from a test-suite, making a change and getting the expected
outputs from a test-suite proves that you are getting the expected outputs
from your test-suite. It may do a lot of other things but typically proving
correctness is not one of them.

 _Side note: I'm honestly trying to figure out the best way to write code._

I think that's key and merits more than a side note. One of the reasons just
about any methodology can boast of success stories is that given a concerted,
focused and rational effort to improve quality, quality tends to improve,
regardless of methodology. Such efforts are quite hard to get right,
especially for organizations but when they succeed, there's little empirical
evidence the primary cause is the methodology.

So if TDD helps frame your own efforts to be a better programmer, by all
means, TDD away. The TDD griping comes from the often overblown claims of TDD
proponents that TDD improves quality. 'Honestly trying to figure out the best
way to write code' is a more valuable starting point than TDD.

------
MicahWedemeyer
This is a truly excellent article, and I really appreciate the tone. Calling
attention to the false dichotomy between pure TDD and test-last development
especially spoke to me.

In the real world, everything is a shade of grey, and the same is true with
software testing. Puritanical pursuit of theories and practices is silly. TDD
is useful in some instances and not in others. A skilled and experienced
programmer should be able to know when it's appropriate and when it's not.

------
omouse
Both the fib and prime factor examples are shit. The tests are individual
cases which is treating the input as a Cartesian product (a point on a line
since these are single-variable functions). What you want are sets. The inputs
for both are all natural numbers. However, this doesn't tell you much, there's
still a lot of state to work with. So what you want are end conditions, but
again that doesn't tell you much.

Tests are no substitute for reasoning and TDD is just another fad.

------
kscaldef
I'm overall sympathetic to the opinions in this article, but I think he's gone
astray in the section "Complaint: TDD freezes the API too early". One of the
ideas of TDD is that writing tests guides the API, rather than forcing you
into an API. That is, if you write tests based on the actual business
requirements, and write them first, that process will lead you to discover the
best design of the API.

Note that for this to work, you have to be writing tests based on the real use
requirements. Simple synthetic unit tests will not help you here, and may lead
to exactly the problem the author is complaining about.

------
awa
In the code kata's I have been to, we use the following model to TDD:

1\. If (all tests pass), Do one of the following:

    
    
      a. Write a failing test
    
      b. Write a passing test
    
      c. Refactor
    
      d. Start a 2 min discussion 
    

2\. Else (If a test is failing)

    
    
       Fix the code.
    

I would imagine the above model to TDD would fix many of the complaints in the
article.

------
darkxanthos
Great article. I enjoyed how well the author builds his case and argues for it
rather reasonably. While I do disagree with his article on the whole, I do
agree that TDD is not the end all be all. There is good TDD and there is bad
TDD. TDD is a very general practice and what really defines your tests as good
or bad is really your OOD skills.

One thing (really two things that work hand in hand) that I've found to
augment my OOD skills so that I write better tests and better software is
domain driven design and behavior driven testing. Using these techniques
together has helped me to learn to ask better questions of my domain experts,
create more decoupled and flexible software, as well as solve that age old how
do I test the UI when it isn't already in an MVC framework (the correct answer
is to treat it or portions of it as their own domains imo).

This is the kind of article that provokes thoughtful discussions by having a
well thought out opinion and not just blabbering the first idea that comes to
mind.

Thanks and keep it up!

------
pilif
I do agree with the general sentiment of the article, but on the other hand,
you should begin writing tests before it's too late and the code is too
interdependent to test without a LOT of effort.

When you write your tests first, you will naturally never end up in that
situation.

Of course, when you see that you have a hard time at writing unit tests
because, well, the units are too big, then you can (and should!) start
refactoring, but how do you refactor with confidence when you don't have tests
to see whether the refactoring went well?

Chicken and egg I guess.

~~~
dalke
"but on the other hand"? Isn't that the same hand? "My own practice is to have
good, automated tests, but these don't get put into place until the
cost/benefit ratio makes the tests worthwhile; which is rarely at the start of
the code development and always by the end." and "The important requirements
are to have good, complete automated unit tests, to develop code for testing,
..."

------
JulianMorrison
Why do people still write unit tests? You are stabbing in the dark, in a very
nonrandom way, and you're doing all the donkey-work _manually_.

If you care about testing, use a quickcheck clone. Bonus: it's much easier to
refactor tests when you're saying "it takes an X and a Y" instead of "it does
this and then that". And if you discover a bug caused by a corner case of Y
and add it to your Y simulator, then _everything_ that takes a Y gets tested
with that corner case.

------
vgurgov
good article. can give few examples myself when too much focus on code
perfection led startups to dead...

just get things done, man!!

------
gokhan
A classic example of getting TDD wrong. Happens all the time.

TDD is about design, the article is actually criticizes unit tests. TDD helps
you achieve SOLID principles, which really helps in real world. Fibonaccis,
primes etc. are only for demonstration purposes.

And 100% coverage is for sissies, only a TDD noob talks about it in this
context since it's not sane.

~~~
ShabbyDoo
>And 100% coverage is for sissies

Depends on your project and the cost of failure. I once worked on a system
that would automatically instruct semi-truck drivers to move their vehicles
hundreds of miles. I spend over a week building tests for
transactionality/concurrency edge cases to make sure that two trucks could not
be instructed to go to the same place in error. Worth it? Yup. Should you test
all possible behaviors of a utility script that, if broken, can just be re-
run? Probably not.

~~~
gokhan
It sure depends on the project type, I was talking about projects of HN
people. For my site, for example, I have broken the signup process twice so I
now have a couple of tests for it, unit + watin. But I won't bother with
writing tests for, say, privacy settings.

