Hacker News new | past | comments | ask | show | jobs | submit login
Problems with TDD (dalkescientific.com)
79 points by fogus on Dec 29, 2009 | hide | past | web | favorite | 64 comments

I find TDD impossible for all but the most trivial of coding tasks. As the article asks, what should the resulting API look like? The act of solving a problem gives me insight into the actual requirements, and the API results from that knowledge.

Another issue I have with unit testing is that the cases which, for me at least, identify the most bugs pre-production are more integration-ish. Does the timeout setting for the HTTP API work as expected? Does this transaction really roll-back against an Oracle DB with the "thin" driver on JBoss? These are cases which are really hard to catch with traditional, use-the-UI testing, either manual or automated. I should read more about what others are doing for these semi-integration cases.

A couple of years ago, I committed a test case called TestHowDatabasesWork because of a team argument about how Oracle handled concurrency and locking. It got a good laugh, but it also documented a few behaviors specific to the problem at hand.

A couple of years ago, I committed a test case called TestHowDatabasesWork because of a team argument about how Oracle handled concurrency and locking. It got a good laugh, but it also documented a few behaviors specific to the problem at hand.

There is a good possibility that in documenting what you saw working you failed to notice some important details of its optimistic locking policy.

If you're ever again faced with such questions about Oracle I highly recommend picking up an appropriate book by Tom Kyte. For a developer in particular I would recommend http://www.amazon.com/exec/obidos/tg/detail/-/0072230657.

Agree. TDD in the middle tiers of the app is easy, but that's not where the difficulties lie.

Testing the UI layer is quite difficult (at least in the absence of something like MVC, which too many developers can't handle and only want VB-like or WebForms-like coding). And in any paradigm I've ever heard of, real testing of the bottom layers of DB interaction is darned near impossible.

As the OP asserts, imposing a well-defined testing regimen is where most of the bang for the buck derives.

Testing all of the ins and outs of possible DB behavior is out of scope for most software projects. You need to assume your driver works until proven otherwise.

As for testing interactions with the database, one approach that I've seen work is to have data in your test cases with which you build up database objects that you then interact with. If your classes all have appropriate helpers and appropriate ways of inserting reasonable defaults on fields not explicitly set in the testing code, this makes testing the expected database interactions fairly straightforward.

> And in any paradigm I've ever heard of, real testing of the bottom layers of DB interaction is darned near impossible.

Is it that because the code is written in an untestable way? A pretty standard mocking approach should be able to test this.

In my experience, problems "at the bottom" usually occur because underlying third-party components like the database (or the filesystem or the network stack or the remote server) don't behave the way you thought they did. They're buggy or undocumented, or you just misunderstood their specs. When you mock them away for testing, the mock objects act like you expect and don't have the same quirks as their real-life counterparts. If you don't mock them away, then you have hard-to-run or hard-to-reproduce tests, or tests that give different results with different instances of the external dependencies.

Interesting - in the embedded software world we have the exact same problem with hardware. Mocking the hardware can confirm the internal consistency of your ideas about how it works, but most problems stem from your ideas being inconsistent with how it actually works.

Until now I always assumed that the environment from which TDD emerged (presumably database-driven "Enterprise" apps) was different in a way that made it seem like a better idea than it does to me, but your comment is causing me to seriously rethink that assumption.

Ah. That makes sense. I guess I've had the luxury of only working with MySQL, Postres, and sqlite, and have never run into bugs with them.

I think the issue is that a mock of a DB doesn't imitate all the nuanced behaviors of the real thing. Transactionality, concurrency, isolation, error generation, etc. So, while you can prove functional correctness with a DBUnit/etc., you can't prove that the difficult stuff works.

Isn't that the job of the DB package, and not of you, the user?

No, and I'll give you an example. Take a simple relational table with some kind of constraint. Begin a transaction and insert some data into this table that violates the constraint. What happens? Guess what, it depends on the db. Some databases/drivers will throw an exception immediately upon insert. Some will wait until the commit has occurred to throw the exception.

Subtle differences like these exist all across the various relational db world. Not to mention that they all do locking and concurrency differently, which is probably the biggest problem.

It's the job of the DB package to perform as specified. The problem is that one can write code that works correctly under the wrong assumptions of DB behavior. Here's an example I've experienced:

We were building (to be purposefully vague) an auction system where an auction could end at any point in time. Bids flowed in via a queue. It didn't really matter within reason which order the bids were processed, but we had to make sure that a bid which caused an auction to end was processed in isolation from other bids that depended on the auction still being open. Our code relied upon the DB's locking behaviors (SELECT FOR UPDATE) to ensure that the code processing a particular bid laid claim to the state (auction status, etc) used for its work. All fine and good except that we didn't quite understand how Oracle implemented locking at various isolation levels. So, some code which would have worked against, say, DBUnit, failed under Oracle. This wasn't a failure of the DB, but we wouldn't have caught this problem unless we ran an integration test.

Ah. Interesting.

Databases have always been one of my weaker points, and lately, ORMs have allowed me to be pretty lazy on that front. Thanks for walking me through that.

Suppose your code needs to interact with stored procedures in the database. How does a standard mocking approach in your code help you test the stored procedures?

See, I really enjoy freezing the API early. The API should be very, very simple. I tend to code by designing top down, then coding from the bottom up.

APIs should _not_ be from the perspective of "this is how I solve this problem" and should be from the perspective of "this solves my problem."

For example,

    TempCalc tc = new TempCalc();
    tc.f = 50;
    System.out.println("50 degrees farenheit is " + tc.convertToCelcius() + " degress celcius");
is bad design. Yes, you may need to get a farenheit value, and change it to celcius, but I don't care about setting internal state, then calling functions to change it. I just want an "fToC(int f)" function.

This is an admittedly contrived example, but I see this principle in API design all the time. Everything should be from the perspective of the user, not the coder.

"I'd love to use this api if it looked like this."

Anyway... as far as that integration stuff goes, have you used Selenium? I'm not super up on Java testing stuff, as I'm a Rubyist, but I use it, and I know it has Java bindings of some kind...

Selenium 2 (the Google "Webdriver" stuff) works quite well and isn't specific to Java in any significant way. If you can bring up your UI with a bunch of mocks/stubs under it, Selenium is a great way to do automated regression testing of your web UI.

On your API design comments:

My issue with TDD is that it's implicitly big-design-up-front. I used to think I understood a problem before I stated coding, but I don't anymore. So, anything I design in the beginning ends up being modified (if not scrapped entirely). There's no better way for me to understand a problem than by trying to solve it.

Also, for important enough APIs (public stuff, widely used in a codebase, or whatever), the design of the API is dependent on the problem being solved. Let's say you are building an API for a yet-to-be-developed photo storage SaaS. For any solution of significant sophistication, can you really think through all the issues that have API design implications before attempting to create the service itself? Is it better to stream back the bits of a photo found via a tag search or should a "ticket" in the form of a GUID be provided instead? I don't know, but I'll have a better idea once I spend some time solving the problem.

Thanks for confirming on the Selenium thing.

I guess it just comes down to personal taste, then. I feel that I can't understand how the coding should be laid out until after I've designed how it will be used.

I think that TDD is like Wittgenstein's ladder, which can be discarded once you climb through the window, or the Buddha's raft, which can be discarded once you cross the river.

TDD offers novice (or stuck) programmers two powerful nudges: 1) to break tasks down to trivial units, and 2) to think about what you're trying to accomplish before writing the code. Once you get the hang of doing this, TDD is just dogma to cling to.

I have to admit I've always been puzzled by the opposition to TDD. I don't see it as a novice thing, or a dogmatic thing. Yet many in this community see it as such, so I'm wondering what they're seeing that I'm not.

To me, the fundamental benefit of TDD is that it proves that the code is in the state I think it's in before I make a change, and it proves that the change I made did what I expect. I'm not aware of any other technique that does that. And I find that to be compelling enough to use the technique routinely.

It seems to me the article author's basic point is that TDD alone isn't sufficient to prove all the requirements are met. I agree with that, but I also don't think that's sufficient to dismiss the practice.

To me, the fundamental benefit of TDD is that it proves that the code is in the state I think it's in before I make a change, and it proves that the change I made did what I expect.

I think this is the sort of statement that makes some of us foam at the mouth and gesticulate wildly at well-intentioned TDD proponents. TDD doesn't prove anything of the sort. It proves that the code passed your tests before you made the change and passed them after - which is a very, very different thing from proving the change you made did what you expect.

I'm not aware of any other technique that does that.

Because there isn't one, in the non-trivial case but there are many that try to help - from static typing to design-by-contract and of course a zillion other testing methodologies. In fact, chances are very good that what you do is not strictly unit testing. And that's fine and good. TDD didn't invent testing.


Another rather irritating thing is the insistence that TDD is a design methodology. If you haven't yet, read the informative case of an XP adherent trying to write a sudoku solver with TDD. Here's a blog post that links to all the relevant articles - the XP'ers attempt, Peter Norvig's solution and a reference to Coders at Work where Norvig talks about it a bit.


Long story short, if you don't know how to solve a problem and reason about a problem TDD won't help you. Neither will Design Patterns, UML or any particular fashionable technique, of course.

I very much agree with this. I for myself think writing tests first or rather during development for well testable units as sweeping a number of small pebbles out of my way, so tripping due to trivial typos is less likely. Certainly, there might still be big rocks to encounter and I might still fall due to a large crack in the ground, but I certainly won't fall over due to small slippery pebbles.

In other words, I think TDD can be an amazing help at _implementing_ a solution after it was designed and planned and I think the ability to change code while the tests lower the probability of existing assumptions breaking unnoticed allows one to improve the existing design of a solution easily and significantly, while I also agree that designing and solving with tests only is awkward.

In case you want better examples where test driven development will fall flat on its face, take a look at the problem sets on http://uva.onlinejudge.org/ . You have 1-6 testcases in each problem, so there are enough test cases? :)

> I think this is the sort of statement that makes some of us foam at the mouth and gesticulate wildly at well-intentioned TDD proponents. TDD doesn't prove anything of the sort. It proves that the code passed your tests before you made the change and passed them after - which is a very, very different thing from proving the change you made did what you expect.

Would it have helped if the OP had said "made me more confident" instead of "proves"?

It would not have helped at all since it completely changes the meaning. 'proves' is a statement about the code, if true then we can talk about the influence of TDD on the code. 'made me more confident' is a statement about the author's state of mind - it lets us talk about the effect of TDD on the author's state of mind, which is a lot less interesting. Re-reading my code carefully or telling myself I'm awesome might make me more confident in my code changes but you rarely hear much about Careful Inspection Driven or Self-affirmation Driven Design.

I think perhaps in my quest for concision I failed to communicate. I certainly didn't mean to imply that TDD could provide certainty for the entire system.

But I have to differ with you in one key point. TDD specifically proves that the code does not pass your test before you made the change. The red bar is, in my mind, the most important part of the practice.

Given that you have a failing test, you make a change, and then you have a passing test, what more do you feel is required for proof that the change did what you expect? It's possible that "proof" is too strong a word, but I find that process gives me sufficient confidence.

(Side note: I'm honestly trying to figure out the best way to write code. I don't think TDD is the silver bullet, but I think it's a useful tool in the mechanical part of development.)

We're using 'pass' in slightly different senses, what I meant is (and this applies to any test methodology) - in the strictest sense, getting the expected outputs from a test-suite, making a change and getting the expected outputs from a test-suite proves that you are getting the expected outputs from your test-suite. It may do a lot of other things but typically proving correctness is not one of them.

Side note: I'm honestly trying to figure out the best way to write code.

I think that's key and merits more than a side note. One of the reasons just about any methodology can boast of success stories is that given a concerted, focused and rational effort to improve quality, quality tends to improve, regardless of methodology. Such efforts are quite hard to get right, especially for organizations but when they succeed, there's little empirical evidence the primary cause is the methodology.

So if TDD helps frame your own efforts to be a better programmer, by all means, TDD away. The TDD griping comes from the often overblown claims of TDD proponents that TDD improves quality. 'Honestly trying to figure out the best way to write code' is a more valuable starting point than TDD.

I think the important realization in here is: TDD is a /tool/. If we fix the *DD == tool, we can reason that a good programmer has a toolbox in his head, containing a number of such tools, that is, a number of development and driving strategies which can be selected according to the problem at hand.

This is at least what I am working on right now. I have a solid grip on TDD, and I begin to understand where TDD works and where it does not (e.g. simple datastructures are nice, and things like katamorphisms can be well-tested pointwise. Basically the "middle tier" and "inner plumbing" of an application, but complicated algorithms, IO and such are problematic.

On the other hand, I am currently investigating the refinement calculus in order to derive programs from specifications. It is true that RDD (for consistencies sake: refinement driven development) is really, really hard when you try it on big problems and IO is again an evil problem lurking in the dark, but on the other hand, refining invariants has given me some really elegant and clean recursive solutions to some really tricky problems. I have to admit, it took around an afternoon to understand the result from refining, but I was able to throw just about any input I could think of at the problem and it just worked.

Third, there is probably some sort of KDD, knowledge driven development. Knowledge driven development occurs if you see a problem and just know the solution. In that case, you can just write it down, fix a few bugs and be done (or receive really obscure wrong outputs :)). Compare Norvigs sudoku solver for some pure knowledge driven development.

I think this makes sense and at least fits my current observations and explorations in the land of improving ones code-writing.

To me, the fundamental benefit of TDD is that it proves that the code is in the state I think it's in before I make a change, and it proves that the change I made did what I expect.

I think this is the sort of statement that makes some of us foam at the mouth and gesticulate wildly at well-intentioned TDD proponents. TDD doesn't prove anything of the sort. It proves that the code passed your tests before you made the change and passed them after - which is a very, very different thing from proving the change you made did what you expect.

TDD does not prove your change did what you expect, at all. In that you are correct. However, TDD does provide a way to alert you if your change caused a side-effect you were not expecting it to. ie, passing tests does not indicate success, but failing tests alerts you to unexpected failures in edge cases you may not have considered otherwise.

That doesn't invalidate what you said at all, and in fact I agree with it, and think too many people probably equate passing tests with successful feature implementation, which is not ensured. But the fact that it does not serve the purpose you state, does not mean it does not serve any purpose at all.

I think there is some value in it, I just don't know that it is what many of the proponents try to push it as.

but failing tests alerts you to unexpected failures in edge cases: That should be "alerts you to some unexpected failures". That is, your new code could cause a side effect that is not caught by the test.

> To me, the fundamental benefit of TDD is that it proves that the code is in the state I think it's in before I make a change, and it proves that the change I made did what I expect.

That's called unit testing, and it's gets you to the same place. It's just TTD says when you must write the unit tests: before the code.

As an experienced engineer, I love unit tests for the freedom it give me later when changing things, but TDD just feels all wrong during the early phases. It prevents me from seeing the larger picture and adds "inertia" to internal APIs that maybe later turn out to be inelegant. But that's just me, others are helped greatly by TDD in their coding.

So my opposition to TDD is strictly personal. It's just not for me, but feel free to do it all you like.

How do tests written after you've changed the code prove the state of the system prior to the change? Or put another way, what technique do you use to ensure the test is exercising what you think it is?

I understand what you mean about inertia. If I'm not sure what I want an API to look like, I'll sometimes mess around with it without tests, then discard the work and rewrite with tests. I suppose post-hoc testing could be equivalent, but I find I usually write the code better the second time.

I think I pointed that out in my essay when I discussed how one limitation with TDD is that tests must fail, then change the code to pass the test. This doesn't let you add tests which are expected to pass but which are used to assess the validity of the code.

One way is make the tests fail, such as by changing the comparison from the expected value to something else, or by forcing an exception at the end of the code. Once the test fails then you know the test is being executed against the expected code, so then change it to make it pass. This is done without modifying the code. It's similar to what you would have to do if you depend on a third-party package which might be buggy. Code coverage is another way to check if the test is actually exercising the code paths you think it should. And yes, in some cases I will insert failures into the code itself, to convince myself that the test is also working, but that's only one of several possibilities.

While you make only a single iteration when developing APIs, I find that I many more than that to get the API to feel right, and that API development needs feedback based on implementation. I'll end up with a bunch of semi-tested but reasonable code, which I then test through unit tests, guided by my experience and assisted through code coverage (either manual or automated). It is much easier to make this code work right than it would be to start from scratch.

I'm also not sure that I would say "post hoc", since it really is a mixture of different types and quality of tests all throughout the development process.

Thanks for this. I'm going to start paying attention to times when I want to change the API but feel inhibited by the tests. I haven't noticed it happening in the past, but it's possible I dismissed the idea of changing the API quickly enough that it just flew past me.

When the code needs additional functionality, then yes, I'd say that TDD indicates you should create a failing test case.

But if you are adding additional test cases to prove that edge cases work correctly (and no code changes need to happen), then I don't see why you'd need to have a failing test case, and I don't know how you could argue for it.

"I don't see why you'd need to have a failing test case". It's going to depend on the case, but I've had enough times in Python where I've written (via copy & paste) a new test case and forgotten to change the name, so the new test replaces the old, that I slightly wary of trusting that new test code which seems to pass, actually did run.

On the other hand, if I have two parallel lists, one with input parameters and the other with expected results, and I add a new test case to those lists, then I'm not going to worry so much about the new test passing by accident because the likelihood of making an silent error there is quite small.

Under TDD you write tests not after you've changed code, but before you have written any code at all. Under TDD, all tests initially fail, as there is no code yet to allow the test case to succeed. You then write the code necessary to allow the tests to succeed. This is why tests in TDD are commonly referred to as a 'spec.' They serve as the actual specification for what to write next.

My take on it is that this is useful when implementing a well-known spec, like a communications protocol. It is less useful in an iterative design where the assumptions of the spec may not hold true at all later.

Writing tests before the code can have advantages. Specifically, it forces you to look at what the code's external interface is, rather than how it will be implemented. This is important, because the simpler the interface is, the less cognitive load it puts on someone using it, even if that person is yourself.

But I agree that being dogmatic about requiring all code to have tests written beforehand is silly. Being dogmatic about most things is silly.

* I love unit tests for the freedom it give me later*

I'm curious, what freedom is that? I'm not sure I've heard anyone else say that about unit-testing.

If you have through test coverage, you can cut out a sloppy-but-functional prototype and replace it with a cleaner (or carefully optimized, etc.) version, and be reasonably confident that your code will still work. It gives you more freedom to replace parts of a project, rather than getting stuck maintaining horrible legacy code.

Of course, there are a few big ifs there. Writing very thorough test coverage is not a trivial undertaking, and may not be the best use of your time. Also, this says nothing whatsover about when you write the tests.

This matches my experience (legacy code, ugh), and I watched a Google tech talk in which the main SQLite author, Dr. D. Richard Hipp, spoke at length about how SQLite's extensive automated tests (http://sqlite.org/testing.html) have given them room to change the implementation of major internal systems several times.

I'm not dismissing the practice, nor am I opposed to TDD. But your experience and mine are very different if you haven't encountered TDD zealots. In fact, as much as I hate to say it, I think zealotry is the hallmark of the TDD community, such as it is.

zealotry is the hallmark of the TDD community


it proves that the code is in the state I think it's in before I make a change, and it proves that the change I made did what I expect

This advantage is less useful the better you know the system and the better you are as a programmer. One experienced programmer I know (30+years programming) said "When I know what to test I don't need the tests." When I started programming it seemed like I spent most of my time getting the code to do what I want, but that’s morphing so I spend more time trying to understand what I need the code to do. IMO, TDD is like using a good debugger, they can be awesome tools when you have a vague idea of what's going on, but once you really understand the code base you they slowly become less useful.

PS: The guy I am talking about was one of those "old masters" you occasionally hear about. One of my favorite stories about him was: "He once spent a week cleaning up someone’s code and sent it off to testing with some misgivings, which it passed and then failed in production. Apparently the testers had learned to ignore his code as it was normally a waste of their time to look at it, and after a few years this ended up biting them is the ass." So clearly testing is useful, but you need to balance cost and benefit.

"I'm not aware of any other technique that does that."

Well, as pointed out by others, TDD doesn't actually prove that.

There are however methods that do, which fall under the general heading of formal methods or program proof. Read "A Discipline of Programming" by Dijkstra for a reasonable introduction. The techniques work, but are laborious. Modern theorem provers can help, but it still involves considerable effort. Which implies that such methods likely won't provide a return on investment unless the cost of errors in the program is extremely high (avionics, medical control).

I prefer to think of TDD as similar to working a problem backwards and forwards in elementary algebra. You may still have gotten the answer wrong, but you'll have to have made a mistake twice, so it helps your odds quite a bit.

Though I don't have any opposition to TDD as a technique, I have an issue when some developers believe that just because the code has tests it means that it works and if the code doesn't have tests it doesn't work.

Wittgenstein isn't the only one who owns a ladder, and there are many boats and even bridges one can use to cross a river. Some rivers you can even wade or swim across. Testing is not a treacherous expanse of water crossed solely by own's own efforts. But that's just a quippy response.

More seriously, TDD doesn't really have nudge #2. I showed examples where TDD-based solutions weren't thought through well enough to be declared done, and I showed that in the prime factor case that the tests aren't sufficient to be good tests for the implemented algorithm. Specifically, tests should help others verify that the code works (which is not a goal of TDD), and the lack of tests with prime factors greater than 3 is a warning flag that should have been addressed during the development iterations. That the code happens to works (for sufficiently small numbers) is not the issue - the issue is that the tests aren't good enough to remove reasonable doubt.

I absolutely agree that TDD does not guarantee well-thought-out code. (If you find something that does, let me know.)

However, for some people, I do think it does give them a nudge in the right direction. Over the years, I've been amazed at the quantity of code I've seen written by programmers who had only the vaguest notion of what the code they were writing was supposed to do. Any ritual, no matter how silly, that helps the novice (or lazy) coder to stop and think before coding is ok by me.

But, as I said, there's a long way from that to insisting on TDD as "the one true way".

I think this is a great insight. I know that I prefer test-first because I haven't been testing for very long, and if I don't test first, I don't get around to testing.

That's great, if it works for you. I certainly never discouraged my employees from using test-first, if that was what helped them write good code.

But it's a long way from there to a categorical imperative.

Going from Wittgenstein to Kant's categorical imperative - nice!

Don't mistake the instance for the class.

The class of categorical imperatives, per Wikipedia:

A categorical imperative ... denotes an absolute, unconditional requirement that asserts its authority in all circumstances, both required and justified as an end in itself.

Whereas Kant's Categorical Imperative is an instance of that class, specific to morality:

Act only according to that maxim whereby you can at the same time will that it should become a universal law.

This is a truly excellent article, and I really appreciate the tone. Calling attention to the false dichotomy between pure TDD and test-last development especially spoke to me.

In the real world, everything is a shade of grey, and the same is true with software testing. Puritanical pursuit of theories and practices is silly. TDD is useful in some instances and not in others. A skilled and experienced programmer should be able to know when it's appropriate and when it's not.

Both the fib and prime factor examples are shit. The tests are individual cases which is treating the input as a Cartesian product (a point on a line since these are single-variable functions). What you want are sets. The inputs for both are all natural numbers. However, this doesn't tell you much, there's still a lot of state to work with. So what you want are end conditions, but again that doesn't tell you much.

Tests are no substitute for reasoning and TDD is just another fad.

I'm overall sympathetic to the opinions in this article, but I think he's gone astray in the section "Complaint: TDD freezes the API too early". One of the ideas of TDD is that writing tests guides the API, rather than forcing you into an API. That is, if you write tests based on the actual business requirements, and write them first, that process will lead you to discover the best design of the API.

Note that for this to work, you have to be writing tests based on the real use requirements. Simple synthetic unit tests will not help you here, and may lead to exactly the problem the author is complaining about.

Great article. I enjoyed how well the author builds his case and argues for it rather reasonably. While I do disagree with his article on the whole, I do agree that TDD is not the end all be all. There is good TDD and there is bad TDD. TDD is a very general practice and what really defines your tests as good or bad is really your OOD skills.

One thing (really two things that work hand in hand) that I've found to augment my OOD skills so that I write better tests and better software is domain driven design and behavior driven testing. Using these techniques together has helped me to learn to ask better questions of my domain experts, create more decoupled and flexible software, as well as solve that age old how do I test the UI when it isn't already in an MVC framework (the correct answer is to treat it or portions of it as their own domains imo).

This is the kind of article that provokes thoughtful discussions by having a well thought out opinion and not just blabbering the first idea that comes to mind.

Thanks and keep it up!

In the code kata's I have been to, we use the following model to TDD:

1. If (all tests pass), Do one of the following:

  a. Write a failing test

  b. Write a passing test

  c. Refactor

  d. Start a 2 min discussion 
2. Else (If a test is failing)

   Fix the code.
I would imagine the above model to TDD would fix many of the complaints in the article.

I do agree with the general sentiment of the article, but on the other hand, you should begin writing tests before it's too late and the code is too interdependent to test without a LOT of effort.

When you write your tests first, you will naturally never end up in that situation.

Of course, when you see that you have a hard time at writing unit tests because, well, the units are too big, then you can (and should!) start refactoring, but how do you refactor with confidence when you don't have tests to see whether the refactoring went well?

Chicken and egg I guess.

"but on the other hand"? Isn't that the same hand? "My own practice is to have good, automated tests, but these don't get put into place until the cost/benefit ratio makes the tests worthwhile; which is rarely at the start of the code development and always by the end." and "The important requirements are to have good, complete automated unit tests, to develop code for testing, ..."

Why do people still write unit tests? You are stabbing in the dark, in a very nonrandom way, and you're doing all the donkey-work manually.

If you care about testing, use a quickcheck clone. Bonus: it's much easier to refactor tests when you're saying "it takes an X and a Y" instead of "it does this and then that". And if you discover a bug caused by a corner case of Y and add it to your Y simulator, then everything that takes a Y gets tested with that corner case.

good article. can give few examples myself when too much focus on code perfection led startups to dead...

just get things done, man!!

A classic example of getting TDD wrong. Happens all the time.

TDD is about design, the article is actually criticizes unit tests. TDD helps you achieve SOLID principles, which really helps in real world. Fibonaccis, primes etc. are only for demonstration purposes.

And 100% coverage is for sissies, only a TDD noob talks about it in this context since it's not sane.

>And 100% coverage is for sissies

Depends on your project and the cost of failure. I once worked on a system that would automatically instruct semi-truck drivers to move their vehicles hundreds of miles. I spend over a week building tests for transactionality/concurrency edge cases to make sure that two trucks could not be instructed to go to the same place in error. Worth it? Yup. Should you test all possible behaviors of a utility script that, if broken, can just be re-run? Probably not.

It sure depends on the project type, I was talking about projects of HN people. For my site, for example, I have broken the signup process twice so I now have a couple of tests for it, unit + watin. But I won't bother with writing tests for, say, privacy settings.

If the demonstration programs contain problems (ie, the tests aren't good enough and the implementations contain unexpected performance and overflow problems, as I discussed), then why should I expected the practices to scale up to more complicated problems?

SOLID applies to OOD, and an essay I don't plan to write is why I consider OOD to be the wrong solution for most cases. In any case, my point is that TDD adds very little to the process of developing good software.

I mentioned several times that TDD provides weak tests and is insufficient to guide the development process. With examples. I did not criticize unit tests and I said that that's what I use in my own work.

The 100% coverage reference I gave comes from Kent Beck. I'm surprised that he's considered a TDD n00b.

You are more than welcome to reply in more detail.

I couldn't be able to pass the first steps of applying Scrum where I work for years, while web was flooded with positive experiences. Than, I met with a person with firsthand experience and found the flaw in my thinking, I was trying to follow everything in the book.

Some people combines Scrum with their existing process, some set 4 months iterations while others insists on not more than a month. Some can't live with a Scrum Master who is a part time developer. By the book, they all wrong. Yet, they're successful Scrum implementers.

So here's my account of TDD, regardless of what Mr. Beck said in the book (I am actually an Alt.Net follower):

Unit tests and TDD, for me, are totally different things. TDD is a way of construction, while unit tests are for inspection. Unit tests' re-validation power is great, but less and less needed if you have a good design overall.


- TDD should not care to emphasize good test cases. It shouldn't even emphasize a complete test suite against a method. If someone want to test every possible way a function can be called, there are even automated software for doing it. It's a QA thing, not design.

- Worst case scenarios should be left to your test framework. Same as above. Row attribute in MbUnit. Range, Random, Combinatorial etc. attribute in NUnit (I'm from .Net world).

- No approach should give someone confidence that his code works all the time. Even the best QA can't catch all. But "When the bar is green, you know where you stand." means you are safe on what you test. If you test only for the correct input, you're safe for it. You'll know that missing inputs are untested. (Just a couple of unit tests, though)

- There's no 100% coverage. Since 100% coverage might also mean a <div>'s positioning. Screw it and all the UI breaks loose. Critical paths should be covered as much as possible (some 80/20 rule applies here. %20 of tests cover %80 of critical path etc.).

- When you want to change your api, do so. Write new tests, then a new api. The tests you have written are not part of your code real estate. Either feel totally ok for throwing them away, or don't use them. You will break your mental model of the old api when writing the new one without tests (we all write unit tests in our heads when coding), so what's wrong with breaking the digital representation?

- SOLID is nice, don't overuse OOD. But, what's wrong with having OCP everywhere all the time? Need OAuth? Extend your account service. Want to also publish to twitter from your app's main feed? Extend feed service.

Unit tests can be written good or bad. It depends on one's ability, testing framework in use, even the mocking framework. But in my experience TDD only improves your code, makes it more modular, more open to extension, more open to change.

I talked about TDD as a testing system and as a development system. You, like Beck, say it's only the latter, so I'll restrict myself to that aspect of it as well.

Worst case scenarios cannot be left to the test framework neither can good test cases. Leaving them out means the code is going to reflect an optimistic view of the requirement. A more pessimistic view may well entail extensive changes to the code, just like a more optimistic view may well be a lookup table if only two values are tested.

I never promoted the calling a function every possible way. In most cases that would take too long to be called a unit test. On the other hand, if there are only two possible ways then I would go ahead and do those as unit tests.

When I talked about "confidence" I referred to it as minimizing doubt. Other than a few cases of formalized testing or complete enumeration, I agree with you that you cannot be confident that the program works, if the interpretation of confident is "100% certainty." However, confidence has multiple shades of meaning, such as "I'm confident that I'll live to see 2010" when I know that that's not 100% for sure. In software the statement of confidence is more "I'm confident that the software is deployable with an acceptable risk and cost of failure." I don't think TDD leads to that level of confidence.

My position is like yours - 100% coverage is rarely worth the cost. I managed to get 99.8% coverage in a code base last year and failed to get 100% because I couldn't test two malloc failures without a day or two of additional work which clearly wasn't justifiable. My comment was the weaker statement that TDD does naturally lead to 100% coverage.

I wasn't talking about changing a developed API, my example was coming up with the API in the first place. If I develop the tests while I experiment with API and possible implementations then it's extra work with little benefit, and an obvious cost of time spent to develop those tests and modify them as the API changes.

What's wrong with OCP? As far as I can tell, what's wrong is the name. How is OCP different than "code reuse" and "reusable software libraries"?

I've just now done some shallow research on this. Meyer's original form promoted implementation inheritance, which is one of the forms of code reuse I've done. Martin's form of OCP promotes abstract interfaces, and I've grown prefer concrete interfaces. He specifically eschews public attributes and global variables, as being against OCP. I use them, but then again I also use Python, where these have different meaning than C++.

I'll also point out that Martin's form of OCP mentions "software entities" but really seems to mean "classes" based on the examples he gives. Template metaprogramming and Lisp macros seem to be two of many techniques that aren't mentioned in his discussions. For less esoteric topics, neither is Javascript code inserted into a web page during HTML template evaluation, nor dynamic code generation (eg, with LLVM), nor any place which takes an eval(). Those seem to be cases which can be quite useful and yet are counter to OCP.

As for your last paragraph, you've said that unit tests are totally different than TDD, but here you seem to say they are part of the same concept. My point is that good software development styles give the consequences you mention there. While TDD is one method people have used to learn good development technique, it isn't the only. Other development techniques must be learned, and these are not part of TDD as it's taught. My assertion is that TDD adds little to those techniques and as such isn't at best a weak concept.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact