I've never seen untested code that wasn't crap, or didn't require just as much work (if not more) to ferret out all the bugs (including watching everyone like a hawk).
A huge percentage of the code you use every day doesn't have tests - Linux kernel? No tests - TCP/IP stack? No tests - Windows Kernel (up till WinXP anyway) ? No tests - Android? No tests - Quake engine ? No tests
Don't make stupidly broad statements like "I've never seen untested code that wasn't crap". Well unless by "untested" you mean "code that was never even ran or QA'd a little". I'm pretty sure you mean "code that didn't go through TDD" though, in which case - you are very, very wrong.
Even if we make your assumption that he's specifically referring to TDD only, just because good code that didn't go through the more rigorous definitions of TDD exists doesn't mean he's seen it. And plenty of crap code ships.
I'd argue Linux has significantly more than 'no' tests: http://stackoverflow.com/questions/3177338/how-is-linux-kern... -- it's not TDD, and it's not centralized, but then again, neither is Linux development. And while you might convince me that pre-XP kernels went without unit testing, I'd point out we hardly call them bastions of stability and security. Try certifying a 360 or ModernUI title under the Microsoft banner and see if they'll let you get away without writing a single test. I'd wager anyone who successfully accomplishes this has spent far more effort arguing why it should be allowed than writing some tests would take.
And, given the sheer girth of their security processes (patching, QA certification, tracking) and toolset (through virtualization, fuzzing, unit testing, CI build servers), it would take much more effort to convince me that they do absolutely no unit testing whatsoever on their kernel in this day and age. Far more than you could reasonably give: I suspect such convincing would require breaking several NDAs and committing several felonies.
I'm curious as to whether you've actually worked on the Linux kernel, or the UNIX descendants, or TCP/IP stacks.
I have. The only reason they work at all is the enormous number of man years spent banging on the poorly tested code after its written. The number of regressions that pop up in core subsystems of the kernel are staggering. VM bugs still happen in commercially shipping systems like OS X because of small changes to extremely fragile and untested code.
Linux isn't a paragon of code quality, and neither are most Network stacks. If anything, you're proving my point -- code written in that methodology has required enourmous amounts of work to stabilize and prevent/fix regressions going forward.
Case in point; a few years ago, I found a trivial signed/unsigned comparison bug but in FreeBSD device node cloning that was triggered by adding/removing devices in a certain order; this triggered a kernel panic, and took teo days to track down. The most trivial of unit tests would have caught this bug immediately, but instead it sat there languishing until someone wrote some network interface cloning code (me) that happened to trigger this bug faaar down below the section of the kernel I was working in.
This kind of thing happens all the time in the projects you list, and its incredibly costly compared to using testing to not ship bugs in the first place.
That's the statement I was replying to - and yes, code with tests is going to be better. But to call all code without tests crap is directly saying that the Linux code is crap, that the BSD code is crap, tcp/ip code is crap, etc. I disagree completely - the code is awesome and has built billion dollar industries and has made most peoples lives better. Could it be improved with more tests? Sure. Everything can be improved. Calling it 'crap' however is insane.
I, personally, would be more than happy to have produced the 'crap' BSD code that has propelled Apple into one of the most valuable companies today.
> I disagree completely - the code is awesome and has built billion dollar industries and has made most peoples lives better.
It shipped. That doesn't prove your point. Code that is expensive to produce and ships is better than no code at all. That doesn't mean that this is the best way to produce code.
What exactly do you think is awesome about the code?
> I, personally, would be more than happy to have produced the 'crap' BSD code that has propelled Apple into one of the most valuable companies today.
I did produce some of that code.
Nothing that you're saying justifies why producing worse code, less efficiently, is better than producing good code, more efficiently. Your position assumes a false dichotomy where the choice is between shipping or not shipping.
The truth of the matter is that the legacy approach to software development used in those industries has more to do with the historical ignorance of better practices and the resulting modern cultural and technical constraints. In the era that most of that code was originally architected, it was also normal to write C programs with all globals, non-reentrant, and non-thread-safe. Are you going to claim that this is also the best way to write code, just because it produced some mainstays of our industry?
Nothing that you're saying justifies why producing worse code, less efficiently, is better than producing good code, more efficiently. Your position assumes a false dichotomy where the choice is between shipping or not shipping.
Well, you say it's a false dichotomy, but if TDD really does reliably produce better code and more efficiently than a non-TDD approach, how come hardly any of the most successful software projects seem to be built using TDD? It's been around a long time now, with no shortage of vocal advocates. If it's as good as they claim, why aren't there large numbers of successful TDD projects to cite as examples every time this discussion comes up?
The fact that every time I have ever had this discussion, the person taking your position comes back with a question like that instead of just listing counterexamples.
Can you name a few major projects, preferably on the same kind of scale as the examples that have been mentioned in other posts in this discussion, that were built using TDD? Not just projects that use some form of automated testing, or projects that use unit tests, or projects that use a test-first approach to unit tests, or anything else related to but not implying TDD, but projects where they really follow the TDD process?
The development processes for major open source projects tend to be public by their nature, and there are also plenty of people who have written about their experiences working for major software companies on well-known closed source projects, so telling a few TDD success stories shouldn't be a difficult challenge at all.
> Can you name a few major projects, preferably on the same kind of scale as the examples that have been mentioned in other posts in this discussion, that were built using TDD?
Operating systems? No. Sorry. Operating systems, as a rule, predate modern practices by some 1-4 decades. Making a kernel like Linux, Darwin, or FreeBSD testable would be a task somewhere between "massive headache" and "really massive headache". I've done some in-kernel unit testing, but only on our own driver subsystems that could be abstracted from the difficult-to-test underlying monolithic code base.
Outside of kernels/operating systems, just Google. Unit/automated testing is prevalent in just about all modern OSS software ecosystems.
My apologies if I've misinterpreted your posts. I thought we were talking specifically about TDD, since that seemed to be the subject of the original article, but from the examples you gave, you seem to be talking about automated unit testing more generally now. In that case, I would certainly agree that there are successful projects using the approach.
However, I would also say that if you've never seen code untested in that way that wasn't crap then you're not looking hard enough. Some of the most impressively bug-free projects ever written used very different development processes with no unit testing at all. Space shuttle software, which is obviously about as mission critical as you can get, went in a very different direction[1]. Donald Knuth, the man who wrote TeX among other things, isn't much of a fan either[2].
There are also other means to assure high quality of code. Testing is just one of many tools. And even 100% coverage does not guarantee your code still isn't crap. Your tests might be crap as well. By testing you can only prove there is a bug but you can't prove there isn't. You can only hope that if you tested the right things, it is likely your code works fine.
For example static code analysis is sometimes superior to testing, because it can prove for absence of some wide classes of bugs.
And in code reviews we often find subtle bugs that would be extremely hard to write tests for (e.g. concurrency related bugs).
You can also decrease bug rates by writing clean, understandable code. Which is often related to hiring a few great programmers instead of a bunch of cheap ones.
