In the earlier days of the ruby community, I feel TDD was seen as gospel. And if you dared say that TDD wasn't the way (which I always felt), you'd feel like you were ostracized (update: just rewording to say, you'd worry that it would hurt you in a job search, not that people were mean to you). So I never spoke up. I feel like I was in the TDD-bad closet.
I absolutely think _tests_ are useful, but have never found any advantages to test-DRIVEN-development (test-first).
But part of that is probably my style of problem solving. I consider it similar to sketching and doodling with code until a solution that "feels right" emerges. TDD severely slows that down, in my experience, with little benefit.
What I've found works really well is independently writing tests afterwards to really test your assumptions.
Agreed, but I would add there are specific scenarios where doing the test first really helps - when you're building a routine complex enough that you can't think upfront how you'll do it, like a complex calculation for example. In that case, at least thinking about the tests first and making one by one pass helps you breaking down the problem into smaller, easier parts.
This is the best situation for a TDD workflow, and the only situation I myself use TDD in.
If you have complex calculations or data transformations or you know well the expected input and output of what you're trying to do TDD actually speeds you up by allowing you to iterate faster on your solution.
If you're writing glue code between a number of other application components (90% of development in web backends) then TDD loses a lot of benefits.
I think this is the result of code coverage metrics that are often advocated. When you need some arbitrary level of coverage you have to write tests for glue code.
It may be different in dynamic languages, but in statically compiled ones the compiler does all the testing required for glue code.
> In statically compiled ones the compiler does all the testing required for glue code.
It cannot, for example, tell if you have glued things together in the wrong order.
...but a test of the glue code alone will not necessarily tell you that, and you will waste a lot of time mocking the lower-level functions to perform a standalone test on it. It is better to test an assembly as a unit, which will test the glue code (and give you a coverage count for it.) In many cases, it will also test your low-level code without you having to write so many mocks and harnesses.
I think you can find TDD zealots who insist that you have to write tests for every little piece in isolation, but this takes a lot of time, especially if you have to write a lot of mocks to do it (and testing with mocks is error-prone; the mocks may conform to your invalid assumptions.) There is never enough time to test everything, so excessive unit testing means giving up some potentially more informative integration tests. As in most aspects of development, good judgment matters more than strict obeisance to the rules.
I was quite comfortable with refactoring for a very long time before I started doing tests, and I think that colors my approach to the problem space. I write code until it misbehaves the first time, and then I start writing tests, comfortable in the knowledge that in another half hour this code will be unit testable and look better than what everyone else is turning in, and that I'll spend 20 minutes at the end cleaning it up even more.
But I am not everybody, and trying to set team policy on my comfort levels is quite likely a bit dangerous.
I've paired with a few smart people this month and was horrified by how they refactor. Most trust their hands more than the tools. They do a LOT of transcription instead of copy and paste, and don't use the IDE's refactoring tools at all. That's not just inviting human error to the table, that's giving it the seat of honor.
For most processes there are ways to do it that are worse than nothing at all, and unfortunately many of us seem drawn to them almost instinctively. I try not to think about it because it just makes me angry that the distance between what we collectively know and what we individually do is so wide.
> They do a LOT of transcription instead of copy and paste
Transcription makes you think about what you're typing in and avoids the whole class of errors that arise from copy-and-paste. Transcribing code rather than pasting it is not a red flag.
Transcription can lead to the same conditional in two blocks of similar code instead of the two intended. It can lead to fencepost errors due to changing or inverting an inequality. And in some languages it can lead to code that looks up data that doesn't exist (misspell a property name outside the happy path).
The big problem is that you're chewing up short term memory slots that used to hold the next thing you planned to do. For me I generally use only a couple slots for a single refactor because it's all muscle memory.
Additionally, it just takes a hell of a long time, which means you go from virtually no chance of interruption to a measurable one.
When I was young I had an exceptional short term memory and could pull off some crazy big refactor without introducing errors. But now it's normal at best, and also I rely far less on those sorts of feats, in part because I put a higher value on commit history readability.
> Transcription can lead to the same conditional in two blocks of similar code instead of the two intended. It can lead to fencepost errors due to changing or inverting an inequality. And in some languages it can lead to code that looks up data that doesn't exist (misspell a property name outside the happy path).
Funnily enough, all of those are also problems with copy-and-paste. But they're much more likely to occur through copy-and-paste, because it's easy to do that without reviewing the code.
I have the same issue you have. I luckily work in a language with great refactoring tools, but it's like pulling teeth to get people to use them rather than cut and paste.
In this case a bunch of former Java devs touching Java for the first time in a while. There could be something to that.
Could be they've forgotten how, but this isn't the first experience like this. I am doing fine with the tools even though they don't do everything when the language is not statically typed).
Java isn't perfect and when people also treat their IDE as a glorified text editor I think it is no wonder they dislike it.
(If you go with Java, embrace it. This is probably the thing I disliked most about the way programming was taught in school: they used Java, but without the tooling that makes it awesome. Supposedly so we should learn it well.
If anyone here teaches Java: by all mean have your students use javac in the first assignmemt but teach them maven the very next thing you do. Learn them to use an IDE -
it's not harmful as long as you don't accept autogenerated code.)
Nothing. There is nothing wrong with code generation in a production setting with "grown up" engineers who uses version control and knows the pitfalls.
The part in parenthesis is just a rant about education and how teachers use notepad and javac to teach Java.
So my point is the only thing students shouldn't use in an IDE is code generation.
But going from javac to Maven in CS1 would be insanely difficult; there are many prerequisites before one can use Maven. It may only work in top-tier schools.
I too fell in love with NCrunch during my few months with .Net.
It's possibly the single tool[0] I'll miss from the .Net world when I program in Java.
(For anyone who might be wondering why not Resharper that is because that functionality is provided out-of-the-box with all major Java IDEs. On .Net however, Resharper seems to be almost a must-have for many.)
[0]: I'll be missing Asp.Net MVC, parts of Entity Framework and parts of the C# language too.
I think TDD helps or doesn't help depending on the circumstances. The problem is the people who claim there's One True Way.
For instance, if you were working on a big app and there was a bug, writing a failing test for that bug and then looking into the fix is very helpful. But if you're starting a new project from scratch with loose requirements, or worse yet, building a prototype, starting with tests would be a waste of time at best.
> if you were working on a big app and there was a bug, writing a failing test for that bug and then looking into the fix is very helpful.
Well, first of all, that's not TDD. TDD is Test-Driven Development. In your scenario, you wrote the test after the code in question. In TDD, the whole point is to write the test before the code in question.
So the value in your scenario is only insofar as the test is a useful way to quickly exercise the code in question. For example, if we have some sort of customer-facing website and there is a bug in how they save their user profile, it will be easier to test fixes to the problem if we don't have to go through the full flow of logging into the site, navigating to the profile, making edits, and clicking save all the time. But having an XUnit-style framework for running that test is not necessary for that.
>> if you were working on a big app and there was a bug, writing a failing test for that bug and then looking into the fix is very helpful.
> Well, first of all, that's not TDD. TDD is Test-Driven Development. In your scenario, you wrote the test after the code in question. In TDD, the whole point is to write the test before the code in question
It's a regression test, which is definitely TDD. The code in question is the fix for the bug, the test verifies that the bug is fixed and will remain fixed.
Well, that's exactly this case. You would write a test to replicate/prove the bug and then - afterwards write the code to fix the bug, no? So you would write tests first.
This definition of TDD either supposes that it is possible to write software with zero bugs using TDD, or that your project ceases to be TDD as soon as a bug is discovered, or any other change is required. I propose this definition is not very useful.
If you are writing something from scratch yes. But if you inherited a codebase written by somebody else who didn't to TDD so there are no tests, you can still use TDD for refactoring.
So any new changes/bugfixes you will make to this legacy codebase, first you will write a test and then make change. That's pretty standard. Many legacy codebases will have no tests and people who wrote the code will have left long time ago.
It's adopting test driven development during maintenance of an existing system. It's exactly TDD (identify need, write failing test, code to pass test.) TDD is a discipline that can be adopted at any time, not an SDLC that can only sensibly be adopted at the initiation of a project.
It's TDD. Just because the whole app wasn't developed with TDD doesn't mean you can't test drive a bug fix.
There's a bug, you write a test, "this bug shouldn't exist", that test currently fails, then you write just enough code to fix the test, then refactor. That's TDD.
Exactly what I was thinking. The test ends up being written firs this case, before you write/edit the code to fix the bug. So it falls under strict definition of TDD.
You and I have very different opinions on the purpose of TDD. You're presuming it's a design technique - many do. For me, TDD is all about validating a design in a way that gives me faster and better feedback than if I were just coding. I design the code before I write the first test. In fact, my test is derived from the design.
First I sketch a rough implementation design on a napkin or whiteboard. I don't worry about making it perfect, just good enough to create an initial mental model, and perhaps discuss it with someone.
Then, I write a test to express how I would like to interact with a small area of an implementation of this design, typically by calling a function or method and expressing a desired outcome.
I use the feedback from the test to drive out some simple code that resembles the design I sketched. If at any point this becomes hard, I take that as a clue that my napkin design can be improved, so I go back and tweak it.
Then I start over again. This is a quick cycle, typically 10 minutes or so. I get constant feedback on my design as well as my implementation.
I don't design code with TDD. I design code exactly the same way people who don't to TDD do it. Through intuition and experience. This is a skill, not a process. I only write tests to validate my design (not my implementation). Of course, because they are tests, they automatically validate my code as well, but that is not why I am doing it. If I only cared about testing, I might as well write them last.
More often than not my designs are imperfect, but by doing TDD I always improve them significantly, with relatively little time and effort.
It's taken me 15 years to get decent at it, and I'm still learning new techniques and abandoning others. There are many tricks to be learned. I usually only TDD when I care about the design. Quite often I don't, and then I just hack. I'm simply making a choice between going very fast now or going reasonably fast in a year. I haven't found a way to do both yet.
Exactly. I worked at a place where the majority of dev teams had little to no automated tests. Getting them to write tests was near impossible due to it 'taking too much time away from features and bug fixes'.
One team adopted TDD and forced themselves to do it for a year. They admitted it slowed their overall dev-cycle down a ton, but they ended up learning a lot about how to build a well designed API. I think more internal documentation about good design emerged from that then them evangelizing TDD.
Anecdotally, the cleanest, most maintainable code I have ever seen at work was the result of a TDD-mandatory project.
Equally anecdotally, that was the most efficient project management I have ever seen at my workplace.
Economically, our branch office was shuttered, and all employees laid off, just a few months after we delivered it, for almost entirely unrelated reasons. As it turns out, the only reason we did the TDD project at all is because our primary contract had not been renewed, and it was a lower-value contract. After it was done, we didn't have enough work left to do. But more importantly, the main office didn't have enough work to do, so they slurped up our dregs, kicked us to the curb, and sold themselves to a bigger company.
Perhaps the lead dev saw the writing on the wall, and wanted to make a showpiece to boost everyone's resumes. Perhaps he foresaw that one of the several companies we might end up with after the diaspora might get maintenance responsibilities, and that person shouldn't hate their former team for writing crappy, unmaintainable code. Doesn't matter. Did it once, and it turned out great.
I still get nostalgic for it sometimes, when I am working in the sloppy mess I am currently assigned to.
Yes a lot of people kept parroting TDD (as most people parrot the latest fad without thinking for 3 seconds if they should actually do that)
> I absolutely think _tests_ are useful, but have never found any advantages to test-DRIVEN-development (test-first).
Seconded. Especially the crap about "make the test fail first" (making the test fail is useful, especially if you're unfamiliar with some aspect of it), but it should not be an obligation
> Especially the crap about "make the test fail first" (making the test fail is useful, especially if you're unfamiliar with some aspect of it), but it should not be an obligation
Sometimes I write code to fix a bug, then a test the tests my bugfix. Test passes. I'm done right? Well, I make sure my test fails without the bugfix. About half the time, the test still passes. ie I wasn't testing what I thought I was testing. That's why you make the test fail.
Yes, that's why I say "making the test fail is useful" ;)
And yes, I've seen this happen as well, either you're not testing the bug-causing issue or you're using a "Magical Mock Library" where x.does_thing() just returns true and you thought it did the thing but it didn't.
Like the author in the original article, I used to be all about TDD. Now, I like to tell people that "I'm apostate in the ways of TDD". I specifically use the language of religion because I think strict adherence to TDD is itself a religion.
I somewhat disagree with the statement he makes, "we know that most developers do not have great design/refactoring skills". I've certainly worked at places that all but ordered me to never refactor code, and I suspect my experience is very far from unique. They thought "refactoring" was a made-up word that programmers used to cover up dicking around and wasting time. From a non-programmer's perspective, all they see is the programmer spending several hours with the end result being that nothing has visibly changed. Mind you, no visible changes is the ideal case for refactoring, so you have no easy means to explain why that is wrong.
He says that TDD only works if the programmer is good, but I think it's more the case that test-first only works in cases where we have very well-defined problems. For these TDD trainers that he's talking about, they've gone through the same problem set over and over again. For an experienced programmer, if you're encountering the same sorts of problems, you know what the pitfalls are and can look ahead, even if you aren't violating YAGNI, you can still do yourself some favors.
To put it to a specific metric, if you have a specification from which you're working, you basically have an answer sheet for your code. You can then write code that automatically verifies that your code matches that answer sheet. The more like "having a spec" your problem is, the more likely TDD will work very well for you.
But I don't do that kind of work very frequently. Most of my work is extremely experimental, fluid, and in a lot of ways, its behavior is arbitrary. Does a padding of 5px or 10px look better? Should the surface of a polygon be more or less smooth? Should running over the ground be 3m/s or 5m/s? Should the user have two hand tools that are identical, or should they be different tools? A lot of it has to be visually verified, because it would otherwise require a computer vision algorithm of some kind to know that the code I wrote to go from text to screen pixels worked correctly.
It's much more design than development, just that design is expressed in code rather than in Photoshop PSDs.
As a result, I focus much more on REPLs, saved REPL sessions, demo code, and making the iteration time between code change and test run as narrow as possible. What I think TDD provides (poorly) for people in this situation is this REPL-like behavior, in languages that don't particularly provide a good REPL (let us consider RoR a separate "language" from Ruby, in this case, as the expectations tend to be completely separate).
So if I'm writing an implementation of a rope data structure as part of a syntax highlighting text editor that renders into a WebGL texture, yeah, I have tests for that. Ropes and text editors are well-understood data structures, there's lots of information available about them, and there are right and wrong answers to what operations each should have and how they should behave. But that's not where the work ends. I still need to figure out how to let people interact with that text editor, be it with touch gestures or motion controllers or a keyboard or a gamepad or what have you, and that has no easy-mode.
I'm pretty much in agreement with everything you wrote here, but I want reply to one thing you mentioned...
You somewhat disagreed with, "we know that most developers do not have great design/refactoring skills", because developers may have little chance to refactor. Totally on board with this; lack of understanding of technical debt and how it impacts a team's achievement is one of the biggest problems with software management.
But my point - which I clearly didn't make well enough - was that refactoring and design skills are really the same thing, and I've seen enough modified code to understand what the design skills of most developers are.
They thought "refactoring" was a made-up word that programmers used to cover up dicking around and wasting time. From a non-programmer's perspective, all they see is the programmer spending several hours with the end result being that nothing has visibly changed.
The thing is, many refactoring efforts make code worse, not better. Refuctoring is something I've observed many times over, by people who consider themselves quite skilled at the art. The end result of their efforts is that the code is fancier or trickier, not more robust or easier to understand.
So, from a manager's perspective, how are you to discern between a programmer whose refactoring efforts save time down the road, and one whose efforts lose time down the road. Of course, normally you have technical leadership that can make this distinction and makes sure those people get a cold hard reality check, but many organizations discourage technical people from gaining enough power that they can stop other technical people in their tracks before they mess things up.
Even in the honest case, I think people go about it wrong. Refactoring as a separate step is going to get a lot of push-back from a lot of different places. But there frequently is a need to change the structure of existing code to make room for new features (especially if you're on a team having to deal with "refuctoring").
Just put it in the estimate for that new feature. When your boss starts to needle you about why such a "simple" feature is taking so long, that is when you throw the bad code under the bus. You get to demonstrate a real need, rather than just a theoretical one.
The older I get, the more I believe "Big Bang" code is a bad idea, no matter what the bang actually entails. Gotta make a branch to make a feature? Gotta rewrite a large part of the underpinnings of the app to make things able to move forward? Please don't, please develop it in parallel in the mainline.
In a way, refactoring on its own violates TDD. You are supposed to write failing tests first, then write code that passes the test. Refactoring as it is described by the most zealous of TDD advocates is just a hand-waiving over writing code without testing. If we're being strict about it, write a test that proves the code needs to be changed, then you can change the code. If we're not being strict about it, what is even the point?
Regression is some of the biggest harm that can come to users, because it violates their ingrained expectations. Greenfield defects might prevent the adoption of a new feature, in which case the status quo is maintained, but regression is forward progress that is physically lost.
Did you intend to say "Refuctoring"? It could be a typo or it could be clever wordplay. And it leads to two drastically different interpretations of what you're saying :)
I think I'm going to start using that one. Especially on ports of legacy software from one, crappy language to another crappy language. I'd especially guess that the COBOL to OOP COBOL translations involve a lot of refuctoring. I just gotta remember to give Joeri credit for coming up with it.
I was not aware that docstrings were controversial in Python.
Out of inspiration from Python docstrings, I have been occasionally working on a JavaScript structured documentation format that I call Pliny: https://github.com/capnmidnight/pliny
It's all about creating a database of documentation that can be rendered to arbitrary formats. The documentation pages on the website for my big project are all completely generated by Pliny.
I made it executable code rather than comments because I could iterate faster on the idea if there were a simple syntax to construct the database structure at the same time as writing the documentation, as well as by including the right setup script, I could have the database live-queryable in the REPL without any additional processing step.
Unfortunately, there's nothing to verify that the documentation matches the code. There is a lot of type information there that would be super useful to exploit to statically analyze the code, but that'd be a big project to add that. It'd almost be easier to convert all of the code to TypeScript and recreate Pliny to extend a generated .d.ts typings file, so the doc gen could use just the typings file as the database.
My experience is that whether one is writing tests first or not, the code that is being written does get tested one way or the other.
In other words, people (for example web-devs) who don't write tests first tend to "test" their code by clicking around in the browser to see whether they get the desired effect.
Writing tests, running them and then iteratively updating/fixing the code is IMO so much more efficient than manually clicking around to test one's code.
You will miss items if you don't visually inspect even in a ttd application. Tdd is a net to catch some expected bugs but should not be the only tests.
ATDD or design-driven development starts with integration tests first, and can be a great thing if written by the PO, as you can pretty much skip acceptance testing.
It addresses a different concern than unit testing, however.
Maybe the title should be: TDD did not live up to my expectations?
I too, like the author, have been practicing TDD for > 10 years. Test, implement, refactor, test... that's the cycle. If you follow that workflow I've never seen it do anything to a code base other than improve it. If you fail on the refactor step, as the author mentions, you're not getting the full benefit of TDD and may, in fact, be shooting yourself in the foot.
I've read studies that have demonstrated that whether you test first or last doesn't really have a huge impact on productivity.
However it does seem to have an impact on design. Testing first forces you to think about your desired outcomes and design your implementation towards them. If you think clearly about your problem, invariants, and APIs then you will guide yourself towards a decent system.
The only failing I've seen with TDD is that all too often we use it as a specification... and a test suite is woefully incomplete as a specification language. A sound type system, static analysis, or at the very least, property-based testing fill gaps here.
But for me, TDD, is just the state of the art. I've yet to see someone suggest a better process or practice that alleviates their concerns with TDD.
> Testing first forces you to think about your desired outcomes and design your implementation towards them. If you think clearly about your problem, invariants, and APIs then you will guide yourself towards a decent system.
The way I put it: "if I don't know the domain and range of my function, I shouldn't be writing code yet, I should be investigating the problem."
TDD is for unit tests, and unit tests best test functional, stateless code--the functional, central logic of your application is what you should be specifying and writing unit tests for, not all the imperative stuff wrapping it for IO and failure catching and the like (Gary Bernhardt[1] calls it scar tissue).
If you don't know the domain/range of your function, and you are forced to write tests in advance, then you will end up specifying a domain/range. The worry here is that you pick a domain/range that is really inefficient, and you write 'bad' code to support it, and you can't refactor it out without more overhead later.
Now, this is a minor issue. It's probably more worth it to build inefficient interfaces now and try to fix them later than to never build anything because you don't know what the optimal design is. I can't say if TTD gives a real advantage there, but when designing-while-coding-with-no-tests, you end up playing with the interfaces, changing them regularly, and learning something about why they are the way they are. Which is possibly a better process when you don't have any relevant design experience? I really don't know...
I think it probably depends on what your definition of "unit" is. As I stress functional units that encapsulate logic and data model but don't permute state, my units are probably significantly larger (and simpler) than those of people writing more imperative/traditional-OO units (where something under TDD may encapsulate many units due to mounting complexity/complication and so require further decomposition).
"If you think clearly about your problem, invariants, and APIs then you will guide yourself towards a decent system."
This is the most beneficial part of TDD to me. Most devs will deny this, but their biggest issue with TDD is not wanting to take the time to think clearly about a problem. To me that seems a bit sloppy and lazy. We're supposed to be professionals. From some of the comments I've read, it also seems that devs are trying to "test drive" or write unit tests for the wrong things. As I've gained experience as a software developer, I actually look for ways to write less code. The most effective way for me to do that, is to actually think about the problem before I write a single line of code. If I have a clear idea of what it is I'm trying to build, it's very easy for me to write a test first. If I can't identify a way to write a failing test, it's an indication to me that I really don't understand the problem well enough to deliver a dependable solution.
"But for me, TDD, is just the state of the art. I've yet to see someone suggest a better process or practice that alleviates their concerns with TDD."
Design by Contract specifying key properties at interfaces with tests for stuff not easy to specify was state of the art in the 90's. In the 2000's, there's tools to automatically generate tests from such properties or apply formal proofs to code if it's simple enough. I don't see TDD as either strongest method or state of the art. DbC, assertion/property-driven testing, some fuzzing, and proving /model-checking for most important components collectively seem state-of-the-art for realistic software. Most of which is just automated versions of manual methods invented in the 50's-60's. Most being 60's with fuzzing going back to 50's.
One from later time I like since it consistently got low defect rate was Cleanroom. Here's a good explanation of it.
> I don't see TDD as either strongest method or state of the art.
It's not the strongest method. But as far as the SWEBOK[0] is concerned it is state of the art.
The most effective teams I've worked with layer their testing strategies and use multiple approaches to verification depending on the needs of the project. I love quickcheck. I even pull out formal methods when the problem domain demands it or I need to check my thinking.
On the scale of "sketch on a napkin" to "verified and signed-off-by-master-engineer blueprint" I'd say a unit test suite, as far as it is a specification, is somewhere between napkin and the middle. If you use a soundly typed language AND a unit test suite using property-based assertions you're somewhere in the middle; building houses.
However I do see it as a warning sign if a team actively avoids TDD and tries to ship the first thing that works. If I was an actuary that project would represent a high level of risk and I'd probably raise their insurance rates to match that risk depending on the work they're trying to do.
> I've yet to see someone suggest a better process or practice that alleviates their concerns with TDD.
I think that's part of the point. If you want to follow a process like clockwork, test-implement-refactor-test, then TDD might be the best we have.
But what if not having any process is better than even the best formalised process? That's certainly my personal experience: my haphazard, process-free way of hacking delivers solid results very fast.
> Maybe the title should be: TDD did not live up to my expectations?
That is the title. If you are arguing that explicitly specifying "my expectations" would significantly alter its meaning, I disagree. Otherwise, whose expectations were we to assume TDD did not meet?
I don't use them on prototypes because I will throw away that code. But IMHO TDD is at it's best when your design is highly likely to change. The only protection you have against a moving target is your test suite. The requirements or our understanding of the requirements change so we change the code. In large code bases we can only be certain that the code change doesn't break anything if we have good tests. TDD is the only realistic approach to achieve 100% test coverage. You can certainly write good tests to cover your code at 100% but I've never seen it done by anyone consistently without TDD.
> Or other projects where the design is highly likely to change or be discarded?
I don't often just start writing code without thinking about the design of the system or module I'm working on. Unless I'm writing a one-off shell script or some simple elisp glue code... I write the tests first.
I've also never found TDD to really be very beneficial except for all but the most trivial utility libraries.
Most of the time, I have an idea of where I want to go, but not necessarily exactly what my interface will look like. Writing tests beforehand seems to never work our since nearly always, then will be some requirement or change that I decide to make that'd necessitate re-writing the test anyway, so why write it to begin with?
The extent of my tests beforehand these days (if I write any before code) are generally in the form of (in jasmine.js terms)
it('should behave this particular way', function() {
fail();
});
Basically serving as a glorified checklist of thoughts I had beforehand, but that's no more beneficial to me than just whiteboarding it or a piece of paper.
That said, all of my projects eventually contain unit tests and if necessary integration tests, I just never try to write them beforehand.
That's exactly how I've always done it, and it works beautifully for me.
Put "test stubs" out there that define what I'm testing in "human" terms, leave the stub either skipped or failing.
I've also found that if I wait a day or 2 after writing the implementation to write the tests, they catch much more.
If you write the code to match the tests, or you write the tests right after the code the chance that both the test and the code are wrong is much higher in my experience than if you wait a bit and come back to write the tests with fresh eyes.
> necessitate re-writing the test anyway, so why write it to begin with
Because now you have an unambiguous record of the conscious decision to change your interface, because the test demonstrates the correct change of that interface.
I don't write tests first for everything I do, but I try very hard to when I'm writing code that other people may read for this exact reason. Otherwise they have to divine my intent through less significant means.
> the test demonstrates the correct change of that interface.
No, they don't. Tests never prove correctness. They prove that assumptions are met. Whether or not those assumptions are correct is unprovable.
The problem is, there are changes to interface that could make the entire approach of the existing tests completely meaningless. If all you're doing is changing the name of a method, that can be practically automated away in most cases. But if you're changing fundamental behavior, like "we no longer add up a bunch of numbers and show them to the user here, now we print out a bunch of labels", then your leftover, failing tests are completely useless.
I've seen far too many cases in-the-wild where people either just deleted those tests, or faked-it-till-they-maked-it to force the tests to pass, without any thought putting into whether or not the tests prove anything important.
If there are changes to an interface that make irrelevant those tests, then you change those tests (and, if you're semantically versioning like you should be, you bump the major version number). I don't really get what you're objecting to. Implicit in doing TDD effectively is adopting the (light) burden of refactoring.
Yeah, people will delete tests because they refuse to refactor them. They're bad programmers. Don't work with them?
No article about TDD, particularly one that shouts out to the respected Ron Jeffries http://ronjeffries.com/, is complete without mentioning the TDD Sudoku Fiasco :)
Playing devil's advocate here: I'd say that algorithm design (Which is what a sudoku solver is) is a bad fit for TDD.
However, I'd argue that most problems in commercial coding are of a more engineering/architectural type of nature. Here, outside-in TDD has it's place.
Outside-in TDD can help you tackle problems that seem enormous, and slowly but steadily break them into smaller components.
I find TDD useless as a way of poking at problems I don't actually know how to solve (as Jeffries found with Sudoku), in the way and slowing down when writing large new systems, and excellent when bug-fixing maintenance. Test Driven Debugging!
True, but again, most problems I've been given are solved problems I just need to glue the parts together.
I do not design a sorting algorithm from scratch (without consulting any of the literature on writing sort) - I could but the result would probably be a variation of bubble sort. However I stand on the shoulders of giants. That means I already know bucket sort, quick sort, insertion sort, bogo sort (and several more that were covered in CS203). I have never encountered a situation in the real world where calling whatever sort is built into my language is not good enough.
Most problems are variations of take some data, do a little math, and [save for latter use, present to the user, or change some physical control]
In a similar vein, I've found it super-helpful when refactoring. It gives you extra confidence that your refactoring isn't gradually breaking important stuff.
Where's the design? (To me TDD was always championed to help specifically with Design as much as the vague Development. When using the vague Development too loosely you run into a recursive problem because writing tests is itself development so do you use TDD for your tests?)
Depending on how seriously you take the step of writing a unit test that reproduces the bug, you may be forced to refactor quite a lot of code to get that buggy section under test. Writing the test first can help guide your refactoring to avoid mocking the runtime world. But you're not designing anything, and it was all driven by the bug report.
If you are that strict on your definition, yes you cannot TDD when fixing a bug. However all TDD advocates will tell you to write the test for the bug before fixing it. If the code was designed with TDD you will not have to refactor the code to write the test (though the test might be much bigger than it would need to be if the code was designed different). You seem to be describing working with legacy code though, not which is a different situation from TDD.
Test-driven development refers specifically to building tests before implementations such that the impressions of the tests (which are communicating the external intent of whatever you're working on) are not colored by the particulars of an implementation. The tests you are writing are necessarily colored by the implementation you're fixing. This isn't bad (though I'd say it's less effective for most programming tasks than TDD), but it isn't TDD.
I don't find most of the Uncle Bob writes to be clever, and especially when he writes about TDD.
He's just too dogmatic to be credible to anyone working in the industry (and make no mistake, Uncle Bob knows absolutely nothing of the software industry in the 21st century).
His writings are only here to promote himself, his books and his consultancy. That's it.
I weakly agree with this statement. While I think there is merit to TDD under some circumstances (that others have described quite elegantly and succinctly elsewhere in the this thread), my main takeaway from the Clean Coder was that if I ever got super burned out I should just start a consultancy.
[addition]: I also find it weird how the Clean Coder seemed to encourage burnout by prescribing that you use your off-hours to hone your craft. While I agree that software engineering should be treated more like a craft (in particular, I'm thinking of the apprenticeship and craftsman ship culture that is prevalent in Germany / possibly other former Hansa areas), I don't think that it's reasonable to assume that people should sacrifice their personal time for it. I understand that sometimes this might be necessary (the proverbial night class to get up to speed with some new domain of knowledge), but his implying that surgeons constantly practice surgery during their off-hours (and really, short of illegally exhuming bodies, how would they do this?) seemed a bit of a naive and unrealistic ideal.
The 'Fiasco' is a bit of an unfair comparison, because Jeffries is deliberately providing a 'stream of consciousness' writeup while Norvig's is a cleaned-up version with false starts removed.
(You can tell Norvig's code worked differently at one point by looking at the return value of assign(), for example.)
Sure, Norvig knew how to solve the problem from the start.
But that's another reason why the "TDD Sudoku Fiasco" blogpost isn't a terribly useful source of information (or, from another point of view, why it isn't a great way to persuade a TDD afficionado to change their mind).
To be fair (and I'm no TDD fan personally), that mess wasn't really the fault of TDD at all. It was that Jeffries was learning how to write a Sudoku solver while he was writing his tests. If he started from Norvig's level of understanding, he surely could have written tests for an equivalently elegant solver quickly and simply.
Learning how to solve a problem by iterative coding (and throwing away the intermediate solutions) is a time-honored tradition that works very well. TDD may be an imperfect vehicle for doing that, but it doesn't really change the calculus.
The whole thing at the time was 'test driven design' - the idea that by writing tests, you could iterate your way to algorithms that solve problems you couldn't solve by critical up-front thinking.
Test Driven Design fizzled and quietly died just after the Sudoku Fiasco ;)
Well, to me it seems that if you stretch the idea of separation of concerns, you should separate the process finding out what you should do, from the process of how you should do it right.
The problem with TDD is that we flawed humans are writing the tests in the first place. If I suck at writing code there's no reason to believe I wouldn't suck at writing tests to check that code.
I use it on occasion as a good sanity check to make sure I didn't break anything too obvious, but this idea that TDD is a panacea where no bugs ever survive didn't ever make sense to me in the first place.
> this idea that TDD is a panacea where no bugs ever survive
I don't think I've heard many people claim that. Rather the point I've seen made is that it not only ensures you've got tests, doing them up front encourages developers to write code in a way that is easy to test, which usually happens to be higher quality, more loosely coupled code.
When it comes to writing code there are no silver bullets, TDD included.
"I use it on occasion as a good sanity check to make sure I didn't break anything too obvious"
That sounds more like unit tests than TDD. With TDD you should already know that you didn't break anything..
If you suck at writing code, then TDD should help you determine whether your sucky code produces the correct output for a given input; it is not however going to determine if you did that in the best possible way.
This is a good and cool post. TDD is less susceptible to "sucky code" for two reasons:
1) It should be a direct port of your specification. If you don't have a specification, you can't do TDD.
2) It should be written by someone who won't be writing the code. Or, at minimum, checked over by someone who won't be writing the code.
Writing tests after you've written code strongly encourages you to write implementation tests rather than specification tests. Both are important and useful, but for separate things (if you have to change your specification tests it's probably because you need to bump a minor or major version number for your software).
Regarding your second point, it is not only very difficult, but also not even how TDD was designed to work by its original creators at all. It is specifically not written by other people.
TDD, as outlined by the one who created the technique, as well as other big proponents such as Robert Martin, is done in tight 2 or 3 minute loops. As Kent Beck says, if you can't write your test in 5 minutes, you're doing too much.
How "its creators" designed it to work, and how I find it to work in practice, are pretty different. I've found that having two people understand a spec to the level where they can reason about it (and that the best artifact for demonstrating that reasoning is a test suite) is the most straightforward way to build reasonably correct code.
(I mean, heck. Theoretically, Agile's "creators" designed it to be useful and helpful to developers. Cue sad trombone sound.)
100%. This is perhaps one of its most powerful attributes. Whether I have a spec written on paper or one I've informally defined in my head, I very often find myself enumerating the potential failure cases (determining the domain and range of my code, as it were) and going "hey...wait a minute...what about X?".
Writing tests after code is completed, on the other hand, lends itself to reifying an implementation as "correct" and divergence from the implementation as being wrong. No bueno.
Well, if the test is accidentally testing the wrong input/output pairs, your code can pass the test perfectly and still be broken. Especially if you follow the "build to the test" mantra.
> If I suck at writing code there's no reason to believe I wouldn't suck at writing tests to check that code.
Actually, there is ; providing the difficulty is algorithmic.
Take "sort", for example.
// test code
assertEquals([1,2,3,4,5], sort([4, 1, 2, 3, 5]))
// production code
int[] mySort(int[] list)
{
// TODO: implement a sorting algorithm here.
}
Any programmer could write a fairly good test suite for "sort".
However, I highly doubt they could write the implementation.
Of course, this is a very simple example.
Most of the times, the difficulty lies in finding the proper interface between test and production code.
And there, there's no way to compensate for poor software design skills.
Actually, forcing bad software designers to test the code they write might even worsen the situation, by rigidifying poorly-designed-wannabe-interfaces.
> this idea that TDD is a panacea where no bugs ever survive didn't ever make sense to me in the first place.
This is the software industry ; new techniques are confused with silver bullets!
Any developer should know the basic algorithms. If you tell one to write sort they should immediately think to look up merge sort is implemented. My algorithm will be O(n ln n), not because I'm great at creating algorithms but because I know I can do better.
> this idea that TDD is a panacea where no bugs ever survive didn't ever make sense to me in the first place.
I've literally never encountered this idea, in person or otherwise, from anyone advocating TDD, only as a an argument TDD opponents like this attack. I'm not sure if it's a rare argument they've latched onto, a common misunderstanding of the purpose of TDD, or an outright strawman, but it's certainly not the main argument (or one of them) I've seen for TDD.
TDD:
1. Validates that you have a reasonably clear operational understanding of the intended behavior (but not completeness), forcing more questions out early and preventing some portion of the waste of code aimed at something that isn't the intended functionality that would otherwise occur, and
2. Assures that automated tests (which are often most valuable for regression testing later changes) actually get written, which is a significant issue in practice with test-last processes.
> this idea that TDD is a panacea where no bugs ever survive didn't ever make sense to me in the first place.
The idea only exists as a straw man for those trying to attack TDD.
TDD doesn't ensure there are no bugs. TDD ensures that everything you can think of that can go wrong does not, even as you think of new ways something could go wrong and fix that.
I was in one job a few years back without tests. We had a case where someone got a bug which was easy to fix, but they didn't realize their fix broke something else. Then we released and got the bug report which was fixed by reverting back to the original bug - we went through 3 rounds (over 6 years - a different contractor on the bug each time so who didn't realize what was going on) before someone realized what was happening and fixed both bugs.
TDD failed for economic reasons, not engineering ones.
If you look at who were the early TDD proponents, virtually all of them were consultants who were called in to fix failing enterprise projects. When you're in this situation, the requirements are known. You have a single client, so you can largely do what the contract says you'll deliver and expect to get paid, and the previous failing team has already unearthed many of the "hidden" requirements that management didn't consider. So you've got a solid spec, which you can translate into tests, which you can use to write loosely-coupled, testable code.
This is not how most of the money is made in the software industry.
Software, as an industry, generally profits the most when it can identify an existing need that is currently solved without computers, and then make it 10x+ more efficient by applying computers. In this situation, the software doesn't need to be bug-free, it doesn't need to do everything, it just needs to work better than a human can. The requirements are usually ambiguous: you're sacrificing some portion of the capability of a human in exchange for making the important part orders of magnitude cheaper, and it's crucial to find out what the important part is and what you can sacrifice. And time-to-market is critical: you might get a million-times speedup over a human doing the job, but the next company that comes along will be lucky to get 50% on you, so they face much more of an adoption battle.
Under these conditions, TDD just slows you down. You don't even know what the requirements are, and a large portion of why you're building the product is to find out what they are. Slow down the initial MVP by a factor of 2 and somebody will beat you to it.
And so economically, the only companies to survive are those that have built a steaming hunk of shit, and that's why consultants like the inventors of TDD have a business model. They can make some money cleaning up the messes in certain business sectors where reliability is important, but most companies would rather keep their steaming piles of shit and hire developers to maintain them.
Interestingly, if you read Carlota Perez, she posits that the adoption of any new core technology is divided into two phases: the "installation" phase, where the technology spreads rapidly throughout society and replaces existing means of production, and the "deployment" phase, where the technology has already been adapted by everyone and the focus is on making it maximally useful for customers, with a war or financial crisis in-between. In the installation phase, Worse is Better [1] rules, time-to-market is crucial, financial capital dominates production capital, and successive waves of new businesses are overcome by startups. In the deployment phase, regulations are adopted, labor organizes, production capital reigns over financial capital, safety standards win over time-to-market, and few new businesses can enter the market. It's very likely that when software enters the deployment phase, we'll see a lot more interest in "forgotten" practices like security, TDD, provably-correct software, and basically anything that increases reliability & security at the expense of time to market.
No. TDD failed for engineering reasons in addition to economic ones.
The main failing of TDD is the assumption that you know all of the tests that you will need before you know your software. This is true in bridge building as much as it is in anything else. Until you fully know the domain of what you are building, you cannot possibly know all of the things you need to test for.
To that end, if TDD were focused around building things to break them and analyze them, it would be wise. However, in software it is often taught to build a wall of failing tests, and you can then use that as a sort of burn down chart for progress. Note that you don't learn about the software you are building from this burndown. Instead, you only get a progress report.
And obviously, this is all my opinion. I'm highly interested in exploring the ideas. And I'm highly likely to be wrong in my initial thoughts. :)
It's worth noting that Kent Beck ("Father of TDD") himself said that TDD doesn't always make sense. For him, the important thing has always been having a tight feedback loop for determining if he was still on the right track.
For instance, when he was working on an unstructured log parser, there was no way TDD would work because the logging stream didn't have a pre-defined structure. Instead, he developed a process where he could quickly inject his parser into the logging stream, visually inspect the results, tweak, and then test again.
Others (Such as Martin Fowler and Uncle Bob) have also emphasized the value of automated unit testing integrated with a CI pipeline as a way of ensuring that you didn't break existing functionality.
As always, what's important is not the specific practice, but rather the engineering principles behind it. The key questions being:
* Am I on the right track?
* Did I break existing functionality?
* How quickly and frequently can I get answers to the above two questions?
* What's my confidence that the code will work correctly under different situations.
* How maintainable the code is and related how easy is it to make changes.
* How much time/effort does it take to create.
It's funny (or sad) how the XP guys like Kent are always talking about doing the right thing and not being religious. This stops working when the people using it don't know what the right thing is any more. They write terrible tests, they make bad changes to their code to support those tests, it ends up a big un-refactor-able ball of mud that takes longer to create and is more difficult to maintain.
If you're writing a shortest path algorithm or a JPEG decoder then you absolutely should code up some tests to check your code. The tests are relatively simple. They don't need to know anything about the internals of your code. They help you check quickly whether your code is correct and don't interfere with any refactoring. You can rewrite your JPEG decoder from scratch without needing to touch the tests at all.
i'm not a fan of tdd, and a huge proponent of testing in general. what you say about making bad changes to support tests really strikes a chord.
in tdd environments the test harness is used as some kind of crutch to avoid having to have difficult discussions about what the product should* be doing - what makes sense. the test acts as some kind of oracle in the regard, even though it was written by an intern last spring who didn't really have any idea what was supposed to happen either.
i'm basically afraid to say 'the test is wrong' anymore. i always get back a blank look of incomprehension or even outright fear.
Word. I have also worked on projects where existing tests were treated like holy cows. Deleting a test wasn't disallowed as such but it really required a convincing justification. Ironically, this made it especially hard to touch the inevitable subset of tests which had grown unreadable and which were not tracable to specs. Hence, they were often left alone, smelling riper with each passing year.
I like to think of production code as being inherently a liability, not an asset, and the same goes for test code. That's not to say that code, or tests, cannot be valuable; it's just that every line of code carries a cost to the business just by adding complexity to the development process and by taking up disk space and CPU cycles. Not all code offsets this cost, and a lot of tests certainly don't.
We should stop viewing tests like holy scriptures. Some add long term value, some add value during initial development, some should never have been written in the first place.
And on a side note, our tests should test the business requirements, i.e. they should be tracable to specs. There are a lot of tests out there which amount to little more than to "test" some arbitrary decisions which we made when someone initially fleshed out a class diagram.
All successful projects in my career had full functional regression test coverage.
For those successful projects, I made following conclusions:
* My personal TDD process are:
* Script enable - always run every night, every weekends.
Output to html files to see overall green/red in < 1 second.
* When any existing test broke, fix/debug the tests are the highest priority before any more features development.
* The test reports from all the nightly and weekend test should always be green.
* Any red(s) will stop all features dev.
* There are 5 minutes function test scripts that require to run before every commit.
* Normally, I prefer commit in the morning after overnight test runs were all green.
* These were small project with team only 1-2 devs.
* Much easier to enforce the process. (Just need to convince I, me and myself.)
* I worked on big team projects with 100+ devs.
* I can test only my own part of code. For any big team, I can take the horses to the water but can't make anyone to drink it. The overall system SW quality was still bad.
* I Only do full functional tests - I don't do any small unit tests with mocks.
* For CDRW drive, there were 100+ test cases for
close, read, open, repeat,
close, read, write, loop, open, repeat,
for different CD format, disk, drive, etc
* For DNA sequencer machines, there were tests for
Motor controls, 3 axis robot,
CCD camera, laser control, script subsystem, comm sub sys, etc.
(This is old project, 20+ yrs ago)
Deployed in the field for 15+ years with version 1.0 Firmware.
Good for company bottom line - not sure for the SW guy's career. :-)
(Multi billions $$$ of revenue for the company, $500+M first year)
* For video decoder systems with 19FPGA + 10G network: (10+ years ago)
Streaming control, snmp, IGMP, all each has their own test scripts and
combine system level scripts for complex network, HA setup.
Deployed in Comcast for long time without any issue.
Only one issue reported was system has problem after 255 days of
non-stop running.
I did have test case for 10 ms counters (for SW bug happened for 25, 49 days)
I didn't create test case 100ms counters (for SW bug happened after 255 days).
Again, very good for company bottom line.
(Tens millions $$ of revenue for the startup)
But the company don't need me anymore after they shipped that product.
Sounds like you're doing the right thing. Note that not every one would agree that you are doing "true TDD" (due to your lack of "unit" tests) but that doesn't matter. What matters is that:
a) you are disciplined in your commitment to quality
b) you offload as much of the tedious, repetitive work to the computer as possible.
Regarding unit tests and mocking, I have a similar opinion. I tend to test at the larger component/package level rather than the smaller class/module level. Furthermore, I avoid mocking as much as possible. Object-Oriented decoupling through dependency injection makes mocking a lot easier (and removes the need for mocking frameworks) and Functional decoupling[0] makes mocking largely unnecessary.
Mocking is an aspect of testing that grew out of control and the cost can easily outweigh the benefits.
In addition to mocking frameworks, I've also encountered home-grown solutions that were complete time-sinks to use. At some point, you're investing huge chunks of time, not just testing, but mocking in particular.
It can literally layer an entirely different problem domain onto your dev effort, apart from the actual "business problem" the code is intended to solve.
At a certain point, it's like, "wait, what are we doing here again? There has to be another way." Always wondered how people became so dogmatic and determined that they fail to see that they are veering from their original purpose. There is a balance to everything and something about the dev community causes us to lurch in different directions, well beyond that balance.
BTW, there is a post [0] that reminds me of this on the blog you shared. Good blog, BTW, even if it is no longer active.
>Object-Oriented decoupling through dependency injection makes mocking a lot easier (and removes the need for mocking frameworks)
I'm confused by this. In C++, I found that one of the only ways to effectively use mocking was via dependency injection. For us, dependency injection was introduced to enable TDD. That it had other good qualities was a bonus.
A fair question. Let me put it this way: in order to properly do unit tests, you need to decouple business logic from "integration code" (code that manipulates external systems). Dependency injection is one approach. However, with this approach the business logic still depends (now indirectly) on the integration code.
Another approach is to model business logic using pure functions. Instead of business logic calling the integration logic (directly or indirectly) you have the integration logic calling the business logic.
The consequence of this is that when you are unit testing your business logic you do not need to mock anything. It's simply a matter of checking input/output mappings.
>Another approach is to model business logic using pure functions. Instead of business logic calling the integration logic (directly or indirectly) you have the integration logic calling the business logic.
Yes, this is similar to what I did. I ensured business logic dealt only with data structures and algorithms, and knew nothing about the rest of the world. If it needed to report an error, it would call an interface class's "ReportError" function, and pass a string. It did not need to know if the error went to the console, a file, an email, text message, etc.
However, in our case, the communication is still two way - just separated via interfaces. The integration logic does call the business logic, but the business logic may need to do things like report the results, log interesting information, etc. So we still had to mock the interface class (i.e. mock the ReportError function).
But in the bigger picture - yes. Not allowing your business logic to know anything about the details of the integration was really helpful. In principle, our customers could have written the business logic for us.
Sadly, my team did not accept it so we never adopted it.
>Note that not every one would agree that you are doing "true TDD" (due to your lack of "unit" tests)
I think he's doing it right in spite of what other religious zealots may say. In my experience, unit test driven development causes three massive problems:
* Unit tests are intrinsically tight coupling.
* Unit tests require you to build a model for most things that you are testing against (e.g. your database). Building these elaborate models (i.e. using mocks) is prohibitively expensive when you have tightly coupled code but they're often still expensive to build (in terms of developer time) even when you have loosely coupled code.
* Unit tests are intrinsically unrealistic because they are built against models rather than the real thing.
In my experience unit tests work adequately for domains that are heavy on complex logic and light on integration (e.g. writing a parser). In domains which are light on complex logic and heavy on integration they fail badly (e.g. most business apps).
Optimizing for the overall speed of your test suite by using a more unrealistic test that catches fewer bugs is a massive false economy. CPU time is dirt cheap. QA and dev time is very expensive.
> In my experience unit tests work adequately for domains that are heavy on complex logic and light on integration (e.g. writing a parser). In domains which are light on complex logic and heavy on integration they fail badly (e.g. most business apps).
There's a testing methodology we used at my last company called "Change Risk Analysis" or something. Essentially, the point was that you separate business logic and integration code, and you write unit tests for the business logic. There was a metric that combined the the cyclomatic complexity with the test coverage to produce a "risk" score for methods and classes, and all code (or at least all new code) had to be under a certain risk threshold.
I would also add that domains that are heavy on complex logic and light on integration are also highly amenable to more sophisticated verification methods, such as generative testing and model checking.
> All successful projects in my career had full functional regression test coverage.
Regression testing is not the definition of TDD, however.
TDD is this crazy religion of writing tests for everything, including the lowest-level helper functions in a module that don't correspond to anything that would be tested in a regression test suite.
You write these little tests first (which must be shown to fail at least once and then shown to pass). Then you keep them around when they pass and keep running them all the time even though nothing has changed.
For instance, if you develop a Lisp, you must forever have a test which checks that, yup, CAR on a CONS cell retrieves the CAR field, CAR on NIL returns NIL, and throws on everything else. The image is 10X larger than necessary and it takes 27 minutes to get a REPL prompt on a fresh restart. :)
Thats the interpretation of TDD by people who are afraid of test and seek excuses.
By the name, nothing says you need 100% coverage (a stupid measure abyway, cause you can get those 100% but still neglecting thousands of corner and limit cases for onput data, so actually you need another measure) or that it has to be unit tests...
> TDD is this crazy religion of writing tests for everything,
Not true in my experience. You should only test code you wrote that can break. You do not test core functionality, the OS underneath etc. You do not test the test themselves.
"You should only test code you wrote that can break. "
A common source of errors are mismatches between what your dependencies say they'll do and what they'll actually do. There's been huge, costly failures from developers expecting the dependency would honor its documentation or just take a standard approach to error handling. So, either testing them or at least good catch-all error handling + logging should be done.
No, TDD means that if there is a line of code anywhere that was not written to make some failing test pass, that code is by definition broken. ALL code must be against some test, else it's not TDD. And if you want to write code for which there is no test, you must write the test first, run the test suite to see it fail, and then implement the code that will make the test pass.
>No, TDD means that if there is a line of code anywhere that was not written to make some failing test pass, that code is by definition broken. ALL code must be against some test, else it's not TDD.
Robert Martin is likely the biggest evangelist for TDD, and he does not advocate what you say.
Why do people keep claiming you have to test every line of code in TDD?
> No, TDD means that if there is a line of code anywhere that was not written to make some failing test pass, that code is by definition broken.
No, it doesn't. TDD means tests are written to test the code, and the code being written must pass all tests. It's ok if at any point some parts of the code aren't covered by tests, because TDD is a cyclic process where at each cycle new tests are written to accompany changes in the code.
It seems that the only people complaining about TDD are those who pull the strawman card and portray the process as being unreasonably convoluted and picky, to a level which is almost autistic.
Should you write tests for the tests too? What I find is that in any project of a certain complexity, the tests can end up having the wrong scope or outright bugs in them.
I keep asking the same thing, especially that given any non-trivial self-contained piece of code, the test code will necessarily be isomorphic to the actual implementation - at which point one has to ask, why bother with TDD in the first place and not just write the complex code once?
>TDD is this crazy religion of writing tests for everything, including the lowest-level helper functions in a module that don't correspond to anything that would be tested in a regression test suite.
And like all crazy religions, the followers ignore the prophets.
Robert Martin is likely the biggest evangelist for TDD, and he does not advocate what you say.
Do you write the tests first - before the code - and while still exploring the domain? That's the guiding principle of TDD.
Writing tests after the code has been written - to lock down behavior, quickly identify breakage, and support refactoring is a valuable process. But it's not the original vision of what constitutes test-driven development.
Whenever I write tests first, they usually fail. I need to do some iterations of the test code as the "real" code until they work. Is there some reason why test code is expected to work first time, yet we need a ton of tests to verify that the "real" code works?
I've always relied on unit tests with mocks with the rule that I only mock modules that are tested and I mock or stub all db and network calls. I also like integration and end to end tests but not may of them. I rely on unit tests for regression and should be able to run the whole suite in less than an hour. I set my precommit hook to run unit tests with coverage and static analysis.
> * When any existing test broke, fix/debug the tests are the highest priority before any more features development.
This is the single biggest problem everywhere I have looked. Outside the development team itself no one is willing to wait for a new feature just because a test failed. And for many large programs there will be substantial subsets of users who will not be affected by the bug so they will argue that the delivering the feature is more important, and contributes more to the bottom line, than fixing the bug.
> The main failing of TDD is the assumption that you know all of the tests that you will need before you know your software.
I don't think this is true. TDD is also taught as a way of exploring a space and a codebase.
And there is no assumption that the tests created during TDD are the only tests you will need. They're the tests needed to drive out the desired behaviour. You'll also need tests to check edge cases, tests that operate at a higher level, tests that integrate different systems, nonfunctional tests, etc....
In a lot of ways they are a discovered, executable spec.
> To that end, if TDD were focused around building things to break them and analyze them, it would be wise. However, in software it is often taught to build a wall of failing tests, and you can then use that as a sort of burn down chart for progress.
I've never come across TDD taught that way. Hard core TDD as I know it is RED, GREEN, REFACTOR. You never want more than one failing test, and you don't generalise until you've triangulated based on creating tests that force you to.
I don't personally use TDD on all my projects because often it feels fussy and constraining when you are in exploration mode, but I also don't think it failed. I know people who find it valuable even for exploration. I'm also a big fan of using it for certain kinds of things - e.g. building a collection class.
I don't do the TDD like a waterfall. I don't do upfront all the tests, there is a discussion between the code (what express the business need) and the tests that "wraps" it inside a safe zone (what ensures that the behavior is the one intended). As such, as the idea grow, both the code and the test grow at the same time. Look at [0] to have an idea of how the tests extend the safe zone along the code.
Also, I much prefer a lot of nice unit tests over a lot selenium tests (which takes a lot of time to run).
Rereading my post, I see where it sounds more strawman of the "entire wall of tests." I did not mean it to be the final test suite is written first.
Instead, I intended that to be the wall that you will be passing in a given cycle. So, for a small development cycle, you add in some new tests that will be this cycle's "wall" of failing tests that you flip to passing.
And don't take this to be an indication that I think integration tests are somehow superior. They all have advantages and there is a massive cost benefit tradeoff that has to be considered at all times.
It is easy to talk in strawmen. Such that I can confidently say if you have a set of tests that always fails together, that should just be a single test. However, I do not feel that statements like that are informative, precisely because it is a judgement call masquerading as a rule.
Right. In that, you still write all of the tests before the code. If you are objecting only to the rhetoric of "wall" of tests. That just depends on your size of units. Think of it more as hurdles of tests. :)
Everyone is interpreting this as, "write 10 tests then try to get them all to pass at once". That is not how you TDD. You write one, then get it to pass, then write another.
Maybe you mean, "write the test before the code", but when you say "write all tests before the code", it's not interpreted the same way.
> So, for a small development cycle, you add in some new tests that will be this cycle's "wall" of failing tests that you flip to passing.
That's not right either. There is no "wall". Writing tests helps you write the code which helps you write better tests. It's mutual recursion, not a one-way waterfall.
> The main failing of TDD is the assumption that you know all of the tests that you will need before you know your software.
You know what makes this work for me (I am a TDD & BDD dev, and I don't think it's dead) is the fact that you can reorder commits. When I hit a case like this, where the correct code is easier to write than the correct test, I still test it.
I write the test after, then I reorder the commits once the test passes, and make sure it fails predictably in the way you were expecting it to fail. Sometimes you find your test doesn't fail correctly with the codebase you had before the feature was implemented, and then you amend the commit and use rebase or cherry-pick.
Adding BDD to the mix helps a lot too. Sometimes you can write the whole test up-front! Sometimes you can write pseudocode for the test, but you don't have enough information about the details of the implementation to write what you'd call a complete test before the feature is born. Usually you can at least get a vague, english-language description of what the feature is supposed to do.
Then you have, at least, a foundation for your test (which can ultimately be upgraded to an executable proof/regression test), and you have a way to measure how far from the vague, usually simple, plain-english description of the feature you've had to go in order to produce "correctly behaving" code. (And you can amend the behavioral spec to correctly explain the edge cases.)
I am in the habit of telling my team members with less TDD and BDD experience, who are just trying to get the hang of building up a codebase with strong regression tests, that they should always try to write a Cucumber Scenario before they wrote the feature, even if they don't write step definitions until (sometimes a great while) after the feature is done.
I agree that the burn-down chart is not the end goal of TDD, and you should not optimize your process simply around making sure you have a progress chart and a great estimate up-front. Complete coverage of the codebase is more important! And surely there are other things even more important than that.
>> I write the test after, then I reorder the commits once the test passes, and make sure it fails predictably in the way you were expecting it to fail.
Lol, that literally sounds like cheating. You write the code first, tests later, and then reorder the commits so that you can say to everyone (colleagues and managers) that you follow TDD...
It's really important in TDD to know that your test fails when the code you write for that feature it is missing. If it doesn't fail, that's a false positive. The prescribed pattern is, you write your test (it should be red), you write your code (it should turn green), you commit, push to CI, look at the rest of the tests, make sure none of them turned red... take another look at the code, refactor, commit, push to CI, look at tests again, and then start over on the next feature.
(Edit: OK so reordering the commit is not actually the most important part, it's checking out that commit you reordered: with the code for the test, without the matching code for the implementation; and running the test to make sure that it fails until the feature is implemented.)
I don't really care if you think it's cheating to reorder the commits, if it gets the job done faster / better in some way. I'm not writing code for a grade, I'm writing an application that I'll need to maintain later. If that feature breaks and that test doesn't turn red, it's been a complete waste of time writing it (and actually has provided negative value, because I will now be less inclined to test that behavior manually, knowing that there is a test and it passes.)
That's a false positive that you can easily guard against by reordering the commits before squashing/merging everything back and confirming you see some red where it's expected. (You don't need to reorder if you did actually write the test before writing the code for the feature, but forcing people to do this seems to be the most common cause for complaint about working in TDD. If your developers quit their jobs, then TDD has provided negative value too.)
Meanwhile if you have tools for analyzing code coverage, you can make a rule that code coverage must remain above X% to merge work back to master, or that merges must never decrease the percentage of code covered by tests. You don't have to use these rules, but if you're having trouble getting your team members to test their commits, it's not always a bad idea. It has nothing to do with proving things to managers. Code that isn't covered by tests is code that can break silently without anybody noticing. Code that is covered by bad tests is no better than code that is not covered by any test. It can break without anybody noticing. You need to know that your tests work; if you write the code before the test it's just so easy to accidentally write a test that passes anyway, even if the feature was never implemented.
Also, because you can. Forget TDD completely, if you don't know how to use rebase and cherry-pick, you need to learn to use them so you can reorder your commits.
It makes your history cleaner. Nobody cares what commit you were looking at when you started making your change, that's not useful information. Would you rather have a history that looks like this[left] or this[right]?
I guess another way to put it is that TDD is an attempt to do old school "waterfall" in code form.
Way back before anyone really understand how software engineering should be done, people would begin a software product by attempting to pre-specify ahead of time every nuance of the system. They'd write these massive books of requirements. Then they'd get to work coding to these requirements.
This rarely worked out well and usually resulted in huge amounts of wasted effort. When you get coding you realize the spec is either wrong or there's a much, much better way to do it. Do you do it the bad way to stick to the spec or do you deviate from the spec?
The other problem with this approach is that the requirements are often as hard to write as the code and contain as much information. Why not just write the code?
TDD is just "the spec" written in code instead of as a massive design document. Attempting to achieve 100% test coverage often means writing a test set that is a kind of negative image of the code to be written and is equal in complexity. As such the tests often end up containing bugs, which is the same problem as the classical "when we got going we found that the spec was wrong." Do you code to the bugs in the tests?
This is not to say tests should be skipped. Automated tests are great. I just think the 100% test coverage TDD dogma is oversold and doesn't really work well in practice. Test where you can. Test where you must. Keep going.
Of course I am not a fan of "methodologies" in general. I see them as attempts to replace thinking with rote formula. That categorically does not work outside domains that are purely deterministic in nature, and those can be fully automated so humans aren't needed there at all.
There is no formula, or alternately the formula is general intelligence continuously applied.
Edit:
The same evolution has happened in business. Nobody writes a "business plan" anymore. As soon as you get into business you have to toss the plan so why waste your time. Planning is a false god.
TDD is closer to the Spiral model of iterative development of a product always assumed to be incomplete or broken in some way. Constant improvement instead of perfection.
"As such the tests often end up containing bugs, which is the same problem as the classical "when we got going we found that the spec was wrong." Do you code to the bugs in the tests?"
Work in high-assurance field showed it's often easier to assert or check the results of complex software than it is to implement the software itself. Imagine a control system bringing in data, making complex calculations, and deciding whether to accelerate or decelerate. You want to ensure it never pushes past a certain maximum. It's much easier to code something at its output that says (if value >= maximum then StopAcceleratingMaybeNotifySomeone()). The checker is even easier to formally verify if you wanted to go that far. This is one of reasons many proof assistants started building stuff on simple-to-check cores where the generation process is untrusted but the trusted checker is tiny.
Empirically, though, there's confirmation out there in terms of defect detection that tests catch more bugs than they create.
Your description does not resemble TDD as I learned it and practice it.
> As such the tests often end up containing bugs
All code contains bugs, but we do not abandon programming. Having statements of behaviour from outside and inside the solution increases the chances that one will reveal a meaningful difference from the other.
> I just think the 100% test coverage TDD dogma is oversold and doesn't really work well in practice.
I work for Pivotal, one of the most outspokenly pro-TDD shops in the industry. I have, quite literally, never seen or heard any engineer mention test coverage ever.
> Of course I am not a fan of "methodologies" in general. I see them as attempts to replace thinking with rote formula.
I see them as a starting point. The expert mind sees possibility, but the novice mind needs certainty.
Test coverage doesn't make sense to a TDD shop and so they never talk about it. The only code they have not covered by tests are things that cannot be tested at all. They by definition are as close to 100% coverage as you can be.
Test coverage is important in shops that do not do TDD. Such shops are learning the hard way that they need tests to ensure that existing features do not break when they add a new feature. Test coverage is a metric they can use as a proxy for how likely it is that change will break something else (fixing a bug and introducing a new one is a very real problem).
I think the failures are due to a combination of engineering, economic and human reasons.
TDD is counter intuitive to how most people think and work. They think of the features first, implement them and then once it's working want to have tests to ensure it stays working, especially with continuous delivery, testing is mostly about checking for regression as high frequency iterations are done.
Even if things starts out ok, often at some point, some person in the team breaks the flow. You have to be quite rigid about procedure and in my experience most teams don't have the discipline for it. This is especially true when there are business pressures in some companies that put pressure on the development team. We've all experience instances where a program manager or producer asks for estimations and an inexperience developer only considers the coding time. To be honest, unless you're working at one of the top tech firms where everyone is interested in solid engineering and there's adequate revenue to enable this, I still find even basic automated testing in general is still a struggle to have in some companies. The reason for this is because the increased development cost is difficult to measure and so it's absorbed as a cost. They don't realize how much development is costing because they cannot compare it to the alternate reality where they did things a better way. Whereas feature issues are obvious to see for everyone.
TDD works for experienced developers who know the problem domain well. It does NOT work for that new junior dev you hired straight out of college. They need to write functions and features first and see the spectacular ways in which those functions and features can fail (in a dev/test environment, hopefully), and then use that experience to switch over to TDD when they write a similar feature/function in their next project.
> TDD works for experienced developers who know the problem domain well
This is important: it's not only junior devs that fail --
TDD also does NOT work for experienced developers who do NOT know the problem domain well, as the infamous Sudoku solver debacle showed, where Ron Jeffries got too caught up in TDD/refactoring to make any serious progress in actually solving Sudoku.
> the assumption that you know all of the tests that you will need before you know your software
I'm not a TDDer but must object on their behalf: not only is that not what they said, it couldn't be further from it! What they taught was to work in short iterations and work out a bit of requirements, design, and implementation in each one. If you needed knowledge you didn't have yet, they would advise you to make each cycle shorter until you did. Sometimes they'd take this to absurdity, breaking even simple features into design-test-implement cycles so microscopic that you would feel bored like a kid in class who can already see the next dozen steps and has to sit there waiting for the teacher to catch up.
The TDD approach was at the extreme tip of trying to get away from waterfall methods, so it would be quite some irony if it reduced back to that.
Edit: looks like I misread what taeric meant! But as McLuhan said, 'you don't like those ideas, i got others'...
In my experience there were two problems with TDD. One is that the micro-iterative approach amounted to a random walk in which features agglutinated into a software blob without any organizing design. Of course that's what most software development does anyway. But if you remember what Fred Brooks et. al. taught about having a coherent design vision for software that can organize a codebase at a higher level, the way iron filings are ordered by a magnetic field, that is a key anti-complexity vector which the TDD/XP/agile projects I saw all lacked. (Original XP had a notion of 'metaphor' that pointed vaguely in this direction but no one knew what it meant so it had no effect.) The lesson was that a coherent design vision doesn't evolve out of bit pieces, and there is an important place for the kind of thinking that got dismissed as 'top-down'. (Edit: by far the deepest work on integrating design coherence into TDD/XP approaches was Eric Evans' stuff on domain models and ubiquitous language. Eric is my friend but I knew he had the best stuff on this before that :)) (Edit 2: Eric points out that the people who created TDD were software design experts for whom coherent design was as natural as breathing air. The admonition against too much design thinking in TDD was intended for people like themselves, for whom the opposite mistake was never a risk, and didn't have unintended consequences until later.)
The other problem was that practices that started out as interesting experiments ossified into dogma so damn quickly that people started acting like they'd cracked the code of how to build software. A soft zealotry emerged that was deeply out of sync with the playful, adaptive spirit that good software work needs. This was unintentional but happened anyway, and there's a power law in how much creativity it kills: each generation that learns the Process is an order of magnitude less able to think for itself within it, and so must break free. I don't think the creators of the approach were any more at fault for this than the rest of us would have been in their shoes. The lesson is that this ossification is inevitable, even if you know all about the danger and are firmly resolved to avoid it. If there is any real breakthrough to be had at the 'how to build software' level, it is probably discovering some kind of antifreeze to put in there from the beginning to make sure this doesn't happen. Maybe there's a way to interrupt the brain and social circuitry that produce it. Failing that, I think that each project or maybe (maybe!) organization should hash out its own process from first principles and the prior experience of team members. That's ad hoc and involves a lot of repeating past mistakes for oneself but at least it isn't idiotifying. It's how I like to work, anyhow, and IMO that's what all software processes reduce to.
Oh, to be clear. Just because I had intended it in one way, does not mean I was successful at it. I just didn't want to ninja edit a better wording under what others had already read. :)
The property of iterations is that things change, outcomes change, ideas change, requirements change. Iterations are not black boxes that we tuck away as 100% complete; they often require re-examination as the iteration process continues.
That is not at all the assumption. The assumption is that you have some idea what you want a particular piece of code, method, or class to do and you write a test for a small piece of that and then after it passes you do that again.
At no point does TDD suggest you should know all the tests you want upfront. In fact I would expect if you know that then you don't need TDD. TDD is about learning what tests you need and what you want the object or code to look like one small piece at a time.
"The main failing of TDD is the assumption that you know all of the tests that you will need before you know your software."
If that's what TDD is, then it can only fail. Fortunately, TDD does not require you to write all your tests up front (see "The Three Rules of TDD").
I spend a lot of my time interviewing people and talking with people about TDD. In the vast majority of cases, I find people who dislike TDD are often doing "Not TDD, but called TDD." If I were to write software that way, I too would consider that TDD did not live up to expectations.
Aren't you two saying the same thing? TDD works when the requirements are well defined upfront, which in the case of fixing existing system is provided by the system itself. But writing new programs is exploratory in nature: you don't know the exact requirements or shape your software will take, and TDD shifts a lot of precious money, time and attention from the what to the how.
Which to me, I agree with you, is an engineering issue before being an economic one.
I've never heard TDD being that you write all of your tests up front, before you've written any code. I've always heard it as: When you're writing a class/unit, you write a few tests for what that unit is going to do. You then make those tests pass. You add some more tests, make those pass, and so on and so on.
You are still writing the tests first, with no real learning from them other than "they are now passing."
It is thought this is like engineering. You design an experiment and then pass it. However, that is not how experiments work. You design an experiment, and then you collect results from it. These may be enough to indicate that you are doing something right/wrong. They are intended, though, to be directional learnings. Not decisions in and of themselves.
So, I probably described something in more of a strawman fashion than I had meant. I don't think that really changes my thoughts here, though.
You're often learning the optimal API as well. I've had several experiences where I write a library, write some tests for it, and then realize that the library's API is inconvenient in ways X, Y, and Z, and that I could have a much simpler API by moving a few tokens around. When I write the tests first, I tend to get a fairly usable API the first time around, because I'm designing it in response to real client code.
(This all depends on having a clear enough idea of what the library needs to do that I can write realistic test code...I've also had the experience where I write the tests, write the library, and then find that the real system has a "hidden" requirement that isn't captured by the tests and requires a very different way of interacting with the library.)
It’s funny that you say “I'm designing it in response to real client code.“
To me, a test is not real client code. Real client code is code that calls the API in service of the user. E.g. the real client code for the Twitter API is in your preferred Twitter client, not in the tests that run against the Twitter API.
That's the caveat I mention in my second paragraph.
But yes, if the tests are well-designed and built with actual real-world experience, I do treat them like real client code. Someone looking to use the library should be able to read the unit tests and have a pretty good idea what code they need to write and what pitfalls they'll encounter. And when the library is redesigned or refactored, the tests are first-class citizens; they aren't mindlessly updated to fit the new code, they're considered alongside other client code as something that may or may not have to change but ideally wouldn't.
So, this is a slightly different mantra. Working backwards from the customer is an important thing. And if you know that customers will be using your product programmatically, then yes, building the API first is important.
But note, in that world, you aren't necessarily building tests of your software. You are building specifications for your API. I can see how these might be seen as similar. And for large portions of your code, they can be.
However, for the vast majority of your code, it exists to help out other parts of your code. Not to be used as a library/framework by others.
> with no real learning from them other than "they are now passing."
You don't learn much the moment the test passes. But designing and writing the test requires understanding the requirements in detail, which is often a learning process.
This discussion on what happens - or doesn't happen - after the tests are passing is precisely the point of the article.
The idea of TDD is that every test you look at the existing code and improve it. The majority of developers I've seen trying TDD do this with small code but do not with significant codebases, and you end without the benefits.
But this advice should be extended further. Always be willing to throw away the code and the tests. Take the learning.
Be ware, though, that you are not throwing your customers for a bad ride. As soon as you have customers, it is nigh impossible to throw away the code without neglecting them. And they are your ultimate responsibility. Not the code.
Exactly right, it can be very hard to make changes (even the right changes) once it's in customer hands. Sometimes it is easy, sometimes it is not. However, it's always easy to throw out code that isn't yet committed.
If you wrote the test after you coded your unit,you would not know why it passed. It could be a false positive.
When you move on to the next part of the requirement, you can ensure all your tests still pass.
Sometimes the tests give you the starting point of the problem, "what is the simplest way to get result x" with a way to rapidly get feed back that all the requirements are ment as the structure takes form and you refactor all the code.
I sometimes write tests that I know will pass. A common example is when the previous version of an algorithm had a bug in some special case. My new algorithm written from scratch doesn't have that special case at all so there is no test for it. However because the last version had that bug there is a lot of fear so I will write a test that it works even though it is inherent in my new algorithm that it works and at no point in the TDD cycle is that test the correct one to write.
Can you provide a reference? I'm not a big proponent of TDD but have read up on it and experimented with it some, and none of the TDD literature I've read suggests writing all the tests up front. Virtually all the mainstream TDD literature and talks advocate for the "write a test, make it pass, lather, rinse, repeat" model of development.
I mean before you've written any code at all. Like, you have to have the tests for all units created before you start writing the code for any unit. That's what the person I responded to made it sound like.
Writing tests before you work on the unit you're working on shouldn't be a problem because you should know what that unit is supposed to do. If you don't, then it doesn't matter if you do TDD, Agile, BDD, or whatever. You're not going to be able to do it right. Go have a conversation with someone to find out what that is supposed to do.
1. Write test; build fails because code under test DOESN'T EXIST
2. Write the least amount of code needed for the test to pass
3. Refactor & repeat process ad infinitum
The Wikipedia entry for TDD also states this explicitly [0]
This is a differentiating feature of test-driven development versus writing unit
tests after the code is written: it makes the developer focus on the requirements
before writing the code, a subtle but important difference.
>I've never heard TDD being that you write all of your tests up front, before you've written any code.
That's interesting, because as a non-TDD practitioner who occasionally gets it evangelized to me, that's certainly how I was told to go about it by multiple people.
Wow, if that's how people are describing it, no wonder it's seen negatively. That seems very waterfall-ish and horrible. Thankfully that's not TDD as originally intended.
TDD is writing code to automatically validate that a specification/requirement has been met, before writing the code that meets the requirement.
It does one requirement at a time. You might add a specification that the code has to create a list of strings from a data set. You write a test that takes a data set as input and checks the output to see if it is the expected result. This fails, because the code hasn't been written yet. Then you write the simplest function possible to make the test pass.
Once it passes, you check to see if anything can be refactored. You can refactor as much or as little as you like, provided that all the tests still pass. Once you're satisfied, you write another test for another requirement. It's like a spinning ratchet wheel, where the test suite is the pawl. It keeps you from backsliding, but you can't spin too fast, or it flies off and won't engage with the wheel. Turn-click-turn-click-turn-click.
Writing all your tests up front is one of the worst ideas I have ever heard.
> However, in software it is often taught to build a wall of failing tests
People teaching that are not teaching TDD. They may be letting the tests drive the development, but TDD specifically is _all_ about the second-to-second cycle of moving between test and code. If you write all the tests up front then refactoring is expensive and you've locked in your design way too early.
I agree that time to market is really important, but when a company is searching for the product that will work, the most important thing that you can have is code that is easily refactored, which generally means low coupling.
TDD is one way to get there for some people, and it's not 2x the amount of time. My experience is that TDD takes a little bit longer (say 30%) than just writing the code, but my bug rate with TDD code (and I don't do TDD for all my code) is very small. Bugs are so damn expensive that 30% to not have them is a small investment, even if that were the only benefit you got.
I would agree most of the time. However in my experience you often end up with at least some small part you have to figure out how to test-drive - i.e. using a dependency. Then you have a choice of "TDD this and write proxies or some other kind of wrapper around everything", or "just go with it".
You're assuming TDD has failed and are reaching for evidence to make your case.
From the SWEBOK [0]:
5.1.5 Test-Driven Development
[1, c1s16]
Test-driven development (TDD) originated as one of the core XP (extreme programming) practices and consists of writing unit tests prior to writing the code to be tested (see Agile Methods in the Software Engineering Models and Method KA).
In this way, TDD develops the test cases as a surrogate for a software requirements specification document rather than as an independent check that the software has correctly implemented the requirements.
Rather than a testing strategy, TDD is a practice that requires software developers to define and maintain unit tests; it thus can also have a positive impact on elaborating user needs and software requirements specifications.
It really depends on many more factors than getting to market first. If we developed software in this way for fighter planes or search engines I think we wouldn't find them nearly as useful. Perhaps even dangerous.
In the absence of a more formal software specification or testing strategy TDD is the best way for a self-directed team of software developers to maintain that specification.
Beyond that it also has benefits for practitioners working in languages that lack a sound type system to catch domain errors for them at compile time. Coupled with property-based testing TDD can be extremely effective.
I have a colleague who is not a household name but still fairly widely known in the software process space and he has insisted to me multiple times, for more than ten years now, that this is a west coast thing. Other parts of the US are much less allergic to discipline in development.
I'd agree with this characterization. I started my career in Boston - actually, one of the first companies I interned at did high-assurance systems for avionics, financials, and medical devices.
However, look at the size of the West Coast tech industry vs. the size of tech firms in New England or the Midwest. That's what I mean by an economic argument. I got paid 5x more at Google than I did in Boston, and I was employee #45,000 or so, not even an early employee.
Unless the causal relationship is the other direction.
That is, everyone always assumes in our industry that the chaos allows this volume of people to be employed. But usually large number of people leads to chaos all by itself. Maybe we have it backward.
I know I've worked on a lot of projects that got version one out quick and could never get version 3 out the door. It gets old after a while. Why do I want to spend a few years of my life working on something nobody will remember? No thanks.
I actually don't think either of these factors is causal.
Rather, I think money is the primary causal agent. The top 5 most valuable companies on earth right now are all West Coast tech companies, and they got that way by building software systems that are an integral part of the lives of billions of people. The fact that software is an integral part of those activities is a function of it being a million+ times more efficient than humans; any company could've filled that need, but AppAmaGooBookSoft was the one who actually did it first.
Money (and specifically, rapid growth in customers & revenues) causes chaos, and money causes lots of people to be employed, and money lets you pay them a lot. The specific engineering practices - at this stage in the industry - are a sideshow. They're important to practitioners, but not important to customers, as long as the software basically works. And the reason TDD has fallen short is because you can build software that "basically works" without it, so it's just overhead until you need software that "really needs to work in all situations".
Well, MIT, sure. But is Harvard really a remarkable CS, engineering, or applied science research institution comparable to Berkeley or Stanford?
It's interesting what happens when you look at rankings of engineering programs, especially at the grad level. You start to see more large state institutions, as well as more technical institutions, with a heavier concentration in the mid west or west coast.
The ivies are hardly missing in action, especially Cornell (should probably also say Columbia) but Texas, Washington, many UC campuses, certainly Michigan... I'd probably put all these, certainly at the graduate engineering level, at or above the ivies. And the ivy that really seems to be a heavy hitter here, Cornell, again shows how much things have shuffled up once you go to engineering.
> Software, as an industry, generally profits the most when it can identify an existing need that is currently solved without computers, and then make it 10x+ more efficient by applying computers.
>In this situation, the software doesn't need to be bug-free, it doesn't need to do everything, it just needs to work better than a human can.
Well said, these statements underscores importance of FINDING the PROBLEMS/TASKS that are done by HUMANS/MANUAL that can be AUTOMATED with COMPUTERS that can yield 10X efficiency .
The ability to write and refactor code to a state with low coupling, well-separated concerns > TDD Skills
1/ Design on the fly is a learned skill. If you don’t have the refactoring skills to drive it, it is possible that the design you reach through TDD is going to be worse than if you spent 15 minutes doing up-front design.
2/ I think that’s a good summary of why many people have said, “TDD doesn’t work”; if you don’t have a design with low-coupling, you will end up with a bunch of tests that are expensive to maintain and may not guard against regression very well.
That's a great write-up on the situation. There's one thing I'll counter:
"And so economically, the only companies to survive are those that have built a steaming hunk of shit,"
This is true in average case because most of these startups don't care about quality that much since they don't have to for reasons you describe. However, if they did, then they could at least document things, do limited amount of Design-by-Contract, and/or do assertion/property-based testing as they went along. They can also follow principles like cleanly separating modules, protocols that have an update feature, standardized formats for data, and so on that can make rewriting a lot easier later. Easier, not easy, as rewriting is also difficult. That's why I mentioned documentation and Design-by-Contract as they take very little time. Essentially just write down whatever was in your head at that moment.
Most of the time will still go to experimentation and debugging which... wait, styles like Design-by-Contract in safe languages knock out a lot of that debugging. Good docs do, too. Now, we've gotten to the point illustrated by methods like Cleanroom where the QA might cost a little extra, might cost the same, or might even save time/money due to less debugging of new or old code.
Which is another way of saying TDD is somewhat usefull, if you start from a huge pile of code written by someone else.
When you think this might be going on, so if I do this it should work... then writing some tests will help decide if your assumptions or your code is wrong. Most importantly you are probably writing minimal code in those situations so the overhead of tests is minimized.
Or even just a clear spec from a customer who knows what they want.
The challenge in most of the software industry is that the customer does not know what they want, and oftentimes it isn't even clear who the customer is. TDD is great if you can define your problem well enough to write tests for it up-front. Most software problems with a decent economic return are not like that.
I would say that the primary job of a modern developer is actually to figure out what the spec really is. Implementing the spec is easy in comparison. More often than not, "bugs" are errors in the spec that TDD of course has no hope of catching.
TDD is useful if you start from a well defined specification. The better defined, the more useful it is, starting from a big negative utility when you don't know your problem.
Props for mentioning Carlota Perez. I'm familiar with her Technological Revolutions and Financial Capital (which is quite good). Are there any other works you'd recommend?
P.S. If you click the [1] link above you will be taken to an imgur image making fun of Hacker News.
To read the real article you'll need to copy and paste the URL address in your browser. Looks like the site is reading the referrer and forwarding to mean things...
I updated the link to point to the DreamSongs original. Sigh, jwz. Ironically, DreamSongs is the personal site of the guy who actually wrote the essay, but it's way down in the Google results because everybody links to the jwz mirror.
This is not how most of the money is made in the software industry.
Are you sure? I agree that the most profitable companies are not doing that, but at least from what I've seen in a few European markets, the number of developers writing custom software for one client (consulting or in-house) or maintaining stable applications absolutely dwarfs the number of developers writing new products.
I work on a consulting company that customizes a relatively little known open source platform (Odoo) and the number of small and micro companies doing the same over the world is staggering.
> Software, as an industry, generally profits the most when it can identify an existing need that is currently solved without computers, and then make it 10x+ more efficient by applying computers.
Which is exactly how testing should be done. If you have a piece of software that you test so frequently by hand that you could get actual ROI by automating the testing, awesome. If you don't, then what exactly are you getting out of all the automated testing of code that often never changes with tests that essentially assure you that 2 + 2 is still 4?
It's all a balance, but one thing to consider is that future maintenance or new development may require changes to that software. At that time, having good tests will help you avoid breaking things in subtle ways. The problem is that when you reach that point, nobody may remember enough about that code to write the tests, so it's preferable to have already written them.
If a piece of software is trivial or isn't core to your business, though, obviously this doesn't apply.
So there is a tipping point when the technical debt you acquire from not testing catches up with you and cripples the project. I think this tipping point comes much faster than people realize. Most people have no idea how bad their tech debt is until it's too late. Then you see one failure after another while management tries to fix the train wreck. At this point any good Eng is going to bail in this market and you're stuck with the scrubs.
The crippling debt is caused by not refactoring, not lack of testing. Admittedly testing will make the refactoring easier, but thats not the root of the problem.
This was always my issue with TDD. I never tried it, but I know that the way in which I have written software for more than a decade is "get a loose set of requirements, start digging, write some code, figure out what works and what doesn't, change things, repeat until done."
I never saw a way in which I could write a large number of tests up front.
This!! x100, I am constantly working on features that are not even decided if they go into production. Using TDD for these would be a huge waste of time and energy.
Once a feature is accepted I write tests to cover it... after it's already being used and getting feedback from users.
TDD failed? The use case of unknown requirements is precisely what we solve everyday with TDD at Pivotal Labs, as consultants! It works, but I don't have time to outline our process right now.
> Software, as an industry, generally profits the most when it can identify an existing need that is currently solved without computers, and then make it 10x+ more efficient by applying computers. In this situation, the software doesn't need to be bug-free, it doesn't need to do everything, it just needs to work better than a human can.
Unfortunately, the project isn't "finished" after you achieve initial success. You have to maintain it for years after that. Without good tests, you get yourself into a state where one change breaks two things. You need teams of people manually checking it still works. It takes them an order of magnitude longer to check than it would take a computer. Your feedback loop that answers the question, "does it still work?" is in weeks instead of minutes.
You stop changing the bad code because it's too scary and dangerous. The bad code reduces the frequency at which you release new features. You slow to a crawl. The team of people testing gets expensive and eats into your profits. Some competitor eats your lunch.
In my experience, these problems occur earlier than you expect. Usually before you even attain success. I'm too lazy to comb over my app checking for bugs every time I develop a new feature. I get bored too easily to check that the same bug doesn't crop up over and over again. I'm also too impatient to wait for another person to do it. In these conditions, TDD speeds you up.
I've never worked on a manually tested project that avoided these problems. It usually happens around month 2 and gets even worse as time goes on.
Here's why I think TDD fails: The consultants say, "it's so easy! Just red-green-refactor!" No, it's extremely complex and difficult, actually. There's a ton of depth that nobody talks about because they're trying to get you to try it. The typical dev learns about "Just red-green-refactor" and fails. Now "TDD did not live up to expectations".
For example, did you know there's different styles of TDD? The author of this article doesn't seem to (I may be wrong, but he gives no indication). He's talking about Detroit style TDD (also known as classicist). This is the style Ron Jeffries, Uncle Bob, Martin Fowler, Kent Beck, etc. practice and advocate for. There's another style called London style (AKA mockist) that Gary Bernhardt, etc. practice. And then there's Discovery Testing from Justin Searls. There's probably others I haven't heard of yet.
They all have different pros and cons and a very common pitfall is to use a style in a way it's not really meant for. e.g., Using mockist to gain confidence your high level features work. Expecting classicist to help you drive out good design for your code.
To wrap this up, there's so much unknown unknowns about TDD it's usually premature to judge the technique. If you haven't tried all these styles, one might click with you. A lot of people are concluding they hate all sports when they've only tried baseball.
"Unfortunately, the project isn't "finished" after you achieve initial success."
It is if the goal was to sell the company and people bought it. After the success, the remaining problems are the buyer's while the money is going to the sellers. Is that a perverse incentive for quality or what? ;)
Just like other agile rhetoric I've found the benefits are not what the proponents advertise.
I teach it through pairing and here's what I find.
TDD provides two things.
1. Focus
Focus is something I find most programmers struggle with. When we're starting some work and I ask, "ok what are we doing here" and then say "ok let's start with a test" it is a focusing activity that brings clarity to the cluttered mind of the developer neck deep in complexity. I find my pairing partners write much less code, and much better code (even without good refactoring skills) when they write a test first. Few people naturally poses this kind of focus.
2. "Done"
This one caught me by surprise. My students often tell me they like TDD because when they're done programming they are actually done. They don't need to go and begin writing tests now that the code works. They like the feeling of not having additional chores after the real task is complete.
I would agree with those two statements. They are even mentioned by Kent Beck, who created the technique in the first place, in his book Test Driven Development: By Example.
He spends a whole bunch of time hammering home the point that TDD is to help you manage complex problems and focus. He even mentions that if you think you can just power through the code and write it correctly in one swoop, then just do it. Skip TDD - it's not helping you then.
There are also other techniques he talks about regarding focus. For example, leaving a failing test as your last "bookmark" to where you left off for when you come back the next day. That allows you to jump right into where you left off, no ramp-up time at all.
Regarding Focus, I have to agree mostly that pairing is amazing for focus. I've found that I never lose focus when I pair. And we almost always come up with better solutions when we think out loud.
Regarding being done, often this works for more cut and dry elements, which are increasingly more like boilerplate. But this doesn't hold up for what I nowadays spend most to my time on, which is UX and the experience of using the application.
The tests get in the way. Because my design does not have low coupling, I end up with tests that also do not have low coupling.
Not to be smug, but I feel like this is a rookie mistake I learned 10 years ago immediately after starting TDD.
The slogan I use in my head is that testing calcifies interfaces. Once you have a test against an interface, it's hard to change it. If you find yourself changing tests and code AT THE SAME TIME, e.g. while refactoring, then your tests become less useful, and are just slowing you down.
Instead, you want to test against stable interfaces -- ones you did NOT create. That could be HTTP/WSGI/Rack for web services, or stdin/stdout/argv for command line tools.
Unit test frameworks and in particular mocking frameworks can lead you into this trap. I've never used a mocking library -- they are the worst.
There are pretty straightforward solutions to this problem. If I want to be fancy then I will say I write "bespoke test frameworks", but all this means is: write some simple Python or shell scripts to test your code from a coarse-grained level. Your tests can often be in a different langauge than your code.
"How I Plan to Use Tests: Transforming OSH": http://www.oilshell.org/blog/2017/06/24.html -- I want to change the LANGUAGE my code is written in, but preserve the tests, and use them as a guide.
And definitely these kinds of tests work better for data manipulation rather than heavily stateful code. But the point is that testing teaches you good design, and good design is to separate your data manipulation from your program state as much as possible. State is hard, and logic is easy (if you have isolated it and tested it.)
Summary: I use TDD, it absolutely works. But I use more coarse-grained tests against STABLE INTERFACES I didn't create.
This is terrific. You know, I've spent the last ten years perfecting coding strategies around unit test frameworks and mocks. I'm really really good at that. Really. I won't be humble; if somebody wants this, then I'm one of the best in the business. Yet, I'm slowly coming to the realization that that's a perfectly useless skill. Perhaps, even a negative skill.
It depends on the project, but for most projects I've worked on, the most difficult parts are the integrations with external systems. Figuring out what headers you need to pass in a call to Facebook, or what certificates you need to access some third-party API, or what data to send over USB to activate some device. Unit tests / mocks let you blithely ignore all those things. You mock them out, hide behind an interface, write your "application code" that uses these interfaces to do whatever your application does, make unit tests with mocks that behave the way you'd like, and viola your app is done! With almost 100% code coverage even! And it's even fairly well-designed with fairly low coupling. Except, it doesn't work at all, the hard work is still ahead of you, and your interfaces are all probably leaky abstractions that you're going to have to change substantially to make it work for real.
Anyway that's the hole I've dug for my current project. I'm pretty quickly coming to the conclusion that I need to unlearn quite a bit, retrain my instincts. I like your posts. Not even much for the content, but mainly for the concept. Rather than cargo culting onto "unit-testing-in-framework-X-to-achieve-100%-code-coverage-because-that's-how-you-make-sure-your-app-works,-right?", it's more of an intentional approach. Step 1: determine what we really need to make sure of, step 2: determine the best way to achieve that.
Ultimately, what I think is wrong with TDD as most people know it, in a word, is that it's a shortcut. It's easy. You populate your mocks with fake data and it's infinitely repeatable so yay. Populating an actual database with fake data and making sure it's deleted/refreshed/whatever between tests is difficult. But that doesn't mean it's not the right thing to do.
I would response here that you need to consider your facade design pattern. You should have a facade that you test your code with by careful mocks and fakes. Then the facade translates between your code (which now works) and the API. Particularity if the third party is infamous for breaking changes all the time you want to ensure that when things break you only have to figure out what they changed and fix the glue code.
Of course if all you have is glue code (this is likely: a lot of real world problems are just getting data from one system to another) there is nothing to test.
Thanks for this. TDD against coarse interfaces as you put it is the correct approach. Sometimes called end to end tests. The use of TDD at the class level, with mocking hell, is the cause of most of TFA's issues.
TDD has worked well for me exactly once: porting a library from to Python to C. I had a very clear idea of what every function was supposed to do and I could write tests first. Due to the nature of the library I was able to write tests that generated lots of random inputs and check the properties of the outputs. This was a great experience --- it was very easy to change the internals without fear of breaking something. Ordinarily writing C is a bit of white-knuckle experience, but this made it quite pleasant.
TDD never worked for me, I believe because of the nature of my work: research code, very explorative in nature. I do not now in advance how the interface will turn up, so it's hard to anticipate the interface in my tests (or it leads to a lot of wasted).
Nowadays I mostly test with "redundant random generation testing": generate random but coherent input, run logic, then... Either I can reverse the logic, and I do that and verify that I get back the original input. Or I can't and then I simply write a second implementation (as simple as possible, usually extremely inefficient). This finds bugs that classical unit and integration testing would never find.
My former lab tried to use TDD. I've come to the impression that it's not the right approach for science / prototyping and just bogs you down. I don't think that science should eschew testing completely- I just think that something like your "redundant random generation testing" is more in line with what needs to go on. The trickier issue is changing your lab culture so that it actually understands the need to run tests of some kind consistently / with discipline.
This is funny, because as a consultant, I think to myself "TDD really doesn't work for me, because I've got so many 3rd-party dependencies that I have to mock out in my tests (which often makes unit tests seem more like a test of your mocks than of your actual application); TDD must really be best for researchy things where they don't have to deal with those problems".
Why are you mocking those 3rd party things? I find in most cases I can test with the 3rd party thing in an integration style test. When the data I'm testing is trivial those 3rd party things can give me an answer in less than a millisecond and so my whole test is fast enough. I find that even when I need to do things like database query that setting up a database with fake data is enough that mocking the database isn't worth it.
As a bonus if there is a bug in the 3rd party code I'll find it before we go to production. My boss doesn't want me to point fingers and some 3rd party code when we are losing millions of dollars because of a software bug: he doesn't want to lose those millions of dollars in the first place!
When you look at the TDD evangelists, all of them share something: they are all very good – probably even great – at design and refactoring. They see issues in existing code and they know how to transform the code so it doesn’t have those issues, and specifically, they know how to separate concerns and reduce coupling.
I think one of the selling points of TDD, and something I hoped for from TDD, was that causation went the other way, and writing tests would result in code being refactored to separate concerns and reduce coupling. Sadly, I've seen that it is possible to write code that is highly testable but is still a confused mess. What's more, TDD as promoted encourages people to confuse the two, resulting in testability being used as a reliable indicator of good design, which produces poor results because it's much easier to make code testable than it is to make it well-designed.
I've also seen people mangle well-factored but untestable code in the process of writing tests, which can be a tragedy when dealing with a legacy codebase that was written with insufficient testing but is otherwise well-designed. A legacy codebase should always be approached with respect until you learn to inhabit the mental space of the people who wrote it (always an incomplete process yet very important), but TDD encourages people to treat untested code as crap and start randomly hacking it up as if it were impossible to make it worse.
This unfortunate (and lazy) habit of treating testability as identical with good design is not a mistake that good TDD practitioners would make, but I think they did make a mistake in the understanding of their process. My guess is that when those people invested effort into refactoring their code for testability, they were improving the design at the same time, as a side effect of the time and attention invested combined with their natural tendency to recognize and value good design. They misunderstood that process and gave too much credit to the pursuit of testability as naturally leading to better design.
I do think the idea of TDD is not entirely bankrupt, because the value of writing tests is more than just the value of having the tests afterwards, but I think its value is overblown, and people who believe in the magical effect of TDD end up having blind confidence in the quality of their code.
> I've also seen people mangle well-factored but untestable code in the process of writing tests, which can be a tragedy when dealing with a legacy codebase that was written with insufficient testing but is otherwise well-designed.
Have you read Michael Feathers' Working Effectively with Legacy Code? [0]
In his definition of legacy code it is any such code that has no test coverage. It's a black box. There are errors in it somewhere. It works for some inputs. However you cannot quantify either of those things just by "inhabiting the mind of the original developers." The only way to work effectively with that code base in order to extend it, maintain it, or modify it is to bring it under test.
This is far more difficult than it sounds with legacy code than with greenfield TDD for the aforementioned reasons: there are unquantified errors and underspecified behaviours. You can't possibly do it in one sweeping effort and so the strategy is to accept that tests are useful and to add them with each change, first, before making that change and using the test to prove the change is correct.
Slowly, over time, your legacy code base surfaces little islands of well tested code.
You have to be deliberate and careful. You have to think about what you're doing.
This is a much different experience than writing greenfield code. TDD is effortless and drives you towards the answer in this case.
Yeah, I've read it, I know the definition, and I understand the intention behind adding tests to legacy code. Unfortunately, TDD (meaning TDD as I've encountered it in print and in practice) encourages people to think that "no tests" is tantamount to "no value to preserve" and therefore "no risk of harm from refactoring." Maybe that's an exaggeration, but certainly some TDD practitioners think that they can't possibly harm a codebase by adding tests. Unfortunately, testing requires refactoring, refactoring is redesigning, and if you don't understand the design you're modifying, your changes can make the code less understandable, not more. Tests added after the fact impose the test-writer's understanding of how the system works, which results in chaos if their understanding isn't compatible with the understanding embedded in the existing design.
Also, in spite of that definition, there's a lot more to "legacy" than not having tests. Not having access to the original designer and not having access to the the requirements or domain understanding that influenced development are important handicaps of legacy code that are entirely separate from tests. Certainly tests added during development can capture some of this knowledge, but adding tests after the fact does not automatically recreate it.
These are both examples of the danger of trying to elevate one aspect of software development to a primary and sufficient role. "Take care of X and everything else will work out" has no known solution for software development, and any methodology is harmful to the extent that it encourages people to think that way.
Your writeup mirrors my personal experience, which was why I was such an advocate for so long. But it doesn't mirror my team experience at all.
I bought Feather's book when it first came out and liked it so much that I bought copies for a whole team out of my own pocket. And I talked about and taught the techniques to the team members.
I had high hopes, and a good team, but the results that we got were not what I hoped for. The result we got was a lot of small tested functions throughout the codebase ("sprouts" in Feather's nomenclature) - which I expected - but I didn't see any evidence of coalescing of those tests into something better - and by themselves, the sprouts made the codebase worse.
I ran into a number of cases where I would pair with developers who were looking to do something better than sprout - so they could fix a whole class or area to be better - and the best answer I could give them is, "give me a morning and I can probably come up with a good solution", which doesn't really help. Some code is just aggressively hard to test.
And I should mention that this was pretty much the perfect environment to try this; I had a team without shipping pressure and a mandate from above to write better unit tests and sufficient time to train and pair with developers.
The approach we settled on was different. We spent our time looking for "targets of opportunity"; the problematic classes or groups of classes that were leading to real issues - bug farms, hard to modify, that sort of thing - and we took a targeted approach to replace them. The replacements were generally written using TDD.
There are at least several other advantages to TDD the article misses:
* Faster development feedback loop by minimizing manual interactions with the system
* The tests are an executable to-do list that guides development, helping you stay focused and reminding you what the next step is
* Provides a record of the experimentation taken to accomplish a goal, which is especially useful when multiple developers collaborate on work-in-progress
Except in order to have a checklist you have to have your design and implementation up front which is out of reach for most developers.
If were testing high level results in multiple cases thats cool. But a lot of the time youre going to try different approaches that may modify or involve changing your output.
No you don't. You just need to know what the unit you're currently working on does. And if you're working on it, you should know what it does. If you don't, that's a sign that maybe you need to have a conversation with someone.
I was summarizing the parent. Although, a few other comments here better put my point, that TDD and unit testing in general, freeze your interfaces. Because interfaces define the functionality of modules, this forces your choice of modularity upfront.
This is fine, if your already have a fair idea of how to architect your problem. But if you don't, it makes it harder to refactor your interfaces (in the same way that clients depending on your interfaces does). Of course, you can rewrite the tests, since you have control over them; it's just harder. And you don't have the TDD raison d'etre reassurance of working tests for this refactoring, of interfaces. Higher level function tests, yes; unit tests, no.
> The tests are an executable to-do list that guides development
So what generates the to-do list when you're building out the tests (arguably the harder of the two tasks, when the tests drive the design).
> Provides a record of the experimentation taken to accomplish a goal
If you're doing a proper refactor cycle, those tests will vanish (or change) as the code evolves. You're back to relying on your source control, which provides the benefit with or without tests.
There are benefits to TDD; these two are not among them IMO.
> So what generates the to-do list when you're building out the tests (arguably the harder of the two tasks, when the tests drive the design).
I don't necessarily use tests as a todo list like gkop, but for me at least... my first tests usually come from acceptance criteria if available, or some sort of business requirements. They usually end up being my feature or integration tests.
Indeed, as long as there are people caring enough to seek out ways to improve their product/craft, there will be an ample supply of silver bullet merchants to satisfy the demand.
Predictably with discussions along the same fault lines too (works for me, hate being forced to use it, you are doing it wrong etc.)
This is the top-level comment that resonated most with the point I wanted to make.
TDD is good for training you to recognize untestable code. "Oh, yeah, that's a Law of Demeter thing... maybe we should just ask for the narrow object rather than the giant one that knows how to get the narrow one," etc.
Once you have that skill ingrained, it's reasonable to stray from strict TDD and simply remember to defend your code with adequate tests.
tl;dr: TDD is not, on its own, effective. If code is highly coupled, tests become highly coupled and a nightmare. The author advocates that learning to properly refactor and create low-coupled code should be a first priority ahead of following TDD blindly.
Very true. Even Robert Martin himself says that TDD without refactoring is worse than TDD (in one of his books). He mentions that if refactoring is not done you will be worse off than had you not done tests at all. He even recommends simply deleting all the tests.
It doesn't necessarily lead you there at all. "Dependency injection" is a fancy phrase for "pass stuff a function needs into the function", only it does so by adding state to objects where that state probably shouldn't exist. You don't have to faff around with IoC or MVVM to properly isolate dependencies; indeed, it's the core idea behind something like Gary Bernhardt's "Functional Core, Imperative Shell" (which, if you're not understanding this instinctively, makes me fear for your code):
State is a desirable thing in the code that wraps your business logic, for sure! Otherwise you're not actually doing anything. That's distinct from state in your business logic, though.
That's interesting although after a brief look it just looks like he is splitting the code into two parts, one part he tests and the other part he doesn't. I don't really disagree with this, my UI code has very few tests.
"which, if you're not understanding this instinctively, makes me fear for your code"
Sorry--that was a generic "you" there, not you-specifically. I'm saying that this is something everybody needs to understand because writing state-munging code is intrinsically harder to do correctly.
He's not just "splitting the code into two parts", though. He is consciously and specifically defining a core based on functional principles that avoid mutating state (which is harder to measure and reason about). That UI layer, his "imperative shell", is where state is managed, rather than embedding side effects into his business logic. (If you watch the Boundaries talk you'll see he frames larger applications as sets of functional cores surrounded by a layer of imperative routing between them.)
"Functional core, imperative shell" sounds quite dogmatic and not "getting it" is no reason to fear for anyone's code.
Just like the OP article argued that TDD can become an all-solving hammer in the eyes of people, so can functional programming. In this particular case I can think of a whole host of applications where the core should definitely not be functional (and, for what it's worth, immutable state is not inherent to functional programming, or at least it didn't use to be). If it were so, ORMs wouldn't be a thing.
I can't think of a single case outside of a systems or game development context (and a number even there) where anything I have written was improved by tightly coupling state mutation to business logic. Not one. Rather than trying to invoke "dogma" so very seriously, feel free to suggest one. But it's a really long row to hoe: if you are conflating state mutation and business logic, you have entire classes of errors that clean separation doesn't and it's harder for you to meaningfully test for correctness. Not firing that gun into your foot seems very obvious.
And functional programming has never encouraged mutation of data model objects. Not at any point. Mutation of data model objects is and can be nothing but a side effect of a function. There are cases where you can't not mutate a model, such as when that refers to a system resource (a console, a socket, etc.); those are not your data model. Those are dependencies that your application uses to create instances of your data model and feed them through a set of functional processes. Almost like there's an imperative shell that deals with and controls side effects, wrapped around a core of pure functions that manage stateless transforms based on your business logic...?
ORMs (in the mutating, own-your-object sense like an Active Record model) shouldn't be a thing because they're awful creations that aggressively they encourage bad, muddy code that's harder to test and trace and debug. Once more for emphasis: the second you mutate a data object you have made testing an order of magnitude more difficult and now you have to live with that forever. Active Record requires this. There are places where you may choose to embrace this, but it's a good way to shoot yourself in the foot.
On the other hand, data mappers (Anorm, DataMapper, Slick) live outside of the cleanly tested, no-dependency core (and its data models) and are used to feed objects into your business logic and record the results back to an external data store.
If you aren't using plain old objects to encapsulate your data, you're asking for it.
You know what? I had prepared a longer answer addressing most of your opinions here, but I'd rather not[1] so I'll just say I disagree and leave it at that.
[1] After all, I can't conclusively prove an imperative codebase is more maintainable anymore than anyone can do it for a functional one so this would just be another entry in a 50-year-old flamewar.
You might not be "conclusive", but a cogent argument like "I'm adding classes of bugs that don't exist otherwise but I get X for doing so" should be pretty straightforward. If it exists. My contention is that it doesn't, but I'm always willing to change my mind. I'm pretty emphatic about this because I've spent twenty years learning that every other approach in common practice will eventually screw you catastrophically.
> If code is highly coupled, tests become highly coupled and a nightmare.
TDD is supposed to prevent that. What's supposed to happen is, if the test is hard to write, it's telling you something. You're supposed to pay attention to that pain. It's supposed to show you that your code is highly coupled, and make you fix that.
If you just go ahead and write the test, difficult as it is, instead of finding a way to refactor to clean up the coupling, then yes, TDD produces a nightmare of a test. That's because you're not doing it right, though.
(Yes, I recognize that this is a lot like a No True Scotsman. But in fact any methodology, practiced badly, will produce sub-optimal results...)
TDD takes discipline, planning, and creates stability; I think in the day and age of "rewrite it using the latest framework" it just doesn't coincide with the insatiable thirst developers to use the latest bleeding edge XYZ.
Another reason for the failure of TDD is the fact you need to be a very good programmer to be productive with it. Indeed, it requires you to be able to think your general API and architecture ahead.
Junior and 9-to-5 programmers suck at this. They are much better at tinkering until it forms a whole, then shape that into something that look decent and works well enough.
And we live in a world where they represent a good part of the work force.
You can't expect everyone to be a passionate dev with 10 years of experience and skilled in code ergonomics and architecture design while being good at expressing him/herself. That's delusional. And armful.
But you need to know what the working API must look like. Creating an API is a much harder task than you think. You are used to it so it seems natural to you, but I assure you I keep giving training in IT months after months, and most people are not very good at it.
HN is full of 10x devs and they live in an expert bubble.
I develop software for radio towers. This was a very confusing headline and article. I only figured out halfway through that I was thinking of the wrong TDD.
Test driven development is one thing. Time division duplexing is very different. I'll have you know that the latter did in fact live up to its expectations!
Showerthought: I wonder if our TDD codebase is TDD?
Sorry for the aside but I find it humorous that the headline that should read "TDD--, Refactoring++" instead shows "TDD—, Refactoring++".
This is emblematic of that frustrating AutoFormat behavior that replaces double dashes with an emdash. Probably not a coincidence that this appears on MSDN -- perhaps it was drafted on Outlook or Office or some other tool w/this same AutoFormat.
This feature is responsible for countless miscommunications between colleagues à la "I copied and pasted your command just as you had it in the email"...
Having done TDD mostly full-time for well over a decade, I have to agree, and it hits home with some past experiences.
You can end up with fully tested, TDD'd code, that is not well-designed - i.e. unnecessarily coupled, and not cohesive. Cohesion and coupling are the basis of most everything in good software design - e.g. all the letters in SOLID boil down to those two things.
The premise of TDD is that it's supposed to make that too painful to do to a damaging extent. But, if you keep ignoring the pain, perhaps in the name of a "spike" solution, or because you just don't have the experience or background to know what good design is, you will end up with a tested mess of spaghetti.
And that's even harder to untangle and refactor than untested code, because you have to figure out which tests are useful, useless, or missing. That just slows you down as you work towards a better design.
In these situations, scrapping the whole module, including tests, and starting over, is sometimes faster in the long run that trying to refactor incrementally with the safety net of existing tests (another of the main values of TDD).
I appreciate that many of you (including the article author) are coming at this question with a lot of experience. I, however, knew very little about coding a year ago and learned with TDD as part of how I build almost every project. Although I think it's always the case that I might "be doing it wrong", it's hard for me to imagine now writing code without first writing tests. Part of this is, admittedly, that I'm still uncertain if my solutions or code will even work and writing tests helps me to both organize what I want to do and also verify that I haven't made silly syntax mistakes.
Was it harder to learn this way? Absolutely (at least I think so, but my sample size is 1). I can't tell you the state of despondency I was sometimes in learning test suites and trying to figure out how to test certain things all while knowing that I could just write the stupid code and inspect the output to see if it was right.
Also, I love to refactor. How do you refactor if you don't have tests to catch you when you break something?
Not doing TDD does not mean not having tests, it just means writing them after writing the feature (this has been the usual way for a long time before TDD, although automated testing somewhat got very popular only a few years before TDD).
I too became a big fan of TDD when I was introduced to it by rspec, then a brand new testing tool. But those latest years, I grew tired of keeping fixing my tests despite having no regression in my codebase - just because implementation changed. I quit doing TDD, but mainly because I quit doing unit testing.
Nowadays, I only do integration testing. I've made a chrome devtool extension to generate capybara tests while I'm browsing, so that I can write my feature, then generate a test for it while doing visual QA in literally 5 minutes (this was something selenium IDE was doing in its time, I just made something a bit more modern to replace it). And since this is integration testing, my test won't break as often if I change the underlying implementation to do the same thing. I would even argue it makes my refactoring more pleasant : if I refactor correctly and end result is not altered, I don't even have to edit my tests (if it breaks, it may actually be a regression).
That being said, what matters is what makes you more productive. Don't take our word for it, it may not apply to you. Just keep wondering how you could make things better.
You don't refactor if you don't have tests, or rather you do and you break stuff and take on a lot of risk. The reality is a lot of developers aren't working in teams with others and working on software that has a long future ahead of it, hacker news tends towards the smaller teams with short runways and its write it and throw it away as a strategy rather than refactoring.
As far as the beginner angle, it makes sense to me to stick to TDD when learning to code. There are already so many things to learn and understand, that sticking to one way of working would likely be helpful. The question for me would be whether it matters whether that method is TDD or something else; I suspect it wouldn't matter much.
Contract-driven development is the best model for building stable and reusable systems at scale. The flaw in TDD is that it tries to make the tests be the contract instead of supporting the contract.
If anything I think this article makes a great case for TDD. If your developers aren't good at design and refactoring and that is showing up in your tests, that is an indication that your design needs to be refactored to be less coupled. TDD isn't a panacea, developers have to have some level of sense and see the signs of a less than optimal design. Pain in test creation is a great way of showing that as it simulates client code.
I also don't understand people thinking that you have to write the entire test suite up front. You build your test along with your code. You start simply and build up, this way if you don't have concrete specs your tests are helping you with the design by thinking about consumption as well as implementation.
> Never got into it and always thought it is a complete waste of time. I advocate to write tests only for critical algorithmic calculations and nothing else.
What great advice! While we're at it let's advocate surgeons didn't use sterile instruments cos it's a waste of time.. well maybe only for heart transplants.
Yeah, I bet you are one of those who also test your repositories and DB connections while mocking the DB response essentially testing database operations without even having an actual database in the back-end.
I disagree with the assertion that TDD takes more time. TDD takes less time if you factor in the reduction of errors TDD helps prevent.
This article should be renamed, “TDD doesn’t work if you don’t do it right.”
This article seems to argue for Big Design Up Front. Ok, if you do that, then why not write the tests for those designs after you make the design – then the code you write confirms to the design.
I don’t think anyone advocates that writing tests is the same as the design process. Tests are the result of design not the other way around. The gray area is not design up front – but how much design up front.
I get your point, but it's not really about not doing TDD right. It's about TDD evangelists - and I used to be one of those - advocating that people should use TDD despite those people not having the skills to do TDD "right". And my experience is that not only is doing TDD without those skills not a fun experience, the resulting code is worse than if you let those same developers write the code the way they are used to, and maybe spend some teaching time on refactoring because it's a more basic skill.
I think people should in fact test data, not code. Looking at it from purely functional point of view, functions should be either proved that they do what they should, or they should be asserted and QuickCheck-ed. But what really needs to be tested is that the input parameters (i.e. data) conform to some "hidden" assumptions that we had when we wrote the functions. Because as we modify the program, or even why we modify it, is that these assumptions have changed.
I think testing itself is the second-best thing short of proving correctness. It won't guarantee that your code is correct but ideally it greatly helps reducing bugs. TDD, as far as I understand, seems to promise more than just the benefits of testing. It promises an emergent design that produces a solution to your problem. I think this works well for some problem domains, like a lot of web/CRUD/LOB apps, and not so well for others, eg. the Sudoku solver mentioned in this thread. On the other hand a lot of real world problems can be successfully solved by solutions that are adequate but not optimal, ie. good enough solutions and TDD seems to be a viable strategy for these. I personally yearn working on problems where TDD based emergent design is not enough and human ingenuity/intelligence/creativeness is needed. Sometimes I bump into these but at the same time I realise that most of my day-to-day job involves problems that is solvable by TDD alone. That said, while all my production code has extensive tests, probably less than 50% has a test driven design and I'm content with that.
"The tests get in the way. Because my design does not have low coupling, I end up with tests that also do not have low coupling. This means that if I change the behavior of how class <x> works, I often have to fix tests for other classes.
Because I don’t have low coupling, I need to use mocks or other tests doubles often. Tests are good to the extent that the tests use the code in precisely the same way the real system uses the code. As soon as I introduce mocks, I now have a test that only works as long as that mock faithfully matches the behavior of the real system. If I have lots of mocks – and since I don’t have low coupling, I need lots of mocks – then I’m going to have cases where the behavior does not match. This will either show up as a broken test, or a missed regression."
Simply comment them out temporarily?
"Design on the fly is a learned skill. If you don’t have the refactoring skills to drive it, it is possible that the design you reach through TDD is going to be worse than if you spent 15 minutes doing up-front design."
I don't quite understand how TDD means you skip up-front design.
His argument is basically that with tight coupling, TDD is too hard and time consuming to pay off.
But part of the point of TDD is ensure that all code is testable, and testable means loosely coupled.
So you can't start TDD'ing on a bad and tightly coupled legacy codebase. You can do it on a greenfield project however. Greenfield is very much the "lab environment" he talks about. You control everything.
With greenfield projects comes another reality though: you often have to explore and sketch a lot. TDD does not work well for writing a dozen sketch solutions to something and throwing out eleven.
And that to me is the main drawback of TDD: it works poorly for very young code bases and it works poorly for very old ones (that weren't loosely coupled to begin with). It's a very narrow window where you can start using TDD in a codebase and that's when the architecture is first set, but the codebase hasn't yet grown too coupled. Such a narrow window means it's not very popular, for good reason.
I'm not really convinced by this article or by the comments.
You can do exploratory code and TDD at the same time - you just have to write down what you expect the code to do first.
These criticisms of TDD are very weak because they don't spell out an alternative - every critic's vision of proper testing is different - and will respond "that's not what I meant".
I hope nobody uses this article as ammo against TDD. The benefits are not felt immediately but when time comes for maintenance/updates, I'm working on my second port with a company. The first app had fantastic testing and I was confident in the work I delivered. This second app however was led by a developer who "needed to get things done" and I now have to wrap the v1 app in functional tests to validate that I'm delivering a solid port. If the company had enforced better practices sooner, they would have saved the time I'm spending on retesting the original app. This second iteration is test driven, hopefully the next dev has a better experience.
Also testing helps alleviate QA's workload by ensuring developers have not broken any tests and regressed functionality before we hand off to QA.
If you're hacking on an idea or learning, I can understand not testing, but if someone is paying to deliver code, deliver it with tests, period.
This logic is flawed, and I'm not surprised it's coming from Microsoft. If the expectation is that (repeatedly?) making blind guesses quickly and (optionally) cleaning up the mess later is better than expressing an understanding of the problem domain before writing 'real' code, then yes TDD will not live up to those expectations.
Note the site is http://blogs.msdn.microsoft.com. Any MS employee can start a blog here, and none of the content posted by an employee goes through a vetting process.
That being said, the logic is not really flawed at all. There will always be the real scottsman fallacy to deal with (what is real code? When does code become real?). TDD promotes writing tests first, whereas you might not even know what the spec is until you've gone through multiple iterations of the code, and along the way you'll have to refactor and rewrite to reduce coupling and reflect better understanding of the spec. Tests aren't bad, but writing the tests at the beginning doesn't really make sense in that context.
TDD supports the paradigm of Software Engineering as an Engineering field. Design, plan, build, test.
Chartered Engineers have qualifications for their skills to do this - whether it's building bridges, designing circuits, or making cars.
Most programming is not Engineering. It's scripting. Hacking together a quick solution to meet the user's immediate needs.
Huge businesses (including the company I work for) have some really weak points in their production flow. They're planning factory operations using some shoddy macros in Microsoft Excel thrown together by some businessperson with no programming experience. Management won't change it, because "it works".
Other fields of Engineering (civil, electronic, mechanical) have serious life-threatening consequences if they fail. Software rarely has that risk. (Insert comment about healthcare systems and WannaCry here).
For times when software carries serious risk, then TDD is still important! The rest of the time, it's a burden.
Sometimes lowering your expectations are a good thing for everyone. Now we know the pros and cons and can use it appropriately. No one particular habit is going to solve all your problems.
Would it still be TDD when talking about functional tests? So no mocking. Or does strict TDD definition only include unit tests.
Because the general principle of writing a test case and then writing/editing code still applies with functional / integration tests.
And I have always preferred to use functional tests to test bigger components / packages based on their public interface then to write a unit test every small function inside the package.
Unit tests seem to be much more useful in situations when you know exactly what should your inputs and outputs be, for example if you are writing a function to transform data from one type/object to another. This is where unit testing shines.
But a lot development usually involves integrating / gluing together several higher level components together and passing data between these components and I much prefer functional tests there.
One thing I've noticed. TDD takes longer. It just does. You can argue that you are racking up less technical debt in the long term but every consulting gig I've been on where TDD was the "directive" often deteriorated because the business does not want to factor in between 40 and 100% extra time to allow proper TDD coding. They want the same somewhat arbitrary and bonus driven deadlines that they always do, and in order to meet them, we usually end up tossing out TDD halfway through and reverting to just having skilled developers get the job done as quickly as possible.
THIS is the economic reality of TDD failing. A manager wanting to reduce quarterly spend so he gets his bonus doesnt care that TDD will cost him less over 5 years, he cares that he can get a project delivered on time and under budget...
It's not just you, there's ample evidence[1][2] that TDD is a tradeoff between delivery time and quality. I think most experienced practitioners would agree this makes sense: when the schedule slips, you can cut testing but you can't cut implementation. Works at all beats trumps works in all cases.
In the TDD model, you make that decision up front, as well as the investment in tests. When the schedule slips in development, there's nowhere to cut but the most painful part, so you don't. The schedule slips.
To take a contrarian position, TDD fails because it doesn't allow management to revisit decisions as reality deviates from the plan N months ago. This seems like a surprisingly non-agile approach at the high level.
It always startles me when people assume that one programming technique either works for every type of programming or doesn't work for every type.
Working on Perl 6 compilers, an extensive set of tests was our best friend. It was (probably still is, I haven't had time to help the last few years) utterly routine to write tests first and then write the code to make them work. It was a perfect way of working on it.
On the other hand, one of my personal projects in Perl 6 is abc2ly. It has lots of low level unit tests, great. But almost everything really interesting the program does is really hard to test programmatically. How would I write tests to make sure the sheet music PDF generated has all the correct notes and looks nice? That problem is significantly harder than generating the sheet music in the first place!
It's interesting to me that programming is becoming a fundamental part of many (maybe most) things humans do. And yet we treat it as if there was one single correct way of programming and all other ways are wrong.
I feel like it's much more akin to human language. Though the language primatives might be the same, it's used differently in different contexts are appropriate.
In formal, legal speech, you might want to be explicit.
In informal, everyday speech, not so much.
Similar analogs exist in programming. 10 time script to reformat data for one time use, careful documentation, tests etc. not really required.
Aircraft flight control system, very different...
But then why is it that so many programmers seem to focus on "the one true way" (TDD, agile etc). Beats me.
I think TDD works great if you have a very clearly designed input or interface. A programming language falls under this category: here's a piece of code, make sure it compiles and returns the correct output is a perfect test. And one which you can defined beforehand and is achievable to write as a test.
However, programmers are often not in such a luxurious position. Business requirements are unclear, it is unclear what the solution even should look like, and the requirements will change over time and are changed based on the software's progress. In this case TDD only slows you down. It's also why short iteration times (i.e. agile) seem work relatively well in software development.
It's exactly the opposite. TDD was a put forward as a solution to unclear, changing business requirements. It allows you to adjust course very quickly when the business realizes its requirements were off and new set of requirements are needed.
Robert Martin has even mentioned that it's a waste of time if you have a full 100% specification available to you, or when you already know what the final solution is and simply need to implement it.
I'm not an expert on TDD, but how does TDD help in this situation?
If your requirements change, your existing tests and code become (partly) useless. This would indicate the most efficient way is the exact opposite where you test minimally until the requirements have stabilized.
The goal of TDD is not to have tests, or a huge test coverage. Those are just important side-effects. One goal, ignoring how it improves focus and divides complexity up nicely, is to make cleaner code. If your code respects the single-responsibility-principle, is not duplicated, and doesn't have dependency issues, you can adjust the requirements faster. The test suite allows you to be confident your changes to some parts of the system don't break others.
You can keep more code, not even having to touch many parts of the system. You know that by changing x, you don't also have to change w, y and z. Of course code still must change, but its vastly simpler and quicker to change.
Now, that doesn't mean you can't have those benefits without TDD, it's just much easier and clear.
Because there is only so much talking is going to do, at some point you just need to build a prototype and see if it works and how it can be improved.
If pre-planning, talking and requirements solved this problem we wouldn't see failed (software) projects and companies. As a matter of fact, extensive requirement gathering before prototyping and iterating is a good way to fail a project as at some point you're just adding new towers to your castle in the air.
Yeah, but if you're prototyping, then that shouldn't be the actual code you use. You write a prototype to test your assumptions, have the people bang on it for a bit, then throw it away. You take the lessons you've learned from that, and now you have more of a "spec" than you did before. Thus, you can write your tests.
People have a constrained view. They see their own problems and have seen the "old" way do nothing but fail and the "new" way do nothing but succeed. What other reaction would you expect?
If they actively researched and looked beyond their personal experience then dramatic 0% to 100% Success rates would turn into more reasonable assessments.
I think my view on TDD is more reasonable than most of the zealots, but I still like mant of the ideas of TDD a great deal. I don't think it much matters if the tests come before or after the code as long as they come early enough to save time and effort. This means that specification of some kind is required, without specification you can't do TDD. Once you have tests and some code, you can add more test to increase confidence and with confidence you can aggressively refactor.
Sometimes its enough to write a minimal viable product and then write tests just so that refactoring for version 2 can be done reliably. I am doing something like this now for some C++ build system stuff... That sounds odd saying it loud, but yes I have cmake modules complex to need tests so I can be confident in refactoring them. But I didn't write any tests until the minimal viable version sort of work enough to have bugs filed against it. Now TDD on ethat project seems like great idea, when I didn't use any tests in the beginning (Partly because we couldn't specify what we wanted).
" have seen the "old" way do nothing but fail and the "new" way do nothing but succeed."
When I look at TDD I see a lot of success but also a lot of fail once people start pushing it too far. To me a fail is when the mock becomes more complex than the real system or a system can't be refactored because the refactoring of the tests takes too long or the tests are just so complex nobody wants to touch them.
This reminds me a little of the situation with OOP. It solved a lot of real problems but then people said "only objects" and you ended up with stuff like replacing a simple switch statement and 100 easy to read lines of code with command patterns that had the implementation spread out over 10 different files.
I think you are totally correct both in your view that some take goods things too far and that this isn't the first time this pattern has been repeated.
I wonder what things I am expounding now that I will pull back on in the future?
This looks more like "Misunderstood TDD did not live up to expectations." It's obvious from the article that tests were written after the code. That explains the excessive coupling in the code and tests being hard to write and constantly getting in the way of code development. This is not "test driven development". This is bolting tests on top of already (poorly) designed and implemented code. Tests are an afterthought. They did not drive the design. No wonder it doesn't work well. The high coupling in the code has to be repeated in tests and it's a rather painful and fruitless exercise. Have tests been written first, it would have been clear which design led to lower coupling: the one it's easier to write tests for.
It’s important to not assume that tests and code will be written by the same person.
When tests are being created early, it’s actually a good excuse to have at least a couple of minds looking at the same problem, instead of bottling it up into one person who ends up quitting next month. It’s an excuse to not just discuss the approach to use but have some code, where each person may realize that they hadn’t really thought about the whole problem or maybe didn’t understand it at all.
Other criticisms in this thread are still fair. It is certainly possible to waste a lot of time on tests for instance, and to build something that is too restrictive. Ultimately though, if you’re more than a one-person project, some form of “sketch it out first” is a good thing.
I think it all depends on the context of what you're writing. For example, we use TDD when making additions and modifications to a tax engine. For this use case it's incredibly useful as the relative inputs and outputs are both predictable as well as repeatable.
TDD is a nice idea, however TDD can add quite a bit of overhead. Upfront overhead, before writing any code. This isn't a problem if it is justified/ needed. Plus the tests can become more cumbersome when code changes, more baggage to carry.
Testing often becomes a KPI, and therefor is commonly gamed. Doing the bare minimum to tick '100% code converge!'. I'm a fan of contracts to test the spec and boundaries(whether human or other application) of software.
TDD requires discipline and experience; You could spend an infinite amount of time writing tests and never deliver, or the other extreme becoming incapacitated fighting bugs.
Our first priority should be crash-free software, THEN start to think about making it bug free.
For me TDD is mainly three things.
Firstly it’s about testing myself so I’ve understood the task properly before writing any code at all. In this step I also can tell if my code is doing too much and therefore can tell if my method or function conforms to the “single responsibility” rule.
Secondly it’s about maintainability and logical reason. A codebase where I don’t know what it’s supposed to do, or forgot about it, I can always rely on the tests to skip the parts that’s not interesting making me to move faster.
Thirdly it’s about the ability to refactor and therefore evolving design. Even if this is a step in TDD you will always need to refactor since requirements changes over time. The solution you had is not the optimal solution anymore and therefore you must refactor anyway. Evolving design is a strength where you continuously strengthen your code while getting work done faster and faster since you can offload the reasoning on the tests, covering your back.
AFAIK TDD is actually the only way to produce code in a systematic way. When reading TDD driven code I can make certain assumptions which I cannot do with randomly produced code (there’s no system to the code). TDD code is developed automatically with tests in mind and are always a lot easier to test when you need to do it, and you always need to do it. (I would argue that there is possible code which is not testable as is, unless you refactor and then you don’t know what the code is doing the same thing as it did before).
If your tests get coupled with the code, I’d say it’s because either your method/function is doing to much or it’s a language problem, not giving you the necessary tools to ignore implementation details, which mocks usually are an indication of.
Since TDD is a systematic way of producing code (at least more systematic than not doing it), code which isn’t produced with TDD will not play well with TDD produced code since it won’t follow the same conventions, designs and possibilities.
TDD doesn’t automatically make the code more bug free, but I don’t believe that TDD cause more bugs just because you use TDD.
If programmers cannot learn or deal with TDD, you have a different problem on your hands.
Failed at what exactly? Who would think 1 methodology would give them all they need to be a great software engineer?
If TDD supposedly failed, can we hear the process that succeeded at what TDD failed? Please extend an olive branch and enlighten the rest of us.
Because I tell yea, I can't even count the amount of "senior software engineers" I've encountered, that deploy untested production code daily to systems that help you guys buy your coffee in the morning, manage your money and pensions. Oh yea, and they all seem to think TDD is bollocks too.
When that percentage decreases and engineers like that become a rare occurrence, then we can talk again. Peace.
"developers are quick to pick up the workflow and can create working code and tests during classes/exercises/katas – and then failing in the real world."
The anathema of TDD is that people equate having well defined test cases to a solid product that brings the solution that the user wants. I have seen far more software projects boasting of >85% of code coversge for tests, but still failing spectacularly.
TDD failed because it was assumed as the magic wand that aligns the end product with what's on spec and the process that it would cover, but the reality of human behavior being merely coded to test cases is far out from reality.
Title should be: "TDD did not live up to MY expectations".
TDD is sort of an art more than a science. You have to know when and how to do it. You also have to know how much as the marginal utility diminishes very fast.
I remember an old Microsoft analysis where they measured empirically time to delivery and actual defects and found TDD to not reduce defects while increasing time to delivery. Can't find it anymore.
The main benefit for TDD in my mind is that it mostly makes it painful to write spaghetti code. When I'm reviewing something that looks too integrated, I just ask for a unit test for that particular piece of functionality, and the author is effectively forced to go back and refactor. After going through this a few times, they learn to think about their design before they write code. Of course, many dynamic languages defeat this by offering hacks like Python's mock.patch which let you nominally test spaghetti code...
Interesting that you mention dynamic languages. I think that a lot of why people do TDD is that you can't reliably know if a program in a dynamic language has basic issues unless you run it.
The same is partially true for manual memory managed languages.
You need to push towards 100% coverage in order to replicate the advantage you get in a statically typed language like Rust.
Yes, the better static analysis you have, the less you need to depend on runtime checks, but Rust makes a lot of tradeoffs which ultimately make it a poor choice for most application development (steep learning curve notwithstanding). There's a whole suite of static functional languages that give you the same strong static guarantees, but without the headaches of borrow checkers, lifetimes, boxes, etc, etc. Beyond that, there is Go which gives you some rudimentary static typing (a 99% improvement over dynamic languages), world-class tooling, a great deploy story, etc. In my mind, Go is the sweet spot for the demands of modern application development.
TDD is about creating tests for your requirements, and then writing code to make them succeed. The reason it doesn't work is because most medium/large feature changes don't fully realize their scope and requirements until they're well along into development, making the earlier tests an unnecessary drag.
Writing unit tests at all is independent of TDD. After you get the feature working, you write those to verify the program does what you think it does and to think through failure scenarios. Most importantly they make sure that future changes do not break the functionality you just implemented. I think asking for unit tests is important for any true bit of functionality in a large system.
Fair enough. The benefit of TDD them in my mind is that it keeps undisciplined developers from wedding themselves to a bunch of untestable code and then getting frustrated when asked to add tests in a code review. Whether or not this is the stated purpose of TDD is a different matter.
Microsofts own study of TDD showed that it definitely improved defect rates and they went fully into developing with it for Vista which is part of the reason for the delays. Nowadays large chunks of the API is automatically tested and this has allowed Microsoft to release changes must more often with a lot less manually testing.
So while this individual is finding his local team isn't getting the full benefits, Microsoft appears to be based on its own report on the technique and its outwards software release cycle changing.
Personally, I found that TDD works very well for small modules / classes, etc, when there's little design to be done. In this case, you can focus on writing down the spec (test cases) and be fairly confident that it works by the time all test cases pass. Also, I agree with the author about complexity involved if one decides to go down the TDD way for large, complex systems. So, essentially, it boils down to picking the right tool for the job, and TDD is just a tool, like any others.
TDD works exceptionally well. The secret sauce is to find the correct scope for tests. In my experience, integration tests are the most suited kind of tests for successful TDD.
Any programming language suitable for business application development is going to have static analysis tools that can reveal your percentage of test coverage.
As long as 95% (or whatever) of your logical branches are covered by tests, I don't really care whether you wrote the tests beforehand or after the fact.
However, TDD being hard is not a justification for not writing the test coverage at some point in the dev cycle. Too many developers, and managers, make that fallacy leap.
> As long as 95% (or whatever) of your logical branches are covered by tests, I don't really care whether you wrote the tests beforehand or after the fact.
Do you care that those tests reflect the business goal or just the implementation? 'Cause TDD is way, way more likely to do the former than test-last development.
What are you talking about? TDD generally deals with unit testing. Which by definition is more fine-grained than end-to-end integration tests against non-mock dependencies.
All of which is unrelated to UAT. Where you show the alpha to the business, and it's only after seeing it live for the first time that they start to get on the same page about what their business goals were to begin with.
TDD totally deals with unit testing. Are your units not reflective of business goals? Mine tend very strongly to be, perhaps because I extract IO and other state crap out into the less functional part of my program (and I don't really unit test that).
When reading the title, I had hoped for data. Like when Microsoft analyzed their own developer's data to find out if remote work impacted productivity or bug counts.
Instead, just another piece of anecdote. Sure, anecdotes from 15 years, still not what I hoped for.
Doesn't Microsoft have hundreds of dev teams, and can compare things like development speed and bug counts, and correlate with whether those teams practice TDD? I'd read that article immediately!
Also, when a test relies on mocks it doesn't test the real thing, and doesn't guarantee proper behavior in the real world. I suppose this is obvious from the nature of mocks. And yet, if you can figure out a clean and fast way to test something without mocks, I think you're better off.
Along with the coupling problem mentioned in the article, these are the two reasons why I am writing a lot fewer mocked tests (e.g. Rspec) than before.
What about a kind of middle ground, doing a 'spike' when figuring out stuff, what kind of thing you should build, what the design should look like, etc then follow up with TDD to stabilize the identified interfaces and produce tests that act as a system health check?
Ok, for consultants, perhaps they end up doing the same kind of things for different clients to the extent that they can just jump in doing things TDD from the very beginning.
That's exactly how I work (web development) and I am really happy with it so far.
First, I am playing with a throw-away code to understand how I want it to be.
Then, I write some tests based on the acquired understanding and then finally working code. I can even copy-paste some parts of the throw-away code if they were correct initially.
I see my designs being damaged by tests sometimes, though but in tolerable amounts, and I am happy to compromise.
I've seen TDD used once really well in a university setting where it was used only for shared libraries/services that could be used by multiple other teams or departments but not on individual (front facing) projects.
Probably because only the best developers on the team worked on the shared services it eliminated the refactoring issue as well as ensuring shared services could be a lot more reliably and safely updated.
TDD should not be taken as a religious dogma. The way I like it is, central business logic as pure functions, which have tonnes of unit test. While integration with other services and components sits on edge which do not have unit tests, instead integration tests. If I have a key piece of code, I wanna test, but would require lots of mocks, it is time to refactor.
Defining the interface before you write the code is the major advantage of test-driven development and what it added to that way of thinking was very valuable especially to novices. It also makes your code more modular and reusable. Writing code as if other people were going to use it is something we don't talk about enough.
I found it just flew in the face of iteration. I get something simple working, then iterate a few times, adding complexity. If you have to iterate your tests as well, you're just walking in treacle.
Either that or you end up with a load of stuff in your interface that you thought you'd need, but when you actually got to it you either didn't, or it wasn't important to implement at the outset.
>The tests get in the way. Because my design does not have low coupling, I end up with tests that also do not have low coupling. This means that if I change the behavior of how class <x> works, I often have to fix tests for other classes.
At this point you should be realizing your code is untestable and needs to be refactored.
Tl, dr: When we treat unit tests as an ends in itself, this leads to writing clunky tests for clunky code. If an engineer doesn't understand modular, reusable code then that engineer can't won't be able to write code that can tested easily. Thus understanding design is a prerequisite to effective TDD.
One of the key benefits for me is the mindset (fostered by TDD) to make as much as possible of the code (unit) testable. This naturally leads to less coupled code, because otherwise it is not possible to test it in isolation. So the fact that you start with the aim of unit-testability leads to better designs.
Tests are good for detecting code that is not working as expected, using them as a investment/insurance, based on a budget. However, in my opinion, TDD is often more about an obsesive-compulsive religion built on wishful thinking on programmers reaching the excellence by writing tests ad nauseam.
Software obeys its own dynamics. Some things work well. Some not quite the same. It's nice to see someone admitting that testing is good, rigid TDD; like rigid Agile or rigid Waterfall, are bad. Corollary, what our parents said still holds true; too much of a good thing is bad.
I thought TDD meant doing tests before implementing code rather simply testing code.
So I always assumed someone testing his code wasn't necessarily TDDing, and that someone not TDDing wasn't necessarily not testing, just that she tests after implementation.
Seems a little arrogant to say most developers don't know how to refactor or do it poorly. Maybe it's true. I really can't say one way or the other because what I see is most devs believe they don't have time to refactor.
Let me ask you - and by "you", I mean everybody reading this - a question.
If you had poor design or refactoring skills, how would you know?
Let's just say you've finished working on some code that involved writing new code. How do you know whether it's well-designed?
Well, you start by using your own sense of what is good design. But all of us can only see the things that we already know about, which us thinking that the code has decent design is only a sign that it is up to our level of understanding. This is a fundamental part of how knowledge works.
Code review can give some additional design feedback, but 1) code reviewers generally don't have time to give in-depth design feedback and 2) "you should redesign this" is generally not well received (see previous point about everybody thinking they are good designers), and I both have to work with my teammates and their comments factor into the kind of review I get. So, code review is not a good way to improve design change, even assuming that you have people on your team who have useful design advice to pass on and you are in the kind of environment that lets you be thoughtful about design.
If you are in a team with decent designers and your team pairs, then you are in luck; you have a good chance to expand your design skills. Leverage this at every opportunity.
Finally, it is possible to spend time (almost always your free time) learning more about design, doing katas, going to code retreats, etc. This also works.
And I'm sorry if this sounds arrogant; I've tried to find other ways of saying it.
As I remember, one Microsoft was one of the original TDD pushers. One person who worked there for over 20 years told me that "the peak TDD" at Microsoft was right around time of Win ME release
As I remember, one Microsoft was one of the original TDD pushers.
You may be mistaken: There was a well-known incident back in 2005 where Microsoft published guidance for TDD that was flat-out wrong (or at least not fitting with the mainstream understanding), and they had to retract it (in fact it was so far-out that they had to expunge the article from MSDN).
One person who worked there for over 20 years told me that "the peak TDD" at Microsoft was right around time of Win ME release
Of course, Microsoft is big, and the broken guidance may not have reflected practices elsewhere in the company, but it seems unlikely that they were early TDD pushers if they hadn't grasped the concept (as commonly understood) by 2005.
Since when did TDD fail? Which is not to say it needs to be applied systematically to everything. But there are often bits of code which are better off being correct, and TDD works well for those.
Off topic: But who's in charge of typography at Microsoft? The typesetting of their blogs and documentation are horrific. And what little I see of Windows looks just as bad.
I'm not sure if the OP is advocating against using tests and ci completely, or just the process of writing tests first and then code... Any one got any thiughts on that?
Stylistic critique of the article: Is it too much to ask that you enumerate your acronym at least once throughout the entire article? The acronym "TDD" appears 16 times throughout the article, and not once do we get "Test-driven development" spelled out.
I get that it's a technical blog, but "TDD" isn't exactly a household name. You can't utter it in the same breath as SSL or RSA and expect people to know what it means without context.
As a test (no pun intended), try reading the article with the premise that you have NO idea what "TDD" stands for. Can you reasonably infer it from the rest of the article?
if requirements are not clearly understood then the tests will not be complete - now does that invalidate the need to write tests? I don't think so.
Now this problem is amplified when writing tests on top of mocks; if you don't understand the requirement of the next level (mocked level) then your tests will be very incomplete.
Still, having unit tests that are run with each build is much better than having no tests at all.
too bad about the RE on the REOI. But TDD really failed because it didn't identify SLG parameters. I don't know if I'd call it a failure though, SLG parameters are usually hard to know before the start of a project and, even, throughout.
Test Driven Development? All these acronyms and not once is it spelled out in the article or the discussion! Is it like saying HTTP or DNS to most people? I'd honestly never heard of it...but the concept seems logical from a high level.
Salesforce.com Developers are the highest paid in the IT industry and TDD is hardcoded into their development process. (All code is required to have 75% unit test coverage).
Over 405 comments, I am late to the party here, especially since this is just my 2cents.
I think TDD "failed" largely for creative reasons. And it didn't actually fail.
The reason I was willing to use the word failed, in quotations, is that I do think that TDD is dead in the sense that it's a stick that can be used to beat people into submission. There was a long series of debates on youtube with DHH and a few TDD advocates, titled "Is TDD Dead", and it's funny that I think DHH largely won the debate considering that I believe the answer is, clearly, "no, TDD is not dead". TDD remains relevant and useful.
And yet, I think the TDD proponents suffered a severe setback in that debate, severe enough that I'd consider it a pretty bruising defeat.
Why? Because the debate showed that debate is reasonable. That the position that TDD is dead can actually be defended. Here's the thing - a lot of TDD proponents denied the existence of a legitimate debate. There was right, and wrong. Blog posts saying that people who even question the value of TDD should be unemployable, that TDD might rightly one day have the force of law behind it, that questioning TDD is the modern day equivalent of medieval doctors denying the importance of sanitary conditions and washing hands. That sort of thing. I think that by the end of the debate, there were too many cracks in the TDD argument to deny that not doing TDD may, in fact, be a good way to write software. TDD may not be dead (or even close), but that sort of browbeating certainly was put to rest.
Some of it was what DHH called test driven design damage. But the biggest one was creative. TDD may simply not work for the creative flow required for many types of software development. It's like, to contrive an analogy of my own, requiring a writer to outline the next page before writing the current one. It's just too disruptive. You can justify it a hundred ways from Thursday, but if people doing it can't write software as well as people who don't, TDD will lose.
None of this is to understate the importance of test coverage. But write some code, write some tests, repeat - yeah, I think that works. Trying to force everyone to do TDD through a campaign of shaming and intimidation was a horrendous fail. That's was the outcome of the youtube debates - TDD advocates actually did defend the practice quite well, but they fell far short of a standard that would mean DHH shouldn't be employable because he question TDD.
Perhaps not all TDD advocates were that extreme, but it was a strong enough faction in the TDD movement that I don't think I'm finding extremists and using them as a straw man. That sort of browbeating really was part of the TDD culture, and I think that even the good parts of TDD, the parts we should keep and even evangelize as developers, are harder to defend because of these early tactics.
Boston used to have a software craftsmanship meetup. One month, on the train going home, a few of us discussed "how to describe TDD". Someone had a teaching gig coming up. That night, I attempted to distill the views expressed by this couple of experienced TDD folks. Here it is, FWIW.
# What is TDD?
TDD is JIT-development, built on tests.
It's not developing things before you need them. That's too easy to get wrong. It's not built on reviews and approvals. They're too slow and fragile.
## TDD's JIT development with tests is:
1. Live in the present.
Focus effort on what is clearly useful progress now, not speculation.
Don't do planning or development before you need to. Because later, you will better know what is actually needed, if anything. Be restrained but thoughtful in judging how much of what, needs to be done now.
Don't put off integration. Until then, usefulness and spec are only speculative.
2. JIT-spec
Capture each behavior you care about as a test.
Keep them simple, small, and clear. A new spec is a failing test. A passing test means "done with that -- next!".
Don't stuff your mouth. Don't do lumps. Keep it bite-sized.
Don't spec it until you need it, even if you (speculate) you know where you are going later.
Don't worry about the spec having to change later. They usually do. If the speced behavior is clearly useful now to make progress, that's good enough. If it's something you don't really care about long-term, you can remove it later.
3. JIT-implementation
Keep implementation minimal.
Don't create speculative code. Do reactive implementation and refactoring. If you "might need it later", write it later, when you have clearer need and spec, and more tests available.
## About tests.
Programs have a few behaviors you care about, and many more that are implementation details. Test the behaviors you care about.
Tests redistribute development flexibility and speed.
* Behaviors pinned down by tests, are harder to change. Because you have to update the tests too. They're transparent but rigid, and change is slower.
* All other behaviors, become much easier to change. Because everything you care about is tested, implementation changes can be done energetically, without careful cautiousness and fear of accidental hidden breakage. They're opaque but flexible, and change is faster.
Distribute your transparency and flexibility wisely.
Some topics I'm notably unclear on include test refactoring and management.
* when does one delete tests?
* how are lines of development pivoted?
* how are different classes of tests handled? (eg: external spec commitment; less critical spec I still care about; sentinel spec, which I don't mind changing, but I don't want it to happen accidentally/silently; spec that's transient development scaffolding, and should be removed later; and so on)
Opportunities include:
* broader coverage of the strategy, test, and implementation layer activities
* description of how test suites and implementations change longer-term
* specific discussion of cross-cutting issues like risk mitigation
* tighter characterization of core (eg, all tests and code are a burden, and start with a high time discount, so create and retain only those which are clearly and currently useful)
Most developers are bad at refactoring because most of the tools for it are terrible. Even at Google, something as trivial as renaming a function can be a monumental task.
I don't really have solid arguments against it, I just never found it particularly helpful, or ended up with a result that seemed to justify always working this way, or even defaulting to working this way.
That's great. In my experience, most TDD has looked like "We're going to need a function or class to do this, so let's write a test that calls all those methods first." Which is absolutely ridiculous when we could just be using languages that have concise semantics to describe what they actually do.
For example, the last HTTP API I wrote in Haskell has no tests. The proper request/response is ensured by the types alone. I've had multiple jobs where I spent weeks writing tests to do the equivalent thing with a nodejs API (the body should contain this, the headers should look like that, etc.)
TDD could be the reason for the bad system design. If you are writing to pass a test if you do not refactor you end up with a mess. Why not write it properly as a first step and start adding the tests later in the process.
I absolutely think _tests_ are useful, but have never found any advantages to test-DRIVEN-development (test-first).
But part of that is probably my style of problem solving. I consider it similar to sketching and doodling with code until a solution that "feels right" emerges. TDD severely slows that down, in my experience, with little benefit.
What I've found works really well is independently writing tests afterwards to really test your assumptions.