Hacker News new | comments | ask | show | jobs | submit login

TDD failed for economic reasons, not engineering ones.

If you look at who were the early TDD proponents, virtually all of them were consultants who were called in to fix failing enterprise projects. When you're in this situation, the requirements are known. You have a single client, so you can largely do what the contract says you'll deliver and expect to get paid, and the previous failing team has already unearthed many of the "hidden" requirements that management didn't consider. So you've got a solid spec, which you can translate into tests, which you can use to write loosely-coupled, testable code.

This is not how most of the money is made in the software industry.

Software, as an industry, generally profits the most when it can identify an existing need that is currently solved without computers, and then make it 10x+ more efficient by applying computers. In this situation, the software doesn't need to be bug-free, it doesn't need to do everything, it just needs to work better than a human can. The requirements are usually ambiguous: you're sacrificing some portion of the capability of a human in exchange for making the important part orders of magnitude cheaper, and it's crucial to find out what the important part is and what you can sacrifice. And time-to-market is critical: you might get a million-times speedup over a human doing the job, but the next company that comes along will be lucky to get 50% on you, so they face much more of an adoption battle.

Under these conditions, TDD just slows you down. You don't even know what the requirements are, and a large portion of why you're building the product is to find out what they are. Slow down the initial MVP by a factor of 2 and somebody will beat you to it.

And so economically, the only companies to survive are those that have built a steaming hunk of shit, and that's why consultants like the inventors of TDD have a business model. They can make some money cleaning up the messes in certain business sectors where reliability is important, but most companies would rather keep their steaming piles of shit and hire developers to maintain them.

Interestingly, if you read Carlota Perez, she posits that the adoption of any new core technology is divided into two phases: the "installation" phase, where the technology spreads rapidly throughout society and replaces existing means of production, and the "deployment" phase, where the technology has already been adapted by everyone and the focus is on making it maximally useful for customers, with a war or financial crisis in-between. In the installation phase, Worse is Better [1] rules, time-to-market is crucial, financial capital dominates production capital, and successive waves of new businesses are overcome by startups. In the deployment phase, regulations are adopted, labor organizes, production capital reigns over financial capital, safety standards win over time-to-market, and few new businesses can enter the market. It's very likely that when software enters the deployment phase, we'll see a lot more interest in "forgotten" practices like security, TDD, provably-correct software, and basically anything that increases reliability & security at the expense of time to market.

[1] https://dreamsongs.com/RiseOfWorseIsBetter.html




No. TDD failed for engineering reasons in addition to economic ones.

The main failing of TDD is the assumption that you know all of the tests that you will need before you know your software. This is true in bridge building as much as it is in anything else. Until you fully know the domain of what you are building, you cannot possibly know all of the things you need to test for.

To that end, if TDD were focused around building things to break them and analyze them, it would be wise. However, in software it is often taught to build a wall of failing tests, and you can then use that as a sort of burn down chart for progress. Note that you don't learn about the software you are building from this burndown. Instead, you only get a progress report.

And obviously, this is all my opinion. I'm highly interested in exploring the ideas. And I'm highly likely to be wrong in my initial thoughts. :)


It's worth noting that Kent Beck ("Father of TDD") himself said that TDD doesn't always make sense. For him, the important thing has always been having a tight feedback loop for determining if he was still on the right track.

For instance, when he was working on an unstructured log parser, there was no way TDD would work because the logging stream didn't have a pre-defined structure. Instead, he developed a process where he could quickly inject his parser into the logging stream, visually inspect the results, tweak, and then test again.

Others (Such as Martin Fowler and Uncle Bob) have also emphasized the value of automated unit testing integrated with a CI pipeline as a way of ensuring that you didn't break existing functionality.

As always, what's important is not the specific practice, but rather the engineering principles behind it. The key questions being:

* Am I on the right track?

* Did I break existing functionality?

* How quickly and frequently can I get answers to the above two questions?


I absolutely agree. It's also nice to know:

* What's my confidence that the code will work correctly under different situations.

* How maintainable the code is and related how easy is it to make changes.

* How much time/effort does it take to create.

It's funny (or sad) how the XP guys like Kent are always talking about doing the right thing and not being religious. This stops working when the people using it don't know what the right thing is any more. They write terrible tests, they make bad changes to their code to support those tests, it ends up a big un-refactor-able ball of mud that takes longer to create and is more difficult to maintain.

If you're writing a shortest path algorithm or a JPEG decoder then you absolutely should code up some tests to check your code. The tests are relatively simple. They don't need to know anything about the internals of your code. They help you check quickly whether your code is correct and don't interfere with any refactoring. You can rewrite your JPEG decoder from scratch without needing to touch the tests at all.


i'm not a fan of tdd, and a huge proponent of testing in general. what you say about making bad changes to support tests really strikes a chord.

in tdd environments the test harness is used as some kind of crutch to avoid having to have difficult discussions about what the product should* be doing - what makes sense. the test acts as some kind of oracle in the regard, even though it was written by an intern last spring who didn't really have any idea what was supposed to happen either.

i'm basically afraid to say 'the test is wrong' anymore. i always get back a blank look of incomprehension or even outright fear.


Word. I have also worked on projects where existing tests were treated like holy cows. Deleting a test wasn't disallowed as such but it really required a convincing justification. Ironically, this made it especially hard to touch the inevitable subset of tests which had grown unreadable and which were not tracable to specs. Hence, they were often left alone, smelling riper with each passing year.

I like to think of production code as being inherently a liability, not an asset, and the same goes for test code. That's not to say that code, or tests, cannot be valuable; it's just that every line of code carries a cost to the business just by adding complexity to the development process and by taking up disk space and CPU cycles. Not all code offsets this cost, and a lot of tests certainly don't.

We should stop viewing tests like holy scriptures. Some add long term value, some add value during initial development, some should never have been written in the first place.

And on a side note, our tests should test the business requirements, i.e. they should be tracable to specs. There are a lot of tests out there which amount to little more than to "test" some arbitrary decisions which we made when someone initially fleshed out a class diagram.


TDD always worked for me.

All successful projects in my career had full functional regression test coverage.

For those successful projects, I made following conclusions:

  * My personal TDD process are:

    * Script enable - always run every night, every weekends.
         Output to html files to see overall green/red in < 1 second.


    * When any existing test broke, fix/debug the tests are the highest priority before any more features development.

      * The test reports from all the nightly and weekend test should always be green.

      * Any red(s) will stop all features dev. 
    
    * There are 5 minutes function test scripts that require to run before every commit.

      * Normally, I prefer commit in the morning after overnight test runs were all green.




  * These were small project with team only 1-2 devs. 

    * Much easier to enforce the process.    (Just need to convince I, me and myself.) 

    * I worked on big team projects with 100+ devs.  

      * I can test only my own part of code.  For any big team, I can take the horses to the water but can't make anyone to drink it.   The overall system SW quality was still bad.


  * I Only do full functional tests - I don't do any small unit tests with mocks.

      * For CDRW drive, there were 100+ test cases for 
          close, read, open, repeat, 
          close, read, write, loop, open, repeat, 
          for different CD format, disk, drive, etc

      * For DNA sequencer machines, there were tests for 
          Motor controls, 3 axis robot, 
          CCD camera, laser control, script subsystem, comm sub sys,  etc. 
          (This is old project, 20+ yrs ago)  
          Deployed in the field for 15+ years with version 1.0 Firmware.  
          Good for company bottom line - not sure for the SW guy's career.  :-) 
           (Multi billions $$$ of revenue for the company, $500+M first year)


      * For video decoder systems with 19FPGA + 10G network:  (10+ years ago)
         Streaming control, snmp, IGMP, all each has their own test scripts and 
         combine system level scripts for complex network, HA setup.  
         Deployed in Comcast for long time without any issue. 
         Only one issue reported was system has problem after 255 days of 
         non-stop running.  
         I did have test case for 10 ms counters (for SW bug happened for 25, 49 days) 
         I didn't create test case 100ms counters (for SW bug happened after 255 days). 

 
         Again, very good for company bottom line.   
           (Tens millions $$ of revenue for the startup)
           But the company don't need me anymore after they shipped that product.


Sounds like you're doing the right thing. Note that not every one would agree that you are doing "true TDD" (due to your lack of "unit" tests) but that doesn't matter. What matters is that:

a) you are disciplined in your commitment to quality

b) you offload as much of the tedious, repetitive work to the computer as possible.

Regarding unit tests and mocking, I have a similar opinion. I tend to test at the larger component/package level rather than the smaller class/module level. Furthermore, I avoid mocking as much as possible. Object-Oriented decoupling through dependency injection makes mocking a lot easier (and removes the need for mocking frameworks) and Functional decoupling[0] makes mocking largely unnecessary.

[0]http://prog21.dadgum.com/131.html


Mocking is an aspect of testing that grew out of control and the cost can easily outweigh the benefits.

In addition to mocking frameworks, I've also encountered home-grown solutions that were complete time-sinks to use. At some point, you're investing huge chunks of time, not just testing, but mocking in particular.

It can literally layer an entirely different problem domain onto your dev effort, apart from the actual "business problem" the code is intended to solve.

At a certain point, it's like, "wait, what are we doing here again? There has to be another way." Always wondered how people became so dogmatic and determined that they fail to see that they are veering from their original purpose. There is a balance to everything and something about the dev community causes us to lurch in different directions, well beyond that balance.

BTW, there is a post [0] that reminds me of this on the blog you shared. Good blog, BTW, even if it is no longer active.

[0] http://prog21.dadgum.com/56.html


Along the same lines, checkout "Functional Core, Imperative Shell".


>Object-Oriented decoupling through dependency injection makes mocking a lot easier (and removes the need for mocking frameworks)

I'm confused by this. In C++, I found that one of the only ways to effectively use mocking was via dependency injection. For us, dependency injection was introduced to enable TDD. That it had other good qualities was a bonus.


A fair question. Let me put it this way: in order to properly do unit tests, you need to decouple business logic from "integration code" (code that manipulates external systems). Dependency injection is one approach. However, with this approach the business logic still depends (now indirectly) on the integration code.

Another approach is to model business logic using pure functions. Instead of business logic calling the integration logic (directly or indirectly) you have the integration logic calling the business logic.

The consequence of this is that when you are unit testing your business logic you do not need to mock anything. It's simply a matter of checking input/output mappings.

See this series of articles for a deeper explanation: http://blog.ploeh.dk/2017/01/27/from-dependency-injection-to...


>Another approach is to model business logic using pure functions. Instead of business logic calling the integration logic (directly or indirectly) you have the integration logic calling the business logic.

Yes, this is similar to what I did. I ensured business logic dealt only with data structures and algorithms, and knew nothing about the rest of the world. If it needed to report an error, it would call an interface class's "ReportError" function, and pass a string. It did not need to know if the error went to the console, a file, an email, text message, etc.

However, in our case, the communication is still two way - just separated via interfaces. The integration logic does call the business logic, but the business logic may need to do things like report the results, log interesting information, etc. So we still had to mock the interface class (i.e. mock the ReportError function).

But in the bigger picture - yes. Not allowing your business logic to know anything about the details of the integration was really helpful. In principle, our customers could have written the business logic for us.

Sadly, my team did not accept it so we never adopted it.


>Note that not every one would agree that you are doing "true TDD" (due to your lack of "unit" tests)

I think he's doing it right in spite of what other religious zealots may say. In my experience, unit test driven development causes three massive problems:

* Unit tests are intrinsically tight coupling.

* Unit tests require you to build a model for most things that you are testing against (e.g. your database). Building these elaborate models (i.e. using mocks) is prohibitively expensive when you have tightly coupled code but they're often still expensive to build (in terms of developer time) even when you have loosely coupled code.

* Unit tests are intrinsically unrealistic because they are built against models rather than the real thing.

In my experience unit tests work adequately for domains that are heavy on complex logic and light on integration (e.g. writing a parser). In domains which are light on complex logic and heavy on integration they fail badly (e.g. most business apps).

Optimizing for the overall speed of your test suite by using a more unrealistic test that catches fewer bugs is a massive false economy. CPU time is dirt cheap. QA and dev time is very expensive.


> In my experience unit tests work adequately for domains that are heavy on complex logic and light on integration (e.g. writing a parser). In domains which are light on complex logic and heavy on integration they fail badly (e.g. most business apps).

There's a testing methodology we used at my last company called "Change Risk Analysis" or something. Essentially, the point was that you separate business logic and integration code, and you write unit tests for the business logic. There was a metric that combined the the cyclomatic complexity with the test coverage to produce a "risk" score for methods and classes, and all code (or at least all new code) had to be under a certain risk threshold.

I would also add that domains that are heavy on complex logic and light on integration are also highly amenable to more sophisticated verification methods, such as generative testing and model checking.


> TDD always worked for me.

> All successful projects in my career had full functional regression test coverage.

Regression testing is not the definition of TDD, however.

TDD is this crazy religion of writing tests for everything, including the lowest-level helper functions in a module that don't correspond to anything that would be tested in a regression test suite.

You write these little tests first (which must be shown to fail at least once and then shown to pass). Then you keep them around when they pass and keep running them all the time even though nothing has changed.

For instance, if you develop a Lisp, you must forever have a test which checks that, yup, CAR on a CONS cell retrieves the CAR field, CAR on NIL returns NIL, and throws on everything else. The image is 10X larger than necessary and it takes 27 minutes to get a REPL prompt on a fresh restart. :)


Thats the interpretation of TDD by people who are afraid of test and seek excuses.

By the name, nothing says you need 100% coverage (a stupid measure abyway, cause you can get those 100% but still neglecting thousands of corner and limit cases for onput data, so actually you need another measure) or that it has to be unit tests...


> TDD is this crazy religion of writing tests for everything,

Not true in my experience. You should only test code you wrote that can break. You do not test core functionality, the OS underneath etc. You do not test the test themselves.


"You should only test code you wrote that can break. "

A common source of errors are mismatches between what your dependencies say they'll do and what they'll actually do. There's been huge, costly failures from developers expecting the dependency would honor its documentation or just take a standard approach to error handling. So, either testing them or at least good catch-all error handling + logging should be done.


No, TDD means that if there is a line of code anywhere that was not written to make some failing test pass, that code is by definition broken. ALL code must be against some test, else it's not TDD. And if you want to write code for which there is no test, you must write the test first, run the test suite to see it fail, and then implement the code that will make the test pass.

Anything else is not TDD.


>No, TDD means that if there is a line of code anywhere that was not written to make some failing test pass, that code is by definition broken. ALL code must be against some test, else it's not TDD.

Robert Martin is likely the biggest evangelist for TDD, and he does not advocate what you say.

Why do people keep claiming you have to test every line of code in TDD?


> No, TDD means that if there is a line of code anywhere that was not written to make some failing test pass, that code is by definition broken.

No, it doesn't. TDD means tests are written to test the code, and the code being written must pass all tests. It's ok if at any point some parts of the code aren't covered by tests, because TDD is a cyclic process where at each cycle new tests are written to accompany changes in the code.

It seems that the only people complaining about TDD are those who pull the strawman card and portray the process as being unreasonably convoluted and picky, to a level which is almost autistic.


Should you write tests for the tests too? What I find is that in any project of a certain complexity, the tests can end up having the wrong scope or outright bugs in them.


I keep asking the same thing, especially that given any non-trivial self-contained piece of code, the test code will necessarily be isomorphic to the actual implementation - at which point one has to ask, why bother with TDD in the first place and not just write the complex code once?


>TDD is this crazy religion of writing tests for everything, including the lowest-level helper functions in a module that don't correspond to anything that would be tested in a regression test suite.

And like all crazy religions, the followers ignore the prophets.

Robert Martin is likely the biggest evangelist for TDD, and he does not advocate what you say.


Do you write the tests first - before the code - and while still exploring the domain? That's the guiding principle of TDD.

Writing tests after the code has been written - to lock down behavior, quickly identify breakage, and support refactoring is a valuable process. But it's not the original vision of what constitutes test-driven development.


Whenever I write tests first, they usually fail. I need to do some iterations of the test code as the "real" code until they work. Is there some reason why test code is expected to work first time, yet we need a ton of tests to verify that the "real" code works?


I've always relied on unit tests with mocks with the rule that I only mock modules that are tested and I mock or stub all db and network calls. I also like integration and end to end tests but not may of them. I rely on unit tests for regression and should be able to run the whole suite in less than an hour. I set my precommit hook to run unit tests with coverage and static analysis.


> * When any existing test broke, fix/debug the tests are the highest priority before any more features development.

This is the single biggest problem everywhere I have looked. Outside the development team itself no one is willing to wait for a new feature just because a test failed. And for many large programs there will be substantial subsets of users who will not be affected by the bug so they will argue that the delivering the feature is more important, and contributes more to the bottom line, than fixing the bug.


Aren't you conflating TDD with just automated testing?


Do you have a link to when Kent Beck was working on that log parser?


The "Is TDD Dead"[0] discussion between Martin Fowler, Kent Beck, and DHH.

[0]https://martinfowler.com/articles/is-tdd-dead/


> The main failing of TDD is the assumption that you know all of the tests that you will need before you know your software.

I don't think this is true. TDD is also taught as a way of exploring a space and a codebase.

And there is no assumption that the tests created during TDD are the only tests you will need. They're the tests needed to drive out the desired behaviour. You'll also need tests to check edge cases, tests that operate at a higher level, tests that integrate different systems, nonfunctional tests, etc....

In a lot of ways they are a discovered, executable spec.

> To that end, if TDD were focused around building things to break them and analyze them, it would be wise. However, in software it is often taught to build a wall of failing tests, and you can then use that as a sort of burn down chart for progress.

I've never come across TDD taught that way. Hard core TDD as I know it is RED, GREEN, REFACTOR. You never want more than one failing test, and you don't generalise until you've triangulated based on creating tests that force you to.

I don't personally use TDD on all my projects because often it feels fussy and constraining when you are in exploration mode, but I also don't think it failed. I know people who find it valuable even for exploration. I'm also a big fan of using it for certain kinds of things - e.g. building a collection class.


I don't do the TDD like a waterfall. I don't do upfront all the tests, there is a discussion between the code (what express the business need) and the tests that "wraps" it inside a safe zone (what ensures that the behavior is the one intended). As such, as the idea grow, both the code and the test grow at the same time. Look at [0] to have an idea of how the tests extend the safe zone along the code.

Also, I much prefer a lot of nice unit tests over a lot selenium tests (which takes a lot of time to run).

[0]https://github.com/charlesthk/python-resize-image/commits/ma...


Rereading my post, I see where it sounds more strawman of the "entire wall of tests." I did not mean it to be the final test suite is written first.

Instead, I intended that to be the wall that you will be passing in a given cycle. So, for a small development cycle, you add in some new tests that will be this cycle's "wall" of failing tests that you flip to passing.

And don't take this to be an indication that I think integration tests are somehow superior. They all have advantages and there is a massive cost benefit tradeoff that has to be considered at all times.

It is easy to talk in strawmen. Such that I can confidently say if you have a set of tests that always fails together, that should just be a single test. However, I do not feel that statements like that are informative, precisely because it is a judgement call masquerading as a rule.


That's not how you're supposed to do it. Write one test, get it to pass, repeat.


Right. In that, you still write all of the tests before the code. If you are objecting only to the rhetoric of "wall" of tests. That just depends on your size of units. Think of it more as hurdles of tests. :)


> write all of the tests before the code

Everyone is interpreting this as, "write 10 tests then try to get them all to pass at once". That is not how you TDD. You write one, then get it to pass, then write another.

Maybe you mean, "write the test before the code", but when you say "write all tests before the code", it's not interpreted the same way.


> So, for a small development cycle, you add in some new tests that will be this cycle's "wall" of failing tests that you flip to passing.

That's not right either. There is no "wall". Writing tests helps you write the code which helps you write better tests. It's mutual recursion, not a one-way waterfall.


> The main failing of TDD is the assumption that you know all of the tests that you will need before you know your software.

You know what makes this work for me (I am a TDD & BDD dev, and I don't think it's dead) is the fact that you can reorder commits. When I hit a case like this, where the correct code is easier to write than the correct test, I still test it.

I write the test after, then I reorder the commits once the test passes, and make sure it fails predictably in the way you were expecting it to fail. Sometimes you find your test doesn't fail correctly with the codebase you had before the feature was implemented, and then you amend the commit and use rebase or cherry-pick.

Adding BDD to the mix helps a lot too. Sometimes you can write the whole test up-front! Sometimes you can write pseudocode for the test, but you don't have enough information about the details of the implementation to write what you'd call a complete test before the feature is born. Usually you can at least get a vague, english-language description of what the feature is supposed to do.

Then you have, at least, a foundation for your test (which can ultimately be upgraded to an executable proof/regression test), and you have a way to measure how far from the vague, usually simple, plain-english description of the feature you've had to go in order to produce "correctly behaving" code. (And you can amend the behavioral spec to correctly explain the edge cases.)

I am in the habit of telling my team members with less TDD and BDD experience, who are just trying to get the hang of building up a codebase with strong regression tests, that they should always try to write a Cucumber Scenario before they wrote the feature, even if they don't write step definitions until (sometimes a great while) after the feature is done.

I agree that the burn-down chart is not the end goal of TDD, and you should not optimize your process simply around making sure you have a progress chart and a great estimate up-front. Complete coverage of the codebase is more important! And surely there are other things even more important than that.


+1, that's a pretty cool idea

>> I write the test after, then I reorder the commits once the test passes, and make sure it fails predictably in the way you were expecting it to fail.


Lol, that literally sounds like cheating. You write the code first, tests later, and then reorder the commits so that you can say to everyone (colleagues and managers) that you follow TDD...

Seriously, why bother reordering the commits?


It's really important in TDD to know that your test fails when the code you write for that feature it is missing. If it doesn't fail, that's a false positive. The prescribed pattern is, you write your test (it should be red), you write your code (it should turn green), you commit, push to CI, look at the rest of the tests, make sure none of them turned red... take another look at the code, refactor, commit, push to CI, look at tests again, and then start over on the next feature.

(Edit: OK so reordering the commit is not actually the most important part, it's checking out that commit you reordered: with the code for the test, without the matching code for the implementation; and running the test to make sure that it fails until the feature is implemented.)

I don't really care if you think it's cheating to reorder the commits, if it gets the job done faster / better in some way. I'm not writing code for a grade, I'm writing an application that I'll need to maintain later. If that feature breaks and that test doesn't turn red, it's been a complete waste of time writing it (and actually has provided negative value, because I will now be less inclined to test that behavior manually, knowing that there is a test and it passes.)

That's a false positive that you can easily guard against by reordering the commits before squashing/merging everything back and confirming you see some red where it's expected. (You don't need to reorder if you did actually write the test before writing the code for the feature, but forcing people to do this seems to be the most common cause for complaint about working in TDD. If your developers quit their jobs, then TDD has provided negative value too.)

Meanwhile if you have tools for analyzing code coverage, you can make a rule that code coverage must remain above X% to merge work back to master, or that merges must never decrease the percentage of code covered by tests. You don't have to use these rules, but if you're having trouble getting your team members to test their commits, it's not always a bad idea. It has nothing to do with proving things to managers. Code that isn't covered by tests is code that can break silently without anybody noticing. Code that is covered by bad tests is no better than code that is not covered by any test. It can break without anybody noticing. You need to know that your tests work; if you write the code before the test it's just so easy to accidentally write a test that passes anyway, even if the feature was never implemented.

Also, because you can. Forget TDD completely, if you don't know how to use rebase and cherry-pick, you need to learn to use them so you can reorder your commits.

It makes your history cleaner. Nobody cares what commit you were looking at when you started making your change, that's not useful information. Would you rather have a history that looks like this[left] or this[right]?

http://www.bitsnbites.eu/wp-content/uploads/2015/12/1-nonlin...

You may not care now, but I'll bet you care again if you're stuck doing the conflict resolution when it's time to merge!


Agreed! That actually makes it much more feasible to insert into my current workflow!


Yes! When you know how to use git, you can honor the spirit and basic practice of TDD without following it to the letter.


In TDD you don't build a wall of tests, you build your tests as you build your code. Test a little, code a little, refactor, repeat.


I guess another way to put it is that TDD is an attempt to do old school "waterfall" in code form.

Way back before anyone really understand how software engineering should be done, people would begin a software product by attempting to pre-specify ahead of time every nuance of the system. They'd write these massive books of requirements. Then they'd get to work coding to these requirements.

This rarely worked out well and usually resulted in huge amounts of wasted effort. When you get coding you realize the spec is either wrong or there's a much, much better way to do it. Do you do it the bad way to stick to the spec or do you deviate from the spec?

The other problem with this approach is that the requirements are often as hard to write as the code and contain as much information. Why not just write the code?

TDD is just "the spec" written in code instead of as a massive design document. Attempting to achieve 100% test coverage often means writing a test set that is a kind of negative image of the code to be written and is equal in complexity. As such the tests often end up containing bugs, which is the same problem as the classical "when we got going we found that the spec was wrong." Do you code to the bugs in the tests?

This is not to say tests should be skipped. Automated tests are great. I just think the 100% test coverage TDD dogma is oversold and doesn't really work well in practice. Test where you can. Test where you must. Keep going.

Of course I am not a fan of "methodologies" in general. I see them as attempts to replace thinking with rote formula. That categorically does not work outside domains that are purely deterministic in nature, and those can be fully automated so humans aren't needed there at all.

There is no formula, or alternately the formula is general intelligence continuously applied.

Edit:

The same evolution has happened in business. Nobody writes a "business plan" anymore. As soon as you get into business you have to toss the plan so why waste your time. Planning is a false god.


TDD is closer to the Spiral model of iterative development of a product always assumed to be incomplete or broken in some way. Constant improvement instead of perfection.

https://en.wikipedia.org/wiki/Spiral_model

"As such the tests often end up containing bugs, which is the same problem as the classical "when we got going we found that the spec was wrong." Do you code to the bugs in the tests?"

Work in high-assurance field showed it's often easier to assert or check the results of complex software than it is to implement the software itself. Imagine a control system bringing in data, making complex calculations, and deciding whether to accelerate or decelerate. You want to ensure it never pushes past a certain maximum. It's much easier to code something at its output that says (if value >= maximum then StopAcceleratingMaybeNotifySomeone()). The checker is even easier to formally verify if you wanted to go that far. This is one of reasons many proof assistants started building stuff on simple-to-check cores where the generation process is untrusted but the trusted checker is tiny.

Empirically, though, there's confirmation out there in terms of defect detection that tests catch more bugs than they create.


Your description does not resemble TDD as I learned it and practice it.

> As such the tests often end up containing bugs

All code contains bugs, but we do not abandon programming. Having statements of behaviour from outside and inside the solution increases the chances that one will reveal a meaningful difference from the other.

> I just think the 100% test coverage TDD dogma is oversold and doesn't really work well in practice.

I work for Pivotal, one of the most outspokenly pro-TDD shops in the industry. I have, quite literally, never seen or heard any engineer mention test coverage ever.

> Of course I am not a fan of "methodologies" in general. I see them as attempts to replace thinking with rote formula.

I see them as a starting point. The expert mind sees possibility, but the novice mind needs certainty.


Test coverage doesn't make sense to a TDD shop and so they never talk about it. The only code they have not covered by tests are things that cannot be tested at all. They by definition are as close to 100% coverage as you can be.

Test coverage is important in shops that do not do TDD. Such shops are learning the hard way that they need tests to ensure that existing features do not break when they add a new feature. Test coverage is a metric they can use as a proxy for how likely it is that change will break something else (fixing a bug and introducing a new one is a very real problem).


I have never seen anyone propose writing all the tests before writing a single line of code.


I think the failures are due to a combination of engineering, economic and human reasons.

TDD is counter intuitive to how most people think and work. They think of the features first, implement them and then once it's working want to have tests to ensure it stays working, especially with continuous delivery, testing is mostly about checking for regression as high frequency iterations are done.

Even if things starts out ok, often at some point, some person in the team breaks the flow. You have to be quite rigid about procedure and in my experience most teams don't have the discipline for it. This is especially true when there are business pressures in some companies that put pressure on the development team. We've all experience instances where a program manager or producer asks for estimations and an inexperience developer only considers the coding time. To be honest, unless you're working at one of the top tech firms where everyone is interested in solid engineering and there's adequate revenue to enable this, I still find even basic automated testing in general is still a struggle to have in some companies. The reason for this is because the increased development cost is difficult to measure and so it's absorbed as a cost. They don't realize how much development is costing because they cannot compare it to the alternate reality where they did things a better way. Whereas feature issues are obvious to see for everyone.


Yes, precisely.

TDD works for experienced developers who know the problem domain well. It does NOT work for that new junior dev you hired straight out of college. They need to write functions and features first and see the spectacular ways in which those functions and features can fail (in a dev/test environment, hopefully), and then use that experience to switch over to TDD when they write a similar feature/function in their next project.


> TDD works for experienced developers who know the problem domain well

This is important: it's not only junior devs that fail -- TDD also does NOT work for experienced developers who do NOT know the problem domain well, as the infamous Sudoku solver debacle showed, where Ron Jeffries got too caught up in TDD/refactoring to make any serious progress in actually solving Sudoku.


Amen. And this is what spikes are for :)

I TDD, but I don't TDD my spikes. Sometimes I write an almost working system before I start actually working...


> the assumption that you know all of the tests that you will need before you know your software

I'm not a TDDer but must object on their behalf: not only is that not what they said, it couldn't be further from it! What they taught was to work in short iterations and work out a bit of requirements, design, and implementation in each one. If you needed knowledge you didn't have yet, they would advise you to make each cycle shorter until you did. Sometimes they'd take this to absurdity, breaking even simple features into design-test-implement cycles so microscopic that you would feel bored like a kid in class who can already see the next dozen steps and has to sit there waiting for the teacher to catch up.

The TDD approach was at the extreme tip of trying to get away from waterfall methods, so it would be quite some irony if it reduced back to that.

Edit: looks like I misread what taeric meant! But as McLuhan said, 'you don't like those ideas, i got others'...

In my experience there were two problems with TDD. One is that the micro-iterative approach amounted to a random walk in which features agglutinated into a software blob without any organizing design. Of course that's what most software development does anyway. But if you remember what Fred Brooks et. al. taught about having a coherent design vision for software that can organize a codebase at a higher level, the way iron filings are ordered by a magnetic field, that is a key anti-complexity vector which the TDD/XP/agile projects I saw all lacked. (Original XP had a notion of 'metaphor' that pointed vaguely in this direction but no one knew what it meant so it had no effect.) The lesson was that a coherent design vision doesn't evolve out of bit pieces, and there is an important place for the kind of thinking that got dismissed as 'top-down'. (Edit: by far the deepest work on integrating design coherence into TDD/XP approaches was Eric Evans' stuff on domain models and ubiquitous language. Eric is my friend but I knew he had the best stuff on this before that :)) (Edit 2: Eric points out that the people who created TDD were software design experts for whom coherent design was as natural as breathing air. The admonition against too much design thinking in TDD was intended for people like themselves, for whom the opposite mistake was never a risk, and didn't have unintended consequences until later.)

The other problem was that practices that started out as interesting experiments ossified into dogma so damn quickly that people started acting like they'd cracked the code of how to build software. A soft zealotry emerged that was deeply out of sync with the playful, adaptive spirit that good software work needs. This was unintentional but happened anyway, and there's a power law in how much creativity it kills: each generation that learns the Process is an order of magnitude less able to think for itself within it, and so must break free. I don't think the creators of the approach were any more at fault for this than the rest of us would have been in their shoes. The lesson is that this ossification is inevitable, even if you know all about the danger and are firmly resolved to avoid it. If there is any real breakthrough to be had at the 'how to build software' level, it is probably discovering some kind of antifreeze to put in there from the beginning to make sure this doesn't happen. Maybe there's a way to interrupt the brain and social circuitry that produce it. Failing that, I think that each project or maybe (maybe!) organization should hash out its own process from first principles and the prior experience of team members. That's ad hoc and involves a lot of repeating past mistakes for oneself but at least it isn't idiotifying. It's how I like to work, anyhow, and IMO that's what all software processes reduce to.


See my response in https://news.ycombinator.com/item?id=14664831

Basically, I'm not intending something different than what you are saying. I just picked a phrasing that can go both ways, unfortunately. :(


Ah I see. I still can't see how to read the GP the intended way but maybe it's just me.


Oh, to be clear. Just because I had intended it in one way, does not mean I was successful at it. I just didn't want to ninja edit a better wording under what others had already read. :)


> iterations

The property of iterations is that things change, outcomes change, ideas change, requirements change. Iterations are not black boxes that we tuck away as 100% complete; they often require re-examination as the iteration process continues.


Sure. I think the idea was that you'd incorporate that into the next iteration. But what do I know—I'm sure we were all doing it wrong.


That is not at all the assumption. The assumption is that you have some idea what you want a particular piece of code, method, or class to do and you write a test for a small piece of that and then after it passes you do that again.

At no point does TDD suggest you should know all the tests you want upfront. In fact I would expect if you know that then you don't need TDD. TDD is about learning what tests you need and what you want the object or code to look like one small piece at a time.


"The main failing of TDD is the assumption that you know all of the tests that you will need before you know your software."

If that's what TDD is, then it can only fail. Fortunately, TDD does not require you to write all your tests up front (see "The Three Rules of TDD").

I spend a lot of my time interviewing people and talking with people about TDD. In the vast majority of cases, I find people who dislike TDD are often doing "Not TDD, but called TDD." If I were to write software that way, I too would consider that TDD did not live up to expectations.


Aren't you two saying the same thing? TDD works when the requirements are well defined upfront, which in the case of fixing existing system is provided by the system itself. But writing new programs is exploratory in nature: you don't know the exact requirements or shape your software will take, and TDD shifts a lot of precious money, time and attention from the what to the how. Which to me, I agree with you, is an engineering issue before being an economic one.


> The main failing of TDD is the assumption that you know all of the tests that you will need before you know your software

One of the main failings of TDD is that developers think they need to know all of the tests up front.


I've never heard TDD being that you write all of your tests up front, before you've written any code. I've always heard it as: When you're writing a class/unit, you write a few tests for what that unit is going to do. You then make those tests pass. You add some more tests, make those pass, and so on and so on.


You are still writing the tests first, with no real learning from them other than "they are now passing."

It is thought this is like engineering. You design an experiment and then pass it. However, that is not how experiments work. You design an experiment, and then you collect results from it. These may be enough to indicate that you are doing something right/wrong. They are intended, though, to be directional learnings. Not decisions in and of themselves.

So, I probably described something in more of a strawman fashion than I had meant. I don't think that really changes my thoughts here, though.


You're often learning the optimal API as well. I've had several experiences where I write a library, write some tests for it, and then realize that the library's API is inconvenient in ways X, Y, and Z, and that I could have a much simpler API by moving a few tokens around. When I write the tests first, I tend to get a fairly usable API the first time around, because I'm designing it in response to real client code.

(This all depends on having a clear enough idea of what the library needs to do that I can write realistic test code...I've also had the experience where I write the tests, write the library, and then find that the real system has a "hidden" requirement that isn't captured by the tests and requires a very different way of interacting with the library.)


It’s funny that you say “I'm designing it in response to real client code.“

To me, a test is not real client code. Real client code is code that calls the API in service of the user. E.g. the real client code for the Twitter API is in your preferred Twitter client, not in the tests that run against the Twitter API.


That's the caveat I mention in my second paragraph.

But yes, if the tests are well-designed and built with actual real-world experience, I do treat them like real client code. Someone looking to use the library should be able to read the unit tests and have a pretty good idea what code they need to write and what pitfalls they'll encounter. And when the library is redesigned or refactored, the tests are first-class citizens; they aren't mindlessly updated to fit the new code, they're considered alongside other client code as something that may or may not have to change but ideally wouldn't.


So, this is a slightly different mantra. Working backwards from the customer is an important thing. And if you know that customers will be using your product programmatically, then yes, building the API first is important.

But note, in that world, you aren't necessarily building tests of your software. You are building specifications for your API. I can see how these might be seen as similar. And for large portions of your code, they can be.

However, for the vast majority of your code, it exists to help out other parts of your code. Not to be used as a library/framework by others.


> with no real learning from them other than "they are now passing."

You don't learn much the moment the test passes. But designing and writing the test requires understanding the requirements in detail, which is often a learning process.


Author here...

This discussion on what happens - or doesn't happen - after the tests are passing is precisely the point of the article.

The idea of TDD is that every test you look at the existing code and improve it. The majority of developers I've seen trying TDD do this with small code but do not with significant codebases, and you end without the benefits.


You can throw away the test if you don't find value in it. This can be very useful for fleshing out an idea.


But this advice should be extended further. Always be willing to throw away the code and the tests. Take the learning.

Be ware, though, that you are not throwing your customers for a bad ride. As soon as you have customers, it is nigh impossible to throw away the code without neglecting them. And they are your ultimate responsibility. Not the code.


Exactly right, it can be very hard to make changes (even the right changes) once it's in customer hands. Sometimes it is easy, sometimes it is not. However, it's always easy to throw out code that isn't yet committed.


If you wrote the test after you coded your unit,you would not know why it passed. It could be a false positive. When you move on to the next part of the requirement, you can ensure all your tests still pass.

Sometimes the tests give you the starting point of the problem, "what is the simplest way to get result x" with a way to rapidly get feed back that all the requirements are ment as the structure takes form and you refactor all the code.


I sometimes write tests that I know will pass. A common example is when the previous version of an algorithm had a bug in some special case. My new algorithm written from scratch doesn't have that special case at all so there is no test for it. However because the last version had that bug there is a lot of fear so I will write a test that it works even though it is inherent in my new algorithm that it works and at no point in the TDD cycle is that test the correct one to write.


> I've never heard TDD being that you write all of your tests up front, before you've written any code.

That's the only way I've ever heard it described. I thought that's why it's called "Test Driven"


Can you provide a reference? I'm not a big proponent of TDD but have read up on it and experimented with it some, and none of the TDD literature I've read suggests writing all the tests up front. Virtually all the mainstream TDD literature and talks advocate for the "write a test, make it pass, lather, rinse, repeat" model of development.


I think this is a case of two ways to hear "write all the tests first."

If you read that as "write every single one of your tests, then write the code", then you see it as a strawman and not the point of TDD.

If you read that as "before you write any specific line of code, you write the test for it first", then you are likely to see that as the same as TDD.

I intended the second reading. I can see how both are accurate ways to read it, though. Apologies for the confusion.


Ah. Exactly the misunderstanding I was labouring under.

Turns out we are all in agreement after all.


I mean before you've written any code at all. Like, you have to have the tests for all units created before you start writing the code for any unit. That's what the person I responded to made it sound like.

Writing tests before you work on the unit you're working on shouldn't be a problem because you should know what that unit is supposed to do. If you don't, then it doesn't matter if you do TDD, Agile, BDD, or whatever. You're not going to be able to do it right. Go have a conversation with someone to find out what that is supposed to do.


You are correct.

Not only this, but it's a blatant violation of TDD. TDD is having a single test go red->green, not a suite.


Exactly this. It's Red, Green, Refactor.

1. Write test; build fails because code under test DOESN'T EXIST

2. Write the least amount of code needed for the test to pass

3. Refactor & repeat process ad infinitum

The Wikipedia entry for TDD also states this explicitly [0]

  This is a differentiating feature of test-driven development versus writing unit
  tests after the code is written: it makes the developer focus on the requirements
  before writing the code, a subtle but important difference.
[0] https://en.wikipedia.org/wiki/Test-driven_development#Test-d...


It's a very common way to evangelize TDD. Ruby Koans is a tutorial entirely based on that principle.


But Ruby Koans is a tutorial on Ruby, not on TDD.


>I've never heard TDD being that you write all of your tests up front, before you've written any code.

That's interesting, because as a non-TDD practitioner who occasionally gets it evangelized to me, that's certainly how I was told to go about it by multiple people.

Possibly TDD has a marketing problem?


Wow, if that's how people are describing it, no wonder it's seen negatively. That seems very waterfall-ish and horrible. Thankfully that's not TDD as originally intended.


TDD is writing code to automatically validate that a specification/requirement has been met, before writing the code that meets the requirement.

It does one requirement at a time. You might add a specification that the code has to create a list of strings from a data set. You write a test that takes a data set as input and checks the output to see if it is the expected result. This fails, because the code hasn't been written yet. Then you write the simplest function possible to make the test pass.

Once it passes, you check to see if anything can be refactored. You can refactor as much or as little as you like, provided that all the tests still pass. Once you're satisfied, you write another test for another requirement. It's like a spinning ratchet wheel, where the test suite is the pawl. It keeps you from backsliding, but you can't spin too fast, or it flies off and won't engage with the wheel. Turn-click-turn-click-turn-click.

Writing all your tests up front is one of the worst ideas I have ever heard.


> However, in software it is often taught to build a wall of failing tests

People teaching that are not teaching TDD. They may be letting the tests drive the development, but TDD specifically is _all_ about the second-to-second cycle of moving between test and code. If you write all the tests up front then refactoring is expensive and you've locked in your design way too early.


Author here...

I agree that time to market is really important, but when a company is searching for the product that will work, the most important thing that you can have is code that is easily refactored, which generally means low coupling.

TDD is one way to get there for some people, and it's not 2x the amount of time. My experience is that TDD takes a little bit longer (say 30%) than just writing the code, but my bug rate with TDD code (and I don't do TDD for all my code) is very small. Bugs are so damn expensive that 30% to not have them is a small investment, even if that were the only benefit you got.


Well, there's time you save on testing.

TDD gives me quick feedback. Its faster to run unit tests vs deploying and testing the functionality. I'd say TDD adds very little or no overhead.


I would agree most of the time. However in my experience you often end up with at least some small part you have to figure out how to test-drive - i.e. using a dependency. Then you have a choice of "TDD this and write proxies or some other kind of wrapper around everything", or "just go with it".


You're assuming TDD has failed and are reaching for evidence to make your case.

From the SWEBOK [0]:

  5.1.5 Test-Driven Development

  [1, c1s16]

  Test-driven development (TDD) originated as one of the core XP (extreme programming) practices and consists of writing unit tests prior to writing the code to be tested (see Agile Methods in the Software Engineering Models and Method KA).
  In this way, TDD develops the test cases as a surrogate for a software requirements specification document rather than as an independent check that the software has correctly implemented the requirements.
  Rather than a testing strategy, TDD is a practice that requires software developers to define and maintain unit tests; it thus can also have a positive impact on elaborating user needs and software requirements specifications.
It really depends on many more factors than getting to market first. If we developed software in this way for fighter planes or search engines I think we wouldn't find them nearly as useful. Perhaps even dangerous.

In the absence of a more formal software specification or testing strategy TDD is the best way for a self-directed team of software developers to maintain that specification.

Beyond that it also has benefits for practitioners working in languages that lack a sound type system to catch domain errors for them at compile time. Coupled with property-based testing TDD can be extremely effective.

[0] http://swebokwiki.org/Chapter_4:_Software_Testing


I have a colleague who is not a household name but still fairly widely known in the software process space and he has insisted to me multiple times, for more than ten years now, that this is a west coast thing. Other parts of the US are much less allergic to discipline in development.


I'd agree with this characterization. I started my career in Boston - actually, one of the first companies I interned at did high-assurance systems for avionics, financials, and medical devices.

However, look at the size of the West Coast tech industry vs. the size of tech firms in New England or the Midwest. That's what I mean by an economic argument. I got paid 5x more at Google than I did in Boston, and I was employee #45,000 or so, not even an early employee.


Unless the causal relationship is the other direction.

That is, everyone always assumes in our industry that the chaos allows this volume of people to be employed. But usually large number of people leads to chaos all by itself. Maybe we have it backward.

I know I've worked on a lot of projects that got version one out quick and could never get version 3 out the door. It gets old after a while. Why do I want to spend a few years of my life working on something nobody will remember? No thanks.


I actually don't think either of these factors is causal.

Rather, I think money is the primary causal agent. The top 5 most valuable companies on earth right now are all West Coast tech companies, and they got that way by building software systems that are an integral part of the lives of billions of people. The fact that software is an integral part of those activities is a function of it being a million+ times more efficient than humans; any company could've filled that need, but AppAmaGooBookSoft was the one who actually did it first.

Money (and specifically, rapid growth in customers & revenues) causes chaos, and money causes lots of people to be employed, and money lets you pay them a lot. The specific engineering practices - at this stage in the industry - are a sideshow. They're important to practitioners, but not important to customers, as long as the software basically works. And the reason TDD has fallen short is because you can build software that "basically works" without it, so it's just overhead until you need software that "really needs to work in all situations".


> Other parts of the US are much less allergic to discipline in development.

I see TDD as a practice that requires a lot of discipline.

The TDD equivalent of crossing every t and dotting every i requires sustained attention and intention.


Research and startups are on the west coast, government and finance on the east coast. Q.E.D.


Research is all over the place. East coast has MIT and Harvard.


Well, MIT, sure. But is Harvard really a remarkable CS, engineering, or applied science research institution comparable to Berkeley or Stanford?

It's interesting what happens when you look at rankings of engineering programs, especially at the grad level. You start to see more large state institutions, as well as more technical institutions, with a heavier concentration in the mid west or west coast.

The ivies are hardly missing in action, especially Cornell (should probably also say Columbia) but Texas, Washington, many UC campuses, certainly Michigan... I'd probably put all these, certainly at the graduate engineering level, at or above the ivies. And the ivy that really seems to be a heavy hitter here, Cornell, again shows how much things have shuffled up once you go to engineering.


Other parts of the world are, too :-)


> Software, as an industry, generally profits the most when it can identify an existing need that is currently solved without computers, and then make it 10x+ more efficient by applying computers.

>In this situation, the software doesn't need to be bug-free, it doesn't need to do everything, it just needs to work better than a human can.

Well said, these statements underscores importance of FINDING the PROBLEMS/TASKS that are done by HUMANS/MANUAL that can be AUTOMATED with COMPUTERS that can yield 10X efficiency .

Here are few gems from the original article https://blogs.msdn.microsoft.com/ericgu/2017/06/22/notdd/

The ability to write and refactor code to a state with low coupling, well-separated concerns > TDD Skills

1/ Design on the fly is a learned skill. If you don’t have the refactoring skills to drive it, it is possible that the design you reach through TDD is going to be worse than if you spent 15 minutes doing up-front design.

2/ I think that’s a good summary of why many people have said, “TDD doesn’t work”; if you don’t have a design with low-coupling, you will end up with a bunch of tests that are expensive to maintain and may not guard against regression very well.


That's a great write-up on the situation. There's one thing I'll counter:

"And so economically, the only companies to survive are those that have built a steaming hunk of shit,"

This is true in average case because most of these startups don't care about quality that much since they don't have to for reasons you describe. However, if they did, then they could at least document things, do limited amount of Design-by-Contract, and/or do assertion/property-based testing as they went along. They can also follow principles like cleanly separating modules, protocols that have an update feature, standardized formats for data, and so on that can make rewriting a lot easier later. Easier, not easy, as rewriting is also difficult. That's why I mentioned documentation and Design-by-Contract as they take very little time. Essentially just write down whatever was in your head at that moment.

Most of the time will still go to experimentation and debugging which... wait, styles like Design-by-Contract in safe languages knock out a lot of that debugging. Good docs do, too. Now, we've gotten to the point illustrated by methods like Cleanroom where the QA might cost a little extra, might cost the same, or might even save time/money due to less debugging of new or old code.


Which is another way of saying TDD is somewhat usefull, if you start from a huge pile of code written by someone else.

When you think this might be going on, so if I do this it should work... then writing some tests will help decide if your assumptions or your code is wrong. Most importantly you are probably writing minimal code in those situations so the overhead of tests is minimized.


Or even just a clear spec from a customer who knows what they want.

The challenge in most of the software industry is that the customer does not know what they want, and oftentimes it isn't even clear who the customer is. TDD is great if you can define your problem well enough to write tests for it up-front. Most software problems with a decent economic return are not like that.


I would say that the primary job of a modern developer is actually to figure out what the spec really is. Implementing the spec is easy in comparison. More often than not, "bugs" are errors in the spec that TDD of course has no hope of catching.


TDD is useful if you start from a well defined specification. The better defined, the more useful it is, starting from a big negative utility when you don't know your problem.


Props for mentioning Carlota Perez. I'm familiar with her Technological Revolutions and Financial Capital (which is quite good). Are there any other works you'd recommend?

http://www.worldcat.org/title/technological-revolutions-and-...

There've been a few HN submissions over the years:

https://hn.algolia.com/?query=carlota%20perez&sort=byPopular...


TDD failed for economic reasons, not engineering ones.

And yet, engineering is inseparable from economics.


P.S. If you click the [1] link above you will be taken to an imgur image making fun of Hacker News.

To read the real article you'll need to copy and paste the URL address in your browser. Looks like the site is reading the referrer and forwarding to mean things...


I updated the link to point to the DreamSongs original. Sigh, jwz. Ironically, DreamSongs is the personal site of the guy who actually wrote the essay, but it's way down in the Google results because everybody links to the jwz mirror.


This is not how most of the money is made in the software industry.

Are you sure? I agree that the most profitable companies are not doing that, but at least from what I've seen in a few European markets, the number of developers writing custom software for one client (consulting or in-house) or maintaining stable applications absolutely dwarfs the number of developers writing new products.

I work on a consulting company that customizes a relatively little known open source platform (Odoo) and the number of small and micro companies doing the same over the world is staggering.


> Software, as an industry, generally profits the most when it can identify an existing need that is currently solved without computers, and then make it 10x+ more efficient by applying computers.

Which is exactly how testing should be done. If you have a piece of software that you test so frequently by hand that you could get actual ROI by automating the testing, awesome. If you don't, then what exactly are you getting out of all the automated testing of code that often never changes with tests that essentially assure you that 2 + 2 is still 4?


It's all a balance, but one thing to consider is that future maintenance or new development may require changes to that software. At that time, having good tests will help you avoid breaking things in subtle ways. The problem is that when you reach that point, nobody may remember enough about that code to write the tests, so it's preferable to have already written them.

If a piece of software is trivial or isn't core to your business, though, obviously this doesn't apply.


So there is a tipping point when the technical debt you acquire from not testing catches up with you and cripples the project. I think this tipping point comes much faster than people realize. Most people have no idea how bad their tech debt is until it's too late. Then you see one failure after another while management tries to fix the train wreck. At this point any good Eng is going to bail in this market and you're stuck with the scrubs.


The crippling debt is caused by not refactoring, not lack of testing. Admittedly testing will make the refactoring easier, but thats not the root of the problem.


This was always my issue with TDD. I never tried it, but I know that the way in which I have written software for more than a decade is "get a loose set of requirements, start digging, write some code, figure out what works and what doesn't, change things, repeat until done."

I never saw a way in which I could write a large number of tests up front.


This is basically the workflow with TDD though, and if you're writing a large number of tests up front, you're certainly doing it wrong.


This!! x100, I am constantly working on features that are not even decided if they go into production. Using TDD for these would be a huge waste of time and energy. Once a feature is accepted I write tests to cover it... after it's already being used and getting feedback from users.


TDD failed? The use case of unknown requirements is precisely what we solve everyday with TDD at Pivotal Labs, as consultants! It works, but I don't have time to outline our process right now.


> Software, as an industry, generally profits the most when it can identify an existing need that is currently solved without computers, and then make it 10x+ more efficient by applying computers. In this situation, the software doesn't need to be bug-free, it doesn't need to do everything, it just needs to work better than a human can.

Unfortunately, the project isn't "finished" after you achieve initial success. You have to maintain it for years after that. Without good tests, you get yourself into a state where one change breaks two things. You need teams of people manually checking it still works. It takes them an order of magnitude longer to check than it would take a computer. Your feedback loop that answers the question, "does it still work?" is in weeks instead of minutes.

You stop changing the bad code because it's too scary and dangerous. The bad code reduces the frequency at which you release new features. You slow to a crawl. The team of people testing gets expensive and eats into your profits. Some competitor eats your lunch.

In my experience, these problems occur earlier than you expect. Usually before you even attain success. I'm too lazy to comb over my app checking for bugs every time I develop a new feature. I get bored too easily to check that the same bug doesn't crop up over and over again. I'm also too impatient to wait for another person to do it. In these conditions, TDD speeds you up.

I've never worked on a manually tested project that avoided these problems. It usually happens around month 2 and gets even worse as time goes on.

Here's why I think TDD fails: The consultants say, "it's so easy! Just red-green-refactor!" No, it's extremely complex and difficult, actually. There's a ton of depth that nobody talks about because they're trying to get you to try it. The typical dev learns about "Just red-green-refactor" and fails. Now "TDD did not live up to expectations".

For example, did you know there's different styles of TDD? The author of this article doesn't seem to (I may be wrong, but he gives no indication). He's talking about Detroit style TDD (also known as classicist). This is the style Ron Jeffries, Uncle Bob, Martin Fowler, Kent Beck, etc. practice and advocate for. There's another style called London style (AKA mockist) that Gary Bernhardt, etc. practice. And then there's Discovery Testing from Justin Searls. There's probably others I haven't heard of yet.

They all have different pros and cons and a very common pitfall is to use a style in a way it's not really meant for. e.g., Using mockist to gain confidence your high level features work. Expecting classicist to help you drive out good design for your code.

To wrap this up, there's so much unknown unknowns about TDD it's usually premature to judge the technique. If you haven't tried all these styles, one might click with you. A lot of people are concluding they hate all sports when they've only tried baseball.


"Unfortunately, the project isn't "finished" after you achieve initial success."

It is if the goal was to sell the company and people bought it. After the success, the remaining problems are the buyer's while the money is going to the sellers. Is that a perverse incentive for quality or what? ;)




Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: