
Why Most Unit Testing is Waste (2014) [pdf] - pmoriarty
http://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf
======
avip
Ok let's stop writing UT and see what happens. Wait... we've already tried
that, and we know the result pretty well:

1\. In dynamic languages, simple type mismatches, wrong variable names etc.
are now caught in "top level system level test". Yes these are bugs that
should have been caught by a compiler had we had one.

2\. There's no documentation as to how something should work, or what
functionality a module is trying to express.

3\. No one dares to refactor anything ==> Code rottens ==> maintenance hell.

4\. Bugs are caught by costly human beings (often used to execute "system
level tests") instead of pieces of code.

5\. When something does break in those "top level system tests", no one has a
clue where to look, as all the building blocks of our system are now
considered equally broken.

6\. It's scary to reuse existing modules, as no one knows if they work or not,
outside the scope of the specific system with which they were previously
tested. Hence re-invention, code duplication, and yet another maintenance
hell.

Did I fail to mention something?

UT cannot assert the correctness of your code. But it will constructively
assert its incorrectness.

~~~
gaius
In OCaml (or any similar language) 95% of what you "unit test" for in
Javascript just goes away.

If people cared about software quality, they would use strongly typed
languages before even thinking about "unit tests". What people _actually_ want
is a) to hack it out quickly and b) to have a lot of busywork to do as a form
of job security. Hence 50,000 lines of "unit tests" per 10,000 lines of
application...

~~~
sametmax
Here is a list of basic errors in Python. Only builtin ones, I'm not talking
about ones from other libs. And I'm talking of error categories, as we all
know one exception can be used in a lot of different contexts:

    
    
      BaseException
       +-- SystemExit
       +-- KeyboardInterrupt
       +-- GeneratorExit
       +-- Exception
            +-- StopIteration
            +-- StopAsyncIteration
            +-- ArithmeticError
            |    +-- FloatingPointError
            |    +-- OverflowError
            |    +-- ZeroDivisionError
            +-- AssertionError
            +-- AttributeError
            +-- BufferError
            +-- EOFError
            +-- ImportError
            +-- LookupError
            |    +-- IndexError
            |    +-- KeyError
            +-- MemoryError
            +-- NameError
            |    +-- UnboundLocalError
            +-- OSError
            |    +-- BlockingIOError
            |    +-- ChildProcessError
            |    +-- ConnectionError
            |    |    +-- BrokenPipeError
            |    |    +-- ConnectionAbortedError
            |    |    +-- ConnectionRefusedError
            |    |    +-- ConnectionResetError
            |    +-- FileExistsError
            |    +-- FileNotFoundError
            |    +-- InterruptedError
            |    +-- IsADirectoryError
            |    +-- NotADirectoryError
            |    +-- PermissionError
            |    +-- ProcessLookupError
            |    +-- TimeoutError
            +-- ReferenceError
            +-- RuntimeError
            |    +-- NotImplementedError
            |    +-- RecursionError
            +-- SyntaxError
            |    +-- IndentationError
            |         +-- TabError
            +-- SystemError
            +-- TypeError
            +-- ValueError
            |    +-- UnicodeError
            |         +-- UnicodeDecodeError
            |         +-- UnicodeEncodeError
            |         +-- UnicodeTranslateError
    
    

52 exceptions.

Now, on this list, how many can be linked to language typing ?

    
    
            +-- TypeError
            +-- ValueError
            |    +-- UnicodeError
            |         +-- UnicodeDecodeError
            |         +-- UnicodeEncodeError
            |         +-- UnicodeTranslateError
            +-- AttributeError
    

And ALL those errors can also be triggered with something a type system would
no catch : runtime class generation, unexpexted user input, corrupted
database, wrong data header, etc)

So basically, relying only on typing will cover about 10% of the error types,
most of which are caught with linters such as flake8 and a code intelligence
tools such as Jedi. You are making a very weak case.

~~~
Artemis2
The bugs are way more subtle than that. You're lucky if your bad programming
is caught somewhere at runtime. In most cases, Python and other weakly-typed
languages will do everything they can to fit the data you're providing with
what the code is expecting. This means converting strings to numbers and the
other way around, plus a lot of other oddities you would not always expect
(see PHP's intval for a crash course in bad language design). All that results
in bugs that are far from obvious, and that don't always raise neat
exceptions.

EDIT: I stand corrected, I don't know enough about Python in particular. The
argument is still valid for other languages with weak types.

~~~
zephyrfalcon
Python is not "weakly typed", nor does it ever automatically convert strings
to numbers or the other way around.

~~~
brianwawok
Heck with Python3 it won't even convert bytes to String.

------
erikb
The funny thing about unit tests is that it's actually possible to write unit
tests that don't really help you at all and that this is the way how many unit
tests are written (a pessimistic me would say "most"), but nobody thought that
would be possible.

The initial people who came up with the idea thought about writing down the
execution of a usecase, or a small part of that, as a test. Then they ran
their code against it while developing it. That gave them insight into the the
usecase as well as the API and the implementation. This insight could then be
used to improve tests, API and implementation.

But most professionals aren't about making quality. They are about paying
their rent. So when they started to learn unit tests, they just wrote their
code as always, and then tried to write tests, no matter how weird or
unreasonable, to increas the line coverage of their test suite. The proudest
result for them is not to have a much more elegant implementation, but to find
the weird test logic that moved them from 90% coverage to 91%.

I believe that's how you get a lot of clutter in your unit tests. However what
is described in the document are sometimes example of people really trying,
but that are just early in their development. Of course when you learn to do
something by a new method you will first do crappy, inefficient stuff. The
idea here is how much do you listen to feedback. If that team that broke their
logic to get higher coverage learned that this was bad, then they probably
adapted after some time, and then they did exactly what unit tests are there
for.

~~~
sametmax
Yes, I was the bit of a shock the first time I realized how misleading
coverage was. Being executed doesn't mean tested. Tested doesn't mean it
works. Working doesn't mean it does what you want. Doing what you want doesn't
mean it will do what the client wants. Doing what's the client want now
doens't mean it will do keep doing it in the future.

But anybody who edited a 100 000 lines long project, ran the unit test, and
saw red lines, fixed them, ran it again and then saw green now the feeling :
it's great. You are way more confident.

~~~
madgar
> But anybody who edited a 100 000 lines long project, ran the unit test, and
> saw red lines, fixed them, ran it again and then saw green now the feeling :
> it's great. You are way more confident.

This is the key insight: unit tests are about _feelings_ more than anything
else. People get so defensive in unit testing threads because eliminating unit
tests means eliminating their source of confidence.

------
xahrepap
I'm currently at a phase where I'm just not as worried about TDD or Unit
testing. I've realized that most of my buggy code is in integration points.

For example, I'm working on an internal project that creates VMs with some
provider (be it Virtual Box, AWS, etc) and then deploys a user defined set of
docker container to it. I've found that I don't have bugs in situations I
would typically test using mocking/stubbing/etc in traditional unit tests. I
usually need to have the real AWS service with the docker service running to
get any value out of the test. And at that point it's more work to mock
anything else than it is to just start up the app embedded and do functional
testing that way.

I'm becoming more of a fan of verifying my code with some good functional
tests in areas that feel like high risk and then some contract testing for
APIs other apps consume. Then if I find myself breaking areas or manually
testing areas often I fill those in with automated tests.

------
vdnkh
Abandoning unit tests was a thing many companies were proud to tell me while I
was interviewing earlier this year. I always thought that was ridiculous - but
i suppose their products didn't have so many users, or thought they could
tolerate some bad behavior. I'm happy I ended up at a place where we're big on
tests and even have SDETs embedded in within our team. Besides being useful, I
often use a unit test as my main method of completing a feature. I also
believe it's a code smell when it's hard to write a test. Maybe it's a
regional thing - I've heard it said that here in nyc were much more strict
with testing (a carryover from financial roots).

~~~
pacala
The article top recommendations are strong:

* Keep system level integration tests [for up to a year].

* Keep unit tests that test key algorithms for which there is a broad, formal, independent oracle of correctness, and for which there is ascribable business value.

In my experience, an overarching test layering strategy works best in
providing maximum coverage for minimum friction: Add tests bottom-up, at each
layer testing the core functionality of that layer, using directly the layers
beneath. The most valuable tests are the system-level tests, which, hopefully,
exercise a large swath of the underlying codebase. This reduces to the
author's recommendation for most cases.

Some people are proud to ditch UnitTesting[TM] to avoid the unit test cargo
cult, in particular, the proliferation of "decoupled" tests via overuse of
mocking libraries. There is very little value in "unit tests" that reproduce
the code flow via mocking asserts, alas a lot of codebases that are heavy on
unit testing degenerate into gratuitous mock fests, which become incredibly
onerous to work with.

~~~
je42
No.

It should have been.

* Keep system level integration __until requirement for system level integration change __

* Keep unittests. And increase code coverage. Any code that is not executed in a unittest will eventually break. Especially in interpreted languages.

* Obviously code coverage is not enough. But in my experience code coverage is minimum requirement. Not something to be happy if reached.

* The biz cases also need to be tested, which is usually done by a combination of unittest, integration system, system tests and end 2 end tests.

~~~
ryanbrunner
Honest question. What's the value of a test like this?

    
    
        class MyObjectBuilder
          def create(object)
            if validator.valid?(object)
              factory.save(object)
            else
              raise "invalid"
            end
          end
         end
    
         def object_builder_test_success
           object_builder.mock(validator, mock_validator)
           object_builder.mock(factory, mock_factory)
     
           expect_method_call(validator.validate).return(true)
           expect_method_call(factory.save).return(true)
    
           object_builder.create(object)
        end
    
        def object_builder_test_failure
           object_builder.mock(validator, mock_validator)
           object_builder.mock(factory, mock_factory)
     
           expect_method_call(validator.validate).return(false)
    
           assert_raise(object_builder.create(object))
        end
    
    

This test is both _extremely_ typical for strong advocates of TDD and
"testable" designs, and also almost completely useless in my mind. The test is
literally a more pained version of the implementation of the method (not to
mention that there's typically a lot more ceremony around setting up mocks
than my pseudocode indicates)

It adds value in the sense that if someone randomly types garbage into that
file it will break, but it acts as a barrier to refactoring or business
requirements change, as pretty much nothing but the exact implementation of
the method will satisfy the test, and offers no documentation benefit over the
code itself.

~~~
billsix
Lol I was on a Ruby project like this, I said these tests are tautologies
since we're verbosely rewriting the implementation as a test. My yearly review
from that project? "Bill does not understand unit testing"

Nobody gets fired for writing too many tests.

------
heisenbit
James O. Coplin has a long of experience and is steeped in theory and
practice. His list of publications is about 200 entries long:
[https://sites.google.com/a/gertrudandcope.com/info/Publicati...](https://sites.google.com/a/gertrudandcope.com/info/Publications)

Unit tests are not free as they are also code that much is obvious. Coplin
however delves also into less obvious aspects of impact of unit tests on
design and also the organizational aspects. Ultimately coding patterns are
going to reflect the incentives that govern the system.

Software development is a lot about trade-offs. There is plenty to be learned
here how to do it. A addendum by him can be found here: [http://rbcs-
us.com/documents/Segue.pdf](http://rbcs-us.com/documents/Segue.pdf) but the
meat is in the 2014 article.

~~~
LoSboccacc
one thing that is really imposed by unit testing is the ability of exercise
single part of code independently.

it really brings out code smells: if you need mocks injected everywhere
instead of being able to use dependency injection cleanly, it shows. if you
have code paths that can only be triggered within events, it shows, etc.

having "wasteful" unit testing is more an investment for the future: when
users came with real bugs, the ability of reproducing their step in code and
fixing that in a no-regression suite is invaluable, but requires your app to
be testable in the first place, lacking which you are stuck with manually
testing stuff or even worse _cough_ selenium _cough_

~~~
ryanbrunner
A lot of times, the argument for why a lot of things are "code smells" turns
into circular logic:

Code that is hard to test is poorly architected. Poorly architected means it's
hard to test.

Example. Take this code (I'm reusing an example from this thread since I think
it represents typical "well-factored" code, and isn't an obtuse example to
prove a point):

    
    
       class FooBuilder
         def create(object)
           if FooValidator.valid?(object)
             FooFactory.save(object)
           else
             raise "Can't save an invalid object"
           end
         end
       end
    

It's reasonably easy to intuit the purpose of this code, and what exactly
makes for a valid object (since the validator is an explicit dependency of
this class.) I've often seen this called "poorly architected code", however,
since an isolated unit test needs to depend heavily on mocks to implement, and
you end up with something like this:

    
    
       class ObjectBuilder
         def initialize(foo_validator, foo_factory)
           @foo_validator = foo_validator
           @foo_factory = foo_factory
         end
    
         def create(object)
           if @foo_validator.valid?(object)
             @foo_factory.save(object)
           else
             raise "Invalid - can't save object"
           end
         end
       end
    

From the perspective of a coder coming in and trying to understand what's
happening here, this code is much more difficult to understand, despite being
more "testable". What makes a foo valid? How do I know what a "foo_factory"
is? I suppose I could assume that the class defined in foo_factory.rb is
probably one - but I can't actually be sure.

The code _is_ more extensible, for sure, but in a way that probably doesn't
matter. I can pass in any validator I want! Amazing! Except, in 99% of cases,
I'm going to have one way of validating something. The same goes for saving.

I would posit that at least 90% of the time that I see dependency injection in
a codebase, it's there _solely_ to aid testing and almost never adds practical
(as in, actually being used and not just theoretical) value to a codebase.

~~~
LoSboccacc
That's a false dilemma. One doesn't start criticizing from the most extreme
angle, because then it'd be fair to assume the alternative is equally extreme,
and I don't see anyone advocating for testing backends only exercising them
from the ui level.

Iow "if we take things to the extreme bad stuff will happen" contains its own
solution.

~~~
ryanbrunner
I don't think my point was extreme at all. The example I posted was an
extremely typical, even tame example of dependency injection and architecture
for the benefit of testability. I've seen countless variations of pretty much
exactly that or something quite similar (pulling out explicit dependencies to
be injected that should never reasonably change in normal circumstances).

Regarding "should we only exercise them from the UI level" \- I'm not 100%
sure what you're getting at - but if your point is that we should focus our
testing on business-facing use cases and not trivialities of what class calls
what method, then we're speaking past each other and are in complete
agreement.

------
partycoder
There are things that a machine can do in a much better and reliable way than
a human. Comparing lists of items is one of them (e.g: outputs of functions to
be verified against expected results). And that's the entire point of using
computers in the first place. Otherwise we can go back to pen and paper and
process forms by hand.

Does it make more sense for a human to do all the aspects of the testing by
hand? Of course not. Nobody has budget for that. It's much better to automate
as much testing as possible so testers can focus in higher level tasks. Like
the risk assessment involved in marking a build as releasable.

Then, unit testing encourages people to construct their software for
verification. This software construction paradigm in itself is enough of a
benefit even if unit tests are absent.

Construction for verification diminishes coupling, and encourages developers
to separate deterministic logic from logic depending on unreliable processes
that require error handling. Doing this frequently trains you to become a
better developer.

Unreliable processes can be mocked and error handling can be tested in a
deterministic way.

~~~
zby
I have not read the whole article - but I don't think he argues against
automated testing in general. Unit testing is only one type of automated
testing.

------
n72
Unit tests function as a kind of REPL for me and allow me to code considerably
faster than without them. Without them, it takes me considerable time each
time I want to test the smallest code change since in order to get my app to a
testable state I have to click around in the UI, enter a few values in inputs,
etc. This is just a waste of time. Moreover, there's a slightly costly context
switch which happens when I go from coding the feature to setting up my app to
test the feature. With judicious mocking, however, I save a ton of time
getting my app to a state where I can actually test the functionality I'm
coding and do away with that context switch.

~~~
ryanbrunner
That's great, and I do the same. But you really need to ask yourself what
value keeping those tests around in your repository is adding. In some cases,
absolutely they will have value, but I don't think there's anything wrong with
saying at least some of the tests you write were there as a tool to help you
build.

For example, I almost always do "gold master" testing when refactoring a large
unit or module of code (test the big picture input / output given a few cases
without regard for fine-grained tests within, refactor away as long as you can
keep the tests green). It's an amazing way of refactoring as it acts almost
like a safety harness - you have immediate feedback when you've done something
wrong and changed the behaviour of the class. After the refactoring is done,
however, those tests are almost useless, as they don't test the purpose of a
class but just dumbly look at the input and output.

I think a lot of the tests done via TDD should be looked at in the same way.

------
jondubois
I share a similar sentiment about unit tests. When you have a lot of code with
fast changing requirements, unit testing can be a HUGE waste of time. In my
previous job, we spent much more time writing and fixing unit tests than
actually adding new features. I was operating at like 1/10th productivity.

It might make sense if you're working for a huge corporation with a LOT at
stake. Unit tests then become a form of risk management - It forces employees
to think REALLY LONG AND HARD about each tiny change that they make. It's good
if the company doesn't trust their employees basically.

I MUCH prefer integration tests. I find that when you test a whole API/service
end-to-end (covering all major use cases), you are more likely to uncover
issues that you didn't think about, also, they're much easier to maintain
because you don't have to update integration tests every time you rename a
method of a class or refactor private parts of your code.

About the argument regarding using unit tests as a form of documentation
engine; that makes sense but in this case you should keep your unit tests
really lightweight - Only one test per method (no need to test unusual
argument permutations) - At that point, I wouldn't even regard them as 'tests'
anymore, but more like 'code-validated-documentation'; because their purpose
then is not to uncover new issues, but rather to let you know when the
documentation has become out of date.

I think if you're a small startup and you have smart people on your team (and
they all understand the framework/language really well and they follow the
same coding conventions), then you shouldn't even need unit tests or
documentation - Devs should be able to read the code and figure it out. Maybe
if a particular feature is high-risk, then you can add unit tests for that
one, but you shouldn't need 100% unit test coverage for every single class in
your app.

------
dtheodor
We often see criticism against unit testing that is based on facts such as
tests are badly written, unmaintainable, incomprehensive, and they drag
development down.

Which may very well be true! But I am amazed at the conclusion: That because
tests are badly written, writing tests is a bad thing. No! Any code can be
badly written, it doesn't mean that writing code is a bad thing. Tests, like
any other piece of code, also need to be designed and implemented well. And
this is something you need to learn and get experience with.

As to whether well-written unit tests are worth it, I cannot imagine how
someone could efficiently maintain a codebase of any size without unit tests.
Every little code change is a candidate to break the whole system without
them, especially in dynamic languages.

------
nuggien
Previous discussion:
[https://news.ycombinator.com/item?id=7353767](https://news.ycombinator.com/item?id=7353767)

~~~
shilman
Thanks for the link!

------
scotty79
I think I know why I don't enjoy writing test. Most of my enjoyment from
programming comes from feeling of power when I am able to write concise code
that does a lot for me.

Testing spoils the fun as now I need to write another piece of code for each,
single thing that my original piece of code is doing.

I am no longer a wizard casting fireball into a room. I'm also the guy that
has to go over the corpses and poke each one with a stick strong enough and
for long enough to check if they are absolutely totally dead.

~~~
xtreme
Extending that analogy, I enjoy having a robot that will tell me if all the
enemies are dead. Then I can try out different spells to my heart's content
without having that nagging doubt about their effectiveness.

~~~
scotty79
Except it's not a robot. It's a robot per body. Sure some components are
common but you still have to build and grease each bugger yourself and adjust
them whenever you change the spell in any meaningful way because enemies will
drop differently. I'm too lazy for that. I'll just walk with the people I
should get through the room and kill the enemies that are meaningfully not
dead. Besisdes, after this room, there's another and who has the time.

From time to time some of the people I lead will die due to my sloppiness but
such is life. It's not like I'm leading royal family through this dungeon.

------
MichaelBurge
If I'm writing Perl or something, I'll write unit tests just to verify the
code runs in the basic cases.

I like Haskell because I can skip most of the unit tests. Integration tests
are still good, and some unit tests like "confirm that test data are equal
under serialization and then deserialization" help with development speed. But
I can usually refactor vast swathes of code all I want without having to worry
about breaking anything.

If you do write unit tests and your test passes on the first try, make sure
you change the output a little bit to ensure it fails. It's more common than
you'd think to accidentally not run a test.

~~~
soaringmonchi
I couldn't agree more. I am doing a lot of nodejs development these days and
it's usually a pain to verify that the code works the way it should. Writing a
quick bunch of unit tests is very superior to loading your code in an
interactive console and manually fiddling with inputs / outputs. These unit
tests often break at the first refactoring and we rewrite them at that point.
I'm a bit surprised that people argue that unit tests facilitate "aggressive"
refactorings. Aggressive refactoring to me means architecture changes and unit
tests almost never survive those. Integration testing is what ensures proper
functioning in those situations for me.

------
keithnz
I once had a talk with Kent Beck way way back in the day (early 2000s) about
how much unit tests there should be, etc... and I think its nicely captured by
his reply here

[http://stackoverflow.com/questions/153234/how-deep-are-
your-...](http://stackoverflow.com/questions/153234/how-deep-are-your-unit-
tests)

A lot of people seem to miss many of Kent's subtly, but intentionally phrased
advice. Unit Tests are a liability, so use them responsibly and as little as
possible, but not at the expense of removing confidence in your software.

Also, delete tests that aren't doing you any favors.

~~~
misja111
Thanks for the link! I never knew that Kent Beck, one of the fathers of TDD,
said such a thing already in 2008. And in the mean while herds of developers
have started to chase goals like 90% test coverage, thinking that this is the
way that guys like Kent Beck do TDD. If only they would have known earlier!

------
bluejekyll
I generally try to keep my HN comments positive, but this is total Bullshit,
yes, with a capital 'B'.

Unit tests are not albotrosses around the neck of your code, they are proof
that the work that you just did is correct and you can move on. After that
they become proof that any refactor of your code was correct, or if the test
fails and doesn't make sense, that the expectations of your test were
incorrect. When you go to connect things up after that, and they don't work,
you can look at the tests to verify that at least the units of code are
working properly.

I am no TDD fan, but I do believe that writing your code in a why that makes
it easy to test generally also improves the API and design of the entire
system. If it's unit testable, then it has decent separation of concerns, if
not, then there may be something wrong (and yes this applies to all
situations). I use this methodology for client/server interactions as well
where I can run the client code in one thread and the server in another, with
no sockets, to simulate their functioning together (thus abstracting out an
entire area of potential fault that can be tested in isolation from network
issues).

The article/paper raises good points about making sure that the tests are not
just being written for the sake of code-coverage, but to say they are useless
is just sloppy. Utilize the testing pyramid [1], if you adhere properly to
that, everything about your system will be better.

I have a serious question, given that this was written by a consultant, is it
possible that tests get in the way of completing a project in a timely manner,
thus causing a conflict of interest in terms of testing?

[1] -
[http://martinfowler.com/bliki/TestPyramid.html](http://martinfowler.com/bliki/TestPyramid.html)

~~~
Someone
_" Unit tests are [...] proof that the work that you just did is correct and
you can move on."_

Unit tests only can proof your software to be buggy. Even for the extremely
simple " _two_ , a function that, given no arguments, returns the number 2",
your unit test can't verify that to be _always_ true. It may fail to return on
Tuesdays
([https://bugs.launchpad.net/ubuntu/+source/file/+bug/248619](https://bugs.launchpad.net/ubuntu/+source/file/+bug/248619)),
internally use a web service that doesn't work on leap days
([http://www.wired.com/insights/2012/02/leap-day-azure-
outage/](http://www.wired.com/insights/2012/02/leap-day-azure-outage/)), other
calls may overwrite the 'constant' it returns
([http://programmers.stackexchange.com/questions/254799/ever-c...](http://programmers.stackexchange.com/questions/254799/ever-
change-the-value-of-4-how-did-this-come-into-hayes-thomas-quiz)), etc.

~~~
sangnoir
> Even for the extremely simple "two, a function that, given no arguments,
> returns the number 2", your unit test can't verify that to be _always_ true

That is true, I don't think anyone argues against that point. What unit tests
do, is verify that under the code works under _specific_ conditions (as
defined in the test's set up). When unit tests fail _under those specific
conditions,_ then you know there is a problem.

Another thing I like about unit tests is that it is much easier (and faster)
to test different combinations of conditions

------
shilman
Hilarious:

 _They informed me that they had written their tests in such a way that they
didn 't have to change the tests when the functionality changed._

~~~
maxxxxx
We have a test team that requires the product to stay backwards compatible
with tests, not the other way.

------
henrik_w
There was also a follow-up article [1]. My take on the two articles is that he
argues that integration tests should be able to replace unit tests in most
cases. However, in my own experience, both kinds of tests have their palces.

Why unit tests are good: \- You get well-tested parts that you can use in your
integration tests, so that the integration tests truly catch the problesm that
couldn't be caught at a lower level. This makes trouble-shooting easier.

\- Decoupled design - one of the key advantages of TDD

\- Rapid feedback. Not all integration tests can be run as quickly as unit
tests.

\- Easier to set up a specific context for the tests.

There are more details in the blog post I wrote as a response [2].

[1] [http://rbcs-us.com/documents/Segue.pdf](http://rbcs-
us.com/documents/Segue.pdf)

[2] [https://henrikwarne.com/2014/09/04/a-response-to-why-most-
un...](https://henrikwarne.com/2014/09/04/a-response-to-why-most-unit-testing-
is-waste/)

~~~
maskull
If the code is decoupled it's easier to reason about. The tricky part is
breaking things down to the right level of granularity.

------
testingnut
People start writing tests like they start coding - badly. They forgot all the
DRY principles and don't architect their tests to reduce their coupling on the
code they are trying to test. The result is tests that are a drag on
additional new functionality. Even these are better than no tests as you can
refactor the tests to reduce the coupling.

The debate about whether UT or system tests or something in the middle is
better is missing the point. A test should be understandable at any level. 5+
mocks per test generally doesn't help the next guy understand what you are
trying to test.

If you can abstract your system behind an API to drive and test it, you'll
have much longer lasting tests that are more business focused and importantly
are clearer for the next person to understand.

I can see great value in identifying the slow and rarely failing tests and
running them after the quick / more information producing tests. Aee there any
CI support for such things? I know TeamCity can run failing tests first...

~~~
fahrradflucht
That would be true also if the next one trying to understand what you are
trying to test is actually a girl.

------
jacquesm
Operative word 'most', and then only when done by someone who doesn't
understand the goal of unit testing. Any tool can be abused, including
testing.

I became a 'convert' after having to clean up a fairly large mess. Without
first writing a bunch of test code there would have been no way whatsoever to
re-factor the original code. That doesn't mean I'm a religious test writer and
that there is 150% test code for each and every small program I write. But
unit testing when done properly is certainly not wasteful, especially not in
dynamic languages and in very low level functions. The sooner you break your
code after making changes the quicker you can fix the bug and close the black
box again. It's all about mental overhead and trust.

Unit tests are like the guardrails on the highway they allow you to drive
faster confident that there is another layer that will catch you in case
something goes wrong rather than that you'll end up in the abyss.

------
powera
Why are so many people saying that if _some_ developers write bad unit tests,
then all unit tests are pointless and a waste of time?

Yes, I've seen thousand line files of boilerplate unittests that don't
actually say anything useful about the system. I've also written unit tests
that tell me in 2 minutes rather than in 3 weeks that somebody has broken my
code.

If your standard for a system of testing is that it _guarantees_ that people
can only write good code, you're insane.

------
kriro
Just to add another perspective. There is a (pretty sad but real) business
case for unit tests (don't need be good just need to exist :( ). In some
b2b/enterprise fields it is an excellent sales pitch to be able to say we have
X% test-coverage or 500 unit tests OMG. The people that make the buy decisions
want to sleep well no matter if it is based on sound reasoning or not. Test
coverage is almost a "feature" (in the bad enterprise IT sense). Sounds less
risky to buy the product with more test coverage. Less risk I can understand
it's like buying insurance...which is great since insurance turns risk into a
budgetable item...yes please sell me this awesome software with that high test
coverage.

/cynic

------
planetjones
Throw away tests that haven't failed in a year. That just a ridiculous point
IMO. Written properly tests act as perfect docimentation for the system. So
just because a screen or a process hasn't changed in a year, so therefore the
tests have been consistently passing does not mean you throw the tests away!
You never know when someone will need to maintain that piece of the system.

------
ebbv
There's actually a section of this where he basically says that a test that
always passes is not useful (provides no information) and a test that fails
sometimes is useful (provides lots of information.) I'm not sure exactly what
failure of reasoning lead to this conclusion but it's totally bogus. They are
both useful.

The ideal case is that your codebase is entirely made up of code that never
fails and tests that always pass. Obviously sometimes you are going to have
tests that fail and introduce bugs that cause tests that used to pass to fail.
But that's the reason that you write those tests, to find those problems.

The author gives the silly example of a method that always sets x to 5, and a
test that calls it and makes sure x is now 5. That seems like a bad test but
anyone who's actually done work as a developer understands why it isn't. If
you skip the tests that are simple and straight forward and seem like a waste
of time and only write more complicated tests then you will have a hard time
reasoning what failed when the complicated test fails. Was your x = 5 method
faulty? You don't think so but you don't have proof since it wasn't tested.
Having the test, as silly as it seems, lets you know that method is working.

Anyone who has been on a team that skips easy/simple tests knows what a
mistake it is. And if you don't, you will eventually.

------
tigershark
I think that the author never understood why unit tests have been introduced.
Nowadays, with BDD slowly gaining momentum,unit tests are even more
imprescindible and they really _do_ add business value. In the past TDD used
correctly added business value, but BDD takes it one step further helping to
write more valuable tests. When he speaks about testing all the registers
states and all the possible combinations of inputs it is apparent that he is
missing the point. The only thing that a well written unit test should
accomplish is to make sure that a requirement is implemented correctly. And it
_must_ be written while designing the code. When I write some new feature I
try to understand the requirements and model them into scenarios. At the same
time I start writing the code to implement the feature required that is
complete only when the respective scenario is green. During this workflow I
change multiple times both the code and the tests and, contrary to what the
author experienced, my velocity is not hampered by the tests, but it's
actually increased. This happens because I can improve my design thanks to the
tests that force me to see the problem from the users point of view. I don't
really think that a person that only focuses on writing code without writing
any scenarios can really tackle the problem in the best way.

~~~
kevan
>The only thing that a well written unit test should accomplish is to make
sure that a requirement is implemented correctly.

A common side effect of writing these tests is discovering missing or
conflicting cases. A recent example for me in eCommerce is our algorithm for
determining if a particular product can be added to your cart. Things start
simple (is it in stock?), but they get complicated really quickly. Is there a
dropship vendor with stock? Do we expect to receive more stock in the next 24
hours? Is it discontinued? Clearance? Does it look like we'll run out in the
next day and accidentally oversell the stock we have? (realtime inventory is a
little fuzzy still).

When we unit test the module that handles all of this we don't care about
internal state or exhaustive input testing, we care about whether the user
sees "In Stock" or "Out of Stock" for the situations we encounter

------
lokedhs
The beginning of the essay discusses traditional bottom-up programming to top-
down programming (using object orientation).

I have written a very large amount of Java code in my career, but after having
spent a lot of (personal) time on a Common Lisp project (web application) I
can safely say it's still possible to build modern applications using a
bottom-up approach. I recommend people try it, it can be quite refreshing.

~~~
taeric
Any recommended reading to describe the bottom-up approach you are taking?

~~~
lokedhs
Unfortunately there isn't much that I know of. There are several frameworks
out there, but I built my own. It only requires very little code to get
something that mimics, for example, Ruby on Rails.

As I mentioned in another reply, I intend to do some writeups on this stuff,
but unfortunately it doesn't have as high priority as it should. But at least
a video or two should be doable soon enough.

------
kabdib
Mandate from Above: "All code that is checked in must have seventy percent
code coverage"

Developer Response: Keep tests the way they are. Puff up shipping code by
adding layers that just check and transpose arguments; enough of these layers
and you get your 70% metric of code that's basically guaranteed not to have
issues, and check in.

My response, trying to take that stuff and port it: Rip out about 75% of the
junk code while cursing clueless management and developers who, ultimately,
didn't write very good tests (none of the tests ran anyway, because they
required an elaborate lab setup that involved undocumented APIs, installs of
binaries meant for OSes that were nearly end-of-life, a SQL server and a bunch
of other shit infrastructure. Beware test environment creep).

------
lazyfunctor
I've read this advice couple of times "convert your unit tests to assertions".
What does it actually mean? Say in the context of web dev, you add assertions
to the code and when they fail you log an exception and move on?

Any links related to it will be helpful.

~~~
majewsky
Assertions are usually only run in development or test mode. So if you run an
integration test, the unit tests happen inline through the assertions.

------
OJFord

        > When I look at most unit tests — especially those
        > written with JUnit — they are assertions in disguise.
    

Very often they're assertions not at all in disguise. Python's `unittest`, for
example.

I'd always assumed the point was to run these assertions at 'test-time', prior
to distribution, and not have that code in the 'real' program.

Besides, most (?) of the time we probably want to fail more gracefully than
that. (Okay we could `except AssertionError`, but typically it's going to be
better to return something else, or raise and handle a more specific
exception.)

~~~
wongarsu
>I'd always assumed the point was to run these assertions at 'test-time',
prior to distribution, and not have that code in the 'real' program.

That's the point of assertions, as they are usually not included in release
mode builds in compiled languages. In Python you could just write an assert
function that does nothing if some global "release" variable is set to true.

You only gain something from unit-tests if they test something that can't be
equally well tested by writing the assertions into the regular code and
running the software with typical input.

------
dewiz
Is it possible to do TDD without unit tests? I ask because a place where I
interviewed had no concepts of mocking, injecting, and would just test the
whole chain of classes. But dev leads would insist on TDD. Hmm?

~~~
crdoconnor
Yep:

[https://en.wikipedia.org/wiki/Acceptance_test-
driven_develop...](https://en.wikipedia.org/wiki/Acceptance_test-
driven_development)

I much prefer this approach. The looser coupling gives you more freedom to
undo your initial architectural mistakes.

You also can't effectively do unit test driven development on big balls of
mud.

I dislike Gherkin-based languages though. The syntax design was not
particularly well thought through.

------
svckr
Wow.

> That means that tests have to be at least as _computationally complex_ as
> code.

My BS sense is tingling. No matter how complex the code, in the end it comes
down to comparing the output of a function (or state after execution) against
what you expected.

Granted, OO regularly leads to design where unit testing goes straight to hell
and after way to many lines of test-setup you essentially only test that your
mocking framework works. But spare me these incorrect blanket statements –
they don't help – thank you.

------
fsloth
My kneejerk reaction is that this is BS but the text has good points and can
function as an anecdote on how not to do things.

 _"... Large functions for which 80% coverage was impossible were broken down
into many small functions for which 80% coverage was trivial. This raised the
overall corporate measure of maturity of its teams in one year, because you
will certainly get what you reward. Of course, this also meant that functions
no longer encapsulated algorithms. It was no longer possible to reason about
the execution context of a line of code in terms of the lines that precede and
follow it in execution,"_

Unit tests which break code are stupid. Refactoring is good, but just
splitting a large function into smaller pieces does nothing to improve the
value of code unless it's done so that there is an understanding of the
algorithm available and communicated.

Everything can be abused if not used craftly.

------
bengalister
At my company, we (developers) tend to develop mainly automated functional
tests and we test our product in blackbox mode. I also prefer favouring
functional tests and found that actually many integration tests become
useless. For unit tests, I keep that for pieces of code that are used
thoroughly in different contexts, i.e, actually code that could be turned into
an external library and/or code that is hard to test in integration/functional
tests => it makes non-regression much faster to test.

For blackbox mode, I am not that convinced that it is the proper strategy,
especially when the product is being built incrementally. The typical example
is when an entity state is being modified in a UC and the function to test the
state is not yet developped. I'd prefer in that case to have the test verify
that the state has been properly updated directly in DB.

------
snarfy
With modern compilers and debuggers, a good set of integration tests is all
you need. You can refactor large parts of the code base and if the integration
tests break, thanks to debuggers etc you can easily identify the breaking
changes within minutes. There is no need for 100% code coverage in unit tests
to catch these same changes. If you do need it your tooling is lacking.

Writing code is expensive. If you have more test code than real code it means
you value correctness over features. If I can skip unit tests entirely and
have a 95% functioning system, I'm not sure that 5% is worth an extra 500% or
so lines of code needed in unit tests for 100% code coverage.

Unit tests might seem more important in dynamic languages like JavaScript, but
that really just points to poor tooling.

------
mokn80
"Throw away tests that haven’t failed in a year" What a load of crap. We have
a test that asserts creating a new user doesnt fail, it hasnt failed for over
a year, but the day it does..

------
WalterBright
I'm not giving up unit tests anytime soon. I've had excellent success using
them - success being much faster development time and far fewer bugs found in
the field.

------
tux1968
Not sure what this says about unit testing in general, but Perl 6 as a
language is specified by a test suite. Ie. the only official language
definition is the test suite:

[http://doc.perl6.org/language/faq#What_is_the_Perl_6_spec%3F](http://doc.perl6.org/language/faq#What_is_the_Perl_6_spec%3F)

------
sivanmz
If I write testable code, coverage comes naturally and I don't worry about it.

I do think it's misguided to focus on coverage.

To me unit testing is a development tool. I write my class inside a test suite
and extract it when ready.

------
ensiferum
Wow so much fail. Where do these mickeymouse programmers repeatedly spring up?
"Oh, I'm a supermanprogrammer and I can get it just _right_ the first time
because I use my brain!"

Sure I believe that there are people who can do this and get the code right by
just thinking it through. But I have a question for even these people (and
their organizations).

\- What happens when these people leave and the new junior dev becomes the
maintainer? \- What happens when the code is migrated/reused somewhere else?
\- How do you make sure your code works at least the same as it did before
after you: * modify it * update some dependent component somewhere, say some
open source library that your code uses internally. These have bugs too you
know...

Simply you don't unless you write the tests. The real value of testing (unit
testing, regression testing, system testing) comes once you have that nice
test suite and you can automate it and make sure that on every change nothing
breaks. This is a beautiful thing to have, because simply no human can
understand these software systems fully to make sure that some innocent
looking change doesn't break something. These things unfortunately happen.

Sure testing is hard if you make it so. I said before that I get an average
ratio of about 2/5 of actual code/unit testing code. This doesn't mean that
"this is so bothersome let's not do it", what it should mean is that you
_design_ your system with testability in mind. A primary architectural goal in
any design should be _testability_ How easy it is to write unit tests for your
classes and methods. The easier it is, the smoother the unit tests are to
write and maintain and the better code quality you will have in the end.

Also note that when you design your software to be testable you get a few
other abilities for free such as reusability. One of the most important
starting points is to find the _right_ separation of truly orthogonal
components and define their interfaces right. Way too often I see code where
the engineer didn't understand this and clumped several unrelated concepts
together. This really makes the testing hard and painful.

~~~
Const-me
> the new junior dev becomes the maintainer

this shouldn’t happen. Maintaining a complex project is the job for a senior
developer.

> What happens when the code is migrated/reused somewhere else

someone else need to carefully test what’s he/she reusing.

> The real value of testing comes once you have that nice test suite and you
> can automate it

Sure there’s _some_ value, especially for dynamic languages and/or for
developers who don’t use asserts and other runtime checks.

But that value isn’t free.

Not only they’re expensive to develop, they also make very expensive to change
things: you change a minor thing in the code, and suddenly dozens of tests are
broken.

If your code is implementing some 25-years old spec that’s not going to change
for another 40 years, unit testing will likely pay off. If you’re creating
something new and rapidly changing, using a statically-typed language that you
know how to use correctly, and you understand the value of asserts — unit
tests are IMO waste of time.

~~~
ensiferum
>>this shouldn’t happen. Maintaining a complex project is the job for a senior
developer.

Oh but it happens. Even the other senior dev might not be a superman of
similar competence.

>> someone else need to carefully test what’s he/she reusing.

Sure. Oh...look there's a unicorn!

>>But that value isn’t free. So that's your excuse for bad code quality?

>>Not only they’re expensive to develop, they also make very expensive to
change things: you change a minor thing in the code, and suddenly dozens of
tests are broken.

Sounds like bad orthogonality again. If i don't make any interface changes my
tests continue to run as expected. Even with interface changes it comes down
to design again. With bad starwars/spiderweb design with lots of coupling it
gets harder. Which is my main point. You have to think this through starting
at the design level.

~~~
Const-me
> the other senior dev might not be a superman of similar competence

Superman? Any experienced developer should be able to work with other people’s
code. If you can’t, you’re either inexperienced, or need to learn the
language/frameworks used.

> look there's a unicorn!

IMO 70% of third-party code is crap, regardless on unit tests. Moreover,
nearly 100% of the third-party libraries comes with a license saying “provided
as is, without warranty of any kind” or similar.

When using any third-party code, you have to carefully test whether it fits
your particular purpose. You’ll need to test that code as a part of your
system, feeding your data, and measuring the performance characteristics you
need. Unit tests don’t help with that.

> You have to think this through starting at the design level.

Waterfall model only works if you’re building a space shuttle or similar.

Architect isn’t a prophet. Design changes through development as requirements
change. Between major versions, they can change dramatically. Also there’re
external factors that often affect your design: language, it’s runtime, host
OS, sometimes even CPU architecture — they all change.

Eventually, you gonna need to update your design even if at the time of your
version 1.0 it was just flawless.

~~~
ensiferum
>> Superman? Any experienced developer should be able to work with other
people’s code. If you can’t, you’re either inexperienced, or need to learn the
language/frameworks used.

You'd think so, but thats not always the case. There are many boundary cases
and "gotchas" that might not be even visible in the code. (I.e. the stuff that
_isn 't there_). Secondly the follow-up dev might be not that interested in
code quality. Personally I even feel that if the code doesn't have unit tests
in the first place the original developer didn't care and the shop didn't care
then why should I bother?

>> Unit tests don’t help with that.

Ofc they do. If I write a class that uses for example boost.filesystem, and I
have my unit tests that test my class. I upgrade my copy of boost. Ofc, my
unit tests won't tell me that some other part of boost.filesystem has
regressed but at least if some API that I'm using has changed and behaves
differently I have a chance of catching it.

>> Waterfall model only works if you’re building a space shuttle or similar.

Nobody said anything about a waterfall.

>> Eventually, you gonna need to update your design even if at the time of
your version 1.0 it was just flawless.

Personally I don't rush to write the unit test _immediately_ , but only after
I've put a bit more code together and verified that the design works. Call it
a _gut_ feeling or intuition. But once it's stable I write the unit tests.
That doesn't mean that I'd ignore the "testability" aspect when I do the
design. Quite the opposite it's an important aspect of the whole process.

~~~
Const-me
> I even feel that if the code doesn't have unit tests in the first place the
> original developer didn't care

I saw unmaintainable projects of very low quality with near 100% unit test
coverage.

I saw projects with outstanding quality without unit tests.

To me, presence/absence of unit tests means nothing.

> uses for example boost.filesystem

Why boost doesn’t change API when changing the implementation to something
incompatible? If they did, you’d catch (and be forced to fix) most of those
compatibility issues even earlier than unit tests — at compile time.

Also, unit tests are IMO nearly useless with IO heavy code like filesystem
access. If you use mockups instead of actual IO, you won’t get those errors
from the OS i.e. will test nothing. If you don’t, the tests will be too
complex to deserve “unit tests” name.

> Nobody said anything about a waterfall.

A development model where you first design the entire system, then implement
the entire system is waterfall. The approach only works well for some
projects.

> But once it's stable

Many software projects evolve constantly: you stabilize your design, release a
version, then based on e.g. market performance you redesign your system and
create a next version of it.

Unit tests work well when you’re doing many local changes, when API stays
stable and you only replace or fix the implementation. They are also useful
when you code business logic, or lower-level code with lots of math (esp.
SSE/AVX code).

However, if you have high unit tests coverage, API-breaking redesign is
expensive. Unit tests might cause you stick to your initial 1.0 design, but
because you need those new features you’ll start write hacks introducing
technical debt. Even if you take your time, redesign, stabilize the design,
and write new tests — this costs time + money.

Just like most other aspects of software engineering, unit testing is a
tradeoff: improves something, worsens something else.

------
yason
The final points at the end of the article are gold. They match my own
experiences exactly.

------
V-2
_" Unit tests are unlikely to test more than one trillionth of the
functionality of any given method in a reasonable testing cycle. Get over it.
(Trillion is not used rhetorically here, but is based on the different
possible states given that the average object size is four words, and the
conservative estimate that you are using 16-bit words)."_

It's the most bizarre reasoning I've seen for a while. Well, yes, of course,
and my string trimming function will always have coverage of essentially 0% no
matter what I do, since strings can be of any length, and there's way more
possible strings than atoms in the universe... we clearly have a different
concept of "functionality" though

 _" When developers write code they insert about three system-affecting bugs
per thousand lines of code. If we randomly seed my client’s code base — which
includes the tests — with such bugs, we find that the tests will hold the code
to an incorrect result more often than a genuine bug will cause the code to
fail! Some people tell me that this doesn’t apply to them since they take more
care in writing tests than in writing the original code. First, that’s just
poppycock."_

Of course, because in reality it's not about being more careful or attentive.
It's about the fact that tests don't (or aren't supposed to) contain any
logic! (Meaning conditional statements of any sort, loops, etc.) It's logic
that breeds an overwhelming majority of bugs. If your tests contain any, it
means they're written badly.

 _" even if it were true that the tests were higher quality than the code
because of a better process or increased attentiveness, I would advise the
team to improve their process so they take the smart pills when they write
their code instead of when they write their tests"_

No, it's not about "smart pills" (this condescending tone is so annoying),
it's about the fact that - otherwise than with my tests - I can't keep logic
out of my production code.

 _" Most programmers believe that source line coverage, or at least branch
coverage, is enough. No. From the perspective of computing theory, worst-case
coverage means investigating every possible combination of machine language
sequences, ensuring that each instruction is reached..."_

...oh God, enough.

I don't know the author, but it's quite clear he's some CS professor rather
than a real-life full time software dev

Remember this article? [http://blog.triplebyte.com/who-y-combinator-companies-
want](http://blog.triplebyte.com/who-y-combinator-companies-want)

[https://phaven-
prod.s3.amazonaws.com/files/image_part/asset/...](https://phaven-
prod.s3.amazonaws.com/files/image_part/asset/1669427/kXHIcd9WPsdYClR3CsnR9lipkDk/medium_who_yc_co_want.png)
\- that's who YC startups want. When I saw it for the first time, I wondered
why "academic programmers" rated the worst.

------
em3rgent0rdr
Copied from end: In summary: • Keep regression tests around for up to a year —
but most of those will be system-level tests rather than unit tests. • Keep
unit tests that test key algorithms for which there is a broad, formal,
independent oracle of correctness, and for which there is ascribable business
value. • Except for the preceding case, if X has business value and you can
test X with either a system test or a unit test, use a system test — context
is everything. • Design a test with more care than you design the code. • Turn
most unit tests into assertions. • Throw away tests that haven’t failed in a
year. • Testing can’t replace good development: a high test failure rate
suggests you should shorten development intervals, perhaps radically, and make
sure your architecture and design regimens have teeth • If you find that
individual functions being tested are trivial, double-check the way you
incentivize developers’ performance. Rewarding coverage or other meaningless
metrics can lead to rapid architecture decay. • Be humble about what tests can
achieve. Tests don’t improve quality: developers do.

~~~
taeric
This set of takeaways seems quite solid to me.

In particular, I've pushed the "remove tests that haven't failed in years"
before, but never had any traction. Are there teams that actually do this?

~~~
apsdsm
I really hope not. That would be kind of like saying "this pre-flight
checklist hasn't failed for years, so let's just skip them before we take
off."

~~~
taeric
Ha! I always assume some good faith in the judgement of the people that would
be removing things. Didn't realize that would be withheld here.

So, in short, no. It is not just a time bound thing. However, if you have not
had a test fail in years, it is a good time to audit the test to make sure it
is still relevant. If it isn't, get rid of it.

It is this kind of logic that insists on the pre flight video that everyone
ignores. We can, and should, do better.

------
kmiroslav
I'm really annoyed there is no date and no way to date when this was written.
Coplien has been around for a long time, this could have been written
yesterday or thirty years ago.

~~~
dang
Googling and searching on Twitter both suggest 2014, which is also when the
previous HN thread was posted, so we'll go with that.

------
perfunctory
Excuse me. Did you read the article?

~~~
dang
This comment breaks the HN guidelines. Please (re-)read them:
[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html).

We detached this comment from
[https://news.ycombinator.com/item?id=11799791](https://news.ycombinator.com/item?id=11799791)
and marked it off-topic.

------
csours
Watching the Dragon Innovations videos on Design for Manufacturing [1]
earlier, Bill Drislane said that anything you can afford to test must not
fail.

I think the corollary is that if you cannot afford to test something you
cannot rely on it.

1\.
[https://www.youtube.com/watch?v=zCnTUOxMl_4&list=PLNTXUUIxHy...](https://www.youtube.com/watch?v=zCnTUOxMl_4&list=PLNTXUUIxHyNwrlAh2ZkaMTSBrgk86wC-a&index=14)

