
Lean Testing or Why Unit Tests Are Worse Than You Think - saranshk
https://blog.usejournal.com/lean-testing-or-why-unit-tests-are-worse-than-you-think-b6500139a009
======
guygurari
The article makes some fair points for performing end-to-end tests. There is
however a benefit of unit tests over end-to-end tests that is not mentioned:
code locality. The cost of fixing a discovered bug seems to grow exponentially
with the amount of code in which the bug may appear. Unit tests typically
cover a small part of the code. If a unit test fails, isolating and fixing the
bug is typically cheap. On the other hand, if an end-to-end test fails (due to
the same bug) then isolating and fixing the bug can be quite costly because
there are many different parts of the code that need to be checked. To me,
code locality is the main advantage of writing unit tests.

At the end of the day, there is a spectrum of tests going from unit tests to
end-to-end tests. The spectrum represents several trade-offs such code
locality vs. coverage. In my experience, the most economical approach is to
write a balanced mix of tests the lie along this spectrum.

~~~
mosselman
I hear this argument a lot, but I do not think it is valid. What happens is
that you simply put in the time before a bug occurs with every unit test that
you write. Which isn't very efficient, seeing as not all of your units will
lead to bugs. So yes, once you have all of your Units covered 100% it will be
easier to find bugs, but you have invested a lot of time in order to get here
and in order to keep the unit-test-suite maintained.

~~~
pron
But the goal of tests isn't just to find bugs in newly-written code. It's to
defend against regressions, and a regression is even harder to localize, given
the team may not even be well-familiar with the failing code.

~~~
bluGill
This is both correct and incorrect.

Correct that the goal is to prevent regressions. I claim (I don't know how to
study this) 80% of your tests will never fail and so they could safely be
deleted - but I have no insight into which tests will fail so I say keep them
all.

Incorrect because in fact it isn't hard to localize failures: it is something
in the code you just touched!

~~~
abecedarius
I don't understand how you could live the experience that the problem is
always with the just-edited code. The closest I can come is supposing that
you've always worked with thoroughly unit-tested code (and correct, well-
documented libraries, etc.).

~~~
bluGill
I'm lucky enough that the code base I work on was a "big rewrite" not long
enough with the goal of having everything tested. Also we are an embedded
system where we know exactly which version of each library to support and
upgrading any library is in itself a big deal done as a separate exercise.

------
paulmd
Integration testing and unit testing aren't direct substitutes. You would
never ever say "I have unit tests, I don't need to run the product before
shipping it". Furthermore, automated integration testing is strictly less
capable than a human doing integration testing.

Unit tests also have the advantage of being cheap to set up and cheap to run,
so their limitations in whole-system testing are offset by the fact that you
can have many, many more of them. You can easily blow through a couple
thousand unit tests in a matter of 10 minutes or less, whereas even a modest
battery of integration testing will be an overnight process at best. That
means your OODA loop is a lot faster on unit testing.

(the ideal is of course structuring your application so you can deploy many
tests in parallel, but that's not always possible.)

Fixing Selenium so WebDriver instances don't take 60+ seconds to launch would
be a major help. Until then, integration tests that work at a controller level
(minus the UI) are probably the best compromise between speed and integration.
You can pass in a mocked request as input, and assert on the output. Runs much
faster than a "true" integration test with a driven browser client, but hits
the whole stack better than a unit test. They are also really easy to set up,
since you can just have a tester or an integration test go through the motions
and then dump request data to JSON, then feed it back as a test. The lack of
state persisted between requests makes implementation a lot simpler.

~~~
discreteevent
"Unit tests also have the advantage of being cheap to set up and cheap to
run".

They are not cheap to develop and maintain however. And given enough
developers a certain percentage of them will completely distort the design of
the code to enable unit testing making the code itself much more complex.

~~~
luord
I'm having trouble thinking of how unit tests are "expensive to maintain"; you
don't really need to do anything other than having them there.

If it's because you need to change them after a feature changes or breaks,
isn't that the entire purpose of testing and not exclusive to unit tests?

~~~
jonotime
Unit tests can be hard to maintain for several reasons. When you refactor your
code (renaming methods, changing parameters, splitting methods up, etc), you
will need to update those tests. Also some types of unit testing can be more
brittle then others. For example if you use mocks to invocation counts,
refactorings can be even more costly.

~~~
tynpeddler
This comes up because there's a secret disagreement about the definition of
the term "unit test". Some people see units tests as per method tests. If you
have 20 methods in a class, all 20 methods get unit tested.

Others see units as the smallest bits of irreducible business or tech
knowledge. So if you have 20 methods in a class, you test the 5 public ones,
or the methods that compose a useful "unit". This is the definition used by
the original test driven development book.

Many of the unit test advocates I've read use the second definition because
testing private functionality incurs all the problems the article points out.

Using the second definition leads to the testing pyramid, which is a thing of
beauty. Unit tests have a few mocks to the unit's partners. The unit tests
test for the correctness of the unit. The integration tests are one level up
in terms of code coverage in a single test, and they ensure that the mocks
used in the unit tests are correct. The e2e tests are then used to test that
the app starts correctly and is configured correctly. The obvious happy and
sad path scenarios are also tested to ensure that dependencies are functioning
correctly (in a complex application there can be hundred of "obvious" e2e
tests).

In my career, every single instance I've seen about someone complaining about
writing unit tests, they're either using the first definition I mentioned
above, or their code is not composed correctly. I'm sure exceptions exist
somewhere, but I've yet to see them.

Using integration tests like the article describes leads to a host of
problems. For starters, it's extremely difficult to get complete code coverage
without huge or redundant test suites. These large test suites are harder to
read and take longer to write because more complex testing requires more
complex mocking. Regression is also more likely during large refactors since
you haven't clearly enumerated expected behavior.

~~~
miceeatnicerice
Fun fact: in the glossary at the end of the original xp book, it describes
unit tests as 'tests from the perspective of the programmer', and that's that
- a woolly cop-out.

------
pshirali
I find this article misleading; with a biased PoV. The author says "empirical
evidence is missing", yet advocates a testing methodology, banking largely on
articles and PoVs of other individuals. Is there any software project the
author would like to quote as a case-study? Is there _any_ evidence to prove
that the Lean method is _better_ OR, the definition of _what is better?_

It is OK to be economical in what one would see as a cost of testing a product
end-to-end. Call it whatever term you like, but functionally testing a product
end-to-end has always followed an _economical_ approach, as all such testing
is a factor of time & resources. You'd perhaps be less economical if you were
testing life-safety systems, but you could choose to be more economical if you
considered your software to be not-that-critical, or, you needn't have to set
exceptionally high standards.

I find it extremely odd that unittests are even considered into this equation
of 'testing-costs', despite decades of improvements in software development
processes. Unittests should be part of 'core' engineering & a part of
development. Not a task that's added to testing costs. If you aren't writing
unittests (irrespective of whether its TDD or not), you are not
developing/engineering your product right. You are only building a stack of
cards _hoping_ that it wouldn't collapse at some point in the future.

It's a sad state of affairs if one gives code 1st level treatment, but treats
unittests, documentation, build scripts and other support infra as something
less important; worth economizing. This is really what _matters_ when it comes
to overall quality of a software.

One must remember that software is always improved, refactored, expanded,
ported or worked on in some way or another. When unittests are missing, then
the very boundaries that were meant to dictate the rules of the software don't
apply anymore. This leads to human errors causing portions of software to
break.

Critical pieces of software that exist today, exist strongly because they were
engineered right (Linux kernel for example). Not because their developers
followed an _economical_ approach to testing. If an OS disto claimed to
perform economical end-to-end testing (of a potential user's most commonly
performed paths), would the author of this article want to use that OS over
one that has had strong ground-up unittests and testing of interfaces where
they matter?

~~~
Brotkrumen
His claim of lacking empirical evidence shows his lack of research on the
subject. There's tons of research that consistently shows comprehensive
testing during development makes your product better and cheaper over its
lifetime.

Just a quick google and not even the paper I was thinking of:
[https://softwareengineering.stackexchange.com/questions/6050...](https://softwareengineering.stackexchange.com/questions/60505/what-
is-the-effect-of-creating-unit-tests-during-development-on-time-to-develop)

~~~
mdip
I'd love to see some research around time cost in this area. My personal
experience is that writing unit, integration and end-to-end testing ends up
either being a wash or reducing the total time it takes to build a product[0].

The rule of thumb I always come back to is "the minute you decide it's time to
attach a debugger, be prepared to burn an hour." I used to not mind firing up
the debugger -- heck, I used to run my code with a debugger attached _every
time I ran it_. These days, I write tests. I don't aim for 100% code coverage,
though I aim for a majority and 100% coverage of critical paths. When I
encounter a problem that _isn 't_ covered by a test (rare), I write a test
centered around the problem. This _usually_ surfaces the bug before I've even
_ran_ the actual test -- and once that test is there, any refactoring that
affects that code will avoid that bug if the test passes.

This rule has been so effective for me that I've now gone through four
different teams, encouraging each to do "the next project" with automated
testing. The result has been a commitment to working that way from that point
forward[1].

[0] Assuming some baseline quality requirements -- automated build, and a
general requirement to publish a mostly functional product. :)

[1] I can't take full credit for the idea -- it was a blog post that convinced
me to try it on myself and I advocated that idea after it convinced me.

~~~
chopin
For time cost, my rule of thumb is that unit tests roughly take the same
amount of test code than the code being tested and about the same (mental)
effort. Therefore I don't strive for 100% code coverage but I like all
critical (or difficult to understand) code being (unit) tested.

------
cies
Interesting article.

I'm a big fan of "economic testing". Test that are not going to "be
economically viable" to write during the dev't of new software, usually do not
get my permission to be written in that phase.

Unit test thus get written for part that we absolutely want to be maximum
certain of that it works correctly. Mostly that functionality can be extracted
into a library. For instance a lib that deal with scheduling of events of
arbitrary length: we do not want to have issues with those getting messed up,
so we try to get 100% unit test coverage of the functionality in that lib.

Another thing with unit test is that I found they mean something different in
dynamic languages than in strongly typed languages. In the first unit test are
often something to provide confidence that the type system can give you in the
latter. Sure both will also test the logic. But this understanding has pushed
me to love strong types (as a rubyist), as I can go with less unit test and
still be more confident about my code base when it is being heavily worked on.
Refactoring becomes so much easier with strong types, as all is "in the code".

When it comes to integration tests I usually want to make a biz case for them.
How much does it cost to have it all manually tested, and how much do the
automatic integration tests cost to make. While including the benefit of being
able to run the automated integration tests at near zero cost (thus several
times per day if we need to).

~~~
pards
> Unit test thus get written for part that we absolutely want to be maximum
> certain of that it works correctly.

I hear this kind of rhetoric a lot, but rarely do I hear of code that doesn't
need to work correctly.

~~~
Fradow
The difference is not between code that need or doesn't need to work.

The difference is having code for which having some bugs is acceptable, vs
having code for which bugs are not acceptable.

There is definitely code for which having some bugs is acceptable: think about
some small feature that the user can totally do without. I had to do one
recently, and bugs were acceptable, as long as it was shipped in time, and
working for the client specific devices. That code shipped with no automated
test (testing is done manually), and a few known bugs not worth solving. On
the other hand, anything that touch core features must be tested.

~~~
bpicolo
> That code shipped with no automated test (testing is done manually)

An issue with that is bus factor when it needs to be changed in the future.
What if the original dev isn't around any more?

~~~
Fradow
There is a test plan, and specs. Acceptance tests (manual) weren't even done
by a dev.

Also, the code has been reviewed and is documented.

Bus factor really isn't a problem in the instance I was talking about. In
fact, the original dev won't be around in a few months, and that's fine. The
rest of the team will be able to modify the code.

------
sverhagen
There's two things that humble me on a daily basis, in no particular order:
collaboration and testing.

While I use all sorts of testing, I still write a good amount of unit tests.
For whatever kind of tests I write: not a day goes by without tests or the act
of writing them exposing bugs that would otherwise have moved on to the "next
level". Doesn't mean they'd have ended up in production, but they'd still have
been a lot more expensive to fix.

------
lucisferre
I kind of lost it at this point:

> Kent C. Dodds claims that integration tests provide the best balance of
> cost, speed and confidence. I subscribe to that claim. We don’t have
> empirical evidence showing that this is actually true

And

> There is the claim that making your code unit-testable will improve its
> quality. Many arguments and some empirical evidence in favor of that claim
> exist

Why rely on evidence when conjecture an opinion are so much easier?

So here's my own limited evidence to toss on the other pile. Our team has
started to unit test everything, 100% coverage. We don't take this approach to
testing lightly, it is a holistic approach that includes the code itself,
focusing on architectures that enable testing and testability. Doing so has
vastly improved code quality in measurable ways.

Code reviews are faster and easier as a starting point. It is easier for
another person to understand quickly what each component does and what it is
expected to do. Code is composed using small autonomous functional components
that are uncompleted and easy to reason about and test. These make debugging
easier because they can be isolated to simple components and tests. Bug fixes
are faster to implement and test since the test infrastructure is already in
place and does not need to be remediated in later.

Yes, we do also write integration tests. They tests the end-to-end scenarios
we are designing the system to support. But unit testing is still the primary
approach to how we design, while integration provides support by verifying
complete scenarios.

However our approach admittedly doesn't have "Lean" in the name, so that's one
downside I guess.

~~~
newgame
Software quality can be measured in different ways. Software quality is not
the only dimension to care about when developing products.

~~~
lucisferre
So what dimensions do you propose to care about? Do the dimensions I mention
not matter at all? If they do matter then I think there is still a point here.

You are engaging in a strawman. I don't claim anywhere that these are the only
dimensions to care about, I simply claim that these are some things we've
noted that testing leads to improvements.

Ultimately though my primary claim is simply that I see less value in opinions
vs evidence.

~~~
newgame
Fair enough.

Though I'd like to think that I presented clear arguments even though I didn't
present _empirical_ evidence for my arguments. I presented ideas and logical
steps how to arrive at them. Which, to me, makes it more than "just an
opinion". Sure, you may disagree with the ideas and or the steps–it's not a
mathematical proof, it's much softer. Sidenote: Nonetheless, I provided two
links to empirical research finding counterintuitive results from TDD and from
unit tests.

Clarification: I am not engaging in a strawman. I didn't claim that _you_ said
these are the only dimensions. I tried to express, generally, that software
quality is one of several important dimensions. I could have phrased it in a
clearer way.

~~~
lucisferre
> Software quality can be measured in different ways. Software quality is not
> the only dimension to care about when developing products

This statement is an implied counter argument. If not then what is it? If so,
then I suggest it is a strawman as I never argued against that.

> I'd like to think that I presented clear arguments

If you are referring to the original article I was not aware you were the
author, so my response above simple responded to your statement in context and
not the article as a whole. I'll do so below.

I don't argue that the article presents it's arguments clearly, though I don't
necessarily agree with them. Actually, my main issue is with the rhetorical
style it is presents it in.

It tries to hard to sell less testing as a solution to testing pain, rather
than exposing it as a possible idea to be discussed. Using mostly a
combination of appeals to authority and association with the term "lean" as
rhetorical devices. This, to me, smells like the last generation "Agile" snake
oil.

On separate level I do disagree with presenting logical reasoning as a counter
argument to empirical evidence. Logical reasoning may be a compelling reason
to seek empirical evidence through trial and experimentation, but reasoning
alone can't invalidate evidence. With that said, I do believe some other
commenters here mentioned that there is evidence on both sides of this debate,
so I won't say your idea's are not worth exploring, though they definitely
contradict my own experiences with testing.

------
rectang
Meh. Thinking about ROI when writing tests is important, but my analysis of
ROI is mostly at odds with this anti-unit-test screed: the greatest ROI comes
from writing thorough unit tests against _stable_ low-level sub-components.

Integration tests, in contrast, require attention more frequently -- because
not all subcomponents are stable and when one changes, multiple integration
tests need superficial readjustment.

I'm skeptical that my own ROI analysis is more "right" \-- it probably varies
by project and team. The conclusion I draw instead is that ROI analysis is
important in general when writing tests, but that one particular ROI analysis
is unlikely to apply universally.

------
crazygringo
Unit tests also help ensure both the coder and readers understand what the
function's defined behavior is supposed to be, 100%. They help with clear
thinking to ensure you write the code correctly in the first place.

------
drakonka
There is an interesting article which responds to this and mentions some
misinterpretations of references: [https://www.tedmyoung.com/looking-at-tdd-
an-academic-survey/](https://www.tedmyoung.com/looking-at-tdd-an-academic-
survey/)

~~~
keithnz
I think that article is a necessary read after the original posted article.
The original posted article seems a bit too keen to convince you of its point
of view that it seems a bit disingenuous.

My feeling about unit testing / integration test is you do as much of it as
you can, trying to sincerely learn the lessons of how to do it it effectively,
until you know why you don't need to do it. Or until you know what the right
mix is. I started with Extreme Programming and testing around 18 years ago and
my thinking has constantly evolved over that time. One of the biggest factors,
I think, is project context and the nature of the software you are developing.

I find in embedded systems, unit tests done extensively are very very useful.
As is integration testing.

In web systems, I find integration tests are often most useful as so much is
all about gluing together various tech, and it's often the gluing the comes
unstuck. However there is often chunks that have some interesting logic that
lends itself to more extensive unit tests.

Sometimes testing is just a mess because of the design. Too many things that
are not really needed, too many layers, too many levels of abstractions, too
much "future proofing". Then you see people start trying to throw unit testing
around it, and it is mostly just useless and explodes out the amount of work
to do. Often I think articles like the original are reactions to this
scenario.

------
dalore
This is making the assumption that writing tests takes longer to produce code
then test without. But it has been shown that TDD can and usually does produce
code faster with it. TDD makes you write code that's easier to test, the tests
are fast (mere ms). And it gives you confidence in your code, plus locality as
you know the error since the tests either pass or fail each step.

Not advocating writing 100% test coverage, but just some code that shows how
you will be using the new feature as you add it is enough to help you
structure your code.

------
mamcx
I write some test when starting, with the full acceptance most, if not all,
will be abandoned. Maybe comment out all the ones I don't need anymore or just
are too broken (because are invalid). So tests are just another way to do
exploratory programming...

Most are related to DBs. I NEVER mockup. NEVER.

NEVER.

I just rebuild the DB each time the tests start (and is PostgreSQL. I found
that DROP/REBUILD the SCHEMA is _fast_ than drop the database and rebuild, so
is ok).

This always leave me with all the data in the DB, no matter it fail or not.
This is the _main reason to never mock_. Mock data get lost after the tests
end. The db instead persist the data and allow to inspect after the fact.

Plus, fill me with the data necessary to let the UI run without re-create
again here.

This simple rule is what help me most. If I wanna the test to run faster, I
can't do anything else that write a good database layer and not call the DB
when not needed...

~~~
marcosdumay
Once you abandon the idea of unit testing everything, plenty of possibilities
open up.

I have a mail server that I test by cloning a root (that's in git) and
chrooting there. I also have created a language for testing network protocols
once.

------
danielovichdk
There is a test for that.

Try building a tennis game, just in code with no gui, and take note of how
much you would either write tests for things like the score, set points etc.,
or how much time you spend debugging to justify if your code works.

Unit tests are a great documentation tool as well, both for developers but
also for the business needs of the software — the value.

------
simula67
I have seen statements like 'even bad tests are better than no tests'. I feel
that for unit tests, 'no tests are better than bad tests'. Failure in unit
tests often break CI and I have seen many tests which tests the structure of
the code and not it's intent. For example, I have seen some tests where the
product code is putting some element to a list and the test code checks if the
element is present in the list. I can understand why people do it ( they are
trying to improve "code coverage" ), however if tomorrow you refactor the code
to not put that element in the list, but still accomplish the same thing, the
test would break. Often, this breaks CI also and now you are sitting late at
office trying to figure out what went wrong so you can push your change into
the release. This is valuable time that could be spent on new features, fixing
bugs or just going home and relaxing. End to end tests ( or integration tests
) are better since they are real user interactions.

If a test fails and it breaks CI, there should be a problem in the product.
Move everything else to some "additional testing" bucket and not block
developers.

~~~
nimblegorilla
In an ideal world developers never push something with broken tests to CI.
When developers don't run the tests locally it seems like a sign that there
are too many bloated integration tests and not enough fast unit tests.

------
hdi
Somewhat interesting but calling an old idea something catchy isn't really
revolutionary.

The article makes a lot of assumptions like "In practice, most agree as most
projects set the lower bound for coverage to around 80%". Most projects where,
which industry, who are these 80% that agree? This is just taken out of thin
air in this case. It might be true but should we take the author's word for
it?

    
    
      There are also some psychological arguments. For 
      example, if you value unit-testability, you would prefer 
      a program design that is easier to test than a design 
      that is harder to test but is otherwise better, because 
      you know that you’ll spend a lot more time writing tests.
    

This is very subjective without examples. The opposite argument can also be
made that code which is easier to test is sometimes better. It felt like
reading a collection of quotes and articles by other people.

I think TDD, unit, integration and E2E testing works. How much of which to
apply is entirely project and industry specific and it's up to teams to decide
what testing strategy works best for them.

~~~
LoSboccacc
"The outlook for the effectiveness of testing used by itself is bleak. Jones
points out that a combination of unit testing, functional testing, and system
testing often results in a cumulative defect detection of less than 60
percent, which is usually inadequate for production software."

[https://medium.com/@TuckerConnelly/94-gems-from-code-
complet...](https://medium.com/@TuckerConnelly/94-gems-from-code-
complete-a1e1b69f9f8c)

personally I like write some cursory unit test because it ensures the code
logic is correctly decoupled from data retrieval, but most of my unit tests
are written from user reports, as part of a large no regression suite.

------
billsmithaustin
"This is because an end-to-end test covers a much greater area of the code
base." I disagree. A single unit test will cover less area than an end-to-end
test, but unit tests in aggregate will cover a greater area.

~~~
marcosdumay
> but unit tests in aggregate will cover a greater area.

If integration tests can't cover some part of your code, it's because that
part does not matter. Get rid of it.

~~~
billsmithaustin
I agree that some scenarios are more important than others, but that seems to
be orthogonal to the claim that I was objecting to.

I also disagree that whether an integration test can cover some part of your
code is the litmus test for whether the code is relevant. There are types of
errors that are relevant but complicated to simulate in an integration test,
or at least in an automated integration test.

An example would be a network partition. It's involved to code a test to cause
the partition, wait for the various timeouts to take place, then heal the
partition, then wait some more for things to start talking and recovering. And
depending on who owns the test environment and how your organization is
managed, you may need to negotiate with a DevOps / Operations / Security
person to get permission to run iptables in the appropriate machines.

On the other hand, unit-testing what happens when a connection fails is
probably pretty easy.

~~~
marcosdumay
> It's involved to code a test to cause the partition, wait for the various
> timeouts to take place, then heal the partition, then wait some more for
> things to start talking and recovering.

Yet, on the layer that is charged with surviving a partition, that's exactly
what you should be doing.

Your unit test can verify that some machine sends the designed message down
the wire, it can check if it responds to the expected message, but it can not
even check if the software can detect a partition, even less that it can
survive one.

I would recommend you make those timeouts configurable from the user point of
view.

------
kozak
"Nobody got fired for writing unit tests", and this makes it hard to justify
writing less of them, especially to people that are exposed only to
development side of software business.

------
Jtsummers
> _Imagine you have three components, A, B and C. You have written an
> extensive unit test suite to test them. Later on you decide to refactor the
> architecture so that functionality of B will be split among A and C. You now
> have two new components with different interfaces. All the unit tests are
> suddenly rendered useless. Some test code may be reused but all in all the
> entire test suite has to be rewritten._

That's not a refactor. That's a rewrite or rearchitecturing. Of course you
have to redo unit tests, you've eliminated the units that were being tested.

> _What if you decouple two components that are always used together._

If they're always used together, then they were a unit. Why were they
decoupled? See the other front page post about Dijkstra's parable for a rather
apropos example of where coupling (perhaps not his intended moral) makes
sense.

------
GuB-42
It reminds me of a project I worked on. It was for critical embedded software
in aircraft ("make an emergency landing" critical, not "we are all dead").
Obviously, the code is certified, with specific requirements (DO-178B level B)
and third party audits.

And we made the choice not to have unit tests. Everything was done through
integration tests. What made up for it is that we had very good traceability.
So for we knew which lines of code were related to which requirement and which
test. So if we changed some code, we knew which test to run, and if a test
fails, we knew where to look at.

For the edge cases, we had the ability to inject data directly into memory,
triggering the otherwise unreachable error conditions, and the very last cases
where done through code analysis (we needed 100% coverage).

------
guru4consulting
I guess it also depends on the nature of tech stack and business domain. For
e.g., if you develop a standard CRUD application without much complexity, and
use a static lang like Java, you probably need less unit tests. If you use
framework features, annotations, validation expressions, then most of the
simple data validations (like numbers, dates, future date check, etc) are
automatically handled at framework level. In those scenarios, you would
generally need a very little unit test case coverage.

On the other side, if you use dynamic lang (Ruby, Javascript, etc), or if you
are developing a library or framework, then a good coverage for unit tests is
a must. I guess the developer's context (whether he/she is developing a
framework, library or a simple CRUD app) plays a bigger role.

------
2T1Qka0rEiPr
This article makes lots of good claims as to why integration tests can be the
most useful area to focus on. I do however think that unit tests can be
_really useful_ in encouraging junior developers to improve their plan vs.
write approach to software development (if properly guided).

------
theptip
> unit tests increase maintenance liabilities because they are less resilient
> against code changes. Coupling between modules and their tests is
> introduced! Tests are system modules as well.

Every time I hear this kind of statement, I think of Bob Martin's article:
[https://blog.cleancoder.com/uncle-
bob/2017/05/05/TestDefinit...](https://blog.cleancoder.com/uncle-
bob/2017/05/05/TestDefinitions.html)

> The author tells of how his unit tests are fragile because he mocks out all
> the collaborators. (sigh). Every time a collaborator changes, the mocks have
> to be changed. And this, of course, makes the unit tests fragile... the
> author of the above blog has given up on badly written micro-tests, in favor
> of badly written functional tests. Why badly written? Because in both cases
> the author describes these tests as coupled to things that they should not
> be coupled to. In the case of his micro-tests, he was using too many mocks,
> and deeply coupling his tests to the implementation of the production code.
> That, of course, leads to fragile tests.

> What the blog author does not seem to recognize is that first class citizens
> of the system should not be coupled. Someone who treats their tests like
> first class citizens will take great pains to ensure that those tests are
> not strongly coupled to the production code.

Testing is hard for a number of reasons, not least because given five
engineers you'll get six definitions of Unit vs. Integration test. I think the
term "micro-test" is useful as a contrast; most beginners sit down and write
tests for every significant function, be it public or private (i.e. they write
micro-tests), and this produces tests that are tightly coupled to the shape of
the code at the time of writing.

However if your code is well-encapsulated, and just tested through the public
API of your class, then you should be able to make fairly significant changes
to the structure of that class without actually changing the tests. (Of course
if you change the API then the tests must change; it must always be so).

------
djhworld
I'm working on a ticket at the moment that involves a lot of refactoring work
to support a new feature.

In the initial run on this code base we went a bit unit test crazy, which was
fine at the time, but I'm seeing the drawbacks now where I'm doing a lot of
surgery on the components.

I've ended up just beefing up the current integration tests with some extra
edge case testing and, after the refactoring work, they still pass, which has
given me a lot more confidence.

I think a lot of this comes to head when you unit test the glue logic that
orchestrates the entire program, unit testing those becomes painful with mocks
etc because they become fragile whenever you do any refactoring work.

I'm not sure what to think really.

------
karmakaze
The highezt cost of unit tests is when they are refs tired with code. The
value of a unit test is greatest when written before the implemantation. When
refactored it is usually just made to pass by any means necessary to stay
green without much/any thought. They are also brittle in these cases where
refactoring the test can take meant times longer than the code it tests.

My advice is usually to do tdd for new dev have good coverage of full function
tests and delete unit tests if the get in the way/lose value on refactoring.
No one likes deleting tests but they must add value or its just more debt.

------
throwaway5752
This is ... not even basic test engineering 101. This is remedial: understand
the testing pyramid.

If you can take a 5ms unit test and verify with 80% confidence for a given
feature, run that first before running at 5s system integration test that
verifies it with 100% confidence.

You run both, because you run unit tests with a build/linting stage (something
that can run synchronously with every commit, or close to it) and you run the
integration test less frequently because of time/cost constraints (stable
branch merges).

Even though they test they same thing, they serve different needs.

------
edejong
Edsger Dijkstra: "The first moral of the story is that program testing can be
used very effectively to show the presence of bugs but never to show their
absence." [1]

"I suggest that the programmer should continue to understand what he is doing,
that his growing product remains firmly within his intellectual grip." [1]

[1] E.W. Dijkstra:
[https://www.cs.utexas.edu/users/EWD/transcriptions/EWD03xx/E...](https://www.cs.utexas.edu/users/EWD/transcriptions/EWD03xx/EWD303.html)

~~~
nradov
The trouble is that advice isn't really actionable. In the real world of
commercial software often the only way to satisfy customer requirements is to
build a product more complex than the programmer can really understand. Most
customers would rather have buggy software (which eventually gets fixed)
rather than no software at all.

~~~
edejong
But it is actionable. Each and every time you chose a certain technology (FP
over OOP, immutable vs. mutable state, static typing vs. dynamic typing,
shared vs. specific terminology), we make a choice. These choices together
drive our community towards a certain mode of programming.

So, your argument is true for individuals, but for the community it is void.
Together we should focus on identifying correct, effective, simple and
reliable building blocks on which we can build our larger systems.

Perhaps you haven't seen "Simple made easy" by Rich Hickey, which illustrates
this concept rather well.

~~~
nradov
Those technology choices all carry trade-offs. Optimizing for correctness
doesn't always maximize business value.

And in my experience most of the defects which aren't caught by our quality
control process wouldn't have been prevented by switching from OOP to FP, or
from mutable to immutable state, or anything like that. So I'm not convinced.

------
emodendroket
I'm not sure if I agree with having a lot of DB tests, but at the same time I
definitely have seen the thing where overly specific unit tests make it so
painful to change anything that nobody wants to do it.

I think "Why Most Unit Testing is Waste" is a better article on this subject.
[https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-
Waste...](https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf)

------
Kagerjay
I started unit testing for the first time several months back. I tried
incorporating it into my general workflow

Here's an isolated overkill example of how unit tests should not be done on
one of my codepen files (its a javascript calculator). All tests passed but
the calculator is partially broken still. I made about 20 unit test case
scenarios when in reality it should have been significantly less (~ about 5
given the complexity of the problem)

[https://codepen.io/vincentntang/pen/XqNGqv?editors=1100](https://codepen.io/vincentntang/pen/XqNGqv?editors=1100)

I ended up getting very little useful feedback on codereview / stackexchange,
but this is how I felt about unit-testing in general.

[https://codereview.stackexchange.com/questions/193128/shunty...](https://codereview.stackexchange.com/questions/193128/shuntyard-
javascript-calculator-with-unit-tests)

I think unit testing is great in some applications (e.g. working with teams as
more things can be missed, where a bug can be exponentially cause issues,
systems with larger complexity). Doing it in isolated solo projects is such a
waste of time IMO, good commenting / structurally organizing code/ naming
conventions suffices here. Most things don't work in a functional way anyways
that makes unit-testing easier. E.g hardware calculators don't have logic the
way I wrote it in my code above.

Unit testing is OPPURTUNITY COST. For a business to implement this the risks
of exponential failure and its demand for risk-mitigation should far outweigh
the time and developmental practice of implementing it in your training /
management / workflow / toolchain.

...disclaimer: I might have no idea what I'm talking about I'm a fairly
inexperienced dev

~~~
marcosdumay
You won't get very far without tests. You will notice it, once you start to
create larger programs your productivity will decline. That happens even if
you use safe languages and verify your code.

This article is to be about the kind of test you should use. It's just that
_unit_ tests are often not powerful, but always expensive. So, you should use
other kinds of tests.

------
h0l0cube
I'm a huge fan of the following essay. It provides a set of very rational,
well considered and compelling arguments for less unit tests:

[https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-
Waste...](https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf)

------
rbongers
Your testing strategy should be situational.

Unit tests can efficiently handle, for example, libraries, error handling
which cannot normally be reached, complex algorithms which are not AI based
and require a large number of test cases, fragile pieces code and code
patterns, and program design patterns which can be tested using inexpensive
common tests. Consider the business value of quick tests in each of these
situations. It is high.

While your units have to come together in a way that allows your users to
actually use your software, your business logic has to be represented in whole
or in part by one or more unit. Testing this logic earlier on in the process
and faster than other types of tests is better.

------
shinylane
For me this depends on what I'm working on, and these days my unit tests are
generally for API regression. When I'm writing my APIs, generally my
"integration" test would simply involve an external request and assertions of
the response. This is fantastic for testing what a client expects, not as good
at specific units of business logic where regression can be rather expensive.
I would agree many folks take unit testing too far, but in some cases it's
saved my neck at a pretty small cost of creation.

This is probably not the same story for something like a react or swift
application. Good read, thanks for the solid points.

------
miga
I have also been advocating economic approach to testing, but I find this
article to be misleading at best.

Long-time experience of debugging proves that untested code is worthless. And
indeed most people start programming by testing some test case or use case.

I would rather recommend starting with use case testing. Data driven testing.

Never believe that you can get a code without testing. I guess I finally need
to write the article about my _diamond testing_ routine, so that people have
something better than unit test swarms ;-)

------
usrusr
Another rule of thumb I like to apply when mix-and-matching different levels
of tests: how visible would a bug be in higher level tests? If things will
fail in obvious ways, a unit test will give no benefit besides narrowing down
the bug location in case of a failure.

But if symptoms would likely be too subtle to stand out, e.g. a typical bug
would manifest itself in results that seem plausible despite being wrong, then
it's a strong indication for unit tests.

------
IanCal
> Integration tests lie somewhere between unit tests and end-to-end tests so
> they provide the best balance. Therefore, they have the highest ROI.

Being in between does not mean it gives the highest ROI or the best balance.
They can easily share the worst of both.

> Lean Testing takes an economic point of view to reconsider the Return on
> Investment of unit tests.

But does not in any way actually try and work this out, which means that you
can't make the conclusions that are in this.

> The Return on investment (ROI) of an end-to-end test is higher than that of
> a unit test. This is because an end-to-end test covers a much greater area
> of the code base. Even taking into account higher costs, it provides
> disproportionally more confidence.

And yet no figures or evidence.

Many e2e tests can cover the _same_ path segments, meaning adding a new test
may not increase your confidence by much at all. But they're still taking
longer to run, and despite the articles insistence that integration and unit
tests are brittle, I've found E2E tests are also brittle, just in different
ways.

I shouldn't have to change my string normalisation tests just because the
website changed moving the resulting output field to a new place on the page.
They shouldn't break because the login flow changes.

I have definitely seen (and written) E2E tests that provide almost no real
value, and unit tests that provide huge amounts.

> Plus, end-to-end tests test the critical paths that your users actually
> take. Whereas unit tests may test corner cases that are never or very
> seldomly encountered in practice

Unfortunately low _proportions_ of errors still can account for extremely
large _actual numbers of_ problems. 1/1000 bugs will happen _all the time_ if
you've got a reasonable number of users. Those bugs also may happen to most of
your users if they're somewhat random, or worse may heavily impact one group.

> For many products, it is acceptable to have the common cases work but not
> the exotic ones (‘Unit Test Fetish’). If you miss a corner case bug due to
> low code coverage that affects 0.1% of your users you might survive. If your
> time to market increases due to high code coverage demands you might not
> survive.

Or if roughly one in 1000 customers keep taking your production server down
due to a bug you might not, and you may have been just fine waiting another
week to go live.

I agree with the idea, you should consider what it is you're trying to achieve
and what the best ways of doing that are. What you absolutely shouldn't do is
use terms like ROI and economic without any analysis, purely to justify not
doing something you don't really want to do.

------
pletnes
The article argues that you should test larger units (integration tests)
rather than unit tests, but also that you should use a statically typed
language. This leads to less flexibility and more «ossification» in my
experience, much like the overly detailed unit tests. Seems to be a
contradiction, in my opinion.

------
thecleaner
Perhaps this approach of end to end testing is great for startups. Although
e2e tests can be hard to write, they can be a significant boost when moving
with high velocity.

------
peterwwillis
There are two kinds of end-to-end tests: 1) the software works, 2) the
software works in production. The difficulty usually lies in implementing the
latter.

------
actionowl
Unit tests are better than no tests.

------
mdip
Great article and I'm somewhat surprised at how often I have to repeat the
things mentioned in it -- heck, sometimes it's difficult to even convince
developers of the value of unit testing, in general.

I'm a huge advocate of automated testing and with the available tools, like
docker, it's relatively painless to get the pieces together to sort out
automated testing. Often the tooling you use to enable automated testing is
tooling you end up needing _anyway_ \-- it's dual purpose. Before the "first
run" of a set of code, I'll create Dockerfiles that make up a complete, local,
development instance of an application along with some boilerplate tooling
that I include to make debugging easier on me. When setting up the production
build, the final version is usually _this same environment_ with fewer lines
in the file. Because my environments tend to be similar, I have a zsh script
that strips out lines in the Dockerfile to get me 90% of the way to a
production container.

For me, it's _always_ worth it. I came to this conclusion after spending a few
months _forcing_ myself to test rigorously[0], starting with unit tests
written often and early and ending with a small number of integration tests
and a much smaller number of end-to-end tests. I don't find any of these
particularly difficult to write.

The benefits, however, are vast: (1) Avoiding the debugger time-sink: The #1
thing that I always come back to is that I generally end up _never_ having to
fire up a debugger. I noticed that every time I encountered a bug in code that
was poorly covered, the first instinct was to attach a debugger and peek at
locals to see what was going on. This rarely resulted in spending less than an
hour troubleshooting. Sometimes you get lucky and you find more than one issue
in that debugging session, but often it landed in at burning an hour on ever
bug and way too often it was an hour spent debugging _production code_ and the
bug was _customer impacting_ [1]. At the same time, it's rare for an automated
test to have a time cost that high. (2) Refactoring - Since "premature
optimization is the root of all evil", that necessarily means that a
performance bug is going to involve injecting complexity into a running
codebase and this often comes with high-impact refactoring. Unit tests,
specifically, are incredibly helpful here. This is often an argument _against_
integration/end-to-end test automation since refactoring regularly breaks
these brittle tests, however, I've found in practice that this isn't the case
at least half the time. Of the times that it _does_ affect those tests, the
practice of refactoring can surface subtle bugs (on a few occasions it
surfaced a subtle race condition that might have been missed if a few of the
integration tests covering a subset of the functionality hadn't broken). (3)
Design - more for integration and unit tests, thinking about testing while
writing code can result in a less brittle design[2]. On integration tests, it
means writing SQL scripts and migrations to ensure that a fresh environment
can be spun up on-demand instead of using GUI tooling (or, using the GUI
tooling to generate said scripts/migrations). (4) Build automation - I'm
somewhat surprised at how often I encounter a customer project where I have to
follow a 20 step process to get things functional in a development
environment. It seems like if CI isn't involved, people figure a README.md
with a mess of shell commands and button clicking is OK. Scripting out
environment configuration and build was already one of the first things I did
when I began running the code I'd written, however, I find I no longer have to
argue in favor of this when testing is involved -- everyone wants a single
command to execute tests and once integration and end-to-end tests are
involved, it just makes sense to add standing up the docker parts, too[3].

I get why there's resistance to doing these things -- getting people to simply
write _any_ automated tests seems to be the most difficult hurdle. Throw in
"learn docker" and other technologies to make automated end-to-end testing
easier and the barrier is even higher. And hey, there are some times that the
time spent writing tests doesn't pan out to a time savings. For the
unconvinced, I can only _strongly_ recommend: try it on your next big project.
There's no need to change the way you think -- skip TDD if it doesn't work for
you -- but write unit tests over your, public facing surface area. Write
integration tests over the most important parts of your codebase -- those
which if a bug were to be encountered would have the greatest impact on
reliability. Write a few end-to-end tests of major functionality. Keep track
of the time spent from the first line of code to the final, released, product.
If your experience is like mine and the 4 different teams I've done this
exercise with, you'll end up doing things this way from that point forward. If
not, you have a gift that I lack -- you write incredibly bug-free code "the
first time", every time. :) Then try switching to a single 1080P monitor[4].

[0] I tend to code first, test second. Though, on paper, TDD looks like a good
idea in that it forces you to think about the desired outcome and write
methods in a manner that guarantees the ability to test, I don't find it
difficult to write things in this manner from the beginning and I find it more
natural to write the actual code first and I'll often write a large footprint
of code before writing the first test -- I don't find as much value as others
in frequent, instant, feedback but I recognize the value that others find in
that.

[1] And bugs resolved during an outage are duct-tape "there, I fixed it" kind
of repairs.

[2] Provided you don't like having to figure out new and creative ways to mock
complex god-objects/routines. Maybe you dig that sort of thing?

[3] I reload enough that I have a script that automates _installing and
configuring_ docker in the most common manner if it's missing or the
configuration isn't complete.

[4]
[https://news.ycombinator.com/item?id=14482587](https://news.ycombinator.com/item?id=14482587)

------
jfjfjfjfyy
The dogma around unit testing, TDD and the book selling test hype cycles have
caused immeasurable damage to software development as technical and scientific
disciplines.

~~~
bacro
I agree with you, but could you care to elaborate on that?

~~~
jfjfjfjfyy
Where to begin.. perhaps the projects with code metrics tied into the check in
system - want to commit code changes? You're now on the hook to provide at
least 85% unit test coverage or the source control tool will not accept it. So
what did most people do? They wrote useless tests, assert true equals true,
no-ops or pass statements, basically anything to just get past the arbitrary
metric.

Or how about another project that had lots of real tests, that were old,
unmaintained, did humongous amounts of work in each single unit test and took
a weekend to run, and of course, fail.

Or perhaps the team that obsessed about the testing for each user story, spent
all the time doing that, then running out of time to deliver the actual
stories.

And those were the less egregious examples!

------
kazinator
I think I can articulate what I find distasteful about approaches like TDD:
it's basically a "how to program" recipe for people who have absolutely no
talent. Like "paint by numbers". It's tone-deaf toward the creative aspects of
development: the Art, if you will.

TDD for music composition: add some notes which jar the ear (red test). Then
massage them until they sound pleasing (green test). Make some adjustments for
better flow with surrounding notes (refactor). Now repeat. Gee whiz, you're
gonna be the next Mozart or Bach!

~~~
crazygringo
I may never have disagreed more with a comment on HN than this one.

TDD does absolutely nothing to hinder your creativity. All it does is help you
to think about your code's behavior more thoroughly and catch inadvertent
bugs. Nobody has so much talent they don't write bugs.

For music composition: composers generally start out with an idea of what they
want to achieve, whether at the scale of the symphony, movement, section,
passage, phrase, or bar. Then they try different solutions until they satisfy
the idea they want to achieve. And ensure it flows with the overall structure.
And repeat.

Beethoven didn't write out symphonies fully formed. He spent months and years
experimenting with ideas until they satisfied what he wanted, e.g. the "test".

------
irundebian
Time-to-market is the only valid argument for writing crap code and not unit
tests. I'm not sure if you can seriously argue with economic reasons when
talking about testing. The most costs in software development arise in the
operations/maintenance phase and are probably (there are much papers on this
topic) multiple times higher than the costs of the implementation phase.

I agree that every software development team has to make their own decision to
balance development time and maintenance cost but I don't agree with this low
entitlement mentality of writing correct code in this article.

Of course there are also costs when implementing tests and your unit tests are
rendered useless on design changes. But so what? You're also decreasing code
assurance and it's better to increase it with tests so that you won't
encounter bugs in production which is much more expensive than writing tests.

Software development is not only about adding functionality but also about
ensuring its correctness.

~~~
rc_bhg
sometimes (often times) unit tests make for crap code.

