
Write tests. Not too many. Mostly integration (2017) - AlexanderDhoore
https://blog.kentcdodds.com/write-tests-not-too-many-mostly-integration-5e8c7fff591c
======
hathawsh
My rule of thumb: tests cast away fear. Whenever I get that sinking feeling
that I'll break things when I change the code, I write tests until my doubts
disappear. It works every time.

In response to the article: it's true that "you should very rarely have to
change tests when you refactor code", however, most of the time, most coders
are changing the expected behavior of the code due to changed or added
requirements, which is not refactoring. Tests should change when the
requirements change, of course. I am not contradicting the article, only
clarifying that the quoted statement does not apply outside refactoring.
(Refactoring is improving the design or performance of code without changing
its required behavior.)

~~~
efdee
In my experience, a lot of unit tests are written with mocks, expectations,
and way too much knowledge of the implementation details, which leads to
broken tests simply by refactoring, even if the behavior does not change at
all.

If you test units in isolation, while adhering to SRP which results in many
smaller units depending on eachother to do a task, then simply refactoring
without changing behavior screws up a considerable portion of your tests.

As for "tests cast away fear", that is definitely true. Whether or not the
lack of fear is warranted is something else, and depends heavily on the
quality of the unit tests. I've seen plenty of devs confident of their change
because it didn't break any unit tests, only to discover that it broke
something they forgot to test.

~~~
inopinatus
It is analogous to moral hazard. People with insurance take greater risks, and
vice versa.

~~~
chosenbreed37
> It is analogous to moral hazard. People with insurance take greater risks,
> and vice versa.

Programming with guard rails :)

------
gravypod
> I’ve heard managers and teams mandating 100% code coverage for applications.
> That’s a really bad idea

I hope many managers and programmers out there don't take this the wrong way.
I've been an engineer on a project that was attempting to get 100% code
coverage on a piece of software I was writing. I heard constant remarks during
this period that were similar to "You don't need 100% code coverage, it
doesn't do anything!" These engineers who I was working with had only read
articles like this and didn't stop to think about what the article was trying
to say. From my experience there is no safe rule of thumb for for how many
tests should be implemented for a project. There should be just enough to feel
safe (as hathawsh has said). If you're recommending to engineers on your team
to stop implementing tests when they say "I'm at 60% coverage and it'll take
~2 days to get to 100%" I'd really hope you take the time to understand _why_
they want 100% coverage.

The software I was working on when my coworkers started telling me to stop
writing tests was code designed to trigger alarms when patient's vital signs
met certain criteria set by doctors. I am very thankful that I did hit 100%
coverage because between 60% and 100% there were many small edge cases that,
had they caused a death, I wouldn't have been able to sleep well. Had they
said I was explicitly disallowed from working on the tests I would have come
in on a weekend (or taking a PTO) and implemented them then. It's our ethical
responsibility to know when and where paranoia is worth the marginal time
penalty.

~~~
bulkan
> ...trigger alarms when patient's vital signs met certain criteria set by
> doctor

Most of us aren't working on life or death code like this. My React app
doesn't need 100% code coverage but you put it well when you said "There
should be just enough to feel safe"

~~~
gravypod
What I really want to drive home is that you should listen to the engineers
who are working on the project. Testing is one of those things that is
entirely subject to project needs. Like you say, a React app doesn't need 100%
coverage, neither does code running a CI, something far out of the serving
path, etc. But we shouldn't have a knee-jerk reaction to an engineer saying we
should have 100% coverage.

------
calebegg
It might be obvious to some, but for reference the title (and associated
tweet) is a reference to Michael Pollan's oft-quoted advice for eating
healthy: "Eat food. Not too much. Mostly plants."

[https://www.nytimes.com/2007/01/28/magazine/28nutritionism.t...](https://www.nytimes.com/2007/01/28/magazine/28nutritionism.t.html)

~~~
ggambetta
Does this have a name? Like the overuse of "X considered harmful" after
Dijkstra, or the annoying increase of "X Y and where/how to Z them" after JK
Rowling?

~~~
martinlaz
I don't think it's on the same level with "considered harmful" quite yet but
definitely heading there. BTW, two more for your list:

    
    
      - Everything you always wanted to know about X (but were afraid to ask)
      - X, or how I learned to stop worrying and love the Y

------
munificent
I think one mental model for tests is that they're simply another kind of
automation.

The fundamental question of automation is: _will I repeat this task enough
such that the amortized saving outweighs the cost of writing a script to do
it?_

Whenever you're thinking of adding a test X, quickly consider how often you
and your team are likely to need to manually test X if you don't. Also factor
in the cost of writing test X (though that's tricky because sometimes you need
to build test architecture Y, which lowers the costs of writing many
tests...).

If it's a piece of code that's buggy, changing frequently, brittle, or
important, then you're likely to need to validate its behavior again and
again. It's probably worth writing a unit test for it.

If it's an end user experience that involves a lot of services maintained by
different teams and tends to fall apart often, it's probably worth writing an
integration test instead of having to keep manually testing every time the app
goes down.

If it's an API with lots of important users using it for all sorts of weird
edge cases and you don't want to have to manually repro each of the existing
weird edge cases any time you add a new one, it's probably worth writing some
black box tests for it.

But if it's code that's simple, boring, stable, short-lived, or unimportant,
your time may be better spent elsewhere.

Another model is that your tests represent the codified understanding of what
the system _is_. This can be very helpful if you have a lot of team churn. How
do you make sure a new person doesn't break something when "something" isn't
well-defined somewhere? Tests are a great way to pin that down. If this is a
priority, then it becomes more important to write tests for things where the
test actually very rarely fails. It's more executable system specification
than it is automation.

~~~
chriswarbo
I agree with the idea that we're _mostly_ automating manual work.

The really nice thing about automated tests is that they can also go much
further than I would ever bother to manually. For example, I might manually
check that a couple of product pages are showing, with an image, price and
description. With automated testing I can check _all_ product pages, every
time. It can also test a tedious amount of interactions, e.g. various
interleavings of "add to cart", "remove from cart", "checkout", "press back
button", etc.

Manual testing still gives me confidence that automated tests can't, e.g. that
an "unknown unknown" hasn't taken things down. Yet that's _all_ I need to
check manually; if pages are loading and look right, I don't feel a need to
manually check all of the ecommerce arithmetic, permission logic, etc.

------
notacoward
I _so_ agree with this. I've been fighting this exact battle at work lately.
People on my team have decided to take testing seriously, which is fantastic,
but many team members' understanding of what that means is still at the "watch
unit-test coverage numbers go up" stage. So let me be very clear where I
stand.

 __* 100% unit-test coverage is a garbage goal. __*

I don't hate unit tests. They can have enormous value shaking out edge cases
in a self contained piece of code - usually a "leaf" in the module dependency
graph - and making it future proof. Love it. However, unit tests don't tell
you anything about whether the module's behavior in combination with others
leads to a correct result. It's possible to chain a bunch of unit tests each
with 100% coverage together, and still have the combination fail
spectacularly. In operations, a lot of bugs live in the interstices between
modules.

Even worse, 100% line coverage means a lot less than people seem to think.
Let's say one line to compute an expression will compute the wrong value for
some inputs. You might have a zillion unit tests that cause that line to be
counted as covered, but still be missing the test that reveals the error case.
I see this particularly with ASSERT or CHECK types of macros, which get
counted toward 100% coverage even though no test ever exercises the failure
case.

Striving too hard for 100% unit test coverage often means a lot of work - many
of our unit tests have 3x more tricky mock code than actual test code - to get
a thoroughly artificial and thus useless result. The cost:benefit ratio is
abominable. By all means write thorough unit tests for those modules that have
a good ratio, but _in general_ I agree with the admonition to focus more
effort on functional/integration tests. In my 30+ years of professional
program, that has _always_ been a way to find more bugs with less effort.

~~~
mikekchar
Before you battle too hard, let me introduce you to another way of thinking.
It may not be to your liking, but I hope you'll find it interesting
nonetheless. I'll try to keep it as short as I can. Usually when I type this
same post, it takes me a while, but I'm getting better at it.

Imagine that the word "test" is a misnomer, when talking about unit tests.
Often people think about testing as a way of checking whether or not the code
works properly. This is _great_ for what is known as "acceptance testing".
However, as you no doubt agree, it's not so great with "unit testing".

For some reason, people hang on hard to the words "unit" and "test" and come
to the conclusion that you should take a piece of code (usually a class),
isolate it and show that the class does was it is supposed to. This is a
completely reasonable supposition, however in practice it doesn't work that
well (I will skip the discussion, because I think you're already in agreement
with me on that front).

Instead, imagine that "unit" refers to any piece of code (at any level) that
has an interface. Next imagine that "test" means that we will simply document
what it does. We don't necessarily worry ourselves about whether it is correct
or not (though we wish it to be correct). We just write code that asserts,
"When I do X, the result is Y".

At the macro level, we still need to see if the code works. We do this either
with automated acceptance tests, or manual testing. Both are fine. When the
code works to our level of satisfaction, you can imagine that the "unit tests"
(that are only documenting what the code at various levels is doing) are also
correct. It is possible that there is some incorrect code that isn't used
(which we should delete), or that there are some software errors that cancel
each other out (which will be rare). However, once the code is working on a
macro scale, in general, it is also working on a micro scale.

Let's say we change the code now. The acceptance tests _may_ fail, but some of
the "unit tests" will almost certainly fail (assuming we have full "unit test"
coverage). If they don't there is a problem because "unit tests" are
describing what the code is doing (the behaviour) and if we change the
behaviour, the tests should fail.

For some types of unit testing styles (heavily mocked), often the unit tests
will _not_ fail when we change the behaviour. This means the tests, as a long
lasting artefact are not particularly useful. It might have been useful for
helping you write the code initially, but if the test doesn't fail when you
change the behaviour, it's lost utility. Let's make a rule: if the test
doesn't fail when the behaviour fail, it's a "bad" test. We need to remove it
or replace it with a test that _does_ fail.

The other problem you often run into is that when you change one line of code,
200 tests fail. This means that you spend _more_ time fixing the tests than
you gained from being informed that the test failed. Most of the time you
_know_ you are changing the behaviour, and so you want to have very little
overhead in updating the tests. Let's make another rule: Unit tests must be
specific. When you change specific behaviour only a few (on the order of 1)
tests should fail.

This last one is really tricky because it means that you have to think hard
about the way you write your code. Let's say you have a large function with
many branch points in it. If you give it some input, then there are many
possible outputs. You write a lot of unit tests. If you then change how one of
the branch points are handled, a whole _class_ of tests will fail. This is bad
for our rule.

The result of this is that you need to refactor that code so that your
functions have a minimum number of branch points (ideally 0 or 1).
Additionally, if you split apart that function so that it is now several
function, you have to make each of the functions available to your test suite.
This _exposes_ rather than _hides_ these interfaces.

The end result is that you decouple the operation of your code. When you hear
about TDD being "Test Driven _Design_ ", this is what it means. This is
especially true for things like global variables (or near global instance
variables in large classes). You can't get away with it because if your
functions depend on a lot of global state, you end up having tests that depend
on that (near) global state. When you change the operation surrounding that
state, a whole whack of tests fail.

Committing to writing high coverage unit tests which also have high
specificity forces you to write decoupled code that doesn't rely on hidden
state. And because it doesn't depend on hidden state, you have to be able to
explicitly set up the state in your tests, which force you to write code where
the dependencies on the objects are clear and uncomplicated.

You mentioned code coverage. I'm going to say that I almost check code
coverage when I'm doing TDD. That's because if you are writing tests that
cover all the behaviour, you will have 100% code coverage and 100% branch
coverage. However, as you correctly point out, the opposite is _not_ the case.
The test for the coverage of your code is _not_ a code coverage tool, it's
changing the behaviour of the code and noting that the tests fail.

Most people are familiar with the idea of "Test First" and often equate that
with "Test Driven". "Test First" is a great way to learn "Test Driven", but it
is not the only way to go. When you have full test coverage, you can easily
modify the code and observe how the tests fail. The tests and the production
code are two sides of the same coin. When you change one, you must change the
other. It's like double entry accounting. By modifying the production code and
seeing how the tests fail, you can information on what this code is related
to. You no longer need to keep it in you head!

When I have a well tested piece of code and somebody asks me , "How hard is to
to do X", I just sketch up some code that grossly does X and take a look to
see where the tests fail. This tells me roughly what I'll need to do to
accomplish X.

I see I've failed (once again) to keep this post small. Let me leave you with
just one more idea. You will recall that earlier I mentioned that in order to
have "full coverage" of unit tests with specificity, you need to factor your
code into very small pieces and also expose all of the interfaces. You then
have a series of "tests" that show you the input for those functions with the
corresponding outputs. The inputs represent the initial state of the program
and the outputs represents the resultant state. It's a bit like being in the
middle of a debugging session and saving that state. When you run the tests,
it's like bringing that debugging session back to life. The expectations are
simply watch points in the debugger.

When I'm debugging a program with a good suite of unit tests, I never use a
debugger. It is dramatically faster to set up the scenario in the tests and
see what happens. Often I don't have to do that. I often _already_ have tests
that show me the scenario I'm interested in. For example, "Is it possible for
this function to return null -- no. OK, my problem isn't here".

Richard Stallman once said that the secret to fixing bugs quickly is to only
debug the code that is broken. "Unit tests" allow you to reason about your
code. If you have so called unit tests that are unreadable, then you are
giving up at least 50% of the value of the test. When I have problems, I spend
_more time_ looking at the tests than the production code -- because it helps
me reason about the production code more easily.

I will leave you with one (probably not so small caveat). Good "unit testing"
and "good TDD" is not for everyone. I talked about ruthlessly decoupling code,
simplifying functions to contain single branch points, exposing state,
exposing interfaces (privacy is a code smell). There are people for which this
is terrible. They like highly coupled code (because it often comes with high
cohesion). They like code that depends on global state (because explicitly
handling state means having to think hard about how you pass data around).
They like large functions with lots of branch points (because it's easier to
understand the code as a whole when you have the context together -- i.e.
cohesion). Good unit tests and TDD work against that. If you want to write
code like the above, I don't think unit tests will work for you.

I personally like this style of programming and I think it is dramatically
more productive that many other styles of programming. Not everybody is going
to agree. I hope it gives you some insight as to why some people find unit
testing and TDD to be very productive, though.

~~~
notacoward
Great response. :) I don't actually see anything there to disagree with. Maybe
we have a difference of ... perspective? style? ... on some points, but no
actual disagreement. I rather think the two rants are quite complementary.
Thanks!

~~~
mikekchar
My pleasure. Actually, someone a few seconds ago posted "Keep trying" with
respect to my trying to keep it small and then I guess thought better of it
and deleted the post. I'm a little sad about that because I really think it
was completely on the ball. This stuff is _so_ subtle and it's super hard say
something that has any meaning without burning though a ton of trees. But the
downside is that it requires a _lot_ of effort to follow the discussion. I
think there is a simple message in there somewhere, but I've yet to find a way
to express it.

But I agree completely on the issue of style. I think it really comes down to
that. Conflicting styles is one of the hardest things to combat on a team and
you often end up with some bizarre hybrid that doesn't work at all.

~~~
ahaferburg
That person might have been banned, deservedly so. If you want to enjoy the
internet you either lurk, or you learn to ignore the haters.

FWIW I liked to read you post. Especially your point about balancing cohesion
vs unit testing, as this is something that the TDD evangelists never bother to
mention.

------
stcredzero
The current version of Unit Testing came out of the Chrysler C3 project, which
was written using VisualWorks Smalltalk. (Eventually sparking Extreme
Programming and SUnit, which was the ancestor of JUnit in Java land.) Here's
the thing about Unit Testing in an environment like that. The best way to code
and refactor code would also automatically refactor all of the Unit Tests. In
an environment like that, Unit Tests are pretty nimble. There are no long
waits for compile times. The entire test suite can run at the press of a
button from a widget integrated in your standard development environment.
Though, from what I read, the entire Unit Test suite would take an entire 10
minutes to run. However, you could easily just run the tests for the classes
you were working on at the time, and reserve the whole suite for checkin time.

So what happens when you move the practice away from this particular kind of
Smalltalk environment? Refactorings in most languages are slower without the
Refactoring Browser, and often your Unit Tests effectively double the amount
of work involved. The velocity of change slows down. Unit Tests might be less
nimble to run. A long compile time might be involved. Given those changes, it
makes perfect sense that a larger granularity of tests and fewer tests might
be more convenient.

~~~
keithnz
it may been born in the smalltalk world, but it really was worked fleshed out
and challenged and hashed and rehashed in the world of java ( and mirrored in
other languages like C++, C#, and ruby ). The XP forum was very active in
building what it meant to unit test and how to do test first development, it
was highly motivated by the idea of being able to robustly respond to change
with quick feedback loops. What became apparent is that the smalltalk world of
small modular highly composable designs is kind of critical and much of the
C++ / Java world struggled to achieve that, but there was a lot of advice on
design techniques. Over time what's been left is more of an emphasis on test
than design when talking about that pros and cons of unit testing. So we now
want to validate our software works more than we want techniques to quickly
evolve our designs and adapt to change in a robust way. Both achieve the idea
of working software. So unit testing as a technique to test your software
works may not be as good as integration testing and E2E testing, but that
wasn't its sole goal, adaptable design was. Not that design has been left
behind, if anything we are seeing quite a lot of focus on things like
functional programming and new design ideas for putting software together. The
key thing is not really the dogma of practices but the ideas of working
software and adapting to change and that the idea of robustness and good
design is embraced in the core of what you do.

~~~
stcredzero
_What became apparent is that the smalltalk world of small modular highly
composable designs is kind of critical_

All of the cost/benefit tradeoffs of Smalltalk point towards small
granularity. You had "everything is an object" to a very high extent. This
meant that objects and lambdas had to be super nimble, because literally
everything (with just a literal handful of exceptions) was made out of them.

When these pieces get less nimble, the cost/benefit changes, and the
granularity at which you operate changes as well.

 _if anything we are seeing quite a lot of focus on things like functional
programming_

Some of that is also suitable for small granularity.

Right now, I'm trying to figure out how the above applies to Golang.

------
perfunctory
I believe that contrary to the conventional wisdom one should write tests from
top down. First integration tests, then move down to testing individual
functions if necessary. Not the other way around.

On my current project, for the first time in my long career, I have 100% code
coverage. How I achieved it? By ignoring best practices on what constitutes a
unit test. My "unit" tests talk to the databases, read and write files etc.
I'll take 100% coverage over unit test purity any day of the week.

~~~
fwip
100% coverage doesn't mean much if you aren't testing that the low-level code
is doing the right thing.

For example, imagine you're testing a calculator app. Your integration tests
make sure that it never crashes, the UI works, basic math is working, etc, but
maybe the sin() function is only accurate to two decimal places.

Edit: I do not mean to imply that you, specifically, are missing things.
Rather, it is possible to write tests that have 100% code coverage while
missing many possible bugs, and I think those risks are increased without the
presence of unit tests.

~~~
perfunctory
> 100% coverage doesn't mean much

That's absolutely right. 100% coverage doesn't say much. In fact, by itself
it's a pretty meaningless measure of code quality. But the top-down idea still
holds. In practice bugs are more likely to happen at the seams between
components, and therefore it's where one should test first, not last. It of
course depends on the type of the project but I believe it's true for the
majority of projects out there.

~~~
achamayou
100% means nothing, but 30% means something is wrong. Coverage tells you
nothing, but the absence of coverage is interesting and useful.

------
jb3689
The number of times I've said "oh I bet this works" just to be wrong when I
wrote the tests is countless. For me it's more about having small well-defined
interfaces with very strong tests

Moreover, tests that don't break are useless

~~~
munk-a
Though brittle tests are also useless, we have some snapshot result tests
(take a large set of assumptions, push it through the pipes, get a large
result set - in our case it's a webpage, but this holds for other settings)
and almost every change to code covered by one of those tests breaks it, then
the developer updates the test to work again, was their change logically
correct - it can be hard to tell if 500 lines of a 2k expected result changed
in minor ways.

So a too-brittle test can be valueless.

------
trixie_
Integration tests really are the best bang for the buck. So many times people
think they tested their code by mocking something like the database layer and
are surprised when the application in production breaks. Everything can be
tested in isolation and work perfectly, but often it's how the components work
together that determines if the system works at all.

Integration test what you can. Unit test core pieces of logic. Avoid mocks.

~~~
sime2009
Fully agree. The majority of us are writing code with has to talk to a lot of
other systems which we didn't write and don't control. These interface areas
between systems are huge gathering places for bugs.

Also, mocks are self-deception at best and lies at worst.

------
natashas
In general automated tests are used mostly to save time as the earlier you
find the bug the more savings you have when fixing it. When your testers find
it, they need to create and fill the ticket, assign it to developer, then
developer needs to reproduce, fix it and assign it back and tester needs to
retest it. And it's much more expensive if it happens in production!
Integration/unit tests can catch a lot of those. There are some diminishing
returns so 100% coverage is not needed and integration tests are more
effective in catching bugs so I agree with the article's idea. Use unit tests
for some real units - algorithms, calculations etc, and don't just test
mocking framework. With additional layer of end to end tests and manual
testing system should be able to achieve pretty good quality without spending
unreasonable amount of time for it.

------
vbsteven
This is pretty similar to my approach. In the context of developing an API I
use integration tests more than unit tests. It is straightforward to spin up
an application server against a test database with some test data. Run some
example requests against it and verify the results.

I frequently have Test classes which contain a full scenario related to a
specific resource or action. For example:

* create resource * lookup resource * modify resource * list all resources * delete resource

The JSON files for the integration tests are then used as examples in the API
documentation.

Unit tests are reserved mostly for verifying business logic components. There
is no need to setup a complex ControllerTest with mocked out Services or
Repositories as you don't care about the internals anyway. Just the input vs
output and the integration tests cover those already.

------
edem
My comment from the original post verbatim:

I agree with the part that you should write tests, but I definitely disagree
with the part that most of your tests should be integration tests.

As you pointed out the testing pyramid suggests that you should write more
unit tests. Why? Because if you have ever tried TDD you know that unit tests
make you write good (or at least acceptable) code. The reason for this is that
testing bad code is hard. By writing mostly integration tests you lose one of
the advantages of unit testing and you sidestep the bad code checking part.

The other reason is that unit tests are easy to write. If you have interfaces
for your units of code then mocking is also easy. I recommend stubbing though,
I think that if you have to use mocks it is a code smell.

Also the .gif with the man in pieces is a straw man. Just because you have to
write at least 1 integration test to check whether the man has not fallen
apart is not a valid reason to write mostly integration tests! You can’t test
your codebase reliably with them and they are also very costly to write, run
and maintain!

The testing pyramid exists for a reason. It is a product of countless hours of
research, testing and head scratching. You should introspect your own methods
instead and you might arrive at the conclusion that the codebase you are
working on is bad and it is hard to unit test, that’s why you have chosen to
write mostly integration tests.

------
1337shadow
Maybe but do you think 100% coverage is such a bad experience to have at all ?
This can be a rich exercise, and a rite of passage to know what tests matters,
which should be automated (fuzzers, dbunit etc), which can wait, etc... and
how to get to 90% with the effort of doing doing only 30.

Integration tests should write themselves. It depends on the projects, but
personally i use to get 65% of coverage for the price of 5%. For example you
want to test a bunch of commands, then you can make a function like
autotest('some command', 'some_fixture.txt'): call "some command", capture the
output, and write some_fixture.txt with the captured output, and fail
complaining that it had to create some_fixture.txt. The next run it will find
some_fixture.txt and compare the output and fail only if it differs.

Unit tests should of course be hand written, but to cover everything that
matters like for bugs, or for what you want to refactor in TDD. Of course any
line that's not covered can potentially break when upgrading versions, but
this kind of breaks are likely to be revealed by the 90% of coverage that I
think you can get with minimal effort by applying this recipe. Then, you can
afford 0day updates when underlying libraries upstream make a new release
candidate.

------
kords
100% coverage doesn't prove the code is fully tested, just like having
unit/integration/acceptance/smoke and other types of tests doesn't prove that
application works. But this doesn't mean we stop adding tests to our suite, so
why stopping adding more coverage?

I totally agree with "just stop mocking so much stuff". Most of the time we
can use real implementation and by doing this, we'll also increase coverage.

------
raverbashing
100% unit test code coverage is just Uncle Bob BS and it is infeasible for the
most part

And while it will give some warranties about correctness, it will not
fundamentally guarantee the application does what is supposed to do.

Integration tests are certainly better

I can only see the headline being a "great wisdom" to people who have accepted
Uncle Bob and TDD crap for too long and without questioning it. Because it is
obvious.

Why it is obvious? _Because that 's how things were done most of the time_

When you had 640k of RAM and a C compiler writing unit tests was not
impossible, but pretty hard. Testing that your app reacted to inputs and acted
correctly was doable and easily automatable. And what wasn't automatable would
be tested manually.

Now here comes the "holy highnesses" of testing gurus, gaslighting developers
and saying that code with no tests (which tests? automated? unit? the
definition is purposefully vague) doesn't work or that the only blessed code
is the one that is produced through TDD onanism? No thank you

------
mpweiher
Counterpoint: _Integrated Tests are a Scam_

[https://blog.thecodewhisperer.com/permalink/integrated-
tests...](https://blog.thecodewhisperer.com/permalink/integrated-tests-are-a-
scam)

I found that argument more convincing, and it also aligns better with my
experience.

The point about 100% coverage not being a good goal is pretty solid.

~~~
wenc
For the benefit of others, it's also worth pointing out that the author isn't
talking about "integration tests" but "integrated tests" (as he defines it).

Integration tests = testing individual software modules as a
group/combination.

Integrated tests = instead of testing one behavior at a time, multiple
behaviors are tested simultaneously. (or in the author's words: "any test
whose result (pass or fail) depends on the correctness of the implementation
of more than one piece of non-trivial behavior.")

Integration tests are very important.

~~~
mpweiher
It is also worth pointing out that the author used to use the term
"integration tests" in this post, and only changed to "integrated tests" after
writing.

So a claim that these are completely distinct/disjoint ideas is not really
supportable.

As far as I can tell, he doesn't explain the name change, but having watched
the talk before he changed the name, it looks like "integrated" is a somewhat
more general term.

Specifically: if you are "testing individual software modules as a
group/combination", how are you going to do that without "depending on the
correctness of the implementation of more than one piece of non-trivial
behavior."

I see exactly one way: your "integration" tests have to be trivial, really
more akin to characterisation tests. "When I hit this external API with this
value, do I get this reaction?"

As far as I can tell, he doesn't talk about this in the post, but he does in
the talk/video. These are interface/boundary tests. So you test each unit in
isolation, and you test the boundaries. If your unit tests are sufficient and
your boundary tests are sufficient, plugging things together will work.

So no integrated/integration tests.

Now I am actually not that strict, I let my boundary tests leak a little
towards integration/integrated tests, because it is possible to just not hook
up the components right. So you want to check that with some smoke tests.
However, once they are hooked up, the more complex behaviours Just Work™.

------
neo2006
I agree with the writer about the statement that you don't need a 100% test
coverage but you still need to check your coverage. I mean you need to run
coverage and go through your code and see if there is any gap in your testing.
That said I don't agree with the statement "mostly integration" the reason is
that he don't take into an account the ROI of a test, he only take into
account the outcome. I mean e2e tests are the best to catch bugs but they are
harder to achieve harder to debug and harder to maintain. Same go for
integration test they are harder to debug, maintain and perform than unit
test. Developpers forget that their time is money and that if they spend time
on a test just because it make them feel safe that mean that the project will
cost more money. The general rule I use is simple. When you test, test
behavior not code and do your test at the lowest level possible.

------
thom
The thing that changed my approach to testing, at least in OO languages, was
being shown that use cases/tasks/"things the system can do" should be first
class objects in the model. At that point, you have a single layer that
exposes everything of real value in the application, giving you a really
simple test surface. Underneath that surface you can refactor to your heart's
delight, without worrying about having to maintain lots of pointless unit
tests for trivial behaviours on each object - all that matters is maintaining
the end to end functionality. So yes, I agree integration tests are the key,
and you can architect your application to make this easier. Not over-testing
your underlying model is just good old fashioned information-hiding. Testing
the UI on top I leave as a matter of taste.

------
jondubois
>> Write tests. Not too many. Mostly integration.

If a group of reputable programmers got together and published a book about
programming which had a single page which only contained this line in a large
font, it would be the most useful and valuable programming book ever written.

~~~
edem
Definitely not. See my comment on the parent.

------
SomeHacker44
Even at 100% code coverage it is possible to have missed many code paths that
involve nonlocal control flow. The most common issue here would be exceptions
in languages that support it. In most operating systems there are various
interrupts that a program can also encounter (e.g., a lovely SEGFAULT in many
*nix type operating systems) and other operating system and hardware level
issues (like OOM killer in Linux or an NMI or ECC memory error in embedded
systems).

So, 100% code coverage may not even be enough for some applications as was
discussed in another too level thread (medical device) or other hard to fix
(spacecraft) or life-threatening situations.

------
jamougha
We have a new guy at work who loves unit tests. Yammers about them constantly.

If he's working on a feature that needs changes to classes A, B and C then his
workflow is: make all the changes to class A, along with lots of unit tests.
Then do the same for class B. Finally, do the same for class C. And then,
after days of work during which the application doesn't even compile, he runs
the integration tests and discovers that the application is broken.

Then someone comes over and rewrites his code so it works.

~~~
munk-a
So... that rewrite of his code is pretty quick and easy because the components
you're integrating are well covered by tests at that point?

Unless the tests on A, B and C are ineffective I think this comment isn't
really about tests - some hired developers just aren't great at writing
software but if they all used vim that wouldn't mean the cause of their errors
was the fact that they used vim.

~~~
jamougha
The tests are ineffective. That's because, as the article notes, unit tests -
tests of single functions, with dependencies mocked - give very little
confidence that code actually works. Problems usually appear in the
relationships between parts.

And because they're tightly bound to the internal implementation of the code
they test the tests often have to be rewritten when the code is rewritten.

Does this mean unit tests are bad? No. They way they're frequently used I
believe they are bad, but you're correct that that wasn't the point I was
making. I'm really commenting on the way that common wisdom pushes gimmicky
methodologies but doesn't talk about the fundamentals of how to make code
changes in an effective way.

~~~
munk-a
I agree with you then - there is nothing inherently beneficial from the
quantity of unit or integration tests you have, the quality of the tests to
accurately target specific edge/error cases is what makes them valuable.

As an aside I find TDD a particularly bad trend for these sorts of bad habits,
TDD is a great idea in theory, but you'll often see TDD unit tests come out
mimicking the code under test, i.e. "How to test if this function containing
`return n + n ^ 2` is correct... well, let's generate some random numbers and
then see if the output is equal to that random number plus it's square! That's
like full coverage!" Having tests that duplicate the code under test is a
pretty easy trap to fall into with micro-tests and it should make you
suspicious of any test that covers a very small unit of code.

This is similar to a statistical problem involving over-fitted trend lines,
you can construct a mathematical formula that can go through any number of
arbitrary (X,Y) points, leading to an average deviance of 0, but this formula
will probably break as soon as you take another sample.

------
dang
Discussed at the time:
[https://news.ycombinator.com/item?id=15565875](https://news.ycombinator.com/item?id=15565875)

------
luord
> you find yourself spending time testing things that really don’t need to be
> tested

The thing with 100% code coverage is not that one should write more tests, but
that you should refactor everything that isn't business logic (and, thus,
nothing that would be part of the functionality of the application to test)
into third party libraries, if it doesn't already come from third party
libraries, which it probably should.

------
Dangeranger
As a counter point to this opinion of Kent's, who I respect greatly, I'd like
to propose J. B. Rainsberger's article:

"Integrated Tests Are A Scam" with presentation of the same name.[0]

[0] [https://blog.thecodewhisperer.com/permalink/integrated-
tests...](https://blog.thecodewhisperer.com/permalink/integrated-tests-are-a-
scam)

------
blauditore
I think the root of many testing problems is that people write code and
continuously test manually until they think it works correctly. Then, they
start writing some automated tests.

What I think is much more effective is to write test _instead of_ manual
testing while developing. That is, what you would do by hand just implement in
a test. And only at the very end, double-check manually.

~~~
executesorder66
Isn't that just TDD?

~~~
blauditore
Similar to it, but not as strict. TDD implies that you write tests first,
which also leads the designing of your application code. What I'm describing
is more general, where testing happens alternatingly with writing domain code,
"roughly" at the same time.

------
camnora
I’m not so sure I agree. Often isolated TDD can lead to much cleaner code and
succinct solutions. There is also a benefit of quicker test execution. I’ve
seen a noticeable difference in the quality of software delivered to my
clients with this approach. Gary Bernhardt covers these topics in great depth
and highly recommend to anyone interested.

------
kwhitefoot
I know this is off topic but why is it that people who, presumably, want us to
read their words of wisdom put such a pile of crap at the top of the page that
I have to hit the page down key five times before I get to the article?

------
castlegloom
Yeah this is article comes out and says that it's based on gut, not science.
So speaking for anecdotal evidence, unit tests that don't break and frequent
deploys are way better than finessing one more Gherkin test.

------
garganzol
Those sticky banners and elements injected by Medium are ridiculous. They make
a good article hardly readable on HD desktop monitor. Sad state of things, I
remember how once upon a time it was a great platform to be at.

------
NicoJuicy
At work, the unit tests and data are locked, even the webserver.

Every change of data or requirements causes them to fail or fail because of
human error.

At home, I do integration testing and it seems to work much cleaner and more
complete.

------
aboutruby
It's confusing with Rails, because integration tests are now system tests, and
the kind of integration tests described in this article are named controller
and requests tests.

------
tretiy3
This is good. But russian translation is terrible (were translations some
marketing move?). Please ignore translation. Read original only

------
je42
have to disagree here...

not writing unit tests means architecture change will break your integration
tests. hence, you don't have coverage left during an architecture change.

typing information on a unit (class or function) is nice, except it doesn't
find all bugs. If it were to find all bugs about a unit, it would be a
complete type checker ( which usually comes in a system like
[https://en.wikipedia.org/wiki/Isabelle_(proof_assistant)](https://en.wikipedia.org/wiki/Isabelle_\(proof_assistant\)),
etc, which is a pain to use... )

thus the only "efficient way" to cover the gap between type information and a
full blown type checking system are unit-tests.

integration test run slower, hence, you are also wasting more time.

integration test are more difficult to understand by other people, especially
if they break.

integration test are generally less stable than unit-test because they involve
more units.

integration tests usually don't cover all (error) scenerios of the involved
particular units. you do that in unit-test to get the branch coverage to 100%.

his issue regarding mocking potentially can be solved differently. change your
interfaces and code architecture such that you minimize mocking and make unit-
testing easy. i.e. experiment moving the boundary between units.

------
pmarreck
I would say mostly unit and very few integration, if the app codebase is
designed well, anyway

------
sivanmz
Maybe the author of Socket.io and Zeit.co just happens to work on integration
problems?

------
ttty
> Well, when you strive for 100% all the time, you find yourself spending time
> testing things that really don’t need to be tested

I noticed people skip testing what is hard to test and just test what it's
easy...

Not what really matters

------
maccio92
Wow honestly maybe rethink your upsells here.. literally had to scroll half
way through the page to get to the actual article. And what's the purpose of
having a full screen splash image of a pineapple with sunglasses..

~~~
wishinghand
Does Medium force unrelated images into headers? I've never written on it but
I can't remember the last time I saw a relevant header image.

~~~
dguo
You don't have to add a header image, but doing it no matter what seems to be
the general thinking on Medium, presumably to get people's attention. See
freeCodeCamp's guide on writing for them:
[https://medium.freecodecamp.org/how-to-get-published-in-
the-...](https://medium.freecodecamp.org/how-to-get-published-in-the-
freecodecamp-medium-publication-9b342a22400e#590c)

"Medium will automatically make the first image in your article its featured
image. This means your article’s og-image meta property will be this image.
This image will serve as your story’s ambassador everywhere: social media news
feeds, Reddit, Google News — even RSS readers."

"Start considering images early in the writing process. And never publish
without at least one image. Otherwise your story will be all but invisible in
news feeds."

I agree the header images are rarely relevant, and I wish they weren't
considered must haves.

