
The evil unit test. - wtfdeveloper
http://www.makinggoodsoftware.com/2012/01/27/the-evil-unit-test/
======
MrEnigma
I've found that the higher level the test (i.e. unit -> integration ->
functional) the better they catch things, but the harder it is to figure out
what broke.

For instance we had a rule at a place I worked, where we needed to have 80%
unit test coverage. And what is described in this article happened. Anytime
we'd make a small refactor, we'd have to go update 3-4 test libraries. And if
it was a big one, then you have a lot of tests to update. And since some were
complex (or since there were a lot of them) people didn't dig into that much.
When people finally did it sometimes showed the test wasn't even testing what
it was supposed to anymore.

The other issue is we had a lot of bugs around the integration of data. For
instance if the DB class returned back an empty array vs null vs empty string.
When we did the unit test we mocked it to return what we thought it should,
which may not be the correct one. Integration tests better caught this,
functional tests even more so.

It seems a lot of shops don't use tests at all, now working at places that
have done both, I'd rather error on the side of fewer tests, and do it smart
instead of hitting a metric and assuming your code falls into line.

~~~
wr1472
Sounds like your test were quite brittle, if refactoring application code
causes tests to break. A good mental rule I use is: "test the what not the
how". It also a good reason to program (and test) to interfaces than to the
implementation, although you have to be careful with this as it is all too
easy to go OTT in your use of design patterns. You have to know how to strike
the right balance.

~~~
eaurouge
There are various kinds of tests, from unit tests to behavior driven testing.
Refactoring application code may cause tests, especially unit tests, to break.

If you're writing unit tests, by definition, you are testing the
implementation - that's the point. It seems you're referring to behavioral (or
black box) tests, which are very different.

~~~
wr1472
Agreed refactoring method signatures will cause tests to break, but they
should be relatively trivial to fix.

Black box tests are not limited to behavioral or system integration testing.
you can black box at the unit level. In fact if you are doing proper TDD you
will write the tests first, which help to define the interface of the class
under test, make sure the test fails to ensure you are testing for something,
and then finally write the implementation to make the test pass. Such an
approach implies it is black box as you've defined your interface before your
implementation.

Remember you are testing what it does, not how it does it.

~~~
jsdalton
> Remember you are testing what it does, not how it does it.

While I tend to agree with you (I strongly prefer black-box testing over
white-box testing), this is a hotly debated topic and many people feel quite
strongly the opposite.

You can Google around and find many advocates for white box unit tests. Martin
Fowler's article from several years ago I think does an excellent job of
presenting a balanced look at both sides:
<http://martinfowler.com/articles/mocksArentStubs.html>

~~~
wr1472
I've not come across the article before. The section on coupling tests to
implementation I guess is the most pertinent part
([http://martinfowler.com/articles/mocksArentStubs.html#Coupli...](http://martinfowler.com/articles/mocksArentStubs.html#CouplingTestsToImplementations)).

> Coupling to the implementation also interferes with refactoring, since
> implementation changes are much more likely to break tests than with classic
> testing.

Fowler acknowledges two different styles of tests - state/behaviour also can
be termed as classic/mockist. He doesn't really advocate one over the other. I
do mock when I have to, but only when there is no other way to setup my test.
It is certainly not the default. I find that taking this approach does not
make my tests brittle, and makes them robust enough to handle the majority of
refactoring exercises.

Of course YMMV.

~~~
MrEnigma
Yeah that's a good breakdown of the different approaches.

We started using DI, and then had a rule that all unit tests need to be
isolated. Which means mocking everything they touch.

For small methods it makes them very fragile, although the other way you end
up getting a lot of intertwined tests that can wreak havoc after a lot of
tests are made if you're not careful.

I've found that picking what you want to isolate seems to work the best. Our
unit tests isolated everything, integration isolated any connections (i.e.
DB/Services/etc) which made the tests much less fragile (and much more
valuable), but made the errors harder to track down.

------
Swizec
There is a time and place for unit tests - when you're creating algorithms.

Most of the time, however, we are creating web apps and things that run at a
much higher level of abstraction than the underlying algorithms. We should
probably write tests at the same level of abstraction our solution is.

Although, I guess, when you're writing medical software or something for a
bank, you should be a bit more strict than when you are making the next best
social sharing app.

A guy I know once solved a bug by having a unit test for the md5 function from
the standard library of the language at hand ... turns out a specific version
was buggy.

Either way, I prefer using integration/functional test. What I care about is
that the right output flies in for a certain input. What I don't care about is
_how_. So I don't test for that.

Also, _Unit tests must be created before the code they test._

yes they should. It's very helpful to think about how your code will be used
before you write it, helps you think it through as well.

~~~
nradov
The problem with creating unit tests before the code is that it simply doesn't
work for exploratory programming. I often don't know exactly what I intend to
do or what the tools are capable of until I'm fairly far into coding the
features. Writing unit tests at that early stage would be a complete waste of
time.

If you know ahead of time exactly what the inputs and outputs should be then
sure, writing unit tests first is a good practice.

~~~
ams6110
I don't understand what you are saying. How can you start coding if you don't
even know what you intend to do or what your expected inputs and outputs are?
What are you exploring, and how do you know when you've found what you're
looking for?

~~~
colomon
Another simple example from my work of the moment: my goal is to import points
from a file (that I'm assured does have points) using a 3rd party file import
library. I don't know in advance how the 3rd party library imports points, so
the best test I could possibly write ahead of time is a fairly high-level
integration test, "Did we manage to import a point?"

All my actual programming work here consists of adding printfs at strategic
points to tease out exactly where I should be looking for the point data. Unit
tests are completely useless for this process.

Come to think of it, even once I've got it figured out, it's very hard to see
how to usefully use unit tests here...

------
16s
I used to oppose unit testing because I had a good C++ compiler with all
warnings turned on and I foolishly thought that was all I needed and that unit
testing was only for dynamic languages that did not have compilers.

I'm a big fan now though. If I have written code that's hard to test, I re-
factor the code so that it's testable. I've found that un-testable code is
generally bad code (it works OK, but is hard to modify later). I also run my
unit tests during the build (right after compilation), so I know right away if
something fails.

I've come to really like unit testing. I'm less afraid to change things and
feel better (in general) about the quality of my code.

~~~
franklindholm
I agree that Unit testing is a power full tool. But tools are just tools, and
they can be misused. I see this sometimes at companies where they "make" their
developers write unit tests, they are just writing them because they have to
and this often shows in the quality of the tests. Everything is green in
Jenkins so obviously the functional tester did something wrong :)

------
agentultra
So basically: _don't test the implementation and don't test the framework._

Do test the API. Test the output.

Good, smart tests should be a pleasure to work with. They should give you the
confidence to change the implementation without mercy. They make the feedback
cycle faster, bugs more visible, and documentation more thorough.

The only time I've found unit tests to be a pain are when testing
asynchronous, event-driven code. The tests aren't any more brittle, but they
really need to be supported by good integration tests in order to find the
kinds of bugs that can surface in such systems.

I think the point about unit tests is that they should be the default and not
the exception. 100% coverage is an asymptotic value. But if you write the test
first you can get pretty darn close.

~~~
dspillett
_> So basically: don't test the implementation._

I thought that was the point of unit tests...

 _> Do test the API. Test the output._

... and that this would be integration testing.

Am I drawing my mental lines separating the range of tests in the wrong
places?

~~~
agentultra
I can't really say, but if you read _The Clean Coder_ we might get on the same
page.

fwiw, what I mean when I say "don't test the implementation," is that your
test should _not_ test "how" a function or object performs its task when
called.

tldr; unit tests should test units -- the smallest amount of functionality
possible

A trivial example might be an object that retrieves some data from a
persistent store and prints it out. It might have a "fetch" method whose
argument is a key in the data store.

You could easily write a test case that checks when "fetch" is called that the
data is written to the standard output stream. It would pass and you could go
home after a hard day of work.

But if the requirements change and you need it to write to a socket instead
and format the data for a binary stream -- you'll have to modify the "fetch"
method and the tests.

If we were clever though we'd realize that when we wrote the test to see that
"fetch" output its result to standard out we were actually asking the object
to do too much and our test was testing several different units of work that
could be broken down.

Ideally our hypothetical object would take a couple of callable objects that
handle things like printing data to streams and formatting it. Then our test
for fetch only has to test one small "unit" of functionality -- not several
units (retrieving data, formatting it, printing it to a stream).

hth

------
Peaker
I find TDD to be somewhat redundant when I code Haskell.

The types cover more ground than typical unit tests -- and if you follow a few
simple principles (avoid partial functions and some of the worse part of the
libraries), you can trust the compiled code more than you trust UT'd code.

Production-quality Haskell does usually employ tests, and those are a joy to
write (with QuickCheck), but there's little need to spend time writing tests
at the early coding phases.

When doing large refactorings, you can also trust the type system to guide you
and find virtually all bugs introduced.

------
fuzzylizard
I agree with only one point in the article; the ability to delete tests. Tests
of all kinds should be living, breathing code, just like the code in the rest
of your application. As such, tests, like production code, can become outdated
and obsolete and need to be updated, changed, and sometimes deleted.

However, I do not agree with the idea that tests are evil. If your tests are
failing, or breaking regularly, or are hard to write, then your tests are
trying to tell you something. Tests are a direct mirror of the state of your
code. If your tests are brittle and hard to manage, don't delete the tests,
fix your code. If you need to instantiate a tonne of objects in order to make
your tests work, fix your methods dependencies and learn to test your methods
in isolation of its dependencies.

Any production code needs tests, this is non-negotiable in today's development
eco-system. But I get tired of people who do not understand the purpose of
tests complaining that tests are evil. If there is a problem with your tests,
then your tests are telling you there is a problem with your code. Clean up
your code in order to clean up your tests. However, remember that as code
changes, your tests will need to change as well. It is okay to edit, delete,
change tests to reflect the current state of the application. Just change the
tests first to express how you want to interact with a specific bit of code
and then change the code, not the other way around.

------
yason
The thing with unit tests is that they test _units_. A unit of code _might_ be
a simple algorithm inside one function but more often than not the _unit is a
more complex_ interaction that is better covered by a smart test on a whole
lot higher level.

I prefer fewer tests that try to touch most of the essential features in one
go, rather than several low-level tests that test simple if not idiotically
limited cases. I prefer fewer tests and high code coverage: by being smart you
can do a lot of testing even with a single test if only it touches the
strategically most sensitive parts of your program.

Also, most low-level stuff you will test implicitly by the rest of your code.
Lots of your higher level stuff would fail to work if your lower level
functionality was crappy. Thus, the basic utilities tend to straighten
themselves (bug-wise) early on.

Lines of code have a cost and lots of tests drive up your lines. Less code,
smarter code. Less tests, smarter tests.

------
tikhonj
Actually, his 2x3 example is not very good. The fact that 2x3 is 2 three times
is the _what_ , not the _how_ ; in a perfect world you would just assert that
nxm == n + n + ... n m times for all n and all positive m.

This is, coincidentally, where property based testing like QuickCheck comes
in.

Edit: Just pretend 2x3 has an asterisk instead of an x :)

~~~
prodigal_erik
(FYI, 2 * 3 is okay; HN preserves * followed by whitespace.)

~~~
tikhonj
Ah, thanks. That was probably the issue, because some of my asterisks worked
but others didn't. I'll keep it in mind for the future.

------
3wetwetw
Absolutely - even those of us (like me - disclaimer: I work at Typemock) know
that there is a time and a place for unit tests and TDD (not the same thing).

In fact, we wrote about this before <http://bitly.com/zdyJfl>

We're hosting a webinar on Wednesday about different kinds of testing and when
a unit test is appropriate and when another kind of test, like an integration
test, may be a better choice - <http://bitly.com/xeYSYg>.

Unit tests have a time and place. There are times in which they are a must if
you want to reduce technical debt and test your code - both if you're writing
mission critical stuff or even when you're writing the latest social sharing
app (remember the Fail Whale? We all hate that!). But, no, you don't need TDD
on greenfield code all the time and you shouldn't have to write tests for
logic.

------
owenjones
The title and content don't correlate. This would be like having an article
titled "The Evil Hammer" and then in the article explaining how "Hammer's
aren't actually evil, just you shouldn't use them in situations where a saw
would be more appropriate."

I think anyone falling into these pitfalls just aren't writing good enough
tests, don't dissuade them from furthering their testing skills, promote
skillful testing. And change the title.

------
brown9-2
A lot of these points apply to any sort of coding convention that a group
agrees upon, not just unit testing.

------
jwatte
Unit tests, acceptance tests, and integration tests are different things. Unit
tests test that the implementation behaves as intended, and generally grope
around private internals. These change when implementation changes. Acceptance
tests test that the public API is implemented correctly. These change when the
API/interface change. Integration tests test that the end system implements
requirements correctly, and change when requirements change. Most TDD hate
comes from not separating these correctly, or not having suitable tools for
each type, or from (gasp!) not actually having a clear understanding of
requirements, interface, or implementation.

------
jrockway
Many people write bad tests. So stop doing that, but don't stop writing tests.
The reason why every public method should be tested is because changing how a
public methods works is going to break everything using that public method. If
you must make a change, fix the test along with the method. It broke to remind
you to fix the consumers of your API.

Good tests don't depend on implementation. Your tests should tests interfaces
instead of objects and anything implementing the interface should pass the
tests. If it doesn't, your interface is wrong or your test is wrong.

------
chaostheory
I don't feel that this article is relevant for most programmers using newer
languages such as Ruby or Python. There's a simple change in the way I use
JUnit that I learned from using Ruby (RSpec, Cucumber) about 4-5 years ago:
Don't test methods or functions. Instead test expected behavior e.g. a project
requirement. Then you can pretty much avoid every problem listed in the
article.

------
peteretep
I wrote a similar article not too long ago:

[http://www.writemoretests.com/2011/09/test-driven-
developmen...](http://www.writemoretests.com/2011/09/test-driven-development-
give-me-break.html)

------
lmm
>You shall use a foreach every time you need to perform a loop.

Yes

>You will never access your database if you are not using an ORM.

Hell yes.

(Not necessarily disagreeing with his conclusions)

~~~
tikhonj
What about while loops? That's the problem with blanket statements: they're
very rarely 100% correct.

------
prez
_You shall use a foreach every time you need to perform a loop._

Works pretty well in Python.

Though it doesn't invalidate the point the author tries to make.

~~~
cbs
_Works pretty well in Python_

But xrange() in python is just a roundabout way to get back to having a
regular for loop.

    
    
        for x in xrange(0,100)
        for(x=0, x<100, x++)

------
keithnoizu
Sounds like a bit of a hyperbole,

    
    
      Blanket statements like 100% coverage, unit test per method are not too helpful, and it's important to understand the point where you will start to see diminishing returns with unit tests but that said well written unit tests with ~ 85-95 coverage by line free of conditional logic (use custom asserts) with meaningful test names/descriptions aren't exactly going to put you in league with beezlebub.

------
hashfold
there are really nice comments below in this thread on why testing
functionality is better than writing unit test cases to test the
functions/methods/code. e.g. wr1472: test the what not the how MrEnigma: the
higher level the test (i.e. unit -> integration -> functional) the better they
catch things Keithnoizu: verify requirements not implementation details

we understand that the unit testing is trying to test the functionality and
making sure that the acceptance criteria is also met at the same time. I would
like to mention one more point is that we should also try to write unit test
cases not just only test the functionality but also try to do a good code
coverage. unit test cases are tools for: 1\. functionality testing 2\. comply
with acceptance criteria 3\. a tool to help good code coverage 4\. be
iterative. be backward compatible as long as it could. (anyway to add logic
based on version of the code change? need to think on this). yes it makes it
little bulky but will work and avoid needing code cleanup and redesigns.

let unit tests help come up with min 80% of code coverage.

------
dabit
Was this a sarcastic post?

~~~
Confusion
What makes you think that?

