
It's OK Not to Write Unit Tests - raganwald
http://blogs.msdn.com/b/cashto/archive/2009/03/31/it-s-ok-not-to-write-unit-tests.aspx
======
nopassrecover
Favourite paragraph:

"If you're a Java programmer and want to have a rude awakening, go download
Jester. Jester is an automated mutation testing tool - it goes in and replaces
"<=" with "<", "&&" with "||", "!" with whitespace. And then it re-runs the
tests. And then you get to watch in horror as your tests still all pass,
regardless of what the product code actually does."

~~~
njharman
I don't understand at all. Why would the tests still pass?

~~~
WorkerBee
Because the author wrote poor tests and thinks that you did too.

~~~
cabalamat
Indeed. The whole essay boils down to "bad unit tests are useless, therefore
unit testing isn't worth bothering with".

~~~
dagw
More like "bad unit tests are useless, good and useful unit tests are hard and
take a significant chunk of time, therefore weigh the costs vs the benefits
before deciding whether to bother or not"

~~~
WorkerBee
Hm. Before deciding that good and useful unit tests are too expensive, spend a
week or three doing live support of poorly reported and hard to reproduce
issues where the numbers were slightly out or the result came back ten seconds
later than it should, one time. that will change your perspective on what's
hard and what's not. And possibly damage your will to live.

~~~
cabalamat
Indeed. If you think good unit tests are time-consuming, try debugging code
with lots of subtle bugs in it.

~~~
dagw
If the bugs are that subtle what makes you think you're unit tests are
comprehensive enough to catch them? The hardest bugs to debug are the types
that show only on high latency networks and only when dealing with files
bigger than 2GB and only after the app has been running for at least 50 hours.
Those are also the type of bugs which are almost impossible to write unit
tests for.

I'm not arguing against unit tests, they definitely have a time and a place.
I'm just not convinced that they are an easy and instant cure for all your
hardest bugs.

~~~
WorkerBee
Unit tests may leave you with remaining subtle bugs. This is better than
having both subtle and obvious bugs. Obvious under test, e.g. "the class
returns null when it should return object y", which can have subtle and easy-
to-miss symptoms in a production system

------
gruseom
I don't see why a fanatical anti-rant is any better than a fanatical rant.

 _There's this theory out there that Agile projects can refactor fearlessly
because there's this immaculate suite of tests that can sound the alarm the
second the smallest regression gets introduced. But anyone who's actually
tried it knows that it's mostly just a fantasy._

Poppycock. My project has an extensive test suite that we rely on to make
major design changes with alacrity, just exactly the way that unit-test
advocates advocate. We couldn't dream of doing what we're doing without it.

It's true that not every module of every system responds equally well to this
style of testing. It's true that putzing around revising test infrastructure
for the nth time really sucks. But these are points that belong in a _serious_
discussion, the kind that recognizes tradeoffs.

~~~
douglasputnam
cashto's post was a "serious" effort to disperse the cloud of dogma
surrounding unit testing and TDD. I hardly think that lashing back with
"poppycock" is driving the conversation in the direction of "serious
discussion".

cashto emphasized that unit testing is useful for teams with non-expert
programmers (viz. his comment about the Dreyfus model of skill acquisition).
Many of us are non-experts who benefit from working with the TDD training
wheels on; it's O.K. to be less a Jedi All-Star hacker. But the phenomenal
programmers, the men and women who were 10 times better than me, could not be
bothered with TDD, even through the endorsed it for mere mortals.

To paraphrase the TDD credo of “no test is worse than having no tests”, we can
assert that blind faith is not better than no faith at all.

~~~
Vitaly
this is one of his worst arguments. I find it almost exactly the opposite.
I've been programming for almost 20 years and I am considering myself an
expert. and many people around seem to agree ;). most of the beginning was
spent w/o any kind of unit testing. But I find it that I do more and more
testing with time, and I'm sure as hell I'm not becoming a weaker programmer
that suddenly needs the training wheels ;)

~~~
cabalamat
I've been programming for nearly 30 years, and started doing unit testing
about 8 years ago. I've found it helps me a lot.

------
tetha
Ah, when I first began to read this, I expected one of those rants against
unit tests which don't leave any kinda good thing about tests and I mentally
got ready to comment about that.

However, I am quire pleased that this is a very sensible and mature position
about unit testing (I think a short version would be: well, testing can help,
it won't carry you and you have to stand on your own feet, but it can help),
which is pretty close to my own position. Good Article.

------
10ren
Aside on testing: I was surprised in re-reading Fred Brooks how cloose he
sounds to unit-testing, in _Mythical Man Month_ , 2ed, pp.147-148. He suggests
testing components separately, before integrating, even though this requires
lots of "scaffolding".

Scaffolding includes _dummy components_ , _miniature file_ (minimal eg of a
file format), "generators of test data, special analysis printouts, cross-
reference table analyzers" (jigs and fixtures).

 _"It is not unreasonable for there to be half as much code in scaffolding as
there is in product"_. I end up writing a lot of the above kind of code - not
as automated unit tests, but just as a way to make sure my code is doing what
I think it's doing - just plain ol' testing. It can feel wasteful, so it was
reassuring to hear Fred's take.

------
jwr
As with all things, truth is somewhere in the middle. I've found that unit
tests do help me write better code and find corner cases which I would
otherwise miss. But I don't treat them as a religion, and I don't write them
for every function.

Just don't overdo either way and you'll be fine.

~~~
arethuza
But people really don't want that kind of sensible mature argument and would
rather have set of "best practices" that they can blindly follow.

~~~
abp
Yes, but only if one got zero or not much experience in the field. So actually
you are saying something that is true for everything.

~~~
arethuza
You'd be amazed at the number of experienced developers who tell me "it is
best practice to do XYZ" who when challenged as to what is the underlying
justification for this recommendation can't actually provide a sensible
answer.

------
johnswamps
Original discussion: <http://news.ycombinator.com/item?id=922715>

------
chmike
With my students and used the analogy of unit tests (or asserts) with a fisher
net to catch bugs. As a fisher net, it is made of a mesh that won't be able to
catch small fishes. It may also have big holes in it here and there. So there
is always a limit to its efficiency.

It requires a significant effort to make unit tests more efficient. This
effort must then be balanced with the potential costs of the bugs and across
the application, because some part of the program are more critical than
others.

A good programmer should thus balance his effort in unit tests writing.

I also suggest to write unit tests as needed to be more efficient. If the
software is a data processing pipeline kind, simply write a very tight data
validity checks at the end of the processing pipe and inject test data
(including bogus one). If no problem ever show up with the test data, there is
no point in writing unit tests for all intermediate steps of the data
processing pipeline. If a problem is later discovered, then write the test,
but only at the middle of the processing pipeline. You'll know if the problem
is before or after. Then continue as needed. With this approach, you'll write
unit tests only where it is needed and won't waste you time.

Suggesting as the author that we don't need to write unit tests is like saying
we don't need to put the car safety belt because people still die in car
accident with it, and most of the time we don't have any accident even if we
don't wear it. Is this a smart advise ? I don't think so.

BTW, I use asserts wherever I can by reflex in my code because it saved me
valuable time on many occasions.

------
JoeAltmaier
How about "buckshot testing"? I wrote a file system test that made every
single API call with null arguments, good arguments, illegal arguments and
then logged the error code. We ran this against every release of the OS and
simply dif'd the result against the previous release run. It not only aided in
bugfinding, it identified necessary release notes about API differences. The
bigger point is, you have a computer at your disposal. It can make LOTS of
calls in a short time. Try calling that function with every conceivable
argument. Once you have a baseline, a diff tool can reveal everything.

------
regularfry
"Many of them will not even survive the process at all."

 _That's the point,_ and I think everything else in this article stems from
it. _Unit_ tests are supposed to be transient; you're supposed to be able to
just ditch them and replace them when you ditch and replace the code they're
testing.

 _Functional_ tests, on the other hand, should be longer-lived.

------
j_baker
Personally, if anyone asks me if they should unit test, I'll play the fanatic
and take a hard line. Mostly because the way to learn the usefulness of unit
tests is to write them. You quickly learn what they're good at and what they
are bad at.

Once you've written enough of them, you kind of seem to develop an instinct as
to what unit tests are useful for.

------
philjackson
This whole article could be boiled down to: "bad tests are useless".

------
projectileboy
I'm not sure how valuable these articles - or threads - really are for anyone.
Don't we have to balance cost vs. value for _anything_ we do? Find and replace
"unit testing" with "documentation" in this article and we could be having the
same conversation.

------
cawhitworth
SQLite would like to disagree with this rant, I think.

~~~
alextgordon
The article doesn't seem to be claiming that unit testing is bad, rather that
religious promotion of it is bad, and the benefits and drawbacks should be
weighted up like any other decision.

Obviously SQLite has got a tremendous amount of value out of unit testing. But
SQLite is quite abnormal in many respects. It's used in aviation, so the
standards of testing are ridiculously high. It's also a library, and libraries
tend to be very easy to test. It's also a long-term project with no deadline
to speak of.

~~~
humblepatience
The way I read the article it sounded like he was downplaying the usefulness
of even healthy amount of testing i.e. writing tests along with the code your
writing.

Certainly, all his arguments downplay TDD in general

------
humblepatience
I wrote a response to this on my blog

<http://www.scottschulthess.com/coding/?p=143>

Basically, there are other benefits to TDD than just preventing regression.

Documentation and ease of debugging while your writing code being the biggest
ones.

In a non-compiled language it can serve to verify that your at least runs
without gross failure a little like the compiler does.

