
It's OK Not to Write Unit Tests - bensummers
http://blogs.msdn.com/cashto/archive/2009/03/31/it-s-ok-not-to-write-unit-tests.aspx
======
chadaustin
IMVU did not consistently write unit tests on day one. So why are we all rabid
unit testing / test-driven development advocates now?

<http://www.startuplessonslearned.com/2008/11/five-whys.html>

Every time you ship a bug to customers, ask yourself "How will we prevent this
bug from ever reaching customers again?" and "Why did we write this bug in the
first place?"

Almost every time, unit or acceptance tests would have caught the bug
initially or prevented the regression. That's not to say unit tests are the
only solution.

Sometimes:

* you need better knowledge sharing between engineers

* you should spend more time refactoring your code

* your internal APIs are confusing and inherently cause bugs

Often a follow-up to a bug will involve changes to all of the above.

When you start an experimental project, don't focus on automated tests if you
don't need to... I'm all for lazily pulling in the processes you need as you
need them. :)

Whether you write unit tests or not is entirely context-sensitive. The
author's points might make sense in some context, but they certainly don't
apply to every team or even every engineer.

Maybe this guideline is helpful: "I just wrote a new component. What kind of
test coverage do I need to allow a new engineer to refactor arbitrarily and
still be convinced my code works?"

~~~
jshen
"How will we prevent this bug from ever reaching customers again?"

I think the cost of this is far higher than it's worth.

~~~
j_baker
Are you a programmer presently? That works on a codebase that's at least in
the tens of thousands of lines of code?

Tell me what software you write so that I know to never, ever use it.

~~~
jshen
lines of code is very dependent on the language. I have one rails app that's
in the 5k range, excluding tests, and it's been running for over a year with
no restarts and no need for bug fixes. I think that says something about the
effectiveness of my process.

I question the design decisions of any code base with tens of thousands lines
of code.

~~~
munctional
If your app has been running for over a year, you might want to look into this
(before someone else does)... [http://groups.google.com/group/rubyonrails-
security/msg/7f57...](http://groups.google.com/group/rubyonrails-
security/msg/7f57cd7794e1d1b4?pli=1)

~~~
jshen
thanks for the pointer. the site wasn't affected by that.

------
dschobel
_If you're a Java programmer and want to have a rude awakening, go download
Jester. Jester is an automated mutation testing tool—it goes in and replaces “
<=” with “<”, “&&” with “||”, “!” with whitespace. And then it re-runs the
tests. And then you get to watch in horror as your tests still all pass,
regardless of what the product code actually does._

I've always wanted to write a unit-test testing library which inverts all of
the assertions to catch the false positives. This sounds like it's a lot more
sadistic (and useful) though.

~~~
bbest86
Would be interesting to automatically run something like this and find a way
to assert that the tests _should_ fail. Almost an automated test of your
tests.

~~~
ZeroGravitas
Unless I'm misunderstanding something, that is what these type of tools do.

------
petercooper
I'm a relative newcomer to unit testing and man.. has it saved me a lot of
time already.

Forgetting the gigantic advantages it gives you over mere _bugs_ , I'm just as
impressed at how it helps you resolve higher level issues.

For example, recently I moved a project I was working on months ago from one
machine to another. Tests had lots of failures, despite no changes. Turns out
I had an unpatched library installed, a few libraries of the wrong version, a
slightly different database version, stuff like that. My tests picked all that
up and I could vendorize stuff or tighten up my dependencies. Without the
tests, I'd have been scratching my head for ages figuring out "what" wasn't
working.

~~~
forensic
>Without the tests, I'd have been scratching my head for ages figuring out
"what" wasn't working.

The real question though is how all that time you spent writing the tests
compares to the time you'd have spent troubleshooting.

~~~
jon_dahl
Have you done much unit testing? Done right, writing unit tests often doesn't
take any extra time at all.

~~~
munctional
I disagree. We spend a considerable amount of time writing our behavioural
tests (we have around 1,700 for our large web app), but they have saved us
countless times from having to track down bugs after deploying code.

There was an article published in the last few weeks which showed that teams
who write tests spend 10-15% longer in the development process, but they also
save 25-30% of the development process time by not having to fix regression
defects. Unfortunately, I don't remember which site it was linked from or what
the article's title was. If I can find it, I'll link it here.

------
antonovka
I'm not sure what to take away from this post.

Unit tests just saved my bacon when I accidentally introduced a serious bug in
refactoring some network-related code.

The bug wouldn't have shown up outside of production in some very specific
cases, but it would have had a serious customer impact.

Unit tests catch these things for me all the time. Every 5 minutes I spend
writing tests easily saves me 30+ minutes I'd spend debugging when some code 6
levels down from the apparent point of failure caused the issue.

~~~
jshen
I wonder why the apparent point of failure is so far removed from the actual
point of failure. With my work that is rarely the case, and I wonder which of
us is the anomaly.

Don't get me wrong, I write tests, but not nearly as many as a lot of people
propose. I do a lot of social sites, so the context may be different, but it's
not uncommon for a release to introduce a bug immediately comes to my
attention when a user hits it. I can usually deploy a fix very fast, and I
feel this is a more effective approach than maintaining a test set approaching
comprehensive coverage. People usually fail to mention the time it takes to
maintain such a test set over time, which is higher than the time it takes to
write the initial test.

Note, I always thoroughly test high impact pieces of code. i.e. account
creation, anything involving money or personal information, and data
validation code. Having said that, the bulk of the code is controller and view
code which isn't earth shattering if it breaks for 10 minutes.

~~~
chadaustin
I've noticed that code connectedness has a huge impact on the kind of test
coverage needed. For example, if your website back-end is fairly stable and
the API rarely changes, then you don't typically need 100% code coverage on
each page. As you mentioned, if you fix a bug, it won't randomly break
later...

On the other hand, with a bunch of connected objects and events that fire,
causing other events... it's very possible for a change in one system to
affect others. In this situation, I recommend a ton of unit tests _and_
acceptance tests. In addition, decouple the components in the system and
stabilize your base APIs. This will allow you to write a new feature without
affecting stability elsewhere.

~~~
DrJokepu
I also noticed that (at least in the code I write) bugs are much more
prevalent in "boring code" than in "smart code"; that is, if I got some
elegant but rather advanced piece of code I had put a lot of thought into
(like some smart math stuff or something similar), it almost always works
right away without any bugs whatsoever, however, boring and tedious-to-write
code (such as wiring up GUI, integration glue code etc) tends to have a lot
more bugs in it.

Unfortunately, this first type of code is a lot easier to test automatically
than the second type. Smart code usually has clean and simple interfaces,
while writing tests for "boring code" is also boring and tedious, like writing
a test for each textbox on a form etc.

~~~
chadaustin
Maybe there's a way to automate the boring code? Write a system that lets you
build UI without manually specifying the behavior of every control, etc.

------
jon_dahl
Summary: another post saying that "unit tests aren't a panacea".

Which is true, of course. But that doesn't mean that unit testing isn't
terribly useful.

~~~
joe_the_user
Uh,

He actually makes some subsidiary points that are worthwhile and don't just
fit mold. * A lot of tests people write aren't really unit tests * Unit tests
aren't the armor that protects you from error during refactoring. * Unit test
don't necessarily improve design, indeed they can have a bad effect on design.

So, there's more to the article than the summary - read the article, I think
it has merit.

------
j_baker
Being a huge testing advocatem I agree with this post ...for the most part
(and it's easy enough to disagree with if you didn't read it all the way).
There are just certain parts of code that are just more trouble than they're
worth to test.

However, I'm surprised at just how small an amount of code that really is. In
fact, it's _really_ difficult to say what those portions of code are unless
you stick to TDD.

Another thing that I've noticed is that not only do functional languages lead
to a more testable design, but testable code tends to be more functional in
nature. So perhaps tests are design tools.

Plus, while unit tests really only catch a small number of bugs (I think
studies have shown something like 25%), they're the fastest way of catching
and preventing the bugs they prevent. Words can't describe how frustrated I
get when I deploy to the test server and get an error because I mistyped a
variable name in some obscure part of my code.

Some times I think programming in a dynamic language without unit tests is
like screwing without a condom. Sure it feels good for a while, but eventually
it will come back to bite you (no pun intended).

------
danek
re: pains of unit tests when refactoring --

i.e. my experience with poorly written unit tests

i took over a project once from a crazy person that had a ton of unit tests in
it. about 50% of the tests were dumb, so i deleted those (seriously, things
like instantiating an object and then testing that it wasn't null).

I tried to maintain the rest of the tests as I worked with the code, but what
I found was that there were so many tests, and most of them were testing the
most pointless trivial things, that keeping them around was taking more time
than it was worth, so I axed those too. the remaining tests that actually
tested stuff, were testing parts of the design that were done so ridiculously
that they _needed_ unit tests. So I just got rid of all the tests and
refactored the hell out of the code. I ran it through the debugger a few times
and all was good. the end.

oh, then a few months later the company changed its business model and they
didn't need that subsystem anymore. go figure.

------
Freebytes
The book that changed my life:

[http://www.amazon.com/Clean-Code-Handbook-Software-
Craftsman...](http://www.amazon.com/Clean-Code-Handbook-Software-
Craftsmanship/dp/0132350882)

"Clean Code" by Robert Martin is one of the best programming books ever
written, and it barely has any code in it. However, it is relevant for every
programming language. He treats unit tests like mandatory rituals, though, and
unfortunately, I have not progressed quite that far yet. Still, it is a must
read and gives good insights into this topic.

~~~
plinkplonk
"one of the best programming books ever written, and it barely has any code in
it."

Sounds very fishy, but from Robert Martin, who thinks people who don't do TDD
(not unit testing, TDD) are "unprofessional" and "stone age programmers", I
wouldn't expect any less.

"He treats unit tests like mandatory rituals,"

cargo cult alert!

fwiw _my_ conclusion based on _my_ experience, unit tests are (by and large)a
good idea. Test _Driven_ Development is dubious, especially the "TDD is a
_design_ method" idea. Conflating the two notions isn't very useful but
happens surprisingly often.

~~~
bphogan
Even if you don't get into the TDD stuff he evangelizes, it's a pretty darn
good read. It has some really thought-provoking things to think about in terms
of how to write your code. I wouldn't dismiss it offhandedly.

------
JoeAltmaier
My mantra: "if you haven't tried it, it doesn't work" Unit test, dry run, big
bang, but in the end no significant piece of code works the first time. Ok,
once or twice in my life it has, and I still talk about that experience. But
that just shows how necessary testing is most of the time.

------
wisty
Unit tests have another advantage - they keep you coding and motivated without
an upfront schedule. Sure, you should be motivated already. Why was it you
were reading hacker news then?

~~~
mfukar
I got tired of that argument before I even read it. If you're coding non-stop,
without distractions, for the whole of your work day, teach us how to do it.

------
MartinCron
Wow, people are still writing more and more words on unit testing with
absolutely no new ideas.

Nothing to see here. Move along.

~~~
rbanffy
Quite the contrary. This attitude in a MSDN blog, from a Microsoft employee
explain a lot of problems their users have.

Way to go, Microsoft. Testing is for sissies! Real Men write correct code.

Now I can use Exchange and know why it sucks so bad.

~~~
j_baker
"Testing is for sissies! Real Men write correct code."

I seriously laughed when I read this. I'm willing to sacrifice the extra
couple of inches of programmer penis size by writing tests.

~~~
forensic
But the tests themselves _are_ code. Do you have tests that check the tests?
And then tests that check the tests that check the tests?

At one point or another you just have to learn to live with bugs.

~~~
rbanffy
And even if all code is provably correct, there is always the chance of a
random particle hitting a transistor in the chip and clearing a bit that
should be 1.

~~~
forensic
Just like in math, even if the code is provably correct, there is always the
possibility someone screwed up the proof.

In the end it always comes down to writing the least buggy software possible
and then bug zapping.

Having debug tools is smart of course, but TDD goes a bit further than debug
tools. When I write unit tests I don't think of them as unit tests I think of
them as debugging probes.

~~~
rbanffy
Have you noticed that there is an uncanny number of downvotes in this whole
topic?

