

Unit testing is not (generally) useful - axod
http://prehacked.com/60388.html

======
allenbrunson
The author rejects unit tests because they don't help find bugs. I don't think
his reasoning adds up to much, more like rationalizations for already-held
beliefs.

But if you want to take the argument seriously, I would say that "finding
bugs" is not the primary reason I write unit tests anyway. It's to prevent
regressions, and make me feel safe that I haven't broken anything after a
refactoring. If not for my unit tests, I'd be scared to change much of
anything, for fear of stirring up a whole bunch of trouble that I'll have to
fix later on.

~~~
abstractbill
"preventing regressions" is a subset of "finding bugs", and I've found the
other methods I listed to be more effective (from an ROI perspective) at doing
that too.

~~~
allenbrunson
Apparently you work at Justin.TV. I don't know how big your individual
projects are, but personally, I didn't get unit-testing religion until I
started working on _really big_ stuff, close to the limits of what I'm capable
of. I worked for a year on a 60,000 line program, and it started getting
painful. I worked for another year on a 100,000 line program, and that's when
I finally had to relent and start writing unit tests.

Here's a common scenario that made me want to write tests. I want to implement
feature A, but to make it work, I'm first going to have to write library B and
C. So I write library B, but there is no way to test it in the program yet,
because feature A doesn't exist. Right after I've finished the code, I have a
pretty good idea of where things might go wrong, and what corner cases I want
to exercise. But by the time I finish library C and then get around to
implementing feature A, I've forgotten that stuff. Pain points go unaddressed.

In summary, unit tests really start to show their value when you're working on
a program that's too big to fit into your head all at once.

~~~
abstractbill
Couldn't you have tried to break that 100,000 line program up into ten 10,000
line programs? You might have found the additional benefit that each of the
ten programs could be useful on its own. That's how I would usually attack
this kind of problem, rather than throwing unit tests at it.

~~~
mrtron
I think it would be preferable to do both. Have a few really solid and well
contained modules that are well tested.

Testing is only a piece.

------
bjclark
Judging by his analogy and depth of insight, I'm guessing he's never even
tried to write unit tests on a daily basis and certainly never done TDD. It's
the same old argument that everyone makes when they've never actually learned
to do something properly.

~~~
abstractbill
Actually I wrote unit tests daily at my last company, for about two years, and
in the last six months of that I gave TDD a try.

My claim isn't really that they don't work, it's that (compared to the other
things I listed) they have a _tiny_ ROI.

~~~
bjclark
Unit tests have nothing to do with, and aren't in any way mutually exclusive
to load testing and logging and user reports. Frankly, it just sounds naive to
say that you replaced unit testing with logging, or load testing, or letting
your users find the bugs for you.

People that do unit tests do all that stuff too!

Your lack of specs also has nothing to do with unit testing. It might dictate
the difficultly in writing acceptance tests, but that's a completely different
subject.

I don't feel the need to proselytize unit tests since you've already written
them off, but what's the point to posting about why you don't like something
only to back it up with some sort of rambling, incoherent analogy that _you_
probably don't even believe?

~~~
abstractbill
I certainly do believe the analogy.

The point was that the ROI on unit tests is almost always very much lower than
the ROI of the others. So when time gets tight (as it always is in a startup),
you should generally put effort into all the other things and _not_ into unit
tests.

------
cytzol
I'm currently writing a personal project for university, and I've made sure to
test it all the way through.

When I was part-way into coding this thing, I took a day to go through all the
code and write tests for anything I could. I found a few bugs, so I could at
least say that it helped somehow. But the tests have helped a lot more when I
added to it - it feels -good- to see a big green bar light up whenever I add
something. It's -nice- to run all tests, leave it a minute, and confirm that
it's not going to blow up before I have to demonstrate to anybody. Of course,
finding bugs is nice, and I set up a better testing rig to make sure it
doesn't fail on anything I haven't thought of, but the benefits of unit
testing go further than bug detection.

When you're all crestfallen with no desire to code, finding bugs is only
secondary.

~~~
axod
It's a false sense of security though. It's like putting a big badge on your
website saying it 'validates' - even if it's totally broken in real world
browsers.

In my experience, the more common bugs are integration bugs, scaling bugs,
weird memory usage issues, not to mention interfacing to external systems -
for example some weird browser comes along and does idiotic requests you'd
never expect a sane person to do.

Those things are very hard to test for, and don't really produce a good ROI as
abstractbill says... Best just to log things, graph things where possible,
spot weirdness, investigate and fix.

~~~
biohacker42
Everything you mentioned I count as part of TDD or BDD or what ever automated
testing acronym you like.

What I suspect is that all of us have basically the same idea in mind, and
what we're disagreeing over is mostly semantics and context.

Btw, all of my unit tests include memory usage validation. It's easy in c and
c++, but I'm not sure how it would be done in a web development environment.

------
Hates_
To me Unit Testing is more about testing the expected outcome and making sure
of the status quo rather then finding bugs and testing all the extreme small
edge cases. Then when an abnormal and unexpected situation occurs, writing a
test that induces it and fixing the code so that all the related tests pass.

To use the doctor analogy, I would say unit testing is more about testing
patient outcomes when they are put through varying courses of drugs or
treatments (fixing bugs and adding new features) rather then trying to make
them die before declaring them healthy. Is the patients blood pressure normal?
Check. Now the patient wants to start some treatment for something else. Oh
wait, I'm getting an abnormal blood pressure reading.

------
jcromartie
I didn't appreciate unit testing until I started doing development test-first
in Smalltalk. You just sketch out the skeleton of your classes with instance
variables and accessors. Then you write tests the way you want your objects to
work (design your API by pretending to use it). Then run your tests, and
marvel as the debugger stops on missing methods and lets you create them at
that point and continue the test. It's a terrific way to get your code right
the first time.

I don't know what he's using there at Justin.tv, but most languages that are
more rigid (than Smalltalk or Ruby for example) are certainly not quite as
nice for unit testing.

~~~
abstractbill
Sounds great! But in the meantime, I've put a couple of prototypes online, got
a bunch of feedback, iterated, and figured out that _nobody wanted the thing
we designed in the first place_. I've moved on to an evolution of the original
idea, while you're still focused on getting the first idea to work
"perfectly".

My point is that unit tests aren't cost-free, and their ROI is awful for most
things.

~~~
swombat
Unit tests don't take that long. If it takes you that much longer to write
your unit tests, you must be doing something fundamentally wrong - like
perhaps testing the wrong thing or focusing on the wrong kind of testing.

~~~
Daniel_Newby
It is hard to make inline unit tests when the inputs and outputs are really
complicated, like the request/reply data structures in a web framework.

~~~
swombat
Totally agree, that's why you need good MVC separation and you test those
layers separately.

The models testing is obvious. What you're referring to there is controller
testing. In Rails/Rspec, the way you do that is by mocking/stubbing the models
(so you're only testing the controller logic). Then you pass in the parameters
that would come in from the web, and check that the controller code (which
should be fairly straightforward) behaves appropriately.

This article explains this approach quite clearly:
[http://www.patmaddox.com/blog/2007/9/15/easy-controller-
test...](http://www.patmaddox.com/blog/2007/9/15/easy-controller-tests-and-
expressing-intent-through-expectations)

------
lnguyen
Whenever someone comments on the usefulness of unit testing (or in the
extreme, testing in general), I point them to Feynman's contribution to the
Challenger shuttle report:

Appendix F - Personal Observations on Reliability of Shuttle
<http://www.ralentz.com/old/space/feynman-report.html>

A good summary is at [http://duartes.org/gustavo/blog/post/richard-feynman-
challen...](http://duartes.org/gustavo/blog/post/richard-feynman-challenger-
disaster-software-engineering)

~~~
axod
How many people are writing mission critical software? In startups, hardly
any.

~~~
lnguyen
If you rely on your software to be up and running for your livelihood (aka any
web startup), it is mission critical.

~~~
axod
I disagree completely. A failure might mean you loose a bit of ad revenue, or
maybe a customer.

The consequences of a bug are usually very small. It's also very obvious if a
bug is causing you pain.

------
thenduks
For me the primary use of unit testing is in the prototype stage. I write the
tests instead of the application code to make sure everything is sane. I find
lots of architectural/assumption problems this way.

After you've got real app code the tests serve more or less as regression
tests to make sure I haven't broken anything major deep in the guts of my
models and such.

In short, like some of the other comments here: Unit tests aren't for finding
bugs, they're for helping you not write the bugs in the first place.

------
pmarsh
It all depends on how your development works.

Let's look at a scenario where you are using an outside framework.

Not writing tests for your code? Framework v2.0 comes out and you want to use
a great new feature in that version. Will your app work if you upgrade? Sure
you'll have to test/QA your app regardless, but with tests you might have a
better idea of where to look for problems.

This is what I'm facing at the moment as a single developer. The app has no
test coverage so updating is a laborious process of testing everything by
hand.

But getting into testing has it's own problems. The biggest I have with
testing is when I read discussions about bugs within the test framework. How
much time will you spend working around bugs in the testing framework?

------
vicaya
The article has a rather narrow perspective on unit testing, given his product
is not mission critical. He's basically treating live production as a giant
integration test.

Some of my perspectives:

class/function level unit testing is only needed for common library code.

component level unit testing is critical if you want to do any significant
refactoring.

non trivial integration tests (especially timing sensitive multithreaded code)
can find many subtle bugs that would be too late to find in production.

I've written a few tests (for a database engine) that paid themselves many
times over.

~~~
abstractbill
_The article has a rather narrow perspective on unit testing, given his
product is not mission critical._

I do have a habit of assuming every other hacker is doing some kind of a
startup, so this is a fair point.

------
akeefer
I'd say that it's also important once the team gets beyond a certain size;
codebase and team size tend to scale together, of course, but not always. The
value of tests in catching regressions in code you didn't write, didn't
realize was there, and don't understand is immense. It's one kind of difficult
to not introduce regressions in 100k lines of code that you were deeply
involved with; it's a whole different deal to walk into 100k lines of someone
else's code and try to change it without breaking anything.

Unit testing is like vaccination: it's good for _me_ that I'm vaccinated
against smallpox, but it's also really good for _you_ that I won't be getting
sick and giving it to you, and your ability to avoid vaccination (or testing)
and still function is often directly correlated to how well everyone else is
vaccinated (or how well they've tested their code so you know when you're
breaking it).

Those don't always have to be _unit_ tests in the strictest sense, so you
could perhaps argue that a test that breaks as a result of unrelated changes
isn't really a "unit" test because it wasn't properly isolated, but I really
hate those sorts of language arguments. I'll call something a "unit" test if
it's not an "end-to-end" test, even if I didn't mock or stub out every
possible dependency. And it certainly sounds like the author of this post is
arguing against unit tests in that sense rather than in the stricter sense.

------
dfragnito
It's a money make. We needed some python work done. It was a small project 42
hours 6 hours was unit testing. Unit testing increased the invoice by almost
17%. I can see why consultants like it.

~~~
swombat
I know many people who do unit testing (myself included), some of them
consultants. Not a single one of them does it "to buff up the invoice". In
fact, every single one of them will argue (sincerely) that good unit testing
practices help _reduce_ the time they need to deliver code and increase the
quality of that code. They'd also argue that good unit testing practices make
it easier to then hand over the code to someone else.

------
nradov
This is another case of overgeneralizing based on limited experience.
Automated unit testing may well be of limited value for building the
applications that run a video sharing web site (where failures are no big
deal). But now try building a complex library used as a core component of
several packaged applications. The library developer doesn't really have a
"program" to put through user testing. The only solution to keeping the
library stable while refactoring and adding new features is to build a
comprehensive suite of unit tests that exercise the library's public API.

------
thewileyone
If your unit testing isn't useful, then find a way to make it useful. Author
misses the point completely.

Not unit testing is like building a car door and not check that it fits until
all parts are to be assembled, and then reviewing what's wrong. Dumb.

------
biohacker42
When (or should I say if) JustinTV grows much bigger and hires a lot more
programmers this whole unit tests are not useful thing will come back to bite
them.

But I can see how right now it seems their time is better spent on other
things.

~~~
abstractbill
Out of interest, are you speaking from personal experience?

~~~
biohacker42
Yep, I got TDD religion when I started working for a huge company.

I did some testing, but not much when I worked for a startup.

~~~
axod
For a huge company, I'd say TDD/pair programming etc is also to ensure _some_
productivity from bad programmers.

If one bad programmer writes the unit tests, and another bad programmer writes
the code, then you might end up with a mediocre outcome, instead of a bad
outcome. Same with pair programming.

Likewise, if you get a _good_ programmer to write the unit tests, then give
the programming work to a terrible programmer, given enough time, he'll write
a working solution.

~~~
biohacker42
I'd have to say that two bad programmers are not any better then one bad
programmer even with TDD.

But TDD benefits even the best programmers when they are dealing with a huge,
ugly, old code base.

When on the other hand you're user testing every day or multiple times a day,
like you might with a web startup, then TDD may only slow you down.

~~~
abstractbill
_When on the other hand you're user testing every day or multiple times a day,
like you might with a web startup, then TDD may only slow you down._

These days I'm at a point where I can usually push 4 or 5 new releases into
production every day. It's pretty awesome :-)

------
Allocator2008
Really it's like that bridge in 'Apocalypse Now'. The sergeant is telling
Martin Sheen how the bridge is always bombed and they have to come in and re-
build it, "just so the generals can say the bridge is open." Well, I have done
a lot of unit testing (CPPUnit), and it is like that. We do unit testing so
management can say the product has been unit tested. Coals to Newcastle, to be
sure, but hey, it's a job!

