
Unit Testing: Is There Really Any Debate Any Longer?  - codereview12414
http://www.drdobbs.com/testing/unit-testing-is-there-really-any-debate/240007176
======
lukev
At work, we're definitely turning away from traditional unit tests, which is
_not_ to say we're turning away from testing.

Instead, for complex or algorithmic code we're using generative testing,
barraging the code with random inputs instead of a few cherry-picked test
cases.

Then, we're structuring our systems and operations in such a way that we can
spin up test instances of the entire system and run full, end-to-end
integration tests at various levels of load.

We haven't figured it all out yet, but the results are promising compared to
unit tests, especially tests which contain a lot of mocks and stubs.

~~~
DanWaterworth
This is the approach that I take with my personal projects; I firmly believe
quickcheck style testing is a huge improvement over manually writing out
individual test cases. Do you tend to find, as I do, that when you write you
code to be testable using randomized testing that it tends to look more
functional? (I'm referring to the paradigm rather than the opposite of
dysfunctional)

~~~
kernel_sanders
I find that the same quickcheck style testing is the fastest way to get out
quality code for a project that I am the sole devleoper of.

I usually code in a terminal running screen with many shells going, one of
which is running ipython so I can interactively import and test snippets,
classes, or small functions that I'm writing in a larger module. It works
great.

However - those tests disappear! Once a recent project of mine was handed off
to a team for maintenance and more features, I found that, in particular
places, where the code was subtle, it was subject to breaking by others who
were just trying to fix something unrelated. This was when I, personally,
realized the value of unit testing.

Yes, its slower and takes more time up front, but if your code may someday be
worked on by others, it protects the sections of your code that have test
coverage against breaking - which over a longer time scale can save time.

~~~
lukev
I believe the parent post was referring to QuickCheck
(<http://en.wikipedia.org/wiki/QuickCheck>) as opposed to "quickly checking"
code in the REPL.

Tests definitely need to be repeatable.

------
programminggeek
One thing to consider is that most code that gets written is not built to be
highly testable. This makes testing harder to the point that "it's not worth
it". If you optimize your coding style for testability, writing tests is
faster and easier.

The problem is, writing testable code requires more thought up front not just
about individual objects and methods, but also the overall architecture of
your project. If your code is highly tied to an ORM or framework, then it's
going to suck to test.

Also, not every language is equally easy to test. For example, Ruby has an
obsessive culture around testing, so they have a lot of fantastic testing
tools and the language itself makes things like mocking pretty easy. PHP by
comparison doesn't value testing as much as Ruby does. The testing tools are
worse and there are fewer of them. Also, due to the lack of monkey patching,
you have to write your PHP code with dependency injection and testability in
mind from day one for it to be highly testable.

In short, testing is a useful thing, but most code isn't written to support it
and many languages and frameworks and patterns encourage you to write
unpleasant to test code. If code is hard to test, programmers won't write
tests unless their job requires it.

~~~
cheez
In short: by keeping testing in mind, you make your code better.

~~~
kleiba
...for testing.

~~~
bryanlarsen
In my experience, that clause is not required. Code that's been designed for
easy testing is better code, period. Not just because it's been tested. Code
designed for testing tends to have cleaner interfaces and is better
modularized.

~~~
romaniv
Unit testing can easily lead to _arbitrary_ modularity or code fracturing,
which makes code _harder_ to use and reason about. Good code should be
designed for solving problems, easy use and readability. If your code is
designed primarily for passing code coverage checks, then it probably is not
very good.

~~~
cheez
It does require a degree of latitude and good judgement. This only comes from
experience.

------
arctangent
I see unit testing (and especially TDD) as a barrier to agile development.

If I'm cranking out a prototype to show a customer then I don't want to spend
precious time writing tests for code that stands a good chance of being thrown
away at the start of the next iteration.

Tests often do need to be written, but the best time to do this seems like an
unsolved problem to me.

~~~
bryanlarsen
In my experience, a prototype is never just a prototype. You crank out a
prototype in an hour to demonstrate something. Then somebody says "add this
and then we'll show it to the boss.". Your boss says "that's cool, add this
and this and this and then we'll demo it to a potential customer". You do so,
then play test the demo a few times, uncovering scenarios that need hardening.
You demo to the customer, and they ask a few questions meaning that you add a
few more features to the prototype for the next demo. Rinse and repeat.

After a few successful demos you're given an tight deadline to "finish
things", so you have no choice but to polish up the prototype and ship it.

The moral: code that was written to be thrown away often isn't. You can still
write code quickly, and take shortcuts (such as skipping tests), but recognize
that you'll probably be working on that code for a very long time and may
regret your decisions later.

~~~
arctangent
This is what I was alluding to when I mentioned that it's not obvious when
tests should be written. It simply isn't practical to write them as part of a
quick prototype (or any other rapid iteration). But writing them at the end of
a project may also be difficult due to the need to move on to something else.
The same argument holds for things like documentation...

------
jerf
I'm sitting here significantly refactoring a HTML-generating class hierarchy
that I made a couple of foundational mistakes on a couple of years ago,
causing excessively slow rendering. It has about 100 classes, and is used in
over 1000 screens. Most of them trivial, but the non-trivial ones are
definitely non-trivial. Fortunately, it happens that there is a very clean
input and output that is, by design and by network structure, isolated, making
it feasible to test. I've got decent automated test coverage on the existing
set, and I'm in the process of adding an automated test that will verify that
both the old and the new system emit byte-for-byte identical output. Without
this test, this would be a hopeless exercise and I'd have to just live with
the outcome forever.

I find myself wondering how many people complaining about unit tests work on
leaf-node sorts of programming projects, because for me, many of the tasks I
routinely take on in my environment where about a dozen projects are building
on the code I am writing mean that the question isn't whether unit tests are
"better" or "worse". It's a question of whether the task is _possible_ or not.

------
k2xl
Unit testing is a very very expensive operation. Consider the following: The
developers add unit testing at beginning for specific results. Later on, after
product is released a new design or feature change happens. Often times unit
tests will have to be rewritten else they will fail.

This can be very expensive and take developers and QA engineers extra time
when they try and figure out what's wrong... "Is the test failing because the
test is out of date? Or is our program causing a bug?" I know for a fact HP
deals with this issue constantly. Not saying unit testing is bad (I think it's
usually good) but I don't think it's always good.

~~~
antrix
> "Is the test failing because the test is out of date? Or is our program
> causing a bug?"

IMO, when you change code (extensively), you'd still have to invariably answer
those questions. But in the absence of tests, it becomes much harder to find
the answers.

~~~
justincormack
Indeed. The tests tell you that you are breaking a contract that you once
thought was important. It might not be any more, but good tests reflect good
contracts. Removing the test requires you to think whether any other code
expects this contract.

------
cbs
That debate exists because the people selling the idea of unit-tests were so
goddamn gung-ho for unit testing everywhere and all the time.

I have friends in the medical industry, their tests _must_ achieve 100% code
coverage on multiple coverage metrics, and code coverage is only an
afterthought to even more rigorous test requirements. Even they were pointing
and laughing at the TDD ideologues.

Unit testing is beneficial, if you do the right amount of it and in the right
places at the right time. You might have a totally sensible test plan. But
like it or not, for the time being, plenty of people are unwilling to listen
to you because some other assholes already burned them on the idea of unit
testing.

Its going to take years and years to shake loose all the cobwebs in our
industry caused by half-cocked advocacy. Then we can maybe get to the sensible
discussion about how to analyze a project and develop a sensible test strategy
that fits the project's needs.

------
TianCaiBenBen
1\. Unit Testing is useful ONLY for complex API method/function, which should
not change too often and too frequently;

2\. Ideally, you should only test API not API's implementation but most of the
mock framework will tie your test code directly against your API
implementation. Once your API implementation changed, your test code fails,
you have to rewrite test code, which is waste of time.

Someone may argue that it because you write bad test code; Talking is easier
than doing;

So, usually, I prefer to doing the functional integration test than the unit
test, which is productive for me. (I know people will throw out thousands of
reasons about the benefit of unit test and "correct" way of writing unit
test.)

Anyway, I will avoid unit test as possible as I can and ONLY do my designed
integration test, which works for me.

3\. And writing unit test for code coverage is stupid; Some manager even brag
about the unit test coverage, which is totally a false confidence about the
quality of the code underline; This is even stupid;

4\. Writing unit test for those requirement-always-change area is waste of
time;

5\. TDD is a BIGGEST JOKE, which weights too much on writing auto testable
code over writing working functional productive code;

------
columbo
Purposeless unit tests are a waste of time. These are tests against your
compiler or external frameworks. For example:

    
    
        @Test
        testAddUser() {assertNotNull(new User("columbo").save())}  
      

Wahoo! the ORM works!

The biggest problem I have with unit tests is they tend to be

a) just testing frameworks (crud) and not much else

b) so heavily built on mocking that changing them is like redesigning a wool
sweater. These tests tend to examine how the code was written, not necessarily
what the expected output is (oh shoot, I need to mock up five more objects to
make this test pass... hey look at that, this test doesn't even check what
should be returned. Nice)

c) there is no context into or out of the unit test. The assertions are vague,
it isn't clear how I got there, it isn't clear where I'm going
(testXMLOutWithInProgressUsers ... what is the xml for, who are the in
progress users... what is next in this chain of events)

d) the things that REALLY need unit tests don't have them because they are too
complex to write (ah, that whole system that runs a nightly bash script, takes
in a stack of csv files from the client and converts it in our database...
yeah, that's not tested).

With all that said, tests are invaluable, but they need to actually, you know,
TEST something.

I like nodejs in that respect. It takes me seconds to fire up the webserver
and test the rest api. I don't mock objects, I pass values directly through
the api and check the expected results. I like this because it shows the next
user what the api actually looks like and I'm testing the full stack (In my
experience controllers are notoriously undertested, mainly because you need
the full stack running). Since the application is really just an api this
means that I'm able to test everything equally, from batch processes to CRUD.

~~~
JoeAltmaier
Agreed.

API testing has that wonderful property, its appearance is the whole deal, if
it looks right then its working.

Anything but app code isn't so easily tested. When frameworks are written to
support testing, they often are sub-useful - crippled versions of real
programming languages etc.

Anecdote: I recall a QA guy asking me how to test our OS's file system. I
said, make every filespec-based call with a good spec, a non-existant spec, an
illegal spec, a null spec; log every result. After each release, diff the logs
and explain every deviation.

He said "That's too hard, I'll just shotgun it." and left. Why ? Because his
test-scripting framework wouldn't allow anything as sophisticated as lambdas
etc.

So I spent half an hour writing a table and a loop, and sent it to him. He
used it for the next 5 years to test every release.

My moral: do an exhaustive test when possible. The computer is fast, it
doesn't feel any pain, and you will find more fencepost and boundary issues
than you ever thought existed.

------
damoncali
A rare Anti-Betteridge.
<http://en.wikipedia.org/wiki/Betteridges_Law_of_Headlines>

------
breckenedge
In practice, I find I write code faster and with more confidence after the
tests (usually specs) have been written. So, my productivity increases. I also
sleep better at night.

~~~
mhd
For me TDD is doing the exact opposite. It bores me to tears, which _really_
brings down my edge.

Granted, this is one of the reasons why I'm recently thinking about whether
I've got a slight case of ADHD, but I think a lot of it just stems from the
fact that I like to brainstorm code and _afterwards_ use something like tests
or just good documentation to reason whether that was the most proper
approach.

Never mind that things like cucumber make me cringe big time.

~~~
batista
> _For me TDD is doing the exact opposite. It bores me to tears, which really
> brings down my edge._

That.

It takes all the fun out of using a dynamic language. You might as well go
back to a static typing and write LESS tests for obvious things...

~~~
demallien
Yes, this is an obvious downside to the type of TDD that you sometimes see
advocated, where they write tests for every little thing.

I try to find a compromise solution. I pretty much only write one unit test
for a given module to start with, the nominal case. This is important, because
it forces you to think a little bit about how you actually want your code to
work, which I find to be a rather healthy thing to do.

Once that is done, a little later I will no doubt find a bug. When I'm trying
to debug, and I start wondering whether a block of code is doing what I think
it is doing, I write a test to verify. This does one of two things - it shows
up the bug, which I can then correct, _and verify that I have corrected_ , or
it allows me to remove the tested code as a possible cause for the bug,
allowing me to concentrate on other hypotheses for the cause of the bug,
speeding up the debug process.

By the time I get to the end of the project, I generally have a pretty
comprehensive test suite, without ever once feeling like it was a chore.

------
nicholassmith
Unit testing, like many other good and healthy coding practises is great. In
theory.

In the big mean real world frequently you're shipping things in such narrow
deadlines that if you say "Hey boss, I'm going to take some time to write some
wicked cool unit tests, it'll make my life easier in future because we'll know
this code works!" then Boss' head will pop as it's a non-tangible improvement.

Developer focused companies are great as they'll let you get away with this,
but the large portion aren't.

------
jhrobert
Unit Test, sure, but at what "unit level"?

I usually prefer "automatized system testing" where fewer "high level" tests
involve a lot of code, preferably in a test environment that is close the
actual production environment.

Of course, when a test fails, some more analysis is required to determine what
went wrong. But it's usually related to recently modified code, hence easy to
spot.

That way I get the benefits of automatized tests without the burden of tons of
"micro" tests at the class/method level.

------
languagehacker
There's no debate about testing stateless functions. I think the debate is in
whether the ledger reflects a cost benefit of spending all the time mocking up
different functionalities in order to test stateful components or components
with external dependencies. Since these are hard to calculate ahead of time,
it really does become a question of engineering culture and time to market.

------
ehosca
why do i get the feeling that you probably haven't spent much time "debugging"
unit-tests...

unit testing give you the false confidence that your stuff is working, it has
little to no bearing on stuff actually working when pushed to production
(unless you have 100% coverage and actually thought of and defended againt
every possible use scenario before hand)

~~~
dasil003
Yes, unit tests are code and all code can have bugs, but you're missing the
point. Automated tests are a reflection on the code just like code reviews or
documentation or formal specifications. If they are well written (ie.
orthogonally structured to the code they are testing and with good
readability) then they provide a strong window into the intent of the original
developer, _and this intent can be continually tested for almost no marginal
cost for the lifetime of the code_.

Yes there is a maintenance cost, but I find most people's objection to testing
is based on poor testing skills. Once you get good at testing with a decent
framework, you find that it's often about the same time to write formal tests
as it is to do the manual testing which we all know is necessary to have any
certainty that code is working.

------
koski
Unit tests are good. System wide functional tests are awesome.

------
kmfrk
With CI, it's become a little harder to make the excuse. :)

------
taligent
Of course there is debate.

The reason is that automated tests costs 2-3x as much time upfront. The
assumption is then you will recoup that investment in the future. The problem
is that automated testing doesn't cover everything so you have to augment it
with human testers. Which then begs the question are the human testers being
effectively utilised.

And for those of that do develop have a think about all the time spent on
buggy web UI testing, screenshot comparison testing, tests being flakey due to
software choices etc.

~~~
rwallace
Automated _unit_ tests cost two or three times as much time, sure. That
doesn't mean don't use automated tests, it means don't use unit tests. Use
integration tests instead. Not only do they provide much better coverage
(you're testing all the code from top to bottom, each module in the context in
which it is actually used), but they are also far cheaper.

------
markmm
Yes come on, does anyone still think they are useful and simply not a complete
waste of developer time?

~~~
lucian1900
I can't decide if you're serious and misguided or just joking.

~~~
batista
You know, actual luminaries in our field are also against them. I've seen UNIX
programming gods among men dismiss them. It's not like it's a clear cut issue.

For some things like libraries and such, there are absolutely needed, but unit
testing everything and especially TDD are stupid.

To be frank, they mostly sound like some Java EE-like monstrosity that for
some bizarro reason caught on with the hip crowd too. I'd put them out there
with FactoryOfFactory and XML documentation...

~~~
viraptor
When I think about UNIX programming gods, I think about people who deal with
tools that are never modified, deal with very small problems. They're trivial
to test functionally, but are next to impossible to test internally (have you
ever seen source for tools like tar?)

It's definitely some solution for testing. But when you're writing a
webservice, you should think twice before assuming that anything a "UNIX
programming god" says is a good advice for you.

------
bluedanieru
Ah, the old 'Argument by Dismissal'. Still just as bold, and still just as
unconvincing.

