

Today I Got Burnt by Isolated Tests - abyx
http://www.codelord.net/2011/08/07/today-i-got-burnt-by-isolated-tests/

======
jameskilton
"Generally, I prefer the GOOS school of TDD which includes isolating my
classes as much as possible, _putting mocks and stubs everywhere_." [emphasis
added]

One word: Overmocking. Mocks should be the last thing you add to any test
suite and they should _only_ be used to mock out external services or slow
systems. If you find yourself needing a lot of mocks for simple tests, then
your design is bad and needs to be refactored.

Ryan Davis explains things best when he gets into minitest/mock (for Ruby):
[http://confreaks.net/videos/618-cascadiaruby2011-size-
doesn-...](http://confreaks.net/videos/618-cascadiaruby2011-size-doesn-t-
matter)

I've been bit by this before, multiple times. I've ripped hundreds of lines of
mocking code out of tests in the code I routinely work on (not originally
written by me) and it has given me much more faith in my tests.

~~~
benjiweber
> Mocks should be the last thing you add to any test suite and they should
> only be used to mock out external services or slow systems.

It really depends what you're trying to achieve from your tests. I find a
combination of all of:

1\. Mockist tests that mock /all/ dependencies

2\. Tests that stub out interactions with other layers (e.g. test the entire
model)

3\. Full integration tests that test the whole system with real dependencies
(real db, real services)

works quite well.

The first set help you design and document your implementation through mock
specifications.

The second allows you to assert the intended behaviour of the code without
tight coupling to the implementation. You're less likely to break these when
refactoring.

The third is the most likely to find real bugs in my experience. They're also
generally more difficult to write and take longer to run, but valuable.

Mock-everything tests aren't good at finding defects but they're not without
value.

~~~
prateekdayal
I have seen that over time, when you refactor code and tests that use mocks,
its a good idea to take out the mocks and replace them with actual objects.
Mocks are important early on because you are just discovering the interfaces
but over time, they can be replaced by real objects (this obviously can't be
done for third party interactions).

In the end, there is no substitute for integration tests. We make it a point
to write integration tests for atleast all happy paths.

------
ilcavero
"Isolated tests go much better hand in hand with a few integration tests" I
would say that it is the other way around and that integration tests go much
better hand in hand with a few isolated tests, though I've never been a fan of
the TDD dogma

------
wccrawford
Sounds like the problem wasn't isolation, but failure to actually test the
class. Once they had tests written for it (because they were adding a new
feature) it became glaringly obvious that it wasn't working.

------
boyter
tl;dr

Make sure you use integration tests (or a smoke test) as mocking away the
entire environment will hide environmental and system errors when you actually
run things.

~~~
Yhippa
Also if you have time try to use as much real-world data or a real-world
scenario as possible. I've gotten burned when I've done unit and integration
tests and still that one guy slips through that we never would have thought of
but of course shows up come production time.

~~~
boyter
Agreed. I tend to do the following to test any application that requires a
high level of quality.

1\. Unit tests - for regression testing and to enforce a clean design 2\.
Integration testing against real data - To check that there are no obvious
bugs caused by bad interfaces or data 3\. Smoke tests - To ensure that the
core functionality works as expected

They also get updated in terms of frequency in that order. Smoke tests take
the most effort to make, but they deliver the most value so long as they have
supporting unit and integration tests which catch obvious errors.

------
sleight42
Hi again, Aviv!

This is exactly why I tend to be more of a "statist"l preferring black box
tests. Yes, my unit tests are arguably a form of integration tests themselves.
However, I can usually rest assured that my tests prove the behavior that they
exercise.

The downside is that my tests run more slowly than isolated units under test.

