
Test driven development at Transloadit - felixge
http://debuggable.com/posts/test-driven-development-at-transloadit:4cc892a7-b9fc-4d65-bb0c-1b27cbdd56cb
======
jsdalton
Do you find it hard to change directions with a feature or component, because
your tests have sort of "locked you in" to the structure you were working on?

How religiously do you follow TDD? Do you literally not write a single line of
code unless it's in furtherance of passing a failing unit test, or do you just
worry about the system test you put in place at the start and unit test
aspects as you need to?

Also, you mentioned changes to the API of third-party libraries. How do go
about catching how those changes impact your code? (I would assume you are
mocking these in your unit tests.)

Wow I just asked you a lot of questions.

Seriously though, thanks for posting this article. It's so rare to be able to
read an article about unit testing that isn't just some guy demonstrating how
he'd use TDD to determine if a number is in the Fibonacci sequence or
something.

~~~
felixge
> Do you find it hard to change directions with a feature or component,
> because your tests have sort of "locked you in" to the structure you were
> working on?

Do you mean changing the direction of the implementation, or the outcome of
the feature? If it's just the implementation we usually throw away the unit
tests along with the code we don't like and restart. The system test is
ultimately responsible for making sure the feature itself is working. The unit
tests just allow you to test different parameters without having to write a
system test for each of those.

I have never felt "locked in" so far. Generally I feel very good about the
stability of the Software and my ability to change it as needed.

> How religiously do you follow TDD? Do you literally not write a single line
> of code unless it's in furtherance of passing a failing unit test, or do you
> just worry about the system test you put in place at the start and unit test
> aspects as you need to

Very religiously with almost no exceptions. When I make an exception it is
usually a sign of problems in the code, and I'll revisit it later on / improve
my tools.

I think it is difficult to do anything but full TDD, it's just to easy to drop
the ball once you make compromises.

> Also, you mentioned changes to the API of third-party libraries. How do go
> about catching how those changes impact your code? (I would assume you are
> mocking these in your unit tests.)

System tests are the only line of defense here. That's why we make sure that
all mission critical code paths are covered by them. There are no 100%
guarantees, but 95% certainty beats 20% certainty after a big library upgrade
by a lot : ).

> Seriously though, thanks for posting this article. It's so rare to be able
> to read an article about unit testing that isn't just some guy demonstrating
> how he'd use TDD to determine if a number is in the Fibonacci sequence or
> something.

I never saw the benefits from these articles either. It took the node.js API
to break our app in about 100 different parts for me to see the value in TDD.
I guess it's just like backups ... ; )

~~~
jsdalton
Thanks for your excellent replies. I realize that part of the difficulties I
experience with using TDD in real life is that testing feels "expensive" to
me, e.g. I frequently spend more time futzing around with test code than I do
production code.

Your responses illustrate a different perspective, that testing is cheap (this
probably comes with experience, I would imagine). Anyhow, I can see that if I
thought of tests as cheap, I wouldn't get so hung up on my tests breaking when
I change direction on a feature.

I also write a fair amount of code that consumes web APIs, so I have lost a
lot of hair figuring out to test against those properly without mocking out
everything to the point of meaninglessness.

~~~
steveklabnik
> testing feels "expensive" to me

This is actually one of the bigger 'problems' with {T,B}DD; it's much harder
to figure out how much time it takes to complete things.

BDD: Wrote the test, wrote the feature, checked it in, wall clock time, 1
hour. Next week, something unrelated changes, my tests show right where the
problem is, 10 minute fix. Total time: 1:10.

No testing: wrote the feature, half an hour. Next week, something unrelated
changes, breaks the feature, spend an hour fixing it. Total time: 1:30. And
keep adding time when other things break later, too...

In projects with a high test coverage, I almost never spend any time
debugging. That doesn't mean it's easy to recognize the saved time, though.

~~~
jsdalton
I think the difficulties I've personally had with embracing TDD come down to a
slightly different set of experiences:

TDD: Spend an hour writing the test (sometimes more if I have to figure out
how to properly work with a third-party library). Spend 10 minutes writing the
code. Decide a few days later that I need to rework the API a bit on the
feature, so spend another hour rewriting tests and 5 minutes writing new
production code.

No testing: Write the feature, 20 minutes. "Test" while writing it (refresh a
web page or run something at the CL). Realize a few days later I need to
rework things, spend 10 minutes doing so. In the rare instances where
something breaks, spend 20 or 30 minutes tracking it down and fixing.

Don't read this as an argument against TDD. It's more that I have a very hard
time actually realizing an increase in productivity or code quality when using
it. TDD-backed code generally takes me longer to write and pretty much nobody
on the business team I report to notices any difference whatsoever, except
that feature X took a week longer than it probably would have otherwise. The
few bugs that do creep into production code are usually dealt with promptly,
and I'm not sure TDD would result in entirely bug-free code either way.

Anyhow, that's why this conversation has been quite valuable to me. I am
convinced that if I can clear away real-world obstacles in TDD I can do a
better job of embracing it and realizing the productivity benefits that
everyone always crows about. For now, my personal experience is that it's a
means to marginally improve code quality at a high cost of time.

~~~
Lewisham
I think you're probably not taking into account the amortized cost of TDD
versus what you're doing now: how much testing is really happening when you
"refresh a web page"? How many times do you do that? How confident are you
when you make a change to your code base that you don't have to go back and
manually perform all those tests you were already doing? All that refreshing
takes a lot of time!

I write unit tests so I don't have to refresh web pages all day. If I do find
myself going to the command-line to test something, I figure out what I'm
trying to test, and write a unit test instead; it's something I have missed
when I was writing the upfront tests. Then I can do it again, and again, and
again, and use continuous integration so it runs those tests for me over and
over.

If you do a lot of web dev, Selenium is a really good way of testing web page
features.

~~~
loewenskind
>All that refreshing takes a lot of time!

I have the same problem as the parent and this is exactly why. I've always
thoroughly tested everything but with ad hoc tests that I had to do over and
over.

The main thing I miss with TDD is the interactive nature. If I'm testing in
Smalltalk I can write code right inside the debugger and watch the effect it
has, but TDD always moves me back to what Smalltalkers call "cult of the dead"
programming where I have to stop, run the tests and wait for the output. I
wish there was a way to make it more interactive. It would be easier to force
myself to do it then.

~~~
steveklabnik
There's an awesome Ruby library called 'autospec' that watches for files being
saved, and then automatically runs your test suite in the background, and
gives you a growl notification if they've failed or passed...

------
felixge
A few facts that I didn't get to fit into the article:

\- We have ~1.6x as much test code as we have code being tested.

\- Our unit test suite takes < 5 seconds to run. Our system test suite takes <
60 seconds.

\- We use Hudson for continuous integration

\- The hardest part with TDD for us was reaching 1.0 as you always feel like
it would be "faster" to stop testing.

\- We usually don't go around refactoring stuff just because we can. In fact
we usually feel more confident building upon existing stuff since we have
tests that are saying it works.

And it goes without saying, feel free to ask me anything : ).

~~~
singular
Thanks for this, very interesting.

You didn't mention mocking in the article - to what degree do you use mocking,
and where does it fit in the testing pipeline?

~~~
felixge
I have written a library called gently. This library lets you define a series
of expected function calls spawning multiple objects (since overall order is
important to us).

Whenever you define such an expected call, gently returns a closure that you
can use to inject this expectation in the right place.

From my understanding this is hybrid between Mocking (where you have an object
with an expected call / state sequence), and Stubbing (where you have various
pre-recorded answers to function calls).

But generally I found the semantics of the various TDD methodologies very
difficult to translate into actual code. That's why I'm using as little
abstraction as possible.

~~~
onewland
Is gently open source or could it be?

~~~
felixge
Yes, <http://github.com/felixge/node-gently> .

However, it's still rather minimal and I'll add more stuff to it now that I
actually know what I need to do with it frequently.

------
ericb
Test driven development, to me, starts as the harness you use to write your
code in the first place. I'm not sure how people write solid code with any
kind of speed without a harness testing/driving it. Why not make your harness
your tests?

Everyone keeps talking about how it is more work, which confuses me. I don't
make a religion out of test coverage, so maybe that is the difference? As in,
I don't write a test for every basic thing, like an accessor, for example,
because I feel like there's a point of diminishing returns.

edit: To those downvoting, I'm interested in a dialogue and hearing your
thoughts.

~~~
felixge
The costs for TDD greatly vary. Trying to do TDD for GUI / frontend
programming is _much_ more expensive. I also find TDD very hard when working
within a framework that was not written using TDD.

So of course, if your environment makes TDD as cheap as ours does, I'm with
you and think you have to be a little crazy to not at least try it. But it's
not for everybody / everything by default.

About skipping tests for accessors and stuff: That's a slippery slope, I'm
afraid it would lead me to skip more "obvious" tests which in turn make it
harder to write the right test when this stuff actually is part of a bug in
the future.

~~~
ericb
Agreed, my comment above was not GUI/frontend-centric and that is a different
case.

To me there is another slippery slope for TDD in getting bogged down. I'm
usually interested in TDD in as much as it can be a competitive advantage. The
point where I feel like the returns drop off and I'm burning my man-hours and
paying too much for the TDD insurance on very predictable parts is where my
tests get sparser.

Keep in mind, the degree of TDD insurance I'd pay for is different for every
project--for a Mars Rover you bet I'll test my accessors.

~~~
felixge
> Keep in mind, the degree of TDD insurance I'd pay for is different for every
> project--for a Mars Rover you bet I'll test my accessors.

Sure. If you write Mars Rover software you'll also want somebody to review
your tests to make sure you didn't get the spec wrong. You can go infinitely
deep here, and I'm not telling you which tradeoffs make sense for your app. I
just know which ones I can live with for ours : ).

------
ollysb
One of the most pleasurable benefits of TDD is that I can refactor code and
not worry that I'm going to break something. On larger projects this is really
liberating, I can develop with the same freedom that I did on day 1. It means
that a code base can evolve smoothly and naturally, assimilating new
requirements. Without that safety net you end up with code that gradually
becomes a poorer fit for the requirements, changes becoming huge snaps as they
are delayed over the fear of regressions.

~~~
ddlatham
Is that specific to TDD or just having a comprehensive suite of tests?

~~~
ollysb
There's 2 aspects to it. First, the coverage which gives you the safety net.
Second, specifying behaviour before implementation which allows you to
maintain tight alignment between your requirements and code as they evolve.
According to the BDD guys it's the second aspect that is more important.

~~~
ollysb
I'm not sure why my comment has been downvoted, it answers the question and to
my knowledge is accurate. Perhaps I can expand on something for you?

------
DanielBMarkham
I'm extremely happy to upvote first-hand TDD articles whenever I see them on
here.

There have been quite a few nice stories, both pro and con. I'd like to see
more folks talk about competitive advantage. After all, code just doesn't
exist by itself -- it's supposed to _do something_. Does TDD help make
technology make people's lives better? Or is it just a big PITA that ends up
with more maintainable code? There's nothing wrong with that, but it's the
kind of perspective question can be a critical success factor for startups.

~~~
felixge
Regarding competitive advantages: We are working on a new feature that will
allow us to encode video uploads while they are still uploading. This change
will affect the most critical aspects of our code base and it's very easy to
introduce subtle bugs. TDD is helping us to make fundamental changes like this
without being too afraid of breaking things. I don't think we would undertake
this feature if we didn't have the tests we do. (And TDD is the only way to
get tests you really trust)

~~~
ericwaller
_(And TDD is the only way to get tests you really trust)_

Can you expand on this point? This is one aspect of TDD, which seems to focus
most intensely on unit testing, I've never understood. Why wouldn't full-
stack, acceptance testing be just as effective?

~~~
t_crayford
Full-stack tests will have a combinatorial explosion in the number of tests
you have to write to get decent coverage, wheras isolated unit tests have a
linear (with some constants) increase.

Also full stack acceptance tests take ages to run (by definition they will be
doing some complex computation). As mentioned above, if their test suite runs
in under 5s, it gives a huge advantage. Imagine having light on your computer
that goes red any time you make a mistake. They effectively have that.

------
ericHosick
I think it would also behoove you to start using BDD in your development
process. With BDD, you get a one-to-many correlation between behavior and
code. This also greatly improves on quality. I say this because quality can
only be based against requirements. BDD forces the developer to have the
behavioral requirements upfront.

BDD allows you to take a given behavioral requirement and see all the code
that supports that requirement and visa-versa. If a customer ever asks you,
you can point to exactly where you are fulfilling that behavioral requirement
in the source code.

And BDD languages such as Gherkin are easy to read for most non-technical
people.

Of course, this isn't a replacement for TDD.

~~~
swombat
I take the definition (proposed, usually, by BDD people like David Chelimsky
and the like) that BDD is not fundamentally different from TDD. BDD is just
TDD done right.

As such, it makes little sense to propose using BDD "instead of" TDD. If
you're doing TDD right, there's no benefit to "switching to BDD", other than a
monstrous task of rewriting your tests using another set of sub-frameworks...

~~~
ericHosick
I think there is quite a big difference, fundamentally, between BDD and TDD
(though they are not mutually exclusive and do overlap).

Let's try a quick example of a google maps type app: locating where you are
and where you want to be.

From BDD perspective, you get a mockup of the screen (could even be some non
visual process) and this represents the behavior you want of the product: the
behavioral requirements. From this mockup we can pull out all the behavior on
that screen (or within that system).

Feature: In order to see if I am where I want to be As a traveler I need to
know my location.

Scenario: At My Location Given I want to be at Lon XXXX Lat YYYY And I am at
Log XXXX Lat YYYY Then I should see "You are There"

Scenario: Not there Yet Given I want to be at Lon XXXX Lat YYYY And I am at
Log XXXX Lat YYYY2 Then I should see "You have 4 meters to go".

From this behavior we can then build out the system which inevitably leads to
fine grained software specific behavior which should be tested using TDD such
as:

it should "calculate the distance between two points correct"

In the case of BDD, we don't worry about how the two points are calculated nor
if it is even done correctly (we don't need full fringe test coverage in BDD).
We are able to assume that the underlying calculations will be correct. We do
need to make sure that the behavior of the system as a whole is working
correctly. In this case, we are assuming if the calculation is incorrect we
will not see "You are There" on the screen. Why was it incorrect? Doesn't
matter. That specific behavior, the calculation, was driven by Tests.

Personally, and for efficiency, it is really important to use both in a
project and not look at BDD as "TDD done right."

Just my two cents.

------
olliesaunders
John Hughes, (author of Why FP Matter) is mitigating some of the problems of
TDD with QuickCheck (<http://en.wikipedia.org/wiki/QuickCheck>).

Video: <http://www.infoq.com/presentations/The-Joy-of-Testing>

I've been using it for the last couple of days. You'll have to excuse not
being able to see the demo properly. InfoQ fail. Stick with it, there's some
really cool stuff at the end.

Another video: <http://video.google.com/videoplay?docid=4655369445141008672#>

------
Tomek_
Until recently I was looking at TDD as a necessary evil - yes, it saves your
ass a lot of times, but writing tests doesn't feel fun nor very productive - I
don't know if I can speak for others, but for myself there are two things that
I like most in programming, 1st: creating cool, new features; 2nd:
polishing/refactoring the code until it becomes a state of art.

And what I've found out not a long time ago is that TDD helps _a lot_ with the
2nd point, but only (again, at least for me) if you aim for the 100% code
coverage. Yes, I know it sounds silly, and pointless, and that code coverage
doesn't mean that much, but anything lower than those 100% makes you too
relaxed about quality of the code and leads to skipping the parts that are
harder to test, and most often they are hard to test because they are badly
written (bad architecture, bad API decisions, etc.).

When you go for a 100% CC you have to examine all those parts, and change the
code so it actually can become testable, and it's often hard and requires
making big changes in many places. It's challenging, but challenging means
fun. And you learn a lot, but learning a lot is fun too. So, double win
(triple, if you count easier bugspotting)!

------
omouse
_Writing those tests feels like proofing mathematical theorems_

Uh, not really. Have you ever written tests for functions in math? It's a pain
in the ass once you get past 5 variables. This is why we have proofs in the
first place, to prove that something is true for each and every case without
having to run through every test...

------
johnwatson11218
I really like the author's assertion that writing the test feels like proving
mathematics. I have always been fascinated by this connection. When you look
at the history of mathematics w/r/t the discovery of calculus there was a 150+
year period in which people could use it to do work but nobody could prove
calculus. It wasn't until the early 1800s and the technique involved bounding
errors. I'm referring to the delta-epsilon proofs from analysis. To me this is
_exactly_ like what is happening in computer programming right now. In my
opinion automated testing is some of the most cutting edge stuff going on
right now.

------
steilpass
BTW Felix was interviewed about Node.js at
<http://chaosradio.ccc.de/cre167.html> (German)

------
Ixiaus
I'm building a startup with Python and Pylons - Pylons comes with paster
(python web server) and a small unit testing hook for your app; pylons makes
it so _easy_ to test your application, I love it. You can write two types of
tests: functional tests and unit tests.

I split functional tests into the tests that actually make a web request (by
spawning paster, loading the application, and pretending to be a user) - my
lowest "unit" there are my controllers. I have to have some special trickery
to handle sessions, but it isn't bad, beats having no tests.

My "unit" tests are a bit less fine-grained than yours are. I've had
difficulty narrowing tests down below the module level with anything but
libraries; primarily because models (the culprits here) have a few
interdependencies. I do my best to keep my models orthogonal but not all of
them are, so most of my unit tests are basically testing models as they
operate on the database tables and some of their assistant methods.

If I write a library, I generally try to determine whether it can be its own
"module" or should be a module within the application namespace. For example,
I wrote an authorize.net payment library and ultimately decided that it should
have its own tests - separate from the application, which lead me to splitting
it off as its own project (PayPy now, with adapter support for some other
gateways). Writing some of the library modules in their own namespace with
their own tests keeps it clean orthogonal.

Overall, I would say this: I'm not as OCD about writing tests as you are
because I'm the sole developer and there are too many features to build to be
writing functional tests every time I build a controller. My philosophy is
this, though: the models should _all_ be covered, no matter what; since they
are the messiest bit of logic (usually) in a web application - translation
between _relational_ data into _object oriented_ data has a lot of side
effects. Having all models covered ensures that the majority of my logic that
interacts with the database is solid. It also keeps any schema/model changes
in check that may affect other models I've forgotten about.

After that, I try to make sure any _critical_ controllers/pages/functional
pieces are covered - the really trivial stuff I worry less about and just make
sure it "works right" by doing some manual testing myself before pushing it.

I love functional/unit testing - I will never go back. My old way of
development feels like I was lost in the dark ages of my career or
something...

------
dalore
Is it really a unit test if you make a call to a REST service? Is that not now
an integration test?

~~~
felixge
Yes, that is an integration test. You need both.

------
sreuter
nice one!

