
Clojure and Testing - fogus
http://tech.puredanger.com/2013/08/31/clojure-and-testing/
======
dljsjr
Maybe this is unique to the field that I work in, but I can pretty safely say
that if our organization didn't strive to maintain 60%-80% unit test coverage
at any given time, we'd have been sunk a long time ago.

We don't "ship" a product. We're a humanoid robotics research laboratory, we
have a fairly large and old codebase for what we do (~1MLOC of Java, plus a
whole big whack of C++ for JNI), and since we neither have a customer nor an
OSS community then we make no guarantees about our API nor do we frequently
generate stable, frozen binary releases. As such, the code is _constantly_
being refactored and evolving; especially our numerical based stuff; things
like physics, controls, etc. (we use an in-house simulation environment that
one of our PI's created while he was in grad school, so even stuff like the 3D
engine or the simulator data viz GUI's can change at any given moment).

I realize that this is entirely anecdotal and highly specific to what we do,
but without high coverage and solid CI (We use a self-hosted Bamboo instance
with nightly and every-commit builds), our code would be in complete and utter
disarray. The build routinely breaks a dozen times a day when one of the
control theory guys checks in a parameter tweak to our state estimator or
introduces a new component to one of our modular controllers (we have test
suites that actually run headless versions of our sims and mathematically
verify the walking behavior).

I guess my point is that, as is the case with _anything_ in programming,
choose the right tool for the job. Evaluate what you're doing, what your goals
are, the nature of your code-base, and decide (as is brushed over in the
source article) whether or not the opportunity cost is worth it. For us, we'd
be up shit creek if we didn't have high coverage.

------
siscia
Humm, I completely agree with Alex, however for different reasons.

If you write clojure, you use immutable structure (in most of the case, if you
don't this comment doesn't apply), and so you end up writting only functions
that take some data in, and return some data out.

If you are smart enough to work bottom up and you can create different layer
of abstraction once a layer is finished and once you are sure that it does
what it is suppose to do, then you can really stop to test that layer. (I hope
is clear what I mean...)

The only test I can see having a real sense in clojure is when you use impure
functions (functions that don't only get something in and spit something out,
but that also do some work, writting in a DB maybe...) or when you think that
your function/abstraction will be re-factor someday-somehow.

However in the case of the refactoring, the test need to be written still in a
clever way: you have such functions tunnel A->B->C->D where you know that you
will refactor C so there is no point in test A, what we really need to test is
just C and its direct "neigbourd", or even better making sure that B returns
what C needs and that C returns what D needs.

Or even better, since a clojure functions is way too long if it is more than
10 lines of code, well maybe all this testing is already too much, maybe the
pre-post clojure's condition are enough. (This if we are talking about small
functions, if A, B, C and D are abstraction layer you can't just keep all in
your mind, so tests really makes sense here, still however doesn't makes sense
to test A at all, nor the whole B, nor the whole D)

------
programminggeek
I agree that testing getters/setters and other libraries functionality is a
waste of time, but...

You know, the "pragmatic" approach that many programmers take - "writing tests
is hard and time consuming, so let's write fewer tests" is a recipe for
disaster and frankly the idea that if you just think about the problem more,
that you're make fewer mistakes is very much along those same lines.

Here is the real problem - humans are fallible and we make mistakes. This is
why pilots have flight checklists, why manufacturing has quality checks, why
engineers have building codes, and why at hospitals checklists and procedures
save lives

If pilots and mechanics just "thought more about the problem" they would still
make mistakes. You might be a brilliant surgeon, but thinking you're too smart
or too good to go through a checklist could cost someone their life because of
a small oversight.

Would you feel good going under the knife if you knew that your life was in
the hands of someone who thought they were too smart to scrub in?

We are a young profession with very few actual quality standards. Luckily,
most of the software that we build isn't critical to people's safety, but I'd
be terrified if the software powering medical robots, life support machines,
and airplane autopilot systems were built by people who were "too smart" to
write tests.

When a software bug costs people their lives because some programmer thought
writing those tests was "too expensive" or "a waste of time" how smart is that
programmer?

~~~
yummyfajitas
_I agree that testing getters /setters...is a waste of time, but..._

But it isn't necessarily. Say I have multiple implementations of a class,
e.g.:

    
    
        trait TimeSeries {
            def datapoints: Seq[DataPoint]
        }
    
        case class SequenceTimeSeries(datapoints: Seq[DataPoint]) extends TimeSeries
    
        class ArrayTimeSeries(times: Array[Long], values: Array[Double]) extends TimeSeries
    

It's pretty straightforward and nearly pointless to test that
`sequenceTimeSeries.datapoints` does what I expect. On the other hand,
applying the same test to `arrayTimeSeries.datapoints` exposed an off by one
error in my implementation.

~~~
tel
Out of curiosity—I'm not so familiar with Scala—how does an off-by-1 arise
here? Aren't you just going to zip (times, values)?

~~~
yummyfajitas
I did this a while back, so I don't recall the exact off-by-one error. But I
don't want to to times.zip(values) - that would involve constructing
explicitly a List[DataPoint] which is precisely what I'm trying to avoid with
the ArrayTimeSeries. Rather, I defined a custom Seq[DataPoint], which had the
apply(i: Int) method construct datapoints on the fly from the arrays.

(I admit that the error was entirely stupidity on my part and not the result
of anything complicated. If I was as smart as Rich Hickey, I probably wouldn't
need so many tests.)

~~~
tel
Ohh, I see, array-of-structures/structure-of-arrays trouble.

------
skatenerd
When I test at the REPL, I feel like I spend lots of time trying to re-enter
updated code. This means I do a lot of:

A: Pasting changed code into repl

B: Re-typing code

C: Hitting "previous command" and "return"

The overhead of automating this process (by making a test suite) seems high,
but I haven't ever regretted it.

~~~
emidln
You can also get rid of this overhead by using an editor with an integrated
repl (emacs+nrepl.el/slime/inferior-lisp-mode, vim+slimv/fireplace, LightTable
all do this). I can type ,e to send an updated function to my repl for
instance. ,b to evaluate an entire buffer.

That said, you can also develop your test suite like this as well (which is a
sublime experience, particularly with a generative test tool).

~~~
jamii
It helps to make a sandbox file that has everything imported already and just
reeval the whole file -
[https://github.com/jamii/strucjure/blob/compiled/test/strucj...](https://github.com/jamii/strucjure/blob/compiled/test/strucjure/regression/tests.clj)

Then you can also run the file as a free regression test -
[https://github.com/jamii/strucjure/blob/compiled/test/strucj...](https://github.com/jamii/strucjure/blob/compiled/test/strucjure/regression.clj)

------
tel
Test when you lay down abstraction. If you know your expectations about some
code, then it's worth testing in proportion to the amount of dependency you
have on those expectations holding.

If you don't yet know the expectations/abstraction you have for some code then
you're still deep in the design phase whether or not you like it. Good
tests—like good static types or property checks—can be useful to smoke out the
properties your abstraction will eventually fulfill. If they slow you down,
don't use them.

But once you know your abstraction and start depending on it then please test.
Most likely a good test will, right there, show you that you don't actually
know your abstraction very well.

~~~
puredanger
Good advice. Have you tried using contracts (pre/post conditions, assertions,
invariant checks) in your code? I find them to be a useful way to encode those
expectations in the code before you test.

~~~
tel
I'm a huge proponent powerful static checking, so I haven't written much code
with contracts. My understanding is that they're quite wonderful for dynamic
property elaboration, though, so by that fiat I'd say I'm a fan.

------
thurn
I mean, there's a spectrum here. I don't think anybody is going to argue that
that java.util.LinkedList shouldn't have tests. On the other end of the
spectrum, there are things that obviously shouldn't be tested (getters and
setters, whether some div has 10px of padding, etc). Tests are are how you
lock down your software once it's ready to be frozen. You use them to turn
software into hardware, to use an analogy from Steve Yegge [1].

[1]: [http://steve-yegge.blogspot.com/2007/01/pinocchio-
problem.ht...](http://steve-yegge.blogspot.com/2007/01/pinocchio-problem.html)

------
pbiggar
Could not disagree more. For reference, my company
([https://circleci.com](https://circleci.com)) probably has one of the largest
test suites in Clojure (approx 16kLOC in the test suite, > 5000 assertions).

The value is testing is that it saves time in the long run. If you don't have
a decent test suite, then as your product matures, you become unable to do
anything because each new feature breaks a dozen old things.

Alex is right about the opportunity cost (see my own post "Testing is
bullshit": [http://blog.circleci.com/testing-is-
bullshit/](http://blog.circleci.com/testing-is-bullshit/)). But not testing is
also an opportunity cost.

Basically you can spend T minutes writing tests, or D minutes debugging
problems that tests would have found. If you don't test at all, then D will
rapidly (within weeks possibly) become so high that you can't get anything
done. By contrast, T will rarely go above 50% of time spent on a project (and
if it is, you need to rethink how you test, unless you're building rockets or
medical equipment).

~~~
puredanger
What in particular are you disagreeing with?

You seem to be implying that I'm saying not to write tests or that testing
isn't valuable but that's not what I said. As I said: "I think tests are
highly valuable. I don’t know how I could write high quality code without
writing tests." Every Clojure dev I've ever worked with wrote tests and relied
on them. My point is that there are costs to consider.

I observe that the Clojure community is interested in trying a wide variety of
testing approaches that may mitigate some costs of testing, perhaps more so
than other communities.

I have also worked on a system with a large Clojure test suite (at least
30kLOC), primarily example-based. We found that there was substantial cost
(even with many custom-built tools) to maintaining those tests. I've had the
same experience maintaing large Java test suites. Recently, I've been doing
more with generative testing - I think generative and contracts may give you a
better return on your effort than example based testing. But that's a
hypothesis I look forward to investigating.

------
danneu
Note: I only have experience with my one-man Clojure projects.

My favorite poor-man's test-feedback-loop in Clojure is to use the built-in
core.assert function to put tests at the bottom of the source file in
combination with my editor's eval-buffer command.

Test DSLs, test directory/file boilerplate, and setting up an autorun cycle
are the three things that constitute enough of a barrier to prevent me from
testing, so those are the three things this example eludes.

Here's a file of a `to-slug` function that normalizes drug names into URL
slugs.

    
    
        ;; Production code
    
        ...
    
        (defn to-slug [s]
          (hyphenize (simplify-name s)))
    
        ;; Tests
    
        (defn assert-slug [a b] (assert (= (to-slug a) b)))
        (assert-slug "ocusert pilo-20" "ocusert-pilo")
        (assert-slug "ogen .625" "ogen")
        (assert-slug "ogen 1.25" "ogen")
        (assert-slug "hydralazine hydrochloride, hydrochlorothiazide and reserpine"
                     "hydralazine-hydrochloride")
        (assert-slug "hydralazine hydrochloride w/ hydrochlorothiazide 100/50"
                     "hydralazine-hydrochloride")
    
        ;; Sandbox
    
        (to-slug "00-d-amphetamine")
    

Every time I hit C-c C-k (eval-buffer in Emacs w/ nrepl), the asserts are run
which will throw an error on failure.

And since eval-buffer always returns the result of the last expression
evaluated, the above "Sandbox" section represents whatever case I'm currently
trying to extend `to-slug` support for. The result of which is returned on
every eval-buffer.

In this case, `to-slug` doesn't correctly handle "00-d-amphetamine" yet, so as
I modify the production code to handle it, I can mash eval-buffer after every
change to (1) see if my change introduces regressions and (2) visualize my
progress on handling "00-d-amphetamine". Once it works, I add an
"00-d-amphetamine" assert-slug and move on to the next case.

As to-slug matures, I can then move the asserts to a dedicated test directory
if I want to and replace the slime with minimal case tests. -- Things that
otherwise slow down my exploration phase.

\----

I think core.assert fills the chasm of a function's/file's lifecycle between
"I'm not even sure if I want this function or what its input/output will look
like" and "my source code definitely needs this and this is its API".

In Ruby, I like
[https://github.com/sconover/wrong](https://github.com/sconover/wrong).

~~~
ruricolist
I do this in Common Lisp. It also makes sense as a technique for
documentation: why leave examples rusting in comments when you can incorporate
them into the program?

Add a Quickcheck-ish type-based test case generator and you start to blur the
line between tests and types.

------
i_s
I have a suggestion for people who are very pro unit testing for understanding
people who do not test as much: try working in areas that are traditionally
light on unit testing. For example, graphics, multimedia, and games. People
write stuff in these areas all the time that still work, despite minimal
automated testing. There are other paths we might take, Frodo.

------
davidrupp
[https://twitter.com/davidrupp/status/375290808640884736](https://twitter.com/davidrupp/status/375290808640884736)

"Not all who wander are lost."

