
Unit Testing in C++ and Objective-C just got easier - barredo
http://www.levelofindirection.com/journal/2010/12/28/unit-testing-in-c-and-objective-c-just-got-easier.html
======
dirtyaura
Google's gtest (and gmock) is a wonderful piece of C++ code. It needs very
small amount boilerplate, less than most junit based frameworks (which is
amazing, given this C++ that we are talking about) and great mocking abilities
(for C++). I don't see how this CATCH can match it anytime soon. Headers-only
approach is interesting but can be bad for compilation performance, which is
very very important when projects grow in size.

~~~
ot
The headers are included only in test code which is usually in separate
compilation units, so the compile times shouldn't be a problem

~~~
phil_nash
Arguably compilation time of your test code is even more important! You don't
want to be kept waiting for feedback in a TDD cycle. I agree that compilation
time can be a limiting factor of the header-only approach (actually link time
may be the killer). But it's generally easier to give a header-only library
support for libs than the other way around. I plan to keep header-only as the
default way of using CATCH.

------
timrobinson
What impresses me is this:

 _The assertion macro is REQUIRE( expression ) [...] Don't worry - lhs and rhs
are captured anyway - more on this later._

Phil wrote a bit about this technique -- capturing the two sides of an
assertion -- here: [http://www.levelofindirection.com/journal/2010/5/21/the-
ulti...](http://www.levelofindirection.com/journal/2010/5/21/the-ultimate-c-
unit-test-framework.html)

~~~
ot
Very interesting trick. Basically when you write something like

    
    
      CHECK(a == b);
    

the expression-capturing part of the code is something like (these are not the
actual names)

    
    
      #define CAPTURE(expr) SomeStrangeObject() ->* expr
    

which for a == b expands to something like

    
    
      SomeStrangeObject() ->* a == b
    

Since ->* has highest priority and is seldom overloaded, this is actually
parsed by the compiler as

    
    
      (SomeStrangeObject() ->* a) == b
    

The subexpression in parentheses returns an object which wraps a and overloads
the comparison operators (==, >, < ...), so that it can capture also the
comparison operator used and b.

While very convenient it is kind of easy to "break". For example

    
    
        int one = 2;
        CHECK( one == 1 );
        CHECK( (one == 1) );
    

the two checks are seemingly the same, but in the second case the trick
doesn't work:

    
    
      my_test.cpp(6): one == 1 failed for: 2 == 1
      my_test.cpp(7): (one == 1) failed for: 0
    

Also, expressions not in the '<operand> <comparison operator> <operand>' form
are not supported, for example

    
    
        CHECK ( one + 1 == 2 );
    

causes this obscure error message

    
    
      my_test.cpp:10: error: no match for ‘operator+’ in ‘Catch::ResultBuilder(((const char*)"one + 1 == 2"), 0, ((const std::string&)(& std::basic_string<char, std::char_traits<char>, std::allocator<char> >(((const char*)"my_test.cpp"), ((const std::allocator<char>&)((const std::allocator<char>*)(& std::allocator<char>())))))), 10u, ((const std::string&)(& std::basic_string<char, std::char_traits<char>, std::allocator<char> >(((const char*)"CHECK"), ((const std::allocator<char>&)((const std::allocator<char>*)(& std::allocator<char>()))))))).Catch::ResultBuilder::operator->* [with T = int](((const int&)((const int*)(& one)))) + 1’
    

Same for

    
    
        CHECK( 1 == 1 && one == 1);
    

which fails with

    
    
      my_test.cpp:9: error: no match for ‘operator&&’ in ‘((Catch::ResultBuilder*)Catch::ResultBuilder(((const char*)"1 == 1 && one == 1"), 0, ((const std::string&)(& std::basic_string<char, std::char_traits<char>, std::allocator<char> >(((const char*)"my_test.cpp"), ((const std::allocator<char>&)((const std::allocator<char>*)(& std::allocator<char>())))))), 9u, ((const std::string&)(& std::basic_string<char, std::char_traits<char>, std::allocator<char> >(((const char*)"CHECK"), ((const std::allocator<char>&)((const std::allocator<char>*)(& std::allocator<char>()))))))).Catch::ResultBuilder::operator->* [with T = int](((const int&)((const int*)(&1)))))->Catch::ResultBuilder::operator== [with RhsT = int](((const int&)((const int*)(&1)))) && (one == 1)’
      my_test.cpp:9: note: candidates are: operator&&(bool, bool) <built-in>
    

Not sure if these issues can cause real problems in everyday use, but they may
cause headaches if one doesn't want to dig into preprocessor/template
metaprogramming code.

~~~
phil_nash
You make a good point and I have put in some workarounds for this already.
I'll probably put more in over time. The trade-off is that I didn't want to
over-use expression templates so I have kept it to one level. But my aim is to
at least allow arbitrary expressions to compile and correctly report failures.
It should also detect when the expression cannot be decomposed and report this
in the output so you can rewrite the expression if you do hit it. I have done
this for some operators already.

~~~
ot
I want to say mine are nitpickings anyway, the library is very nice and I
share your dislike for complex unit test libraries. I had to use the Boost
Testing framework recently and _that_ was a pain.

Don't know if it is relevant, but I think you can also work around the
parentheses problem using some preprocessor magic. Boost Preprocessor does
something similar in its handling of lists, but I haven't been using it for a
very long time and I don't remember how it did the trick.

------
jmillikin
I really like INFO(); that's a nice feature I've not seen anywhere else.

What are the differences between a C++ and Objective-C test framework? Is
there anything ObjC-specific about CATCH?

Some of the other design choices here I've used, and later abandoned, in my
own testing libraries:

> _The assertion macro is REQUIRE( expression ), rather than the, now
> traditional, REQUIRE_EQUALS( lhs, rhs ), or similar. Don't worry - lhs and
> rhs are captured anyway - more on this later._

Using C++ operators for assertions is very limiting; some interesting
assertions, like floating-point near-equality, have no operator and thus must
be tested separately. I like to use separate functions to build "assertion
results", so you can get code equivalent to:

    
    
      REQUIRE(equal(lhs, rhs));
      REQUIRE(equal_within(lhs, rhs, 0.001));
      REQUIRE(not_null(lhs));
    

and so on. With this style, you also get multi-variable assertions and value
capture for free.

> _We didn't name the function. We named the test case. This frees us from
> couching our names in legal C++ identifiers. We also provide a longer form
> description that serves as an active comment_

It can be nice to have a description of the test case, but free-form names
like you have are a major pain to write out on the command line if you're
running a particular set of tests during development. If you want a
description, allow it in addition to the test name.

~~~
phil_nash
Floating point comparisons are achieved through the Approx helper class. E.g:

    
    
        REQUIRE( pi == Approx( 3.14159265358 ) );
    

(and yes, I know pi to a lot more digits than that!)

The style you suggest is almost supported out of the box, and I've been
considering fully supporting it - especially for container oriented tests.

As to the test naming - the idea is that you provide a short, hierarchical
name, and a longer description (so the description _is_ in addition to the
test name):

TEST_CASE( "example/test1", "Such and such should do so and so" )

To run a test you need only specify the name (example/test1) - or a part of
the name (e.g. example/*) (allowing tests to be easily grouped into suites
with no extra work).

------
gregschlom
Also interesting to read, a previous blog entry from May 21, where he explains
the birth of this test framework:
[http://www.levelofindirection.com/journal/2010/5/21/the-
ulti...](http://www.levelofindirection.com/journal/2010/5/21/the-ultimate-c-
unit-test-framework.html)

------
socketpuppet
> Most require you to build libraries. This can be a turn off to anyone who
> wants to get up and running quickly - especially if you just want to try
> something out. This is especially true of exploratory TDD coding.

This is absolutely true, and especially painful/noticeable if you target
multiple platforms. To work around this problem, I've started using the stand-
alone version of the Boost Unit Test framework:

    
    
        #include <boost/test/included/unit_test.hpp>
    

The problem with including that, like many other Boost header-only libraries,
is that they wreck havoc on compile time :/.

------
kingkilr
Reminds me of py.test [http://pytest.org/assert.html#assert-with-the-assert-
stateme...](http://pytest.org/assert.html#assert-with-the-assert-statement),
which is a rather good thing :)

------
wccrawford
Interesting, but I want a demo of how mocks and fakes are handled.

~~~
phil_nash
Mocks and fakes are not directly supported, however if you are rolling your
own CATCH allows test code to be called back by your application code.
Although I haven't tried it yet there should be no problem using CATCH with a
third-party mocking framework.

