
Using `make` and `git diff` for a simple and powerful test harness - chrismorgan
https://chrismorgan.info/blog/make-and-git-diff-test-harness/
======
drothlis
This technique is also known as characterisation testing, golden-master
testing, snapshot testing, and probably other names too. I recommend looking
into [https://approvaltests.com/](https://approvaltests.com/) (already
mentioned by another commenter).

I use characterisation testing all the time, in a perhaps unusual application:
Checking the behaviour of "Page Objects" (classes used to model the
application-under-test in GUI-level automated testing) when the application-
under-test has changed. That's right: Automated [characterisation] tests for
my automated [GUI] tests! It makes GUI testing oh so much easier to maintain.
I wrote it up here: [https://david.rothlis.net/pageobjects/characterisation-
tests](https://david.rothlis.net/pageobjects/characterisation-tests)

~~~
drothlis
Another thought related to characterisation testing, this time from Jeremias
Roessler's talk at QAFest Ukraine 2017
([https://www.youtube.com/watch?v=f4PT_u8hjhU](https://www.youtube.com/watch?v=f4PT_u8hjhU)):
Traditional automated tests (with asserts) are "blacklisting" the changes that
aren't allowed, whereas a characterisation test will catch any change in
behaviour, and instead you have to "whitelist" the changes that _are_ allowed
(by providing regexes or some kind of pattern to match the dynamic output that
is allowed to change).

------
jgrahamc
Fun. The rwildcard function here

    
    
        rwildcard = $(foreach d,$(wildcard $1*),$(call rwildcard,$d/,$2) $(filter $2,$d))
    

is very similar to one I wrote in 2011 ([https://blog.jgc.org/2011/07/gnu-
make-recursive-wildcard-fun...](https://blog.jgc.org/2011/07/gnu-make-
recursive-wildcard-function.html)):

    
    
        rwildcard=$(foreach d,$(wildcard $1*),$(call rwildcard,$d/,$2) $(filter $(subst *,%,$2),$d))
    

The only difference is that I wanted to be able to use * in the pattern rather
than % (hence the subst).

    
    
        $(call rwildcard,,*.c)
    

vs.

    
    
        $(call rwildcard,,%.c)

~~~
chrismorgan
I believe it is a direct descendant of yours, which I’ve used from time to
time over the years. I just didn’t feel the need to support *, % was enough
for me.

Thanks for all the stuff you’ve written about Make: I’ve enjoyed reading quite
a lot of it!

~~~
jgrahamc
Glad these things help people. The * for wildcard was a bit of syntatic sugar
to make $(call rwildcard) closer to $(wildcard).

For those who don't know, here's a list of stuff I've written over the years.

[https://blog.jgc.org/2013/02/updated-list-of-my-gnu-make-
art...](https://blog.jgc.org/2013/02/updated-list-of-my-gnu-make-
articles.html)

And the hard copy version:
[https://nostarch.com/gnumake](https://nostarch.com/gnumake)

PS I really like the layout of your blog.

~~~
peterwwillis
I appreciate these posts for "I just need to do X..." google-search-for-my-fix
problems. But I've always found it frustrating that this is what the whole www
is now: a whole lot of spread out individual docs for individual problems,
rather than a concise review of useful information (a "just the useful bits"
database if you will).

Try to learn anything in depth these days and either you're deciphering a
dense yet incomplete manual, or googling until eternity. Make is a great
example, as even though they have a useful manual, good luck figuring out how
to apply it to your problem. Wikis don't quite cut it either. We need a
different data model for knowledge sharing.

~~~
jgrahamc
OK, but I wrote those things as "advanced" uses of make because the actual GNU
Make Manual is very, very good
([https://www.gnu.org/software/make/manual/](https://www.gnu.org/software/make/manual/))
and people should read it.

I keep wondering about doing a video series explaining GNU Make from the
ground up.

------
couchand
I went to an awful lot of trouble to apply roughly the same technique to test
some code generation code [0]. Required pulling in a random dependency and
hacking around for a while to make the output look right. Your solution is
much slicker!

[0]:
[https://github.com/couchand/wayfinder/blob/1dc58a1130bc17941...](https://github.com/couchand/wayfinder/blob/1dc58a1130bc179413018e3a9c374da52b4e9881/wayfinder-
tests/build.rs)

~~~
chrismorgan
What you ended up with has two major advantages: ① it integrates with the
Cargo test harness, so that Rust developers’ expectations will be met; and ②
it does it in _one compiler invocation_. The test harness I describe would be
quite slow for your sort of situation.

I’m not completely sold on using a separate crate and build.rs how you have,
but it looks like it’ll yield a good usable result. A couple of related things
you might be interested in: compiletest_rs, trybuild.

------
d99kris
A bit off-topic, but I like the tty-player you're using for the demo (and
apparently have created). [https://tty-player.chrismorgan.info/](https://tty-
player.chrismorgan.info/)

I haven't come across this kind of terminal playback on the web before, seems
a bit more bandwidth efficient and snappier than video or gifs.

~~~
chrismorgan
Heads up: what’s shown and run at [https://tty-
player.chrismorgan.info](https://tty-player.chrismorgan.info) is currently out
of date (though [https://github.com/chris-morgan/tty-
player](https://github.com/chris-morgan/tty-player) is up to date). I wrote it
years back against Web Components v0 for a project that has been shelved for a
few years, and more recently updated it to Web Components v1 and Shadow DOM,
because I wanted it for this article. Still need to update it to xterm.js.

The most popular thing in this space nowadays is asciinema, mostly used by
embedding it from asciinema.org, but I prefer my thing.

------
errantspark
it's my friday, here's a haiku for you:

    
    
        this is a nice post
        i'm not sober on hn
        fixed width code font please

~~~
chrismorgan
The code font I use is Triplicate:
[https://mbtype.com/fonts/triplicate/](https://mbtype.com/fonts/triplicate/)

I use the Poly variant by default, only disabling it in places where the
monospacedness matters for _layout_ purposes, e.g. the terminal recording in
this article (because of Vim), and Rust compiler output in some of my other
articles. The Poly variant does things like make i, l and space a little
narrower, and m a little wider, which makes casual reading more comfortable
than a strict monospaced font.

I do this because I believe that monospacedness is substantially overrated in
most places, and that most things actually look better not strictly
monospaced. I contemplated not even using a monospace-style font _at all_ but
decided that was probably going too far for most people. (And as a Vim user I
necessarily work in a monospaced text editor; but if it weren’t for that, I’d
probably go full proportional.)

So I’m curious if you have further feedback on this matter and why you find
fault with what I’m doing. It may influence what I do.

~~~
saagarjha
I think one of the best reasons to use a monospaced typeface is that it is a
fairly strong and accurate signifier of code. Of course, in this case you have
special highlighting for it that makes it less useful, but in general I think
that it really helps. (Plus there are a couple of other, minor benefits
probably not worth listing here.)

~~~
throwaway_pdp09
As mentioned elsewhere I've been using proportional for code and it's very
nice but for a small drawback. I'd say give it a try for a couple of days and
see.

------
heinrichhartman
Note that GNU CoreUtils also use Make + shell-scripts + diff for their test
harness:
[https://github.com/coreutils/coreutils/tree/master/tests](https://github.com/coreutils/coreutils/tree/master/tests)

Its not as slick or concise as the solution proposed here, but it shows that
the approach is viable for medium-large projects.

------
code-faster
I've used a similar technique to test, generating an expected output, actual
output and then diff them.

One trick I found helpful was using JSON to serialize test results instead of
unstructured plain text.

Test results stored as JSON are much easier parse and therefore process. You
can quickly whip up programs that verify the tests satisfy invariants, diff
the tests and filter out expected test changes from unexpected test changes.

~~~
qznc
When I did it, we skipped the Make part and reinvented it in Python:
[https://github.com/libfirm/sisyphus](https://github.com/libfirm/sisyphus)

This aspect of expected and unexpected test changes is even more important
than the diff part in my opinion. It allows you add failing tests immediately
once you get the bug report and you notice if you fix something accidentally.

~~~
tom_mellior
The Readme of that project could do with a few examples of what tests and
successful/unsuccessful output look like. I found the examples folder and
still can't visualize what it might be like.

I've been using cram (also written in Python) for a private project and been
mostly happy with it: [https://bitheap.org/cram/](https://bitheap.org/cram/)

------
neuromanser
cram is a very good tool for testing in this manner: a test file is basically
a copy/paste of a terminal window, deviations from expected behavior are
represented using diffs, and `cram -i` will prompt you to update the test file
with actual output. and it supports globbing and regular expressions for fuzzy
matching.

i've been using cram for everything i write for what feels like a decade
(it'll be 10 years old in september), and though it has it's limits, i bitch
and moan about it very little given how much i rely on it. if you'd know me
you'd recognize that this is a huge endorsement, i'm quite vocal about my
disdain for most software in existence. :)

edit: s/i'll/it'll/ rofl

edit: url: [https://bitheap.org/cram/](https://bitheap.org/cram/)

------
twirls
Same concept: [https://approvaltests.com/](https://approvaltests.com/)

------
niknetniko
This is a really nice looking blog, I'm impressed.

------
totony
Major drawback mentioned here is that make breaks when used with filenames
with whitespaces, which is a big blocker for some uses. Anyone know of a
similar alternative which handles this?

------
urda
I’m a big fan of using make in my projects. It’s nice to be able to sit down
another dev or new user and just tell them to `make build` or `make test`. It
also makes finding bugs easier as you can bisect with it.

------
kartayyar
I would suggest stop using a homegrown solution and use something robust like
bazel.

[https://bazel.build/](https://bazel.build/)

------
geoid
You could make use the pattern stem automatic variable in lines five and six
to replace

    
    
        $(patsubst %.stdout,%.stderr,$@)
    

with

    
    
        $*.stderr

------
mkchoi212
Genius. I'm sure Bryan Kernighan would approve :)

------
rjgonza
This is really awesome, thanks for the post!

------
peter_retief
Brilliant, I will definitely use this now. Love the simplicity!

