

Ask HN: What you think about differential unit testing? - yehosef

Our website currently has little&#x2F;no unit tests (we have selenium testing and manual testing).  We would like to start testing more, but we&#x27;re running into a quandary as how to start.  Much of our code is not written in a way that lends itself to unit testing so to be able to start testing, we have to refactor, which might break everything.<p>One approach that&#x27;s come up is called &quot;differential unit testing&quot; where I don&#x27;t need to go in a write the tests - but I measure the inputs and outputs of different functions and I monitor if they changed after my code changes (at least that&#x27;s how I understand it.)  There&#x27;s a paper about it at http:&#x2F;&#x2F;web.engr.illinois.edu&#x2F;~taoxie&#x2F;publications&#x2F;ast07-diffut.pdf.  I first encountered a version of the idea at http:&#x2F;&#x2F;zachholman.com&#x2F;talk&#x2F;move-fast-break-nothing&#x2F; (their case they are running both in parallel so it&#x27;s easier to check each case - but the concept would be the same if I had common input cases and output cases.)<p>Now it&#x27;s question how I implement such an approach. One way might be put in certain waypoints in my code where I can run a stack trace and save it - then use those inputs to different functions as my test cases.  I save a &quot;baseline&quot; and then as I change my code, I run the tests, looking for changes.<p>I can see several issues with the approach.  It might test too many versions - I don&#x27;t know which are important changes to test and which are insignificant (without manual intervention.)  I might actually have a broken implementation to start  - and I&#x27;m comparing agains that. maybe others..<p>I&#x27;m not try to suggest this approach would replace more traditional unit testing - but as a form of &quot;canary&quot; to detect when something is not how it use to be at a more granular and earlier level than doing full integration testing.<p>I&#x27;m curious if anyone else has tried this approach and what your opinions are?
======
speedkills
GitHub has done this, when rewriting their permissions code. They gave a talk
about it at rubyconf 2014 and you can learn more about their experience and
wrapped up the ruby code they used in a gem available at
[https://github.com/github/scientist](https://github.com/github/scientist)

You might find it useful to read their source, the same techniques could
probably be easily adapted for your usage.

Unfortunately I am heading off to work and don't have time to elaborate
further but to very briefly answer your question, yes I have found
differential testing to be hugely beneficial. I don't use it to replace unit
tests but lean on it heavily when doing a rip and replace of a sub-system such
as when I am rewriting a ruby service in scala.

~~~
yehosef
great! thanks for the info and link. If you have a chance, I'd love to hear
more about your experiences and challenges with the approach. thanks!

------
yehosef
links:
[http://web.engr.illinois.edu/~taoxie/publications/ast07-diff...](http://web.engr.illinois.edu/~taoxie/publications/ast07-diffut.pdf)

[http://zachholman.com/talk/move-fast-break-
nothing/](http://zachholman.com/talk/move-fast-break-nothing/)

