On: Best Practices for Testing in Java (2019)

[slavik81]

Your implementation is necessarily complex. That's why it may have bugs, and why it needs tests.

You have many more tests than implementations. In my experience, ~20x more. If your tests had bugs at the same rate as your implementation, you'd spend 95% of your time fixing test bugs and 5% fixing implementation bugs. That's why tests should be simple.

If you're going to be spending that much time on validating assumptions, I think you're better off trying to express them formally.

[asgard1024 → slavik81]

I think I disagree, but it really depends what you mean by "test" or "test case". I assume that test case is for a given input, expect certain output, and test verifies certain assumption, such as for a certain class of inputs you get a certain class of outputs.

I believe that you always test two implementations. For example, if I have a test case for a function sin(x), then I compare with the calculator implementation, from which I got the result. So if the tests are to be comprehensive (and automatically executed), then they have to be another implementation of the same program, you can't avoid it, and you can't avoid to (potentially) have bugs in it.

Now, the advantage is that the test implementation can be simpler (in certain cases); or can be less complete, which means less bugs, but also (in the latter case), less comprehensive testing.

In any case, you're validating the assumptions. The assumptions come from how the test implementation works (sometimes it is just in your head). And to express them formally, of course, that's the whole point.

For example, if you're given an implementation of sin(x) to test with, you can express formally the assumption that your function should give a similar result.

By formalizing this assumption, you can then let the computer create the individual test cases; it is a superior technique than to write test cases by hand.