
Mutation Testing: A Tale of Two Suites - kiyanwang
https://codeascraft.com/2020/08/17/mutation-testing-a-tale-of-two-suites/
======
taeric
I've tried to get mutation testing going on my last few projects. Amusing how
little faith/interest there is in this from most of my colleagues. Would be
interested in hearing how folks keep this up in a project.

~~~
dogweather
It's probably most important in language ecosystems like Ruby which seriously
depend on test suites for catching bugs.

One way to view the problem that Mutation Testing solves is, there's no test
suite for the test suite. And so there's no safety net to enable refactoring
the test code. It's test suites all the way down.

Mutation testing solves this infinite recursion problem by automating the
test-suite-test-suite creation.

~~~
taeric
Yeah, I pitched it as a way to know how good your tests are at catching common
bugs. Still no dice.

~~~
release-object
Did you considered doing it anyway? If a one-off run reveals some previously
unknown flaw in the test suite you could present that.

~~~
taeric
Apologies for not seeing this.

I have added it. It got removed for adding to the build time. Never really
found many issues, so I couldn't argue to keep it. Did point out that the
argument was essentially against testing.

------
dogweather
The logic of refactoring test code is very interesting: IMO one can't just do
that, because it tosses out the guarantee that it still tests what one thinks
it does.

So I actually break my application first to make the tests fail, and then
refactor them while failing to ensure that they still fail. Finally, I undo my
changes to the app code, and the tests should all pass again. I guess this is
essentially a manually performed mutation test.

~~~
DylanSp
I do the same thing, either when refactoring test code or when I'm adding
tests after writing the implementation. I've heard the manual approach
referred to as "refactoring against the red bar"; see
[https://corgibytes.com/blog/2016/09/20/refactoring-
against-t...](https://corgibytes.com/blog/2016/09/20/refactoring-against-the-
red-bar/).

