
Mutation testing based on LLVM (2018) [pdf] - luu
https://lowlevelbits.org/pdfs/Mull_Mutation_2018.pdf
======
gkfasdfasdf
For a great explanation of _why_ you would want to use mutation testing, see
[https://pitest.org/](https://pitest.org/) (a similar tool but for Java). In
short - it is to gauge the quality of your test suite, because ideally any
mutation of your app should cause a test failure.

~~~
amelius
Well, not _any_ mutation. Deleting an assertion should not cause any problems.
Replacing bubblesort by quicksort shouldn't either. Etc.

~~~
AlexDenisov
This is a valid concern, for sure. In fact, these are called Equivalent
Mutants. We observed it several times, here are some examples. You flip the
if/then branches and the code essentially does the same, switching from a fast
to slow implementation of the same algorithm. Another example is mutating the
code in a C++ destructor, which is not easy to test.

With that being said, even despite the equivalent mutants, you can get a
pretty good insight. Here is an example of finding from a proprietory real-
time OS used in the space industry:
[https://gist.github.com/AlexDenisov/b5d2e23457b88813b5ab9d5d...](https://gist.github.com/AlexDenisov/b5d2e23457b88813b5ab9d5d96183ed2)
(this is a part of an email which I could safely post online).

------
eneyman
Is mutation testing different than mutation-based fuzz testing? If so, how?

~~~
jononor
Mutation testing mutates the program (by making changes in the code, like
flipping a condition in an if statement), and then checks whether the test
suite catches it. That is, it tests the tests.

Mutation-based fuzz testing mutates the input to the program (often by
starting from know-valid data and applying a mutation to the data). So it
tests the program, much like unit-tests.

