
Metamorphic Testing - bocklund
https://www.hillelwayne.com/post/metamorphic-testing/
======
sriku
Good categorization. However, the example of sound -> text uses a set of
expected invariants for the problem. Invariants abound in sensory data related
problems as they are inherently high dimensional as measured but our interest
lies in a vastly reduced space. You could squint and look at these examples as
PBT as well. Such powerful invariants have been hard (caveat: for me,
personally, not done research) in general systems dev areas.

Other examples more illustrative would be useful.

Also, this is commonly done as "data augmentation" in ML ... which isn't
surprising given the ML related examples linked in the post. When talking to
software engineers without an ML exposure I often ask them to think of ML as
"extreme TDD" and gave a basic level talk with that perspective [1].

[1]: [http://sriku.org/blog/2019/03/07/talk-machine-learning-as-
ex...](http://sriku.org/blog/2019/03/07/talk-machine-learning-as-extreme-tdd/)

edit: by "ML as extreme TDD", I don't quite mean the algorithms themselves as
they're not central. I refer more to the data work involved.

------
pfdietz
The place where I saw this was in testing compilers. The idea is you get some
piece of deterministic code, and some input to run it on. You compile and run
that code on that input, and get an output.

Now, you modify the code so that the behavior ON THAT INPUT is not changed
(and does not become undefined). For example, you can make arbitrary changes
to parts of the code that weren't executed on that input. You now compile the
modified code and see if it does the same thing on that input. If not, you
have a compiler bug.

------
scribu
The nice thing about metamorphic testing is that it seems easy to implement.
You don't need a complex testing framework, like Quickcheck.

Just alter the input in some fairly simple ways and check the output.

~~~
winkelwagen
But then we are back at example/oracle based testing, the beauty of this and
pbt is the fact that no matter the input, it should always be what you
expected.

I did have hard time follow the audio example, he references a paper about api
testing (didn’t have the time to read the whole thing) but the examples there
are quite good.

for example:

Doing a query requesting YouTube video’s but with location personalization.

A query of New York center with a 50km range, should always be equal or less
then the same query with a 25km/15/5 range.

This would be true for every search query.

That said, I we still having a hard time using PBT, and metamorphic testing
seems like a subset of it. It will take a while before the industry gets
there.

~~~
scribu
Yes, sorry, I meant checking that the output didn't change (instead of
checking that it had a specific value).

------
thanatropism
What's wrong with ACM Queue?

Not every ACM journal is excellent and I know we're supposed to hate
everything that's not arxiv, but TEAC is a good one too.

