
What is property based testing? - hyperpape
http://hypothesis.works/articles/what-is-property-based-testing/
======
santoriv
Here's a question that's tangentially related to the article in that it deals
with property based testing. I'm curious what the software veterans thing
about this:

Should you write property based tests of a third party custom api if you are
on a short term contract and pretty sure that no one is going to maintain the
testing suite? On my last contract the product quote api and the product order
customization UI I was writing were being co-developed at the same time by two
different parties. I found a lot of errors in the schema/third party endpoint
by randomly generating lots of product orders and tossing them at the
endpoint. While this helped a lot in development, it took a fair amount of
time to write. Knowing the parties involved I'm 95% certain that the tests
were never run again and probably ignored after I left. Is it more cost
conscious to just not write the tests and slog through to the hand-off?

I realize that this may be a little hard to answer without knowing the project
intimately. I just sometimes wonder if I write tests too often.

~~~
seanwilson
> Should you write property based tests of a third party custom api if you are
> on a short term contract and pretty sure that no one is going to maintain
> the testing suite?

That sounds like you're doing their work for them. If I'm accepting risk in a
contract, my major assumption would be the third party libraries I'm using are
fit for purpose. You surely can't be held liable if the other contractors
aren't doing their work to an acceptable standard.

~~~
nickpsecurity
It's not quite that simple. Much of the problems of Windows reliability was
from 3rd party drivers and apps (esp killer apps). They couldn't ditch the
hardware or app vendors because their customers demanded them. So, Windows
source was littered with all kinds of provisions to basically work around all
that. They eventually forced driver analysis and safer language onto the
ecosystem when that was too much. But a key part of their path to dominance
and backward compatibility was working around known issues in 3rd party code.

Avoid this when possible obviously. However, it's sometimes a good idea when
it's for tie-in to ecosystem that has benefits worth it.

~~~
seanwilson
Maintaining Windows is a very complex and special case though. I mean for most
contract work you can safely assume the libraries you're going to be using are
mostly bug free.

~~~
fineline
"Safely assume" is an oxymoron. Assumptions are a form of risk taking. The
upside reward is that you save the cost of testing (and thus eliminating) your
assumption. The (potential) downside is the complete failure of everything
resting on that assumption. In many cases, the upside is small and knowable,
the downside is hard to predict or contain - i.e. the worst kind of asymetric
risk. Thus as a rule of thumb, assumptions are very bad practice, and should
always be called out as liabilities in contract terms - e.g. "The cost of
repair or rework due to any bugs found in the provided libraries shall be
borne by X."

~~~
seanwilson
Do you have any concrete examples where a third party library turned out to be
the major factor of a project failing? I've done plenty of projects that rely
on a large number of at least moderately popular open source components for
example and I've never once had a major problem. Obviously all assumptions
carry risk, you should have good contract terms etc. but from experience this
element is not a high risk at all. Clients tend to understand the elements
that are out of your control. Changing requirements is a vastly higher risk
for example.

------
platz
referencing fuzzing so heavily in the definition places a lot of faith in
understanding that there are those subtle varieties of fuzzers that can
generate structured data/inputs.

I didn't know fuzzers did that, so I'd guess that others would miss the nuance
there too; and it's a rather large part of property based testing

This is a definition that aims to be more "categorically correct" without
providing a lot of intuitive explanatory power. (E.g. A monad is a monoid in
the category of endofunctors). Maybe that's ok in that was the aim and PBT
can't be reduced to a single sentence like a lot of abstract concepts

It also assumes a well-defined definition of a fuzzer

I was also surprised to not see the word 'invariants' mentioned once. Maybe
that's very close to "property", but property has other connotations so it
still seems useful

~~~
bbcbasic
To me property based testing is just checking that a function obeys a certain
law by throwing lots of random data at it.

That means you have to define the law in terms of the input and output. Rather
than fix the input like you'd do on a regular test and just assert the output.

The hard part is trying to figure out what laws you should expect from your
functions.

~~~
paulddraper
Maybe you should write a function that does the right thing, and then use it
to tell you the correct output :/

~~~
dbcurtis
Not totally silly. Just last Friday I was writing a Python hypothesis-based
test using constrained-random stimulus generation. The target function under
test is rather complicated because it is coded to be space and time efficient.
It is not so easy to read and reason about, surprise, surprise. But there is a
straight-forward, easy to ready, while-loops-and-simple-if's implementation
that is slow and inefficient, but it's not too hard to convince yourself it
produces the correct answer. So I use that in my test driver to generated
expected results. It's certainly possible for both implementations to contain
bugs, but extremely unlikely for them both to report the same wrong answer for
the same stimulus.

~~~
di4na
Just nitpicking about "extremely unlikely for them both to report the same
wrong answer for the same stimulus".

This assumption is what are "blind implementation voting systems" were based
on, and it was proved to not survive that well in practice. It may totally
work in your case, but you may be interested in looking at the work on Nancy
Leveson at the MIT (all links are from her website)

[http://sunnyday.mit.edu/papers/nver-
tse.pdf](http://sunnyday.mit.edu/papers/nver-tse.pdf)

[http://sunnyday.mit.edu/critics.pdf](http://sunnyday.mit.edu/critics.pdf)

[http://sunnyday.mit.edu/papers/nver2.pdf](http://sunnyday.mit.edu/papers/nver2.pdf)

[http://sunnyday.mit.edu/papers/consistent-
comp.pdf](http://sunnyday.mit.edu/papers/consistent-comp.pdf)

This body of work launched quite a controversy, but well, it is still here...

------
seanwilson
I'm really surprised QuickCheck and similar tools aren't used much outside of
languages like Haskell. QuickCheck is easy to implement, simple to learn, the
test code is concise and it's really good at finding obscure bugs.

~~~
kmicklas
By the time you discover QuickCheck, you might as well just use Haskell :)

~~~
seanwilson
Haskell isn't practical for a lot of projects though just based on where your
code has to run and what existing libraries you want to use.

~~~
willtim
Yes. But this is more the problem of vendor lock-in than anything to do with
Haskell.

~~~
seanwilson
Yeah, not knocking the language. It's frustrating how it takes so long for
good programming language ideas to defuse into the mainstream. It's crazy how
strong type systems have been around for literally decades and we're still
using weakly typed dynamic languages for example. TypeScript is a decent
compromise for me at the moment despite being a fan of actual functional
languages.

------
KurtMueller
Anybody here do any property-based testing with Ruby? If so, what do you use
and what do you test with it?

~~~
hyperpape
I don't do Ruby, but the author of this piece has an article on property
testing in many different languages, including Ruby:
[http://hypothesis.works/articles/quickcheck-in-every-
languag...](http://hypothesis.works/articles/quickcheck-in-every-language/).

------
vram22
Is property-based testing related to equivalence (class) partitioning?

[https://en.wikipedia.org/wiki/Equivalence_partitioning](https://en.wikipedia.org/wiki/Equivalence_partitioning)

------
kristianp
The article mentions that smallcheck doesn't qualify as property based
testing. However it's hard to see what it is from its description. Any ideas
what smallcheck does?

> SmallCheck is a testing library that allows to verify properties for all
> test cases up to some depth.

[https://hackage.haskell.org/package/smallcheck](https://hackage.haskell.org/package/smallcheck)

~~~
paradoja
It's like QuickCheck, but instead of testing random values, it tests all
values in a certain range.

For example, if the function takes an array of 128 bit values, it's test all
the possible arrays of length 0, 1, 2... up to the length you want to test
for.

(Actually, not necessarily all, but all that an exhaustive generator
generates; the generator would normally go through all values, but it
potentially could only test some values, but would not select them at random).

