
Show HN: Python Tests That Write Themselves - timothycrosley
https://timothycrosley.github.io/hypothesis-auto/
======
mrfusion
This actually gave me another idea. What do you all think, I’d be up for
trying to build it.

It would watch your program during execution and record each function calls
input and output. And then create tests for each function using those inputs
and outputs.

You could always go over the created tests manually and fix them or review
them but if nothing else it could be a good start.

~~~
OJFord
Instagram created 'MonkeyType' to do something similar for type annotations.

A lower barrier way of trying this out could be to run MonkeyType in
combination with OP's hypothesis-auto (which uses type annotations to generate
the tests).

~~~
westurner
pytype (Google) [1], PyAnnotate (Dropbox) [2], and MonkeyType (Instagram) [3]
all do dynamic / runtime PEP-484 type annotation type inference [4]

[1] [https://github.com/google/pytype](https://github.com/google/pytype)

[2]
[https://github.com/dropbox/pyannotate](https://github.com/dropbox/pyannotate)

[3]
[https://github.com/Instagram/MonkeyType](https://github.com/Instagram/MonkeyType)

[4]
[https://news.ycombinator.com/item?id=19454411](https://news.ycombinator.com/item?id=19454411)

------
ebg13
Does this do anything other than assert that functions return the right basic
type given extremely basic inputs?

Is naive type testing a thing that people actually bother doing vs testing
that the functions actually do what they're supposed to?

If a function takes two ints and returns an int that is the integer division
of the two, is the evaluation of exceptions and returned types not already
implicit in the evaluation of the actual results?

If you know that divide(a, b) should return c for sufficient candidates a, b,
and c, then you _know_ that divide returns the right type without explicitly
checking. And knowing that the divide function happens to return ints when
given ints doesn't actually tell you that it's doing anything even close to
the right behavior. So this both doesn't reduce the number of tests you need
to write and is also obsoleted by actually writing the tests that you need.

~~~
timothycrosley
By default, the biggest value from this is catching unexpected runtime
exceptions with certain values (the edge cases you don't think of yourself).
To get the most value out of it, you can further specify what you expect from
the result using the _auto_verify=Callable() parameter, or by looking at the
individual test cases.

Simple example:

    
    
      from my_library import add
      from hypothesis_auto import auto_pytest
    
      @auto_pytest()
      def test_add(test_case):
         add_result = test_case() 
         if test_case.params.kwargs['number_1'] > 0 and test_case.params.kwargs['number_1']:
             assert add_result > test_case.params.kwargs['number_1']
             assert add_result > test_case.params.kwargs['number_1']

~~~
ebg13
> _By default, the biggest value from this is catching unexpected runtime
> exceptions with certain values (the edge cases you don 't think of
> yourself)._

So it's a fuzzer?

~~~
timothycrosley
Yes! A smart one though, that limits itself to the type constraints you've
specified to avoid redoing the work of your type checker.

------
rcfox
I like the idea, but I think I would be more comfortable with it generating
test files that I could keep beside my other tests.

Also, `auto_pytest_magic` doesn't seem to exist.

    
    
        ImportError: cannot import name 'auto_pytest_magic' from 'hypothesis_auto' (/home/.../.local/lib/python3.7/site-packages/hypothesis_auto/__init__.py)

~~~
timothycrosley
> I think I would be more comfortable with it generating test files that I
> could keep beside my other tests.

This is a very interesting idea I might explore more!

`auto_pytest_magic` is only made available if you have pytest installed. You
can enforce this by doing:

    
    
      pip3 install -U hypothesis-auto[pytest]

~~~
rcfox
Oh man, for some reason I thought pytest was a synonym for Python's unittest
module.

------
hchasestevens
This is neatly packaged, but it's not immediately clear what advantages it has
over Hypothesis' native offerings for accomplishing this? (
[https://hypothesis.readthedocs.io/en/latest/details.html#inf...](https://hypothesis.readthedocs.io/en/latest/details.html#inferred-
strategies) and
[https://hypothesis.readthedocs.io/en/latest/data.html#hypoth...](https://hypothesis.readthedocs.io/en/latest/data.html#hypothesis.strategies.from_type)
)

~~~
timothycrosley
It's an extension for hypothesis, and its value is 100% just convenience and
accessibility. Really, it's meant to be a sort of gateway to doing full
property-based testing, in the least barrier way possible. I wrote a bit more
about why I created it here: [https://timothycrosley.com/project-5-hypothesis-
auto](https://timothycrosley.com/project-5-hypothesis-auto)

------
ben509
On one job, I had to disable code coverage for a whole suite of tests that
were simply making calls and completely ignoring the results.

I know, coverage is Yet Another Metric, but if you don't game it, it can help
you track down branches you haven't written tests for.

So my hesitation is I can see people running this, gettting 100% code coverage
and thinking, "hooray, it's fully tested!"

------
timothycrosley
Thoughts behind project creation live here:
[https://timothycrosley.com/project-5-hypothesis-
auto](https://timothycrosley.com/project-5-hypothesis-auto)

~~~
djaychela
Thanks for the write up - explains a lot (including what hypothesis is).

I think there's a typo in the last code box? It looks like you repeated "from
hypothesis_auto import"? (that, or I understand even less python that I
thought!)

~~~
timothycrosley
You are correct! Thanks for catching this! I'll fix the example later today

------
iandanforth
I like this idea. One piece of feedback, a parameter with a leading underscore
feels very odd. In python I interpret leading underscores to indicate the
programmer thinks of this as an internal / pseudo-private property. Exposing
it through the api makes it "public" which means (to me) that it shouldn't
have a leading underscore.

This is especially true if the usecase is common enough to put in the top
level examples

~~~
timothycrosley
While I agree, that `_param` generally means private, it is also a common way
to allow parameters to be passed into a function where all other parameters
are passed directly along via _args and_ *kwargs to avoid naming collisions.
An example of this is NamedTuple:
[https://docs.python.org/3/library/collections.html#namedtupl...](https://docs.python.org/3/library/collections.html#namedtuple-
factory-function-for-tuples-with-named-fields).

~~~
duckerude
I believe Raymond Hettinger now considers that a mistake, and wishes he had
gone with a trailing underscore (param_) instead. A trailing underscore is
just as unlikely to lead to a collision, but less confusing.

~~~
timothycrosley
Updated in latest release to use trailing underscore:
[https://timothycrosley.github.io/hypothesis-
auto/CHANGELOG/](https://timothycrosley.github.io/hypothesis-auto/CHANGELOG/)
figured it would be the least painful the earlier I did it :)

~~~
duckerude
Great!

It takes a bit of manual juggling, but if you take __kwargs anyway you could
inspect them to check for the old-style parameters, and use them if they 're
present, perhaps with a warning. That way you don't have to break
compatibility.

------
somada141
While I had really liked the idea of hypothesis in Python I found that the
edge-cases it was uncovering were the ones that were obviously gonna break but
at the same time cases I didn't care to guard against, e.g., using 3-mile long
integers, or cases that wouldn't work with the underlying libraries eg NumPy.
Thus, I found myself spending more time adding constraints on the generated
inputs than fledging out my test-suite. So my adventures with hypothesis were
short-lived.

I don't mean to detract from this library, I think its a great combination of
strong-typing and property-based testing but has anyone had any experience
employing property-based testing on complex functions outside of the whole
add/subtract/multiple stuff? What kinda thing have you used it on?

------
StavrosK
This is great, thank you! Another great example of the benefits strong typing
brings.

------
kburman
Nice concept but does it work with real-world application. I failed to
understand how will it work with methods like `authenticate_user(user)` or
`load_permissions_from_db(user, db)`.

~~~
timothycrosley
Hi @kburman,

For cases like that it would be best combined with a mock:

    
    
      @auto_pytest()
      def test_add(test_case, mocker):
          mocker.patch('db.call')
          test_case()
    

Note that this example utilizes the following pytest extension:
[https://github.com/pytest-dev/pytest-mock](https://github.com/pytest-
dev/pytest-mock)

However, I would also note, that just because you have some methods that have
potentially dangerous side effects, in most large code-bases, not all
functions do. Which is why it operates at a function by function basis. You
can use this for new pure functions, while continuing to write other tests in
the same fashion by hand.

