pytest is by far the most productive testing framework I've ever tried, for any language. It's so good that it switched me from treating tests as a necessary chore to actively enjoying writing them.
Using ctypes to exercise a C module like this is brilliant - especially the mechanisms used here to work around segmentation faults.
Reminds me of SQLite, which is written in C but uses TCL for most of its test suite.
A notable downside that is not mentioned: the Python interpreter is far from Valgrind-clean. Valgrind is generally a powerful tool for debugging memory errors, but if you wrap your C code in Python, Valgrind will be so noisy as to be ineffective. Python startup is also very heavyweight; combined with the ~60x slowdown from Valgrind this is something you are going to notice.
The Python interpreter does not even use malloc()/free() directly by default; it layers its own memory allocator on top called PyMalloc (https://docs.python.org/3/c-api/memory.html#pymalloc). You can disable this by setting an environment variable (PYTHONMALLOC=malloc), but even then you will see many Valgrind warnings inside the Python interpreter itself.
I’ve had success running valgrind on python code with a very small suppression list to cover all of the python interpreter issues (at least for safety, not leaks) and let me focus on the code.
FWIW that's pretty confusing as it gives the impression the library is reloaded for every test, but iirc dlopen() will just return a handle to the existing one. I don't think ctypes has a good way to unload dlls so it should probably be `fixture(scope='session')`.
That gets more relevant when on-the-fly compilation is added to the mix, spawning a compiler for every test is a complete waste of time.
If you're using Linux, I would recommend using TemporaryFile instead of NamedTemporaryFile. It takes advantage of the O_TMPFILE flag, which guarantees that the kernel will clean up the file for you when the process exits:
It's better than relying on your application to clean up the file for you. With application-level cleanup hooks, you're still vulnerable to a resource leak if the process gets a SIGKILL or crashes or otherwise ends before your hooks run.
Does this even work? The documentation states that one should not rely on TemporaryFile having a name. Otherwise why would NamedTemporaryFile exist?
For the rare occurrence of sigkill before cleanup, I think your /tmp has ample of space to keep a few more kb until the next reboot. Servers running tests probably restart more often than desktops nowadays, when wrapped in containers.
That's a good observation. Towards the end the fixture is dropped in favour of a module-like object, which spaws the compiler once per test run. One could enhance this to skip compilation in the sandbox process to avoid unnecessary compilations.
This implementation only works with gcc though probably (it uses the automatic __start_SECTION and __stop_SECTION that gcc generates but clang doesn't seem to... there are likely hacks to make this work anyway though).
In principle, this approach allows interspersing your tests throughout a shared library instead of all in one file (though in that case you wouldn't want the testing function to be called main and you would need a separate driver program for test).
One approach is to build test traversal machinery out of function static variables and have every TEST() macro turn into a function that takes a pointer to some state controlling which test to run next.
You don't need malloc if using local variables either, which is helpful when your target doesn't have malloc or when debugging memory problems on targets that do. It's nice to know the test framework cannot be leaking or use-after-free anything.
Discovery is at runtime. Syntax goes something like
MODULE(foo) {
DEPENDS(bar);
TEST("I like string names") {
CHECK(42 == life());
}
}
where DEPENDS either sends control flow off to bar or onwards towards the test case depending on the value of an argument to the function MODULE created.
...after some thinking, maybe it would be possible to do a manual preprocess using a simple python script that collects all the tests and inserts them in the test main.
Really cool. Several jobs ago, I used cffi to create bindings to a library that controlled a camera's pan/tilt/zoom motors. Those were used to implement a test suite that validates the cameras in the manufacturing facility before shipping. The embedded developers also found being able to use the cffi bindings in the REPL really useful when prototyping changes. Python is a really useful tool for these kinds of interfaces.
I'm just amazed the lengths people go to complicate debugging (two runtimes), building (two things) and deploying (not single executable). Okay, maybe it's worth it for something (e.g. real "C" code used in Python, but just for the sake of testing... hmmm... no)
Depending on the size of your program, investing in test is something that shouldn’t be underrated. For big projects never call it “just” testing.
This example can be useful to generate rich test-data. Imagine you have some C program parsing json. Generating a sample json input in C would be very long and error prone, whereas in python it’s just a few loops for the generation and finally json.dumps.
This also decouples your test from your program, so you don’t do the same mistake in both of them, negating the test.
In same manner, parameterizing tests is surely possibly with GTest but it’s awkward and complicated, a lot easier in py.
Pytest also comes with lots of fixtures and possibility to make your own, such as spinning up a mock-db, although then we are more into integration testing rather than unit.
Not saying everyone should go rewrite their tests like this, but for some cases there is value that can be utilized.
How would you go to do this for c++? Specifically for classes and templates, is there an easy way to call code without having to manually write the demangled names of the functions?
Not that I know of. If I were doing it I'd expose a library / interface with pybind11 to test, but really I find Googles unit testing framework or boosts to both be pretty effective.
pytest is by far the most productive testing framework I've ever tried, for any language. It's so good that it switched me from treating tests as a necessary chore to actively enjoying writing them.
Using ctypes to exercise a C module like this is brilliant - especially the mechanisms used here to work around segmentation faults.
Reminds me of SQLite, which is written in C but uses TCL for most of its test suite.