

Unit testing a TCP stack - djoldman
https://www.snellman.net/blog/archive/2015-07-09-unit-testing-a-tcp-stack/

======
soheilhy
Google has a pretty mature TCP testing framework, called packetdrill:
[http://static.googleusercontent.com/media/research.google.co...](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41848.pdf)

It supports all the features you need, from packet structure to timing:
[https://code.google.com/p/packetdrill/](https://code.google.com/p/packetdrill/)

~~~
acveilleux
It still treats the whole stack as a blackbox. Necessarily the effort to cover
important boundary cases throughout the TCP stack can get overwhelming.

~~~
ori_b
Honestly, I've found black box testing at public boundaries of a system gives
the best payoff vs development time, especially if you can make them run
quickly.

Most issues seem to be catchable with this, and way too many unit tests end up
becoming 'change detector' tests, where you test tautologies about
implementation details.

~~~
gsnedders
Black-box testing as a first move is often a good move, IMO — but it's
important to accept that at some point you're going to stop hitting so many
edge-cases in the implementation that ought be tested. Some of those edge-
cases should probably be tested with black-box tests, and some should probably
be tested with unit tests.

One has to consider the performance cost of black-box tests, though. It will
almost always be higher than that of unit tests, simply because more code will
get run per test (this is _especially_ true of network stuff, as even going
through the loopback device is comparatively _insanely_ expensive). That said,
in some cases, it may well still be the right choice.

------
mokus
Very nice. The fact that the TCP stack isn't in-kernel isn't all that
necessary either. You can perform the same abstraction for an in-kernel stack
as for an in-application stack, though it can often be harder to disentangle a
subsystem that wasn't built from the start with that kind of separation in
mind.

Unfortunately I can't share them, but I have done exactly this sort of thing
in embedded projects at work - compile the same C code (which is coded in a
style that tries to minimize if not eliminate undefined and implementation-
defined behavior) to run on the host that builds it, under a Haskell test
harness that tests behavioral aspects using both unit testing and Quickcheck
property testing, with Haskell code simulating external stimuli such as
interrupts, low-level behavior of storage subsystems, mcu resets, etc.

Testing low-level C code, even code that interacts directly with hardware in
unprotected domains, is absolutely possible. And also extremely valuable. The
usual caveats about test correctness are even more important, though - there's
a much higher burden to ensure that your simulated hardware is both accurate
and comprehensive in the behaviors it can simulate.

