
Building better software with better tools: sanitizers vs. Valgrind - ingve
https://lemire.me/blog/2019/05/16/building-better-software-with-better-tools-sanitizers-versus-valgrind/
======
wyldfire
> Sanitizers are a game changer, they bring C++ closer to languages like Java
> and Rust.

This is a stretch, IMO. I suppose it is ever-so-slightly-closer.

> They do not bring the safety to production, since you probably do not want
> to use sanitizers in production or release

You're right, they're not intended for use in production -- until now! ARM
dedicated some virtual address bits to tagging memory [1][2][3] for ARMv8 so
that we can have sanitizers in hardware. Imagine: chrome on your smartphone
being protected!

[1] [https://developer.arm.com/docs/ddi0487/latest/arm-
architectu...](https://developer.arm.com/docs/ddi0487/latest/arm-architecture-
reference-manual-armv8-for-armv8-a-architecture-profile)

[2]
[https://clang.llvm.org/docs/HardwareAssistedAddressSanitizer...](https://clang.llvm.org/docs/HardwareAssistedAddressSanitizerDesign.html)

[3] [https://arxiv.org/abs/1802.09517](https://arxiv.org/abs/1802.09517)

------
scottlamb
> Sanitizers ... do not bring the safety to production, since you probably do
> not want to use sanitizers in production or release

My team runs ASAN in production. We run lots of servers, and we have load
balancing that keeps CPU utilization equal across servers. ASAN binaries are
slower (maybe 1/2 throughput / 2X "local" cpu-bound latency?) but not so much
slower that running it on a small fraction of our servers means we can't meet
our SLO. And it's been a huge help to us. Once in a while, we have some
segfault that's pretty mysterious otherwise, as the stack trace where memory
corruption is discovered can be totally unrelated to the stack trace where the
problem was caused. If it happens on the ASAN servers, the root cause is
instantly obvious.

TSAN is more problematic to run in production; it's a _lot_ slower.

~~~
boris
Yes, thread-sanitized code is unbearably slow. Plus, you cannot use both ASAN
and TSAN in the same build (but either can be combined with UBSAN).

------
roca
The distinction between "sanitizers" and "Valgrind-like" drawn here is quite
arbitrary. I _think_ the core distinction being drawn is that "sanitizers" use
compiler-inserted instrumentation and "Valgrind-like" tools use dynamic binary
translation, though the author doesn't make this clear.

The conclusions drawn are questionable. The author states that "sanitizers"
are fast and "Valgrind-like" tools are slow, but it actually depends on the
tool. Some sanitizers are much faster than others. Some dynamic binary
translation tools (e.g. DynamoRio's Dr Memory) are much faster than Valgrind.
It wouldn't be hard to come up with examples where a compiler-based sanitizer
is slower than a dynamic binary translation tool. It all depends on what the
tool is actually checking.

Items 3 and 4 are also questionable. It's certainly possible to configure
dynamic binary translation tools with customized function-specific behavior.
Dynamic translation tools are generally even easier to use than sanitizers
because you don't have to modify your build at all, you can just run tests
under some kind of wrapper command.

~~~
jonstewart
He doesn't use the term "Valgrind-like". He only talks about Valgrind, which
he calls a "great tool" (he's right!). But he is evangelizing sanitizers, in
clang and gcc, which are relatively new. And, yes, they do work by adding
instrumentation at compile-time, which allows them to be much faster than
Valgrind (which does all its analysis at run-time). See:
[http://btorpey.github.io/blog/2014/03/27/using-clangs-
addres...](http://btorpey.github.io/blog/2014/03/27/using-clangs-address-
sanitizer/)

Sanitizers are great tools, too. Maybe you can quibble about his item 3, but
he's dead on with item 4. If you have tests, you add a couple compiler flags,
and then you'll find memory problems automatically with your unit tests
(without much slowdown).

~~~
inetknght
I currently work in the DNA analysis industry. We have some pretty massive
data sets and computational challenges. We use C++ for the heavy lifting.
There's no way we'd be able to valgrind anything but toy input (valgrind's
performance cost is prohibitive) -- and there's so many different possible
inputs and code paths that it'd be effectively impossible to automatically
generate all possible toy inputs. So we use sanitizers, as many as we can.
They're a game changer in correctness validation.

I'd love for fuzzing to be a thing, but even that seems impractical on a large
scale (please prove me wrong!) since it's a _lot_ more work than just setting
a compile target in cmake and ensuring that sanitizer flags are set for that
compile target.

~~~
touisteur
The right approach to start fuzzing with AFL, to me, is to recycle existing
unit tests or integration tests. In fact if you already have tests that create
a record (sorry, struct or class instance), inject them in your part of your
application, you're almost. Read your input data from a file and 'there' you
go... Paralellizing or distributing an AFL session on dozens of machines is a
matter of rsyncing some files.

Sanitizers are also very useful there in C/C++ languages because they're
mostly runtime checks. In Ada you had them already for a long time, and you
can be very through with your checks (see for example validity checks
[https://gcc.gnu.org/onlinedocs/gnat_ugn/Validity-
Checking.ht...](https://gcc.gnu.org/onlinedocs/gnat_ugn/Validity-
Checking.html) , runtime stack size checks
[https://gcc.gnu.org/onlinedocs/gnat_ugn/Stack-Overflow-
Check...](https://gcc.gnu.org/onlinedocs/gnat_ugn/Stack-Overflow-
Checking.html#f4) , and the very very powerful Initialize_Scalars that has
saved me many times [https://www.adacore.com/papers/exposing-uninitialized-
variab...](https://www.adacore.com/papers/exposing-uninitialized-variables-
strengthening-and-extending-run-time-check) ).

Shameless plug, but if anyone is interested in an introduction to fuzzing (in
Ada) I wrote something some time ago that could be useful :
[https://blog.adacore.com/running-american-fuzzy-lop-on-
your-...](https://blog.adacore.com/running-american-fuzzy-lop-on-your-ada-
code) .

Fuzzing is fun !

~~~
inetknght
I did a little bit of fuzzing and in general I agree with you about the right
approach being to build off of existing unit/integration tests.

But in practical terms, that's still a _lot_ more than "add this compiler
flag" or "swap out this library".

------
deng
Why "versus"? You use both, of course. And additionally, you use static
analyzers like clang-tidy, cppcheck, PVS, you-name-it. And you're _still_
gonna be miles away from Java and Rust w.r.t. safety. Undefined behavior is
the bane of the C-language family.

------
adrianN
Sanitizers and Valgrind are nice, but you need a comprehensive test suite to
really benefit from them. They only detect problems that are triggered by your
testsuite. This is a far cry from statically proving that your software is
free from e.g. use-after-free mistakes.

~~~
jzwinck
No test suite, no problem. Just run in a production staging area or deploy to
beta users.

~~~
aflag
That's better than nothing, but it would still be no proof that the are no
issues, as uncommon behaviour/access patterns could take some time to appear
(or will only show up after other services are introduced).

------
muricula
Yesterday a coworker had an issue where ASAN wasn't finding a std::vector out
of bounds error. It turns out the code doing the deference was in libc++,
which was not ASAN instrumented even though his program was. Valgrind found
the error right away. They're all great tools with their own pros and cons. If
you have the testing infrastructure, use all of them.

------
wyldfire
Note that valgrind can also detect uninitialized reads (or conditional
considered uninitialized memory). This is possible with MSan but MSan is a
little more challenging to use than ASan+UBSan or TSan.

In the years that I've used the sanitizers, I have yet to discover a false
positive. It's always been a real bug.

~~~
carlmr
I tried getting MSan to run but it never links properly for me.

------
hyperman1
Can someone explain how windbg is valgrind- like? I always considered it a
debugger.

~~~
nn3
windbg has many powerful plugins, and some do automatic verification (or talk
to tools that do so), so it can be a tool kit for verifiers like valgrind is.

~~~
hyperman1
Interestibg. Are there advanced blogs or so out there, demonstrating in the
real world? Raymond chen seems to only touch it's surface.

