
When a Memory Allocator Hides Security Bugs - hannob
https://blog.fuzzing-project.org/65-When-your-Memory-Allocator-hides-Security-Bugs.html
======
nkurz
Instead of switching to the debug allocator, might it instead be possible to
teach ASAN about the pool allocator? I think I did this at some point in the
past for Valgrind. I don't know if it would be sufficient, but with ASAN it's
at least possible to "poison" and un-poison memory manually with macros:
[http://blog.hostilefork.com/poison-memory-without-
asan/](http://blog.hostilefork.com/poison-memory-without-asan/)

~~~
maxlybbert
I really like APR. While pooled allocation isn’t always the best approach, it
makes a lot of sense in server software, and reduces the risk of many classic
memory problems (e.g., I don’t have to worry whether I pair 1000 mallocs with
1000 frees; it’s acceptable to pair 1000 apr_pallocs with a single
apr_pool_free).

ASAN is certainly helpful when using classic memory management techniques, but
since so much C code uses custom memory management of some kind, it has to
handle more than malloc and free.

I certainly hope people improve ASAN instead of abandon things like memory
pools. I would consider “malloc and free with ASAN” to be a step down from
APR’s pools.

~~~
shereadsthenews
ASAN does handle this. As nkurz noted, custom allocators need only
poison/unpoison memory regions to benefit from ASAN and MSAN. For example see
how the absl "SwissTable" does: [https://github.com/abseil/abseil-
cpp/blob/master/absl/contai...](https://github.com/abseil/abseil-
cpp/blob/master/absl/container/internal/raw_hash_set.h#L1587)

------
gwern
Heartbleed was also an example of a bug which was hidden from the various
validation/static checking tools because OpenSSL was using custom memory
allocation, wasn't it?

~~~
nabnob
Yes. There's a good article here explaining why various static analysis tools
didn't catch Heartbleed:
[https://dwheeler.com/essays/heartbleed.html](https://dwheeler.com/essays/heartbleed.html)

>It is known that OpenSSL does not directly allocate and deallocate memory
directly using standard calls. Instead, it used a caching freelist system
internal to OpenSSL to reuse allocated memory. Since OpenSSL does not return
(deallocate) memory back to the underlying system once it was done with it (in
some cases), special tools could fail to detect some common weaknesses such as
use-after-free or double-free that they would otherwise find. It also
sometimes prevented some operating system and run-time mitigation mechanisms
from working. In short, it is widely agreed that this OpenSSL memory
allocation approach prevented many mitigation and weakness detection
mechanisms from working.

------
mattnewport
Somewhat ironically, a major secondary reason we used custom allocators in
games (performance bring the primary reason) was to find and diagnose memory
bugs. Long before ASAN existed I worked with some very powerful in house
memory debugging tools at EA, some of the functionality of which I still
haven't seen matched in commercial or open source tools.

Security wasn't the primary concern back then, it was avoiding memory leaks on
platforms with no virtual memory and tight memory constraints and squashing
bugs.

------
CJefferson
I increasingly feel we need to apply more "sledgehammers" to security. If I
was running servers with security I cared about, I would use a malloc which
cleared all allocations, and the new llvm/clang flags which clear all stack
variables to fix unitanialised problems. Maybe even try turning on the more
advanced clang sanitizers, if I could afford the cpu time.

If we, as programmers, want to be respected, we need to make every reasonable
effort to improve security.

~~~
cyphar
I believe OpenBSD does most of these things. The primary problem is that
security is a tradeoff -- for instance OpenBSD completely disabled
hyperthreading when it was discovered to be fundamentally insecure. That was a
50% performance hit right off the bat -- not everyone can afford to make such
a tradeoff.

------
jedisct1
Expect many of these bugs with WebAssembly.

------
scoutt
Misleading title... why all bugs are "security bugs" nowadays? It's just a
bug.

Also, the allocator does not hide the error. It's not a problem generated in
or from the allocator. It's a problem generated by the programmer and the
impossibility of static analyzers to know every allocator here and there.

------
j16sdiz
> and that indeed it seems some people did run the pool debugger in production
> (OpenBSD).

.. another remote exploit for OpenBSD ?

~~~
chousuke
OpenBSD doesn't include the Apache HTTP server, so while their default
settings might expose Apache bugs if you choose to use it, it's not fair to
say it's a vulnerability in OpenBSD

~~~
cryptonector
This reasoning is getting annoying. If you do nothing with your OS, it's
secure! If you use software it bundles it's not secure and it's not their
fault (but they'll fix it anyways)!

That is just oh so gimmicky. It reflects quite poorly on the project.

~~~
bialpio
I think I'm missing something here. How is a bug in Apache an issue with
OpenBSD?

~~~
mrkeen
I believe what's being referred to is the first sentence on the OpenBSD site:
"Only two remote holes in the default install, in a heck of a long time!"

A more cynical reading of it would be: "OpenBSD is secure as long as you don't
run software on it." I share that cynicism.

The discussion is relevant if you believe that an OS should have some
responsibility for memory-related security issues.

~~~
wahern
> The discussion is relevant if you believe that an OS should have some
> responsibility for memory-related security issues.

Pedantically speaking, your point is valid. But as a practical matter, OpenBSD
is the operating system that takes these issues the most seriously. They
formalized the notion of Unix-based privilege separation, and systematically
employed it across the system. They regularly perform system-wide code audits
(including of bundled software, such as Apache back when they actually bundled
Apache) to identify and remediate high-risk source code patterns (e.g. strcpy
-> strlcpy, realloc -> reallocarray, etc). While people [still] debate the
merits and performance of things like seccomp or Capsium, OpenBSD has
incorporated pledge(2) into the vast majority of binaries on the system. They
were the first to apply stack canaries, ASLR, W^X, and most recently PIE
_globally_ by default. And apropos the article, they were the first to
implement their system malloc in a robust way that maximized the chances of
catching memory bugs, sometimes sacrificing performance in a way that would be
inconceivable for Linux distributions. And all of these things have resulted
in a steady stream of bug fixes upstream to third-party packages from the
ports collection.

One of the reasons OpenBSD forks projects like openssh or libressl, or
implements from scratch services like iked, httpd, ntpd, relayd, smtpd, and
many others, is precisely so they can assume full responsibility and control
for things like memory management.

See
[https://www.openbsd.org/innovations.html](https://www.openbsd.org/innovations.html)

When a researcher points out a potential issue on OpenBSD, they're emphasizing
their point by leveraging OpenBSD's credibility and the hard work put into the
system.

