
The Common Thread: Fuzzing, Bug Triage, and Attacker Automation - jsnell
http://cybersecpolitics.blogspot.com/2016/05/the-common-thread-fuzzing-bug-triage.html
======
jonhohle
I've had very good experiences with AFL on a medium sized C++ code base. I ran
it for 8 CPU months on the code base and was reasonably confident with the
results. It generates samples for you, making the bugs easy to reproduce.
Along with valgrind, most ended up being reasonably shallow - lots of out of
bounds, uninitialized use, use after free, and incorrectly handles return
codes from 3ᴿᴰ party libraries.

I recall getting false positives around hangs, but replaying all the generated
test files made that easy to confirm.

Fortunately, the code base I was testing had an extensive set of input files
for testing already.

~~~
jonhohle
As a quick follow up, one of the things I found surprising were that some of
these bugs could have resulted in DOS for some significantly important
systems, and that the code was written by some of the smartest people I have
ever worked with. If these people were unable to write secure C++, it really
made me question whether anyone could reliably do it in a business
environment.

~~~
microcolonel
I think good code and a business environment just don't go hand-in-hand. I
work for a company where one of our core values is writing clean code. We have
fired clients who compromise too much on quality.

But despite this, corners are always cut when there's an arbitrary deadline;
and in enterprise, there is always an arbitrary deadline.

------
tptacek
The middle part of this post is concerned with the DARPA Cyber Grand
Challenge, which was a contest in which CGC generated dozens of small broken C
programs that did network I/O, and contestants had to use fully automated
systems to find vulnerabilities in them.

The through-line on this Dave Aitel post is that despite a decade of effort on
high-level CS approaches to finding bugs, like lifting binaries to predicate
expressions and running solvers on them, most bugs are still found by
something not much more sophisticated than "cat /dev/urandom | program".

~~~
munin
that kind of reduction is frustrating because not all bugs are equal. are all
bugs found by fuzzing? does the value of the bugs found by means other than
fuzzing outweigh the resources spent to find them?

~~~
tptacek
That's what Aitel is questioning.

~~~
munin
then he should run some numbers or make an argument that takes a position?

