
Compiler fuzzing, part 1 - jjuhl
http://www.vegardno.net/2018/06/compiler-fuzzing.html
======
bufferoverflow
This is a beautiful bug:

    
    
        inline namespace {
                namespace {}
        }
        
        namespace {}
    

[https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84707](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84707)

------
CJefferson
I would strongly recommend fuzzing for testing any software. You are likely to
find interesting things quickly, if the software hasn't been fuzzed before.

I fuzzed early clang for segfaults, just for fun. I found dozens of bugs by
just taking a large preprocessed C++ file, deleting a few lines, and then when
finding a segfault, reducing by deleting lines and tokens until I reached a
minimal example of the same error (make sure you are crashing in the same
place, or you will probably find you keep rediscovering the same bug).

Reported some great fun minimal crashes, like:

:)

:{}

namespace

I also fuzz in my day job. There we want to fuzz valid inputs -- we have a
bunch of ways of generating two inputs which should produce the same output --
generate a pair of inputs, test outputs, repeat. It has found hundreds of
bugs, and almost no bugs (famous last words) have been reported in releases.

~~~
jwilk
Minimizing testcases can (and should!) be automated.

For C(-like) code, there's venerable Delta
([http://delta.tigris.org/](http://delta.tigris.org/)) and more powerful
C-Reduce
([https://embed.cs.utah.edu/creduce/](https://embed.cs.utah.edu/creduce/)).

For binary formats, there's afl-tmin, part of AFL
([http://lcamtuf.coredump.cx/afl/](http://lcamtuf.coredump.cx/afl/)).

~~~
CJefferson
Don't worry, I did automate! However, I did it myself, for the experience.
Some horrible bash and awk scripts did sufficiently for my purposes.

------
bhaak
> I have never had such a pleasant bug-reporting experience. Within a day of
> reporting a new bug, somebody [...] would run the test case and mark the bug
> as confirmed as well as bisect it using their huge library of precompiled
> gcc binaries to find the exact revision where the bug was introduced.

That's a really cool setup. Precompiled binaries to quickly git bisect a
change.

How often did you run into not being able to use git bisect because you
encountered problems due to code changes that were unrelated to the bug you
were hunting?

Quickly being able to reproduce and find the commit for a bug is tremendously
useful.

------
sligor
A little off-topic but... what is the interest for large company like Oracle
to pay employees on such research effort ? What is the value added for the
company ? real product ? or keeping links with advanced research and
maintaining high level of knowledge in the company?

~~~
vidarh
Oracle owns Java, so has a direct reason to be interested in compiler tech,
and other parts of the software suite also would benefit from similar types of
fuzzing (e.g. their query processing for the database).

But apart from the direct value, there's a huge value in keeping high paid
staff motivated and sharpening their skills of course, on top of the good will
it creates among potential hires.

------
boomlinde
I wrote a toy compiler/VM and decided to fuzz test with radamsa. The language
is quite forth-like, so there is minimal syntax and every program is valid so
long as the words that are used are defined and the stack is balanced, which
makes it a perfect subject for fuzzing. After finding some low hanging fruit
almost immediately (segfaults), I let it run for another couple of hours.

Then, the computer started swapping like hell and became unresponsive, which
didn't settle for another 10 minutes after I had shot down the process.
Looking at the case that radamsa had generated, it had found a billion laughs
attack vector. Macros in my language can be defined recursively, and the code
is stored in an array that gets reallocated and grown when the code no longer
fits, unboundedly. Radamsa had created an initial macro and then redefined it
over and over such that it always referred to the previous definition twice.

I was optimistic about fuzzing, but I never really had any expectations of it
finding anything other than stack smashing and segfaults.

------
willvarfar
For the Mill project, we tested a large corpus of test programs and csmith-
generated programs with native output against our simulator. We found lots of
bugs in the various stages of our tool chain and simulator, but so far none in
llvm itself.

~~~
jwilk
Link to the project? The name is not very googlable...

------
ksk
Interesting, but I suppose you still need a human to verify whether its valid
or invalid grammar.

~~~
comex
Not really. I took a look at some of the reported bugs, and all of the test
cases seemed to be invalid code in some way, though often due to type or
semantic errors rather than bad syntax. But the fuzzer is looking for crashes
or internal compiler errors, which represent a bug regardless of how bogus the
input to the compiler was.

~~~
ksk
You're quite right. I used to work on fuzzers/fault-injectors in a previous
job so I was interested because I thought they had figured out a way to find
bugs by generating valid grammar. Not putting down what they've done though..

------
dang
Url changed from [https://www.phoronix.com/scan.php?page=news_item&px=Prog-
Fuz...](https://www.phoronix.com/scan.php?page=news_item&px=Prog-Fuzz-
Compiler-Fuzzing), which is just a short paragraph that points to this.

In such a case where there's a much better original source, could you please
post that instead? This is in the site guidelines:
[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html).

