Fuzzing the .NET JIT Compiler

[matthewwarren]

https://mattwarren.org/2018/08/28/Fuzzing-the-.NET-JIT-Compiler/

[hobls]

> Or in other words, a fuzzer is a program that tries to create source code that finds bugs in a compiler.

This is a very narrow definition of a fuzzer. There are a lot of types of fuzzer that do not generate source code, and are not intended to test compilers.

[seanmcdirmid → hobls]

When I was an undergrad in 1997, I created a fuzzer (well, we didn't know to call it that back then) to insert one byte changes into Java classfiles, and then used that to test our bytecode verifier along with Sun and Microsoft's.

Suffice it to say, we found lots of bugs, the most famous one being what my grad advisor called the vacuum bug that could read a web client's environment variables (among other things): https://archive.nytimes.com/www.nytimes.com/library/cyber/un...

[hobls → seanmcdirmid]

Very cool! I think fuzzers are SO interesting. One of my favorite examples is lcamtuf’s post on generating valid JPGs from nothing [1]. Just so awesome.

1: https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-th...

[kodablah → seanmcdirmid]

Tangentially related, speaking of Java, I wrote a path guided one recently for code running on the JVM: https://github.com/cretz/javan-warty-pig. It's found bugs for me on internal things, but I haven't run it on anything popular to build a trophy case.

[matthewwarren → seanmcdirmid]

Nice! Not many people can say that their university project made it into the New York Times!

[sehugg → hobls]

To test a MS-DOS database program I was writing, I replaced the WaitForKey routine with GetRandom. It was fun to watch! And I think it was a fuzzer.

[matthewwarren → sehugg]

Nice, that's a pretty cool approach! (random user input)

[matthewwarren → hobls]

> This is a very narrow definition of a fuzzer. There are a lot of types of fuzzer that do not generate source code, and are not intended to test compilers

Yeah, that's true, I completely missed out other uses for 'fuzzers', thanks for clarifying that

[kodablah]

I wonder, instead of an AST-generating fuzzer, if they could have used AFL on an instrumented binary that may find more possibilities since it just works on paths and bytes without being constrained by what this fuzzer knows are certain constructs.

[ygra → kodablah]

Can you easily run AFL on .NET (or jitted) code? My understanding was that AFL works by instrumenting paths taken and kinda needed a gcc-produced binary with normal Linux debug symbol stuff.

[0xad → ygra]

What AFL needs is _instrumentation_ and of course the easiest way is to get that at compilation step, however you are not constrained by anything to get that part via other means. Check https://github.com/ivanfratric/winafl that uses DynamoRIO.

So, to answer your question -- it wouldn't be easy but it can be done.

[matthewwarren → ygra]

The .NET Profiling API gives you a way to modify IL code before it's JITted, so that could be one way to do this. See http://mattwarren.org/2018/08/21/Monitoring-and-Observabilit... for a bit more info ('SetILFunctionBody() API')

[kodablah → ygra]

I think there are Windows and VM-based approaches, but it probably isn't to difficult to mimic what AFL does on the CLR.

[tgma]

An obvious next step to consider might be applying EMI techniques: https://mehrdad.afshari.me/publications/compiler-validation-...

[matthewwarren → tgma]

Thanks for the link, I'd not heard of EMI techniques before, they look interesting

[matt_d → matthewwarren]

If you're interested, here's a collection of resources on compilers correctness (testing & fuzzing, validation, and verification): https://github.com/MattPD/cpplinks/blob/master/compilers.cor...

[0xad]

Great article. Kudos.

On a side node, my old project https://github.com/dyjakan/interpreter-bugs along with short presentation I did on WarCon 2017 https://github.com/dyjakan/conference-talks/blob/master/2017... (references might be interesting for others).

[rkagerer]

Nice use of Beyond Compare for the side-by-side IL snapshots.

[matthewwarren → rkagerer]

I'm glad you liked that! (I spent more time that I should've trying to show that in a meaningful way and I was pretty glad when I remembered about 'Beyond Compare')