
AFLFast vs. Erlang - sokoowdevops
http://mindscope.pw/afl-fast-vs-erlang/
======
wyldfire
> Damn. That's terrible. We are never going to get anywhere with 20 executions
> a second

...

> Lucky for us there is a trick we can use to greatly improve that speed.

I just dusted off my fuzzpy project [1] this week and was bringing it up to
date with recent CPython, LLVM/libFuzzer.

> Notice the speed: 373.4/sec, up from 20! And we've already found 100 unique
> crashes!

Most of my test cases run at < 10/sec. I feel like I just got lapped while I
thought I was on a casual solo stroll. O.o

I intentionally left Py_InitializeEx() in the critical section for fear of
contaminating global state, though. But I see that I should've at least
experimented with it out.

[1] [https://bitbucket.org/ebadf/fuzzpy](https://bitbucket.org/ebadf/fuzzpy)

EDIT: sorry, I went a little off the deep end -- this post is in response to
the article listed in the first paragraph "After finding 1800+ crashes in
Python 3.5 using a method shown here ... "

~~~
dalke
I didn't get the point of the 1800+ crashes. Python isn't designed to be safe
from corrupted byte code in the .pyc files.

------
wyldfire

        PyObject *arg = PyTuple_GET_ITEM(varnames, j); 
    

> ... can be null if varnames is an empty tuple (which it can be if the pyc
> file is malformed).

This might be a case of "program works as designed" if the dev team thinks
that's an acceptable reaction to malformed pyc input. But, hey, it's worth
reporting because I would probably be inclined to fix it if I were the BDFL.

------
jlouis
I hope we can get this to run much faster than a few executions per second.
Having a way to speed this up could greatly help eliminate errors in the
Erlang VM.

The OTP team is usually pretty good at working with tooling for correctness
around the VM, so I would think this isn't too hard to get going.

~~~
mononcqc
Wondering if using erlang:halt() rather than init:stop(0) would be better
since halt(0) shuts down the VM instantly while init:stop(0) takes time to
take down the entire supervision trees in the VM in proper order.

~~~
jlouis
When I quickchecked maps, my approach was to use the distribution feature to
run the maps code itself on another VM and trying for force it into a crashing
state. This avoids having to restart the VM all the time.

AFL fuzz is severely limited if you need to boot up the VM, then run a
program, then stop the VM again. It is rarely that part you want to test.

Perhaps we can strike middle-ground. Rip out the ETERM encoder/decoder to
binary data and fuzz that. It should be a far simpler target.

------
rdtsc
Hmm, not bad at all Python 3.5 1800+ crashes, Erlang VM 9.

And I suspect Erlang's VM is more complex. Has a register VM machine. A M:N
scheduler (M cpus, N processes). Fairly intricate memory allocation, hot code
loading and such.

~~~
apoorvkumar
I think the author implies that 9 (unique) crashes are with limited number of
executions due to slow speed. Besides, all 1800+ crashes might not be unique.

~~~
sokoowdevops
Here's my stats on python so far:

(normal afl) [http://mindscope.pw/static/afl-python-
slow.png](http://mindscope.pw/static/afl-python-slow.png)

(aflfast) [http://mindscope.pw/static/afl-python-
fast.png](http://mindscope.pw/static/afl-python-fast.png)

sorry for screenshots ;)

------
_asummers
I hope he's able to work with the Erlang VM devs and get this running at a
faster pace. I'd be curious how many more this is able to find when it's able
to run at a faster pace.

------
jwatte
You can also get QuickCheck for Erlang, although that stays at the Erlang
fuzzing level, not AFL style branch fuzzing. Still, finds bugs!

~~~
jlouis
Fuzzing is different from QuickChecking:

QuickCheck uses hand-crafted known properties and generators. Fuzzing don't.
QuickCheck can be used to find negative tests, but this is the sole target of
a fuzzer. There are some similarities in the approach, but it is not the same
kind of tool.

------
elktea
Did you file bugs for python?

~~~
fulafel
The marshal format used by pyc files is explicitly documented[1] to be unsafe,
just a fast way to cache the compiled byte code to a .pyc file next to the .py
source file. The Python VM is not a sandbox so there has been no need for an
untrusted bytecode format.

[1] see big red warning box at
[https://docs.python.org/3/library/marshal.html](https://docs.python.org/3/library/marshal.html)

~~~
sokoowdevops
That's true, but looking at crashes, the 'exploitable' marks some of them as
exploitable, there are also heap errors. Might be an easy entry point to do...
something ;)

We did similar excercise a year ago with zpaq, and Matt and community was able
to fix it so well that now you can fuzz it pretty much forever :)

