
Afl.rs: Fuzzing Rust code with american-fuzzy-lop - yberreby
https://github.com/frewsxcv/afl.rs
======
ekidd
Randomly looking at the "Trophy Case", it looks like most of these errors are
run-time "panics". We can break Rust errors down into three main categories:

1\. Compile-time errors. This includes most memory-related errors, which are
mostly caught by the borrow checker. These are very serious errors, often with
ugly security consequences. Rust's big selling point is that it can catch many
different kinds of errors at compile time—but _not_ all.

2\. Run-time panics. This includes "index out of bound" errors, integer
overflow errors (in debug builds only), and assertions inserted by the
programmer. This is Rust's second line of defense, so to speak.

3\. Expected run-time errors. These are mostly reported using return values of
type Error, which is the normal way to handle errors in Rust.

Most of the errors caught by AFL seem to be errors in group (2) that ought to
be in group (3). In most cases, these errors couldn't be moved into group (1),
because they're not the kind of thing that's easily caught at compile-time.

So this is a really cool tool for Rust developers, especially ones working on
libraries that parse untrusted input. I was especially impressed by the fact
that AFL could discover overflow errors, which Rust normally only protects
against in Debug mode.

~~~
pslam
> 2\. Run-time panics. This includes "index out of bound" errors, integer
> overflow errors (in debug builds only), and assertions inserted by the
> programmer. This is Rust's second line of defense, so to speak.

I still argue, to every Rust user I meet, that they should turn on overflow
checking in Release builds too.

I strongly disagree with the idea that it's a Debug feature only. If it's not
always on, then it's not actually a language feature. The performance hit, if
you selectively opt-out for profiled critical paths (and thoroughly review
them), is usually very small.

In my opinion the default should switch to "opt-out" instead of "opt-in" for
Release builds.

~~~
ekidd
> I still argue, to every Rust user I meet, that they should turn on overflow
> checking in Release builds too.

Yeah, I remember when the debug-mode overflow checks were added right before
Rust 1.0. There wasn't enough time remaining to thoroughly measure the
performance impact of these checks in release mode or to implement compiler
optimizations to reduce the cost. So IIRC, the decision was made to _allow_
integer overflows to panic, and to turn on checks in debug mode so that the
ecosystem wouldn't come to depend on the absence of checks.

But AFAICT, the debug-mode checks are basically still a placeholder for some
indefinite future version of Rust that wants to fix this issue properly, if
the performance cost can be made low enough. And in the meantime, you can
still turn the checks on for release builds if the performance hit is
acceptable for your application. I'll consider your recommendation when
building binaries in the future!

~~~
pslam
> if the performance cost can be made low enough

I think that's the wrong way to look at it. How about instead:

"Checks can be disabled if the safety cost is worth the performance increase"

The "performance case" is a false baseline. You never had that performance in
the first place, because your code didn't work.

~~~
kbenson
It's a fine line to walk. Performance is a deal breaker for a lot of people,
whether it rightly should be of not. It doesn't matter how much safer Rust is
if it's not used much.

If your goal is purely to make Rust as safe as possible, then you should
advocate the view in your comment.

If your goal is to increase Rust adoption, and/or decrease the use of less
safe languages for development in the future (in whichever prioritization of
the two you like), then IMO the route described in ekidd's comment seems the
quickest route there, IMO.

~~~
pslam
> If your goal is purely to make Rust as safe as possible, then you should
> advocate the view in your comment.

> If your goal is to increase Rust adoption, and/or decrease the use of less
> safe languages for development in the future (in whichever prioritization of
> the two you like), then IMO the route described in ekidd's comment seems the
> quickest route there, IMO.

My view is this is a false dilemma.

The "performance" case is not performance, because the benchmark is _broken
code_. It does not work, because it is not secure/safe/reliable. It is
incorrect to compare the performance of code compiled without overflow
checking against code which is compiled with it enabled. They simply aren't
the same program.

A correct performance comparison would be a program with manually inserted
overflow checks everywhere, against a program with the overflow checks done
automatically (by compiler).

Or, a performance comparison where code is selectively opted-out, where a
critical path bottleneck is identified.

The same can be said of the performance comparison of "array index bounds
checking". One program works, the other does not, so it isn't a fair
comparison.

~~~
kbenson
> A correct performance comparison would be a program with manually inserted
> overflow checks everywhere, against a program with the overflow checks done
> automatically (by compiler).

 _Correct_ is relative to the goals of the person assessing the comparison. If
that person values performance over safety to enough of a degree, them a small
performance loss will outweigh safety (regardless of whether you or I think
this is a useful way to compare). If your goal is get people using Rust, and
you believe people value performance, you can't ignore this and expect things
to turn out how you like. This is the balancing act of a community and of
advocates, making hard choices about what they would like to do, and what they
feel is needed due to the constraints of reality.

> The same can be said of the performance comparison of "array index bounds
> checking". One program works, the other does not, so it isn't a fair
> comparison.

No, for some subset of array bounds checking, one works and the other doesn't.
For the rest, they both work, because it isn't out of bounds. Whether you
think people should program defensively and assume their code could fail, some
will instead choose to assume they can deal with that as the programmer and
choose to do so because it is more performant. You can either write those
people off, or meet them half-way, and hope to escort them the rest of the
way. Rust apparently chose the latter.

------
cm3
Any idea as to why the following requirement? This will limit Afl.rs users
quite a bit.

    
    
        afl.rs needs to compile against a version of LLVM that matches rustc's.
        The easy solution (if you can wait on a slow build) is to build rustc
        from source and put it in your PATH. Then afl.rs's build script will
        find llvm-config automatically. Otherwise, the environment variable
        LLVM_CONFIG should hold the path to llvm-config when you build afl.rs.
    

I was under the impression that Afl can test any application that takes stdin.
I'm underinformed for sure, so what's the idea behind explicitly adding code
to support Afl fuzzing?

~~~
frewsxcv
Author here.

AFL is coverage-guided fuzz testing (fuzz testing that relies on code
coverage).

Someone (not me) wrote an "LLVM pass" that loops through the generated LLVM IR
for any LLVM-compiled program and injects the necessary AFL instrumentation
such that AFL can work on it:

[https://github.com/mirrorer/afl/blob/master/llvm_mode/afl-
ll...](https://github.com/mirrorer/afl/blob/master/llvm_mode/afl-llvm-
pass.so.cc)

Since Rust uses LLVM when compiling a binary, all afl.rs does is incorporate
this instrumentation and AFL just needs to be run on the resulting binary.

There are multiple versions of LLVM though. Rust is compiled using a specific
version of LLVM. Since the LLVM pass above is reading the generated LLVM that
Rust produces, the versions can't be too far off otherwise the LLVM pass might
see some unrecognized symbols.

The README in the project suggests just compiling Rust which also compiles
LLVM, then you'll have a version of LLVM that is guaranteed to be the same
when you setup the instrumentation.

I _think_ it might be sufficient to update the documentation to suggest the
user of afl.rs just download and use LLVM 3.8 since this is the major version
that Rust uses internally. I need to do more testing to confirm this will work
for everyone, then I can update (and greatly simplify) the README.

~~~
jdright
Not sure about this, but isn't this something that could be integrated into
rustc (or cargo at least) to be something ready to use as bench/test?

~~~
frewsxcv
This is what I'm striving for:

    
    
        #[fuzz]
        fn test_fuzz(bytes: Vec<u8>) {
            ...
        }
    

This would live alongside other test cases and everytime fuzzing is desired,
one could just do `cargo fuzz`. Relevant issues and a pull request:

[https://github.com/frewsxcv/afl.rs/issues/24](https://github.com/frewsxcv/afl.rs/issues/24)
[https://github.com/frewsxcv/afl.rs/issues/31](https://github.com/frewsxcv/afl.rs/issues/31)
[https://github.com/frewsxcv/afl.rs/pull/46](https://github.com/frewsxcv/afl.rs/pull/46)

There's still a little more work to do though. Mainly, I want to compile AFL
as a part of the workflow (see the afl-sys crate in the repo) so the user
doesn't need to manually install AFL to use `cargo fuzz`.

------
cpach
Nice! It’s already proven to be useful:
[https://github.com/frewsxcv/afl.rs#trophy-
case](https://github.com/frewsxcv/afl.rs#trophy-case)

~~~
legulere
And all the bugs just caused panics, while they could have been exploitable in
C.

~~~
fulafel
The ASN.1 one is a different kind of crash though still sounds controlled
(SIGILL).

~~~
eddyb
That's LLVM's abort intrinsic - no a lot of those left around, I believe OOM,
panicking while panicking and stack overflow use it or used to use it.

------
pjmlp
> Nightly build of Rust

Oh well....

~~~
frewsxcv
Regarding using stable builds of Rust with afl.rs: If I could, I would.

afl.rs uses a Rust compiler plugin to register the LLVM pass needed to
instrument Rust programs so that AFL can run on them:

[https://github.com/frewsxcv/afl.rs/blob/master/afl-
plugin/li...](https://github.com/frewsxcv/afl.rs/blob/master/afl-
plugin/lib.rs#L7-L16)

Compiler plugins in Rust are currently unstable, and as far as I know, are not
on a path to stability in the near-term.

[https://doc.rust-lang.org/book/compiler-plugins.html](https://doc.rust-
lang.org/book/compiler-plugins.html)

~~~
Manishearth
It doesn't need to use llvm_mode. You can do what gcc does and have an afl-gcc
binary which mucks with the assembly instead of actually inserting hooks
smartly. This isn't as good, but works without nightly :)

~~~
frewsxcv
Huh, wasn't aware that's how afl-gcc worked. I'll have to look into it
sometime, thanks for making me aware of it.

------
loktarogar
I expected this to be something about Australian football, but this is pretty
good too

~~~
frewsxcv
Yes, the shared acronym makes Googling for anything AFL related rather
annoying. The results are usually a mixture of Australian football, American
rabbits, and fuzz testing.

