
Going down the rabbit hole with go-fuzz - Metalnem
https://mijailovic.net/2017/07/29/go-fuzz/
======
morecoffee
Compare Go fuzz to a java fuzzer and there are some interesting differences.
For example, in the Go fuzzer, it is looking for panics, _not_ errors. An
error is okay, it just perhaps isn't interesting. With a Java fuzzer, it is
looking for Exceptions, which aren't as clear. If you get an IOException, is
that working as intended, or an actual failure? It's much harder to tell. I
attribute this not the the fuzzer, but to the language design decisions.

I tried using Go Fuzz on a small parser I wrote, and was surprised that it
found a bug. I was quite pleased, but the input was actually valid, my code
was just wrong. One shortcoming of Fuzzing is that it doesn't find Type II
errors. It can't find invalid inputs that do succeed. For example, if the
input to my parser was actually invalid, but the code didn't crash, the fuzzer
would happily continue on. I'm glad I fuzzed my code, but no errors found
doesn't mean no errors present.

~~~
pcwalton
1\. Errors can be problems in Go as well if they're swallowed and not handled
(for example, writing "os.Chdir("/foo");" will silently cause any errors to be
ignored). Note that a Java fuzzer that detects uncaught exceptions _would_
notice a failing chdir() written in this way, while the Go fuzzer would not
notice this issue.

2\. In Java, it's easy to tell whether, say, an IOException was working as
intended or was an actual failure. Just check whether it was caught or not.
Granted, that won't handle apps that try to catch wide swathes of exception
types, but Go apps can abuse Go's "recover" facility in the same way.

~~~
jen20
There is a useful linter named `errcheck` for Go which looks for swallowed or
shadowed errors [1]. It's included in `gometalinter` [2] (which is worth
running as part of the CI checks on any Go codebase).

[1] [https://github.com/kisielk/errcheck](https://github.com/kisielk/errcheck)
[2]
[https://github.com/alecthomas/gometalinter](https://github.com/alecthomas/gometalinter)

~~~
throwawayfuzz
Unfortunately, there are a number of correct uses of go that swallow errors.
For example, fmt.Println returns an error, but you're just not going to see
that error handled. Sometimes, it doesn't matter whether something has failed,
because there's not a useful, proportionate thing you can do to respond to
that failure. And it's not possible to know from machine examination which
cases are which. I'm sure for some programs (grep?) a failure to emit a line
to stdout is cause for failure. But for most it is not.

~~~
favadi
`errcheck` ignores unhandle error value of `fmt.Println`.

------
wyldfire
> If you didn’t fuzz it, you can’t say it’s correct.

> — Dmitry Vyukov

I would amend this slightly to state instead "If you didn't fuzz it, you can't
say it's [robust]." To invoke correctness seems to suggest that you can do
some amount of verification using fuzzing, but that's not really the case
(unless you have a reference implementation).

But without fail, fuzzing often evokes a handful of shallow defects. And once
those are addressed you can usually dig deeper and find more. I'd be
astonished to find a library or program written in C/C++ that has never been
fuzzed and undergo fuzzing without any newly discovered bugs.

~~~
robryk
> To invoke correctness seems to suggest that you can do some amount of
> verification using fuzzing, but that's not really the case (unless you have
> a reference implementation).

Not necessarily. You just need to be able to verify correctness of the output
and fuzz the following program:

    
    
        y := f(x) 
        if (y,x) are not correct then crash
    

E.g. if you want to verify correctness of the compressor, try to check if a
known-good decompressor can always decompress its output to its input. This is
simpler than having a reference implementation.

