Very much so. Even really common code has issues found via fuzzing on a regular basis. In the past six months I've reported NULL pointer deferences printing fingerprints of SSH keys, segfaults in the GNU awk parser, and most recently segfaults when parsing HTML files with w3m.
Most of the time setting up fuzzing is pretty simple, but even updating programs to read from STDIN rather than sockets is usually possible if you're careful and patient.
The hardest part is waiting for the damn things to run. Right now I've had a fuzzing session going against the text-based browser lynx for 4 days, and still counting. (Specifically looking to see if it can be crashed when converting HTML to text, as is often done in mutt, etc, via 'lynx -dump').
Would it be possible to distribute the fuzzing on a small cluster or some cloud platform thing?
Stop writing AND READING, as what's read can be corrupted and yet written or used elsewhere.
I know what BUG means. There's a reason it exists: to make sure that code in an invalid state doesn't do something really dangerous. assert() is very useful.
See Vegard's discussion with Theodore Ts'o about this: http://marc.info/?l=linux-ext4&m=144898400422842&w=2
"Unfortunately, company policy prohibits me from sharing the actual code." (http://marc.info/?l=linux-ext4&m=145007745502639&w=2)
We have "Time to first bug" for a lot of file systems (ext4, btrfs, hfsplus, NTFS) covering a wide range of OSes & platforms ... and yet not a single word about ZFS?
Come on Oracle... you can do better than that.
Chris Mason wrote btrfs when he started at Oracle (not Sun):
ZFS is not included in the Linux kernel. Much of this work was about building the infrastructure to be able to perform this kind of test.
Trying to do it for file systems that aren't included in Linux is a lot of effort for much less gain.
I guess I should bump reporting/fixing the issues I found on my todo list.
I was able to set it up in a few hours for a moderately sized library. Along with valgrind, I was able to find and fix all of the bugs it uncovered after over a CPU month of testing.
I used it to create a fuzzer for CPython  and it didn't take terribly long to get something going. Majority of my time's been focusing on new test cases.
There's a trade off between development cost, performance, functionality, and correctness. Writing a filesystem with reasonable performance and that never crashes and never does the wrong thing is hard.
No, but our fuzzers were worse, so we didn't know :-)
That would be absolutely unreasonable. Filesystems can be malicious, for example Stuxnet transmitted via USB flash drive.
For example (on a completely hypothetical filesystem/disk), lets say you want to open a file. You read the entry in the filesystem metadata that tells you on what sector that file begins and it gives you a value of 5005. Even if the checksum is correct, the filesystem code still needs to check that a) 5005 is actually within the range of valid sectors, b) that it actually makes sense as a sector value (maybe files must always start on a multiple of 4), c) that the data at that sector actually looks like a file, d) etc.
If you don't update the checksum as well when generating the test data then you limit the amount of depth in the testing since most errors would get caught immediately at the checksum validation stage.
Of course that means your mutations have to be "fair" in the sense that your new values should still be valid. Because as you stated, just stomping on blocks and fixing up their checksums seems like a silly test.
If you assume that, you don't need checksums in the first place.
Other than duplicating the pages (which is certainly a valid approach) how else can you recover from a data stomp?
A reverse time-sort would be
HFS+ & ReiserFS (25s)
Even in the Btrfs case, it may just be a function of the code having lots of awareness of where bodies are buried, so it hits BUG rather than continuing in a bad way and blowing up later. But as yet not enough testing to figure out all the path ways for hitting those bugs. It may explain the edge cases that still pop up with Btrfs, where I go "umm yeah I use this every day and never experience anything like that" but another user suddenly hits something that's a "wow" moment.
You can find the source code here: https://github.com/thoughtpolice/afl/commit/e54c0237e934d734...
It should not be difficult to update this to work on more GCC versions (I only tested on GCC 4.8.x), but that will take some #ifdef'ery. Porting to newer AFLs should be relatively trivial.
EDIT: I initially wrote this for no particular reason, mind you, other than to play around with writing GCC plugins, and the result wasn't so bad, modulo non-existant documentation. I also thought it would be nice to have an identical equivalent to 'afl-clang-fast' for GCC ('afl-gcc-fast'), in the hopes that perhaps one day the hacky, sed-inspired backends could be removed from afl. I initially wanted to use this on a POWER machine as proof of a portable GCC plugin for afl, although I lost interest in porting to a newer GCC, before losing access to the machine. Watching afl fly on 176 cores was fun, though.