
Launch HN: Fuzzbuzz (YC W19) – Fuzzing as a Service - evmunro
Hey HN,<p>We’re Everest, Andrei and Sabera, the founders behind Fuzzbuzz (<a href="https:&#x2F;&#x2F;fuzzbuzz.io" rel="nofollow">https:&#x2F;&#x2F;fuzzbuzz.io</a>) -  a fuzzing as a service platform that makes fuzzing your code as easy as writing a unit test, and pushing to GitHub.<p>Fuzzing is a type of software testing that generates &amp; runs millions of tests per day on your code, and is great at finding edge cases &amp; vulnerabilities that developers miss. It’s been used to find tens of thousands of critical bugs in open-source software (<a href="https:&#x2F;&#x2F;bugs.chromium.org&#x2F;p&#x2F;oss-fuzz&#x2F;issues&#x2F;list" rel="nofollow">https:&#x2F;&#x2F;bugs.chromium.org&#x2F;p&#x2F;oss-fuzz&#x2F;issues&#x2F;list</a>), and is a great way to generate tests that cover a lot of code, without requiring your developers to think of every possibility. It achieves such great results by applying genetic algorithms to generate new tests from some initial examples, and using code coverage to track and report interesting test cases. Combining these two techniques with a bit of randomness, and running tests thousands of times every second has proven to be an incredibly effective automated bug finding technique.<p>I was first introduced to fuzzing a couple years ago while working on the Clusterfuzz team at Google, where I built Clusterfuzz Tools v1 (<a href="https:&#x2F;&#x2F;github.com&#x2F;google&#x2F;clusterfuzz-tools" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;google&#x2F;clusterfuzz-tools</a>). I later built Maxfuzz (<a href="https:&#x2F;&#x2F;github.com&#x2F;coinbase&#x2F;maxfuzz" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;coinbase&#x2F;maxfuzz</a>), a set of tools that makes it easier to fuzz code in Docker containers, while on the Coinbase security team.<p>As we learned more about fuzzing, we found ourselves wondering why very few teams outside of massive companies like Microsoft and Google were actively fuzzing their code - especially given the results (teams at Google that use fuzzing report that it finds 80% of their bugs, with the other 20% uncovered by normal tests, or in production).<p>It turns out that many teams don’t want to invest the time and money needed to set up automated fuzzing infrastructure, and using fuzzing tools in an ad-hoc way on your own computer isn’t nearly as effective as continuously fuzzing your code on multiple dedicated CPUs.<p>That’s where Fuzzbuzz comes in! We’ve built a platform that integrates with your existing GitHub workflow, and provide an open API for integrations with CI tools like Jenkins and TravisCI, so the latest version of your code is always being fuzzed. We manage the infrastructure, so you can fuzz your code on any number of CPUs with a single click. When bugs are found, we’ll notify you through Slack and create Jira tickets or GitHub Issues for you. We also solve many of the issues that crop up when fuzzing, such as bug deduplication, and elimination of false positives.<p>Fuzzbuzz currently supports C, C++, Go and Python, with more languages like Java and Javascript on the way. Anyone can sign up for Fuzzbuzz and fuzz their code on 1 dedicated CPU, for free.<p>We’ve noticed that the HN community has been increasingly interested in fuzzing, and we’re really looking forward to hearing your feedback! The entire purpose of Fuzzbuzz is to make fuzzing as easy as possible, so all criticism is welcome.
======
tptacek
Can you talk a bit about what you're fuzzing for in Python programs? I feel
like I have a good understanding of what cluster fuzzing is accomplishing for
C/C++ libraries, but less clarity about the goals for managed languages.

~~~
evmunro
Sure! Some of the classes of bugs that remain low-hanging fruit for languages
like Python include slowness, hangs, panics, race conditions, assert failures,
excessive resource consumption and other Denial of Service attacks.

Other use cases include using fuzzing to compare implementations of libs that
require the same functionality, detecting invariant violations, testing
implementations that are meant to work together (i.e.
serialize(deserialize(x)) == x).

In general fuzzing C/C++ libraries for memory bugs is the most commonly
described use-case, but I think there are tons of fuzzing use cases that
haven't been thoroughly explored yet.

------
chubot
FWIW I separated my Oil shell parser into a standalone stdin/stdout filter,
which is ready to be fuzzed:

[https://github.com/oilshell/oil/blob/master/bin/osh_parse.py](https://github.com/oilshell/oil/blob/master/bin/osh_parse.py)

[https://github.com/oilshell/oil/issues/171](https://github.com/oilshell/oil/issues/171)

I'm already testing it by running it on more than a million lines of shell
[1], which I imagine should provide a very good starting point for the AFL
algorithm. I've only fuzzed one thing before but that's my understanding.

If anyone is itching to try out Python fuzzing, this might be a nice and
realistic intro.

I made note of Fuzzbuzz on the bug.

[1]
[http://www.oilshell.org/blog/2017/11/10.html](http://www.oilshell.org/blog/2017/11/10.html)

~~~
evmunro
Thanks for mentioning us on the issue! I'd love to help get that project up
and fuzzing

------
evmunro
P.s. Know someone who maintains an open-source project, written in C, C++, Go
or Python, that should be fuzzed? Please send an email to oss@fuzzbuzz.io.
We’d love to fuzz your code for free on our platform and make the world’s
open-source software more secure.

~~~
wyldfire
Can't they do this w/oss-fuzz already?

~~~
evmunro
They certainly could if their project is large enough! Every widely-used C/C++
project should use OSS-Fuzz, it's an awesome service.

We support a couple of languages that OSS-Fuzz doesn't (Go & Python as of
now), which is why I thought this was worth mentioning :)

~~~
treis
>We support a couple of languages that OSS-Fuzz doesn't (Go & Python as of
now), which is why I thought this was worth mentioning :)

I thought the main benefit of fuzzing was finding memory security bugs. If
your program is crashing or otherwise erroring out given crazy input that's
something you want to fix because it is potentially exploitable. With
Python/Go that's not really an attack vector. So what's the benefit from
finding out that some crazy input crashes my Python program?

~~~
evmunro
Memory security issues have been the main focus of fuzzing, but it's really
useful for other use cases as well, such as: slowness/hangs, assert failures,
panics, excessive resource consumption and DOS attacks. We've also done some
work with Go to detect race conditions while fuzzing.

You can also do differential fuzzing to compare 2 different implementations
that solve the same problem, or fuzz for invariant violations/assertion
failures. I think the possibilities extend far beyond just memory safety, and
I'm really looking forward to finding other areas in which fuzzing is
applicable.

~~~
treis
Have you guys run FuzzBuzz against FuzzBuzz? If so, how many bugs did you
find?

------
souprock
I know of a company, called Security Innovation, that tried this in 2002. It
went very badly at times. They added training and pen testing, and today they
bring in just $20 million per year.

They opened an office in Seattle to fuzz for Microsoft. As soon as they proved
that they could succeed, Microsoft hired away all the people, leaving the
company with a lease for an empty office.

Generally, companies don't trust outsiders and/or don't see a need. You're up
against internal politics too. People within the company don't want to compete
with you and don't want to be embarrassed by you.

~~~
tptacek
Companies don't trust "outsiders" to do security testing? Veracode was doing 9
figures a few years ago and was recently spun out from Broadcom for almost
$1Bn.

Also, 20MM/yr is not a revenue number to sneeze at. Enterprise security is a
huge and mature product space and most aspirants in it do not hit that number.
These companies aren't Uber, where every dollar coming in is going back out
the door with an extra couple dimes to boot.

~~~
eganist
Eeeep.

There's a substantial counterargument to this I need to type up. I understand
where you're coming from as someone who once ran a consultancy, Tom, but from
the perspective of someone who hires security firms and consumes their
services—and this is essentially a TL;DR for the opinion I need to flesh out
here—we don't do it because we trust. We do it out of necessity.

TODO: Bryant to flesh this out in between laundry rounds tonight.

~~~
tptacek
My name is Thomas.

~~~
eganist
Well, apologies are due as I can't seem to edit that out of my post now. Sorry
for making the assumption, Thomas.

But considering the gray patina my earlier comment has developed with time,
I'll withhold my point as there doesn't seem to be interest in hearing it.

~~~
tptacek
I actually didn't understand your comment and it was downvoted before I
responded (I didn't downvote it). I just wanted to make sure people knew how
to spell my name.

------
lumengxi
This looks great, congrats on the launch! All enterprise software companies
should take notes, having a "Buy vs. Build" section on the website is
incredibly useful and saves time on both sides.

~~~
andrei
Glad you found it useful - it's a topic that comes up a lot when we talk to
customers and we figured we should just be upfront about it.

Wasn't our idea though - we stole it from Labelbox [0] :)

[0] - [https://labelbox.com/buy-vs-build](https://labelbox.com/buy-vs-build)

------
benatkin
This is a great name, because at least 90% of your customers will find it
funny. Some friends in Boulder made a company called RaffleCopter, and its
meme-inspired name (based on ROFLcopter) seemed to help it get much more
attention than it drove away.
[https://www.rafflecopter.com/](https://www.rafflecopter.com/) Another good
thing about both names is they work even if you don't get the reference. I
think that could be a must.

I don't know much about fuzzing but I'm inspired to give your tool a try
if/when I get a chance.

~~~
andrei
We sort of picked the name as a joke at first, but we noticed that it was very
memorable (as meme-inspired names tend to be), so we decided to keep it. Don't
regret it, yet.

If you're interested in learning more, our docs [0] explain everything from
the ground up.

[0]: [https://docs.fuzzbuzz.io/](https://docs.fuzzbuzz.io/)

~~~
nemo1618
Did you consider "Fuzzbizz?" It's a truer Spoonerism, plus you get the
"business" pun. Although maybe that's outweighed by it no longer rhyming. :P

------
js2
Some feedback going through the docs. For each of the languages you
demonstrate a "BrokenMethod" but I don't understand what's broken about it.
e.g.:

    
    
        func BrokenMethod(Data string) bool {
            return len(Data) >= 3 &&
                Data[0] == 'F' &&
                Data[1] == 'U' &&
                Data[2] == 'Z' &&
                Data[3] == 'Z'
        }
    

What's broken about this? It returns true for strings that start with "FUZZ",
false otherwise, does it not? Python example:

    
    
        def BrokenMethod(strInput):
            if len(strInput) >= 2:
                return strInput[0] == 'F' and strInput[1] == 'U'
    
    

Other than not being idiomatic I don't see what's wrong with this method.

Next, it's not clear to me how you indicate success/failure of a test. Is
success just any program than exits 0 and failure any program that exits non-
zero? That would be my guess but the docs don't say.

Typo:
[https://github.com/fuzzbuzz/docs/pull/3](https://github.com/fuzzbuzz/docs/pull/3)

This page is missing a link to the find-your-first-bug-in-Python example:

[https://github.com/fuzzbuzz/docs/blob/master/getting-
started...](https://github.com/fuzzbuzz/docs/blob/master/getting-
started/introduction-to-fuzzbuzz.md)

The docs site loads slowly for me on an older iPad, and there's even a slight
delay on a recent Macbook. Looks like it's maybe a font loading issue? (Oh,
it's gitbook. How awful. I guess there's nothing you can do about that other
than use a different doc provider.)

~~~
evmunro
Thanks for the questions & feedback! Concise docs are really important so this
is all super useful. To answer your questions one by one:

1) The BrokenMethods are simple examples of programs that crash on buffer
overflows/index out of range errors. If you were to pass "FUZ" into the Go
method, it would check Data[3], thus causing a panic since there are only 3
elements in the string.

ninja edit: that python method in your comment IS a valid method with no error
- a bit of a brain fart on my end when writing out the docs. It's been changed
:)

2) In general a failure is any non-zero exit. We do this to be flexible in the
way you report bugs. For C/C++ and Python this is usually with assertions, and
in Go you can achieve something similar with:

    
    
      if !x {
        panic("Error")
      }
    

We also have other checkers or "sanitizers" that run with your code to look
for certain bugs. For C and C++ code we support tools like Address Sanitizer,
which report memory bugs like Heap Buffer Overflows and UAFs, and for Golang
you can choose to fuzz your code with a race condition checker. These are just
some of the examples of more advanced fuzzing methods we support, and we'll be
making nicer tutorials/screencasts to showcase those over the coming week.

3) Thanks for the fixes - much appreciated. And yeah, we know GitBook is
pretty slow, and we're in the process of moving to another docs provider.

If you've got any more questions please let me know!

~~~
js2
Great thanks for the answers. Maybe on the broken examples just put a comment
on the line that's broken so the reader doesn't have to expend mental effort
understanding why it's broken.

------
aboutruby
That's awesome! Would love to use it on popular ruby gems to make sure they
are secure (including Rails). Also having a free plan is perfect for
experimenting. I predict this company will be a huge success.

~~~
evmunro
Really interesting to see the desire for ruby support in this thread! It's
definitely on our roadmap.

Shoot me an email at everest@fuzzbuzz.io and I'll let you know when we launch
ruby fuzzing.

------
kbeckmann
This is awesome! I wish you all the best and hope that this takes off.

I am curious about how you use AFL under the hood - how do you scale? Do you
use a shared kernel and run a worker process for each physical core, or do you
do some virtualization or perhaps run with a kernel patch such as
[https://github.com/sslab-gatech/perf-fuzz/](https://github.com/sslab-
gatech/perf-fuzz/) ? My experience is that you will hit a wall pretty quickly
unless you start multiple kernels by using virtualization, or simply having a
very slow binary so you don't get a high number of execs/s to start with.

~~~
evmunro
We actually distribute the fuzzing workload across physical machines, for
precisely that reason. Each instance of AFL gets its own kernel & physical
core, and we use a staged synchronization algorithm to make sure all of the
machines' corpuses stay up to date.

All of this was done to try and keep the scaling as linear as possible, so
that when you double your CPU count you're doubling your execs/second as well.

~~~
kbeckmann
Thanks for your quick reply!

This sounds like a good solution. I was trying to solve this problem on my own
as well and ended up making a minimal kernel+afl image that I then boot
multiple times using deduplication features in order to save RAM (I don't have
256 gig ram like you do in a proper server). Each instance ends up eating
quite a lot of RAM even with a limited root filesystem so that's why I wanted
to keep it low. I'm on a 2990wx rig which was kind of a disappointment from a
fuzzing perspective because of the limited memory bandwidth, but that's for
another discussion.

Do think it would be worth trying out that snapshot approach that I linked in
the parent comment or it might not be worth it? I was thinking of rebasing
their patch onto the latest master and getting it to work again - sadly the
authors seem to have abandoned the project, at least there hasn't been any
public changes since a year back.

------
rixrax
Have you thought of integrating some form of exploitability analysis[0][1] for
the crashes|etc. fuzzing locates?

So let's say I upload some FOSS project and end-up finding some
crashes|potential vulnerabilities. Have you considered some sort of tie-
in|integration to bug bounty programs so that I could get a small pay-out
without having to go through the trouble of figuring out how exploitable a
given crash might be, and more importantly to actually have to deal with
trying to get the attention of the project?

[0]
[https://www.microsoft.com/security/blog/2013/06/13/exploitab...](https://www.microsoft.com/security/blog/2013/06/13/exploitable-
crash-analyzer-version-1-6/) [1]
[https://github.com/jfoote/exploitable](https://github.com/jfoote/exploitable)

~~~
evmunro
Yep, we're definitely going to integrate more automated analysis. As of now we
do some rudimentary analysis based off the type of the bug (Heap buffer
overflow, UAF), read/write size, and similar metrics, but we'll be adding more
advanced methods of categorization as the platform matures.

We've been thinking about the best way to use Fuzzbuzz to benefit the OSS/bug
hunting community, and the integration idea is a great one. We're also
providing free plans with extra CPU power for security researchers & bounty
hunters.

~~~
rixrax
Nice. I'm sure you've looked into various backends to use (in addition to
AFL). Just wanted to give a shoutout to radamsa[0]. My [somewhat not-up-to-
date] experience has been that it produced sometimes findings that AFL didn't
(because of e.g. different approach relative to infinite input space).

[0] [https://gitlab.com/akihe/radamsa](https://gitlab.com/akihe/radamsa)

In regard to CPUs - my laptop reports 4 CPUs, my workstation 16 - where the
value for someone involved in fuzzizng would come in my mind would be if you
could take away the hassle of scaling fuzzing 'transparently' to 100 or 1000
CPUs. What I am suggesting here is that on your pricing page you might be off
by factor of 100 in regard to what number of CPUs actually make offering
compelling to someone who would consider outsourcing their fuzzing
infrastructure.

~~~
evmunro
Radamsa is awesome! Definitely agree, and one of the goals for Fuzzbuzz is to
be able to hot-swap between fuzzing backends without any interface changes (or
to use all backends at the same time, to account for differences in findings).

re: pricing, we do offer infinite scalability in terms of CPUs, but that might
not be as clear as we'd like it from our pricing page. Or maybe I'm
misunderstanding you. Either way, if you have any more thoughts/suggestions on
pricing I'd love to hear it.

------
pests
I thought someone had finally done it. Fizzbuzz as a service.

This is even better.

~~~
stcredzero
Actually, asking someone to write Fizzbuzz as a service isn't a bad idea for
an interview question. And now that I've written this comment, it will cease
to be not a bad idea in about 30 minutes.

~~~
jetbooster
Well if you're going to do it as a service you might as well make it
enterprise ready. Allow me to introduce you to Java EE Fizzbuzz

[https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpris...](https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpriseEdition)

~~~
stcredzero
It's 2019. Should be implemented as a Lambda.

------
sk221
This looks awesome. Wish it integrated with Ruby projects.

~~~
evmunro
Ruby is one of the languages on the roadmap, but it's not super high-priority
right now, mostly because we haven't seen a lot of interest. If you have a
specific project/use case in mind I'd be interested to hear more about it.

~~~
mentat
Uploaded CSVs or PDFs would be interesting to me ([https://docs.ruby-
lang.org/en/2.5.0/CSV.html](https://docs.ruby-lang.org/en/2.5.0/CSV.html))
([https://github.com/gettalong/hexapdf](https://github.com/gettalong/hexapdf)).
There's a fair amount of parsing that's probably not been fuzzed
significantly.

------
tanin
This is such an awesome service.

Congrats Everest (and team) for making it easier to make software more secure!

------
tofflos
This looks very nice. I will definitely try it out once support for Java
becomes available.

~~~
andrei
Java is one of the next 2 languages we plan to support (hopefully in the
coming month).

If you send me an email (andrei@fuzzbuzz.io), I can let you know when we
launch Java support

~~~
kodablah
I couldn't find any good AFL-like fuzzers for Java, so I wrote one not too
long ago: [https://github.com/cretz/javan-warty-
pig](https://github.com/cretz/javan-warty-pig). Feel free to take it and
hack/reuse it.

~~~
evmunro
Thanks for the link! We've been looking at all the current AFL-like/AFL
wrappers for Java as we decide how best to implement Java fuzzing in Fuzzbuzz,
and yours looks pretty nice.

Definitely going to play around with this :)

------
technics256
As a somewhat newbie, how would I use this on say my javascript web app and
react native apps?

Thanks!

~~~
evmunro
Since the type of fuzzing you can do right now on Fuzzbuzz is language-
specific, you wouldn't be able to fuzz Javascript code.

We are in the process of building a fuzzer for generic web-apps, so watch this
space :)

------
rixrax
Godspeed!

I recall there is|was a company based out Santa Cruz, CA called Fuzz Stati0n
with a pretty similar concept. Might be good idea to ‘compare’ experiences.

~~~
grajaganDev
I am the founder of Fuzz Stati0n - now a product security engineer at Looker.

Best wishes and happy to talk.

------
yingw787
A number of questions:

Do you guys support fuzzing by protocols? Syscalls, REST, or SQL? It might be
faster to extend protocol fuzzing than fuzzing by language (I'm not sure
though). It'd be cool to have a fuzzer for Apache Calcite; it's a library to
slap on a SQL interface to your database.

Any plans to extend fuzzing to property-based testing?

Do you guys fuzz your fuzzer (dogfood)? Probably useful, but also funny :)

~~~
evmunro
We don't support protocol fuzzing yet, but it's definitely on our roadmap - we
wanted to start in an area that we felt was lacking the most, and then move
into other types of fuzzing. We do have some novel REST API fuzzing techniques
in mind that we're really looking forward to implementing as well.

We're also thinking about extending to property-based testing. There are some
really awesome hybrid testing tools out there, such as DeepState
([https://github.com/trailofbits/deepstate](https://github.com/trailofbits/deepstate))
which combines Symbolic Execution and Fuzzing behind one clean interface, and
we'd really like to push the boundaries of that type of testing.

And yep, we do! Everything is written in Go, which is part of the reason it's
one of the first languages we support.

------
drwl
A small nit: the video is set with these dimensions: width: 940px; height:
529px;

On scaled display it shows up fairly small and so it's hard to see what's
going on without going to full screen. As a quick fix could you enable the
full screen button. And as a longer term fix consider recording another
version where the windows are smaller or there's some sort of magnification?

~~~
andrei
My bad - I accidentally disabled the controls when I was setting it up. Should
be fixed now.

Also, yeah, we're not video experts, so we had a feeling the video might not
be the correct format. We just wanted to put something up that showcased the
platform without forcing you to sign up, but we'll definitely make sure our
next video is in a better format.

~~~
drwl
No problem thanks for addressing. Viewing it on youtube worked for me.
Congrats on your launch and best of luck!

------
jawns
I started learning about this type of testing by writing Hypothesis tests for
Python code:
[https://hypothesis.readthedocs.io/en/latest/](https://hypothesis.readthedocs.io/en/latest/)

One of the things that became a source of frustration is writing the specs
that define the shape of the inputs.

Does FuzzBuzz make that any easier?

~~~
evmunro
That's a problem that we've been thinking about a lot. The way our fuzzing
works right now is that your method consumes an array of bytes, which you can
then use to build up arbitrary structures. It's simple, but manages to be
really generic and flexible at the same time. Of course, it means you do have
to define what your inputs look like.

We do have plans to build some tools to make this easier. I'd like to see a
scenario where defining inputs is as simple as specifying the data types that
your code requires. (Or perhaps even automatic detection, for less complex
cases)

~~~
guidovranken
Some people use Protobufs to solve this, but when I work with C++, I use a
class that takes a (data, size) as an input (to its constructor), and
implements an overridable Get<T> method.

So you can do: auto s = datasource.Get<std::string>();

For each type I need I override Get<T> for that type. If the data source class
is out of data, I throw a specific exception, that I catch at the global
level. This works like a breeze and the advantage over protobufs is that it's
faster (no Protobuf parsing overhead, errors) and you never consume more than
you need.

------
bhargav_
This looks really cool! Fuzzing is something I've been looking into more and
more as I learn about it. I'll definitely keep this in mind for the future.
Have you guys considered Rust support?

~~~
evmunro
We have! afl.rs[1] is awesome, and seeing as it's found some interesting bugs,
I think Rust would be a great addition to Fuzzbuzz. It's on our roadmap.

[1] [https://github.com/rust-fuzz/afl.rs](https://github.com/rust-fuzz/afl.rs)

------
myroon5
Can you add support for social login like GitHub OAuth? Thanks!

~~~
evmunro
We're already sort of integrated with GitHub, since you can integrate your
projects to automatically pull updates from repositories, so GitHub login is
definitely on the roadmap!

------
webgoat
Why do this in the cloud? Wouldn't it make more sense to make standalone
software that does the fuzzing?

~~~
klohto
> It turns out that many teams don’t want to invest the time and money needed
> to set up automated fuzzing infrastructure, and using fuzzing tools in an
> ad-hoc way on your own computer isn’t nearly as effective as continuously
> fuzzing your code on multiple dedicated CPUs.

------
Kkoala
Sounds interesting! At least I learned something new, fuzzing.

------
kdmedev
Can this be done in Rust?

~~~
steveklabnik
Rust has a bunch of fuzzing tools, see [https://github.com/rust-
fuzz](https://github.com/rust-fuzz)

------
jimbo1167
Obligatory plug for the OG of Fuzz Testing, Barton Miller:
[http://pages.cs.wisc.edu/~bart/fuzz/](http://pages.cs.wisc.edu/~bart/fuzz/)

#OnWisconsin

------
maybeiambatman
Fascinating!

------
prcobol
wewlad!

