
The Great DOM Fuzz-Off of 2017 - robin_reala
https://googleprojectzero.blogspot.com/2017/09/the-great-dom-fuzz-off-of-2017.html
======
drfuchs
These results look pretty encouraging as to the general quality of the various
browsers, until you read the fine print: "Only security bugs were counted in
the results (doing anything else is tricky as some browser vendors fix non-
security crashes while some don’t)." So, the numbers they show don't count
random crashes, never mind incorrect behavior.

And who'd have thought that some vendors aren't even interested in fixing
"non-security" crashes; I wonder which vendors those are? And I wonder how
many "non-security crashes" their fuzzer found in each browser?

Even worse, they laud MS for the MemGC technology they use in their browsers,
but MemGc is just a last-minute hack to try to keep use-after-free bugs from
being security holes (by ignoring frees to anything that still has a pointer
to it), without having to actually fix them. So, lots more bugs that really
ought to crash instead go on to other undefined behavior, which hopefully
doesn't amount to a security problem (never mind simply incorrect behavior).

So, it seems that their fuzzer is finding lots and lots more bugs, and they're
just not telling us how many there actually were. Maybe the title should
indicate better that they're limiting themselves to security vulnerabilities.
I suppose there's a call for that sort of measurement, but it would be nice to
also hear about the general quality of the software under test. The notion
that they'd rate a browser that did exactly nothing as having a 100% perfect
rating of zero bugs kind of rankles.

~~~
hannob
> And who'd have thought that some vendors aren't even interested in fixing
> "non-security" crashes;

While I can't answer that I can tell you that I was very surprised Firefox
showed little interest to fix this:
[https://files.hboeck.de/crashff.html](https://files.hboeck.de/crashff.html)
(linux only, crash via notification api, bug open for a year)

~~~
opaque_salmon
It doesn't seem to crash my browser, but I'm assuming it's reproducible due to
the existence of that page.

~~~
ChrisSD
I can reproduce it on Firefox 55 and 56 in Ubuntu Mate, it crashes after I
allow the notification. It'd be useful to have a bug link so I can see what
the status is. It could be an Ubuntu issue for all I know.

~~~
ryacko
My guess it is for specific versions of the Gnome desktop.

------
pcwalton
Note that one of the problems was in Skia. Food for thought for those who
claim memory safety is unimportant in graphics code.

~~~
hannob
> Food for thought for those who claim memory safety is unimportant in
> graphics code.

I have never heard anyone claiming that. Why would anyone say such a thing?
Graphics libraries are a constant source of memory safety bugs.

~~~
pcwalton
Usually, but not always, I hear this from game developers.

To be fair, of course, I'm not doubting that they're correct when the
libraries are used in games. The problem is that increasingly graphics
libraries get used in security-critical contexts like browsers.

~~~
evmar
I bet there are a few security vulns lurking in games that involve user-
provided data (e.g. maybe you can customize your clan by uploading a logo).
The boundary between trusted and untrusted in games with mods is especially
porous ( e.g. [https://security.gerhardt.link/RCE-in-
Factorio/](https://security.gerhardt.link/RCE-in-Factorio/) ).

------
TheAceOfHearts
I wish they included a bit more information on the bug's severity. I've been
using Safari on macOS, and this might make me seriously consider dropping my
usage entirely in favor of Firefox.

What we need is stronger website isolation and permissions. Depending on the
site's functionality and how frequently I use it, I'll modify their
permissions. Some browsers allow you to restrict a couple features, but I'd
argue it's not granular enough.

Why should any random site be allowed to use WebSockets, WebRTC, WebGL, etc?
I'd want to turn off all those features and more on sites that I don't think
should have them.

I'm actually a bit annoyed with Safari 11, which removed the ability to
conditionally disable WebGL. In my experience, most websites that were using
WebGL had absolutely no good reason to do so.

------
Ruud-v-A
It would be interesting to know what the nature of these bugs was, whether
they are mostly memory safety issues or not.

I find it scary that for $1k one can find so many bugs in software exposed to
untrusted content.

~~~
benhawkes
We probably should have linked to this in the post, but you can see the
details of Ivan's findings on our public issue tracker:
[https://bugs.chromium.org/p/project-
zero/issues/list?can=1&q...](https://bugs.chromium.org/p/project-
zero/issues/list?can=1&q=label%3AFinder-ifratric)

As expected from DOM fuzzing there's lots of overflows, use-after-free, and
type confusion issues.

------
pbiggar
If Google needs Mac hardware to test Safari on, CircleCI has a Mac cloud they
could use :)
[https://circleci.com/mobile/osx/](https://circleci.com/mobile/osx/)

~~~
dmoy
Reading the article, it looks like they did test on Mac hardware, just doing
the initial distributed test on cheaper hardware first and then confirming
each one individually on more expensive mac hardware. Or at least that's how I
read OP.

~~~
pbiggar
Yes, I think that's right. I meant they could run the distributed test on the
Mac cloud.

~~~
microcolonel
It'd be way more expensive, they might as well just fuzz with WebKitGTK+ since
it produces the results they desire; then validate the crashes on Safari.

