
MAYHEM – automatically ﬁnding bugs and shell-spawning exploits in binaries [pdf] - wslh
http://users.ece.cmu.edu/~sangkilc/papers/oakland12-cha.pdf
======
wslh
Debian mailing list: [http://lists.debian.org/debian-
devel/2013/06/msg00720.html](http://lists.debian.org/debian-
devel/2013/06/msg00720.html)

Reddit's reverse engineering discussion:
[http://www.reddit.com/r/ReverseEngineering/](http://www.reddit.com/r/ReverseEngineering/)

~~~
dfc
The link to reddit mentioned in parent is to the RE subbredit. The link to the
actual MAYHEM discussion on reddit is:

[http://www.reddit.com/r/ReverseEngineering/comments/1h4vs4/r...](http://www.reddit.com/r/ReverseEngineering/comments/1h4vs4/reporting_12k_crashes/)

------
lysium
While I find the reported results impressive, I'm sad to see another
publication about a closed-source application.

Nobody can reproduce the results or place their work on top of that. Research
was not supposed to work that way (at least in my little universe :-), yet it
gets published by the IEEE (but what does that mean nowadays?).

~~~
tptacek
The paper documents their approach. You could reproduce their findings by
building your own offline/online symbolic execution system.

~~~
lysium
Could I? How long would it take? Would I really get the same results or just
similar results? If it doesn't work, how can I find out where the bug is (the
paper, my interpretation, my implementation)?

Compare this to physics: if you drop this ball from that height under these
circumstances, it will take that long. Everybody (with the right equipment)
can reproduce this result quite easily. Not so in this case, which ironically
could be even more reproducible than any physics experiment if it wasn't
closed source.

~~~
mikeash
Seems to be that rewriting the software is healthier, and a better analog to
physics experiments.

When you reproduce that physics result, you're using a different ball in a
different location and dropping it from a different structure.

If you simply took their program and re-ran it, it's like taking the exact
same ball they originally used, climbing the exact same structure they used,
and dropping it in the exact same spot. When you then get the same result, it
doesn't tell you much, because you don't know if the result is coming from the
ball, the structure, the location, or just a general universal phenomenon.

~~~
luke-stanley
Rewriting from scratch doesn't help run more experiments, it slows things
down.

~~~
mikeash
The point is not to run more of them, but to run them better.

What good does it do to run identical software ten times? These computer-
thingies are pretty deterministic. You're basically guaranteed to get the same
result each time, but you have no idea if the result is because of the
technique being presented, a quirk of the implementation, or just a bug.

~~~
luke-stanley
Of course, access to source allows running better experiments! Writing the
same algorithms from scratch is a clear waste of time! Please stop defending
silliness.

------
contingencies
Gee, they could have sold those for a lot of money. Well done researchers, you
have a strong moral compass.

~~~
dguido
Yes, all of those vulnerabilities in non-privileged applications like 'ls' are
worth millionssssss.

~~~
jerf
Yes, and then they combine them with a kernel exploit and, uh, suddenly your
"unprivileged process" turns out to not be so unprivileged after all.

Resist the temptation to marginalize security bugs. They don't exist in
isolation.

~~~
tptacek
You're right, but so is the parent commenter; there is a popular meme that
security bugs in general are worth large amounts of money, but in reality only
a small subset of bugs command real money.

------
lysium
Impressive!

Where does that 1.2K number come from, though? The abstract mentions only 29
(still a lot) and the results list 'only' 29, too, two of which where 0days. I
admit I haven't read the whole article thoroughly.

~~~
bugfinder
The paper has been published a while ago. We have been improving Mayhem quite
a bit since then, and we recently ran it on debian binaries. It found many
crashes, that are not necessarily exploits. 1.2K is the number of bug reports
we are going to submit to the Debian bug tracker.

~~~
dfc
As a long time Debian user I would like to thank you for choosing Debian as a
target platform. The maintainers might not appreciate the bugs in the short
term but in the long term this is great for everyone involved. Was there any
reason why you chose Debian? I imagine bts made things easier for bug
wrangling/submission.

~~~
bugfinder
The response has been fairly positive in general, and it's great to see that
some of the bugs have already been fixed!

All Mayhem developers are using Debian, and our cluster is running Debian as
well. We've been using it happily for years. That's why we chose to analyze it
first. I believe we will also have an impact on other distributions since bugs
are getting fixed upstream.

~~~
stass
Thanks for the great work!

Do you plan to release the tool one day? It will be a great asset to other
operating systems developers as well.

------
HockeyInJune
The paper above and the post to the debian mailing list describe two different
uses of the Mayhem tool. One with exploits, one without.

------
maerF0x0
Misleading title-- abstract says 29 exploitable vulnerabilities, not 1200 .

~~~
wslh
Look at the reddit thread above.

------
lampe3
keep in mind: bug != security hole

~~~
dfc
I have not finished reading the paper but after the second sentence in the
abstract it seems that the authors clearly define what they consider to be a
bug:

    
    
      Every bug reported by MAYHEM is accompanied by a working
      shell-spawning exploit. The working exploits ensure soundness
      and that each bug report is security critical and actionable.

~~~
bugfinder
Mayhem changed quite a bit since the publication of the paper. It now reports
all crashes, and therefore bugs are not necessarily security critical.

