While I find the reported results impressive, I'm sad to see another publication about a closed-source application.
Nobody can reproduce the results or place their work on top of that. Research was not supposed to work that way (at least in my little universe :-), yet it gets published by the IEEE (but what does that mean nowadays?).
I don't think they're doing anything unethical by not releasing their source code, but the state of the art sure could progress a hell of a lot faster if releasing code were the norm. Just look at all the papers that have built on KLEE since it was open-sourced.
Could I? How long would it take? Would I really get the same results or just similar results? If it doesn't work, how can I find out where the bug is (the paper, my interpretation, my implementation)?
Compare this to physics: if you drop this ball from that height under these circumstances, it will take that long. Everybody (with the right equipment) can reproduce this result quite easily. Not so in this case, which ironically could be even more reproducible than any physics experiment if it wasn't closed source.
The important result of the paper isn't the specific list of bugs they found, which after all: (a) depends on the software they fed to the tool and (b) is verifiable without static analysis (just try to trigger the bug). The important result is the validation of the approach, which makes the paper more important than the tool.
Seems to be that rewriting the software is healthier, and a better analog to physics experiments.
When you reproduce that physics result, you're using a different ball in a different location and dropping it from a different structure.
If you simply took their program and re-ran it, it's like taking the exact same ball they originally used, climbing the exact same structure they used, and dropping it in the exact same spot. When you then get the same result, it doesn't tell you much, because you don't know if the result is coming from the ball, the structure, the location, or just a general universal phenomenon.
The point is not to run more of them, but to run them better.
What good does it do to run identical software ten times? These computer-thingies are pretty deterministic. You're basically guaranteed to get the same result each time, but you have no idea if the result is because of the technique being presented, a quirk of the implementation, or just a bug.
Of course, access to source allows running better experiments! Writing the same algorithms from scratch is a clear waste of time! Please stop defending silliness.
I can start right away changing any one of these to find out, instead of spending time to try to write a program that may well not exist in the first place.
(In this particular case, I admit that posting 1.2k bugs hints to its existence.)
Sometimes research is closed source due to where the funding comes from, I'm working a project that's been funded by two different entities over the years - one of which won't even release the dataset we've been using - so when trying to open source my code I get stuck.
> This research was supported by a DARPA grant to CyLab at Carnegie Mellon University (N11AP20005/D11AP00262), a NSF Career grant (CNS0953751), and partial CyLab ARO support from grant DAAD19-02-1-0389 and W911NF-09-1-0273.
I don't know much about these grants, but DAPRA and NSF sound pretty public to me (read: the public pays for something only a handful of people get access to). Correct me if I'm wrong, though.
The paper documents their approach pretty thoroughly, and honestly, if it were me (and I'm a damn-the-torpedos-full-disclosure-open-access-everything kind of guy), I'd think twice about publishing a program that would give the kiddies something to do on a wet, wet, wet, day.
Well, the authors stated that they will publish the bugs.
Further, publishing a program like this has the potential of finally having software that is not easily exploitable (eg. by requiring every Debian software to pass that program).
Now that it is closed source, only a selected set of people will be able to benefit from it.
You're right, but so is the parent commenter; there is a popular meme that security bugs in general are worth large amounts of money, but in reality only a small subset of bugs command real money.
You could be a better ambassador of our industry by explaining the misconception instead of being condescending. And we wonder why developers think we are just arrogant jerks.
IMHO anyone who contributes to the state of the art with regards to public security tooling is doing the world a favour. Even if the majority of these applications are not well privileged, and therefore of dubious value, applying the same technical knowledge elsewhere could earn them cash. I still applaud their efforts.
The value of a vulnerability is linked to its usefulness in accomplishing the goals held by those paying for the exploit. Based on that information, it should become obvious that the only bugs worth large sums of money are bugs that result in escalated privileges on the system or arbitrary code execution. Bugs that crash services also have value for DDoS attacks, but they're not as valuable as the two previously mentioned types.
While 1.2k bugs sounds like a lot, we can't make an inference about the severity of the bugs based solely on the number of them. Also keep in mind that the Debian repository contains thousands of packages. From the Debian 7.0 Wheezy release page:
"* more than 36,000 other ready-to-use software packages, built from nearly 17,500 source packages"
The best we can infer that given 1.2k bugs, there is some statistical probability that any number of them may be severe. Without studying the data, we can't say what that probability might be though.
Where does that 1.2K number come from, though? The abstract mentions only 29 (still a lot) and the results list 'only' 29, too, two of which where 0days. I admit I haven't read the whole article thoroughly.
The paper has been published a while ago. We have been improving Mayhem quite a bit since then, and we recently ran it on debian binaries. It found many crashes, that are not necessarily exploits. 1.2K is the number of bug reports we are going to submit to the Debian bug tracker.
As a long time Debian user I would like to thank you for choosing Debian as a target platform. The maintainers might not appreciate the bugs in the short term but in the long term this is great for everyone involved. Was there any reason why you chose Debian? I imagine bts made things easier for bug wrangling/submission.
The response has been fairly positive in general, and it's great to see that some of the bugs have already been fixed!
All Mayhem developers are using Debian, and our cluster is running Debian as well. We've been using it happily for years. That's why we chose to analyze it first. I believe we will also have an impact on other distributions since bugs are getting fixed upstream.
I see that it is for moreutils. I am having the hardest time using the interface for forallsecure.com. It seems that you can not search by package. When I browsed through the list I figured I would have come across moreutils around page 49/51 but I did not see it. I am tempted to download:
Our interface is severely lacking. The website is mostly for internal use at the moment. I'll try to add a program search tonight to make it more usable.
As a matter of fact, most programs are not linked to packages, and most packages are not present, including moreutils. You won't find it even by crawling.
Actually, I have not received the one for moreutils yet.
However, we got 1 report for the Debian Installer, which I hope means only 1 problem was found.. which is nice, as the d-i code is mostly not written with this kind of robustness in mind.
I have not finished reading the paper but after the second sentence in the abstract it seems that the authors clearly define what they consider to be a bug:
Every bug reported by MAYHEM is accompanied by a working
shell-spawning exploit. The working exploits ensure soundness
and that each bug report is security critical and actionable.
Reddit's reverse engineering discussion: http://www.reddit.com/r/ReverseEngineering/