

Pwn2Own winner tells Apple, Microsoft to find their own bugs - Sandman
http://www.computerworld.com/s/article/9174120/Pwn2Own_winner_tells_Apple_Microsoft_to_find_their_own_bugs

======
tshtf
I've done some of work in the area. Most of the companies that use, or even
require fuzzing in their software development procedures, have a certain
problem: the same people who write the parsers write the fuzzers. When they
write these fuzzers, they make some too-strong assumptions on the formats of
the data involved.

Writing smarter fuzzers may be needed to fully exercise certain file formats,
say where certain blocks of data need to match a CRC elsewhere. But smart
fuzzers fail where the authors fail to account for edge conditions. Having
both in your toolset is invaluable.

Regardless, it's sad for companies like Apple that they're not doing any of
this.

~~~
tptacek
Dumb fuzzers find things in Microsoft code all the time, and Microsoft spends
huge on external teams to write fuzzers.

~~~
davi
So what's your analysis of why the external teams are ineffective?

~~~
tptacek
They're not.

~~~
davi
OK, so Microsoft finds lots of bugs by this technique (according to you), but
then this guy says that he finds still more bugs, with (according to him)
pretty primitive methods. And then the guy says,

'"I found bugs, lots of bugs. That was both surprising and disappointing." And
it also made him ask why vendors like Microsoft, Apple and Adobe, which have
teams of security engineers and scores of machines running fuzzers looking for
flaws, hadn't found these bugs long ago.'

So given your assertion that these teams are very effective at what they do,
why do you think this guy is very easily (according to him) able to find
additional bugs?

There's a mismatch between what you're saying and the gist of this article.
I'm not saying you're wrong -- far from it. I'm looking to you for a
resolution to the discrepancy.

~~~
jballanc
This guy is in the security business. If you think your software is secure,
he's out of work. If he creates the impression that he didn't even have to try
that hard to find these vulnerabilities, then you get paranoid and want to
hire him.

To be clear, I'm not saying the vulnerabilities weren't real, but to buy his
story about them being "easy" to find, well...

~~~
tptacek
I tend towards the "Charlie is just better than most of us" view of the world,
too.

------
nfnaaron
"He went into the project figuring that he wouldn't find any vulnerabilities
with the dumb fuzzer. "But I found bugs, lots of bugs. That was both
surprising and disappointing." And it also made him ask why vendors like
Microsoft, Apple and Adobe, which have teams of security engineers and scores
of machines running fuzzers looking for flaws, hadn't found these bugs long
ago."

Or, maybe the companies have already found those bugs and more, but focus
their efforts on bugs that surface in the wild and don't spend resources on
the others. If they fix all the bugs, then they spend money on some bugs that
would never have surfaced. Just a guess.

~~~
tptacek
Waiting to fix _bugs_ until they become an issue is smart, if a bit ruthless.

Waiting to fix _security flaws_ until they become public is negligent.

From firsthand experience with at least one of the companies implicated in the
recent stories: they aren't sitting on things. What Charlie Miller is finding
is news to them.

------
tkiley
This is somewhat equivalent to WikiLeaks' release strategy: tell the
organization that you have evidence of XYZ problem, but don't describe the
exact nature and scope of the evidence, then pressure them to come clean and
fix the problem themselves; hopefully this leads to more comprehensive
"cleanup" efforts that have a larger positive long-term effect.

How relevant is the WikiLeaks strategy in the field of security?

~~~
tptacek
It depends on the vendor. There are vendors for which the "announce and hold
back" strategy will probably work: they're the ones who market based on
security. It is a Big Deal if someone has an IIS remote, or a reliable Flash,
or Apache/WebSphere.

We'll see how big a deal it is for Google and Apple. Marketing based on
security means more than just talking about security on your web page. It
means making a business out of people who buy based on product security. I
like my Mac, and I like Apple, but I have no illusions about the resilience of
OS X.

------
djb_hackernews
Anyone able to give more detail? Maybe not on Millers specifics but how does
inserting random data into a program become an exploitable vulnerability?

~~~
evgen
<http://en.wikipedia.org/wiki/Fuzz_testing>

------
pxlpshr
He sounds naive and his logic is quite the cliche. Surprised he didn't close
it with a line from the hacker's manifesto.

I'm not sure what it is with some people acting like the smartest guy in the
room while refusing to share their knowledge for the betterment of society.
Entrepreneurs benefit tremendously from mentors and advice from those who've
made it. I'd be surprised if he didn't learn a vast portion of his trade from
the openness of information. I guess he feels empowered w/ his :15 minutes and
this is how he leverages it? Weak^10.

+++the pxlpshr+++

~~~
potatolicious
I didn't get that sense from the article - he's not refusing to release the
info for his own benefit, or to be a dick; he's ostensibly doing it because he
feels like this just encourages the "we'll fix stuff when other people find it
in the wild" mentality, which doesn't help make software more secure in the
long run.

He wants software producers to actively try to break their software so that
his ostensibly simplistic tools can't find wide open holes like this.

I'm sure some egotism rolls into it somewhere, but I don't think it's
altogether a bad stance... you shouldn't be waiting for your users to tell you
where your security holes are.

~~~
pxlpshr
_you shouldn't be waiting for your users to tell you where your security holes
are._

while I agree with that ideology, in reality you just can't assume that to be
the case when you have multiple people touching the code, and features added
regularly that may inadvertently introduce holes. what software has ever been
100% bullet proof?

Also, I think that statement largely contradicts the notion of 'release early,
iterate often' considering it's grounded on the notion of integrating feedback
and bugs reported by users. Are major corporations held at a different
standard simply because they have more resources? Even with thousands of
employees, it's still a glass box compared to the millions that will use it.

