
How I found a Remote Code Execution bug affecting Facebook's servers - reginaldo
http://www.ubercomp.com/posts/2014-01-16_facebook_remote_code_execution
======
reginaldo
Hi. I'm the one who found the bug. Facebook's side of this story is at
[https://www.facebook.com/BugBounty](https://www.facebook.com/BugBounty)

~~~
oneeyedpigeon
I'm curious: how much time would you say you worked on researching and
identifying this bug? BTW, I don't begrudge you the payout one little bit, no
matter how long you spent on it; such an amount is change down the back of the
sofa for facebook, and the potential impact of the bug means they got a great
deal!

~~~
reginaldo
Well, I originally found the OpenID bug in 2012, but hadn't noticed Facebook
was vulnerable until very recently. After I found their OpenID endpoint, the
hardest part was getting them to make me a Yadis discover request. Then I had
to squash a little bug in the exploit. Most of the time was spent re-reading
the OpenID spec. I'd say total amount of work (including the time it took me
to write the post) was about 2 days.

As I said in the post, I already had a strong suspicion that, once I could
read files, escalating to RCE would be easy. But I decided not to do it
without permission and they fixed the bug very quickly. As much as I'd loved
to actually see the output of an _ls_ or something like that, I think I made
the right call.

------
chmars
The payment was apparently USD 33'500.

Not being in the bug bounty business, I had expected a higher payment.

~~~
hueving
Not bad for 2 days of work.

~~~
nikcub
The part of the work you don't see is the hours, days and months spent,
usually unpaid, spent auditing code to find the bugs.

It is like the anecdote of Tesla and Ford and knowing where to put the X[0],
you aren't paying for time or manual labour - bug value is derived from how
much damage it can cause, what its worth to Facebook to not be exploited and
what the exploit is worth to the bad guys on the black market.

[0]
[http://www.snopes.com/business/genius/where.asp](http://www.snopes.com/business/genius/where.asp)

~~~
epmatsw
He says that it took him about 2 days total

[https://news.ycombinator.com/item?id=7106482](https://news.ycombinator.com/item?id=7106482)

~~~
reginaldo
The 10 years before those 2 days helped a lot too...

------
cheald
Since this sounds like it affects a lot of people in a lot of places, I went
about auditing my own code and found that if you're using libxml2 >= 2.9.0,
you should be safe, _unless_ you're explicitly requesting entity expansion:

[https://mail.gnome.org/archives/xml/2012-October/msg00045.ht...](https://mail.gnome.org/archives/xml/2012-October/msg00045.html)

For the nokogiri users, there are a couple of proofs-of-concept in this
ticket:
[https://github.com/sparklemotion/nokogiri/issues/693](https://github.com/sparklemotion/nokogiri/issues/693)
\- they should be patched with modern versions of nokogiri and libxml2, but if
you're running older versions, you might want to verify their behavior before
someone else does it for you.

------
zobzu
A lot of people think its not enough money. In this case it appears to be
about 30K USD.

Here's an issue with paying more: the security engineers employes are paid
around 100 to 150K a year.

Imagine you'd get paid 100K for a big exploit. It wont be valuable to be a
security engineer anymore (since you can't be paid additionally for finding
the bugs), it's better to be unemployed and spend your time finding the bugs.

I think that's one of the main issue, at least until bugs are extremely,
extremely rare (which really, they aren't - these news are really the tip of
the iceberg).

------
rigelt
Congratulations Reginaldo - great read, great story from A-Z. Don't worry
about the payout. Apparently you got brains and a white hat, that's all you
need for a carefree life.

------
nradov
For those using the Java stack we have found HP's Fortify static analysis tool
helpful in automatically detecting XXE vulnerabilities.

[http://h30499.www3.hp.com/t5/HP-Security-Research-
Blog/Q2-20...](http://h30499.www3.hp.com/t5/HP-Security-Research-
Blog/Q2-2013-HP-Fortify-Software-Security-Content-Update/ba-p/6119297)

~~~
billyhoffman
HP's WebInspect, a blackbox testing tool, also can find XXE's. However, as the
OP shows, XXE's can be tricky and involve a lot of nuance to cox them out.
General dynamic testing tools aren't as good at uncovering XXE's as static
analysis tools.

Disclaimer: I used to work on WebInspect's audit engines

------
raverbashing
So, why is it possible to have XML files reading arbitrary stuff from whatever
place again?

XML is _data_. If your application needs to send a request for reading a file
through XML this should really be explicit, not relying in a "permission
happy" XML library, no?

~~~
billyhoffman
Yes, XML is data, but it allows you to specify where other data is located
that should be included when processing it. These links can be to addition
data, or to definitions of how to process the data in the document (XML
entities). It's a simplification, but it's as if XML can have #includes, where
the source of the #include is a URL, and can even be a file:/// URL.

So the attack looks like this: Server takes input from evil user, inserts it
into an XML document in memory. The input is malicious, and contains not only
XML data, but XML directives to include other documents, specifically
/etc/passwd on the location machine. The XML document is processed, the
contents of /etc/passwd are automatically read by the XML parser/processor.
However the data is not in the correct format, and the XML parser/processor
spits out a detailed error message, showing the data that could not be
processed/parsed, which is the contents of /etc/passwd.

Make sense?

~~~
raverbashing
Sure it makes sense, what doesn't make sense is the library reading anything
it is thrown at it.

I'm thinking there should be a "root path" for the library to be able to
access files. Sure, you want to include "base.xml" it's in a specific
directory and the library is allowed to read only that, and no
"../../../../etc/password" tricks.

------
14th
This was very interesting. What are good reasorces to learn more about this
kind of stuff?

~~~
ladzoppelin
Its hard and I wish I had more info. One thing I do know is that blogs are the
worst form of information. I would think being a whiz at Javascript and PHP
would be a prerequisite but maybe you just need a good understanding of
different specs and protocols.

~~~
hueving
You don't really need to be a master of a specific language. You just have to
understand general programming concepts. The hard part is getting the right
approach to searching for vulnerabilities. Learning to recognize everywhere
the target system is taking input and estimating what it is doing with that
input is where the skill is at.

------
meshko
Wait, so how do you escalate this to RCE?

~~~
meowface
Don't think either party has disclosed that.

>and due to a valid scenario he theorized involving an administrative feature
we are scheduled to deprecate soon, we decided to re-classify the issue as a
potential RCE bug.

I imagine it might be some feature that could maybe be triggered internally
through a file:// or [http://localhost/](http://localhost/) URL, and in doing
so gain access to an interface that can issue shell commands. That's pure
speculation though, and I'm probably way off.

~~~
reginaldo
You're actually pretty close.

------
markshepard
Just curious. Would you be interested in checking (for a fee ofcourse) other
web server products that use similar technology. If so please let me know and
I would love to connect with you.

~~~
hundchenkatze
From the article, he says just shoot him an email.

    
    
        If you find this interesting and want to hire me to do a security 
        focused review or penetration testing in your own (or your 
        company's) code, don't hesitate to send me an email at 
        reginaldo@ubercomp.com.
    

*edit: formatting

~~~
markshepard
Thanks. I had not noticed it. I will definitely ping him!

------
sekasi
Extremely interesting (and open) write-up reginaldo. Congratulations on the
payout.

------
D3_4dl1N3
Very interesting, but there is a good chance that other people have stumbled
upon, but they preferred to wear a black hat. Anyway, hats off.

~~~
cji
from the FB post "In parallel, other members of the Security team investigated
the logs corresponding to this issue and confirmed that it had not been
previously exploited or used maliciously."

------
Thaxll
I'm very surprised that the php code isn't running in a chroot / jail...

------
MrGando
Great work dude, keep em' coming.

------
lijman
Fb is so stingy with payouts.Bugs of a website with over a billion users can
be sold for millions.Is fb ignoring this fact?

~~~
billyhoffman
People here seem to have a strongly misplaced expectations about what bug
bounties pay. Vulnerabilities in web apps/servers tend to be worth less than
vulnerabilities in client computers for a few reasons

First, web app vulns are usually specific to a single site. (Unless obviously
you find an issue in a common underlining framework, say, a session fixation
attack in how PHP or ASP.NET handles sessions).

Second, and much more importantly, the vast majority of site's don't have
financially actionably information. Unless you handle banking/credit card
info, I am limited in what I can do to extract value from the server (compared
to a compromised client). There aren't that many vectors to extract value.

-Dump their list of usernames/passwords? Ok, maybe some of those will also be used on other banking or commerce sites, but I have challenges/risks actually getting money out. And if I want stolen credit card numbers I can just buy them in card forums.

-Serve sleazy advertising? Ok, possible, but ad's are a crappy business to be in and its definitely a high volume/long time approach (ask how well Huffpo pays its writers). You can try affiliate spamming/stuffing, but again, not huge value. Both ads and affiliate approaches are dependent on how much traffic the server you hacked gets. Low traffic, you make no money. The more traffic, the larger the site, the smarter/better equipped the IT/security team. How long do you really think an Alexa top 10 or top 100 site won't notice an IFRAME pointing to .ru or .cn?

-Mining cryptocurrency? Not financially viable

What usually happens when a server is compromised is that an exploit kit is
installed and it's used to attack the visitors (specifically exploit a
vulnerability in the client). And so we are back to attacking clients to
extract value over attacking (most) websites. Why do this? Simple:

\- There are orders of magnitude more desktops/browsers than web servers. \-
They are running tons of diverse plugins and software so the attack surface is
much larger. \- Most of that software will be out of date and have known
vulnerabilities. \- Very few of these clients are "managed" by a personal IT
person like a web server. The user is far less likely to notice anything bad.

All of that mean I can reach more targets, compromise a larger number of them,
_and_ hold them for longer. Why is this better than pwning a server? Because
lots of scenarios to extract value that don't work on a handful of web servers
_do_ work when I have thousands and thousands of compromised clients:

-Show them ads -Stuff affiliate links -Changing their DNS settings and MitM all their traffic (bank.com? Why that's right over here!) -Keylog them to actually steal financial data, credits cards, bank logins, etc -Use them to send spam -Use them as a botnet to DDoS people and get paid protection money -Mining? perhaps?

To put this in prospective, very smart hackers doing crazy stuff to break out
of Chrome's sandbox and exploit clients are getting $50-$100k in public
contests like Pwn2Own. Getting $35,000 for a RCE is pretty awesome

~~~
nikcub
Completely agree. To add, IMO the most valuable application of this exploit
would have been to deliver drive-by malware. There are some groups that pay
$0.5-3 CPM to get their Javascript onto much less targeted web sites. The
Yahoo malware would have cost nearly double that.

Getting drive-by traffic is one of the most expensive pieces of the puzzle for
malware groups. Last time I checked the forums, a thousand visitors was
selling for around a dollar, and that isn't even well qualified traffic.

Having access to Facebook and over a billion pageviews per hour would be worth
millions to any group who is capable of handling that type of volume. If they
were smart about it, they could probably get away with it for up to a day (the
Yahoo malware was active for a day and they didn't obfuscate it much).

Back of the envelope value is around $1M per hour, and that doesn't include
the premium for the higher quality of traffic, but does assume you find a way
to inject across all the servers and somehow not display it to Facebook
internal IPs.

A big group with some fresh browser 0day would have loved to get their hands
on this.

~~~
meowface
Considering Facebook's good security team (and Yahoo's notoriously poor one),
I suspect that they'd probably catch the malware and perform some sort of
emergency action in under an hour.

A much more devious attack would be modifying some of the code to silently
siphon off login credentials, and grabbing the user database. Then once they
were satisfied with that they could go with the malware route.

~~~
nikcub
Facebook must be one of the only large sites that hasn't been hit directly
with serving malware, closest I can recall are incidents like this:

[http://www.f-secure.com/weblog/archives/00002172.html](http://www.f-secure.com/weblog/archives/00002172.html)

------
laxmanm1
It should be 100,000USD. Read this article of microsoft bug bounty program.
[http://blogs.technet.com/b/bluehat/archive/2013/10/08/congra...](http://blogs.technet.com/b/bluehat/archive/2013/10/08/congratulations-
to-james-forshaw-recipient-of-our-first-100-000-bounty-for-new-mitigation-
bypass-techniques.aspx)

~~~
billyhoffman
Not at all. Your link is for someone finding an entirely new way to bypass
protections/sandboxing in IE. That is an enormously impactful issue because it
affects hundreds of millions of desktop PCs. Comparing that to a RCE affect a
few web servers, even at a site as large as FB, is misplaced.

