
Remote Code Execution on a Facebook server - phwd
https://blog.scrt.ch/2018/08/24/remote-code-execution-on-a-facebook-server/
======
mattrobenolt
Hey, developer of Sentry here. We're submitting a patch that prevents showing
any settings on this DEBUG page.

I'd like to mention that we never suggest running Sentry in DEBUG mode, nor do
we document how to do this. Sentry does use Django, so it's pretty easy to put
pieces together and figure out how to do it.

So while we can help obfuscate this specific issue, running in DEBUG mode
always has the potential to surface insecure information due to the nature of
it.

Our patch to suppress this information entirely can be found here:
[https://github.com/getsentry/sentry/pull/9516](https://github.com/getsentry/sentry/pull/9516)

------
matharmin
This is why you (looking at frameworks) should never use a format that may
contain code to store data, especially when the client has control over that
data (even if signed). The same vulnerability has occurred in almost every
language/framework that does this, including Rails and Java-based ones. Just
use something like JSON, which completely avoids code execution
vulnerabilities like this. Except of course for the early JavaScript JSON
parsers that just used eval for parsing...

~~~
collinmanderson
Yes, Django realized this 7 years ago [0], and JSON has been the default for
the past 5 years[1], it's just that facebook was using an old version of
Django, and probably not the default.

[0] [https://groups.google.com/d/msg/django-
developers/YwlZ9m9k1b...](https://groups.google.com/d/msg/django-
developers/YwlZ9m9k1bE/P-bJqoatgH8J)

[1]
[https://github.com/django/django/commit/b0ce6fe656873825271b...](https://github.com/django/django/commit/b0ce6fe656873825271bacb55e55474fc346c1c6)

~~~
js2
Sentry still requires Django 1.6:

[https://github.com/getsentry/sentry/blob/ea8fe10d117f5325f9e...](https://github.com/getsentry/sentry/blob/ea8fe10d117f5325f9e457f4ca727c82c4d63557/requirements-
base.txt#L14)

~~~
dbrgn
Whoa, really? Django 1.6 has been out of extended support since April 2015, if
I read
[https://www.djangoproject.com/download/](https://www.djangoproject.com/download/)
correctly...

~~~
the_mitsuhiko
We maintain our own patches. We want to upgrade but because the migration
framework we use does not support newer versions of Django finding a good
upgrade path has been suboptional. It's likely we're going to just make the
migration framework compatible with newer Djangos (work is under way).

------
Areading314
This is a great concrete example why you should never run debug mode on a
public server. Django can only do so much for redacting private info. This is
also a great example of how insecure pickle is!

~~~
Someone1234
I know this is going to get some jeering, but that's one nice thing about
.Net's machine.config <deployment retail="true" />, you designate the machine
itself as a non-development environment and tracing, debug output, and so on
are disabled for all .Net/ASP.Net applications.

That might not work for all edge cases, but broadly there's a lot of machines
which are only for non-development/production code, and a system-wide setting
makes you a lot safer.

~~~
gandreani
Just a thought. This might actually be trivial to implement in a unix
environment from the administration side of things. I'm not 100% sure but all
child processes inherit the env vars from the parent correct? So setting
`environment=production` high up in the process tree should make it available
to all processes.

There's still chances of this getting overridden down the line and all apps
have to conform to one style but at least it's possible?

~~~
Someone1234
Absolutely. But none of the major players are looking for it, and they all
have unique ways of designating the machine that way.

There's no specific technological challenges, this is entirely political,
getting half a dozen or more different projects with different priorities to
check the same variable for the same purpose.

------
cc-d
Facebook joins Patreon in the "why somebody should make sure our python web
framework debug mode isn't enabled in prod" club.

~~~
makomk
Patreon's screw-up was a lot more embarassing though - they apparently left an
actual Python shell exposed to the web for at least a week _after_ someone
warned them about it, and their entire user database was exfiltrated and
posted on the net as a result.

~~~
meowface
Yep. Every company will face security issues; it's unavoidable. But what
happened to Patreon should make people seriously question trusting them with
your personal information or money. (They probably use a 3rd party payment
processor who has much better security practices, but still. Also, that 3rd
party doesn't do you much good if an attacker with control of your production
web/application servers or CDNs is intercepting credit card form data before
it's sent off.)

Facebook has had vulnerabilities and exposures, but nothing like that.

------
tekromancr
Some surprising takeaways for me; Facebook uses a Django app in their
infrastructure. That app was in debug mode, revealing server secrets. That app
was also configured to use the Pickle based session storage, leading to one of
the few serious RCE vulnerabilities in Django.

I'll have to remember this next time I think an exploit scenario is too
unlikely.

~~~
antoncohen
Facebook runs the largest deployment of Django in the world -- Instagram.

[https://instagram-engineering.com/web-service-efficiency-
at-...](https://instagram-engineering.com/web-service-efficiency-at-instagram-
with-python-4976d078e366)

------
gandreani
Nice job! I also really appreciate the lack of memes and very concise format
of this blog post

~~~
konraditurbe
Also the title. Clear and concise without being click-bait (such as Facebook
RCE for fun and profit, all your Facebook belong to us, etc...)

~~~
alrs
Those kinds of titles are really tired, but clickbait is unfair. "X For Fun
and Profit" can be found in g-files back to the late '80s, a time well before
clicking.

[http://www.textfiles.com/phreak/](http://www.textfiles.com/phreak/)

~~~
kashyapc
The point is, "for fun and profit" is such an overused and utterly boring
cliché. Meaningful titles are pleasant to read and shows that the writer has
put some effort to bring clarity into what they're trying to convey. (As
someone who sits on a major open source conference talk panel, I _cringe_ when
I see one of these clichés slapped into the title without much thought. I
politely suggest to rephrase to convey more "signal" in the title.)

~~~
alrs
I'm in absolute agreement. Nerd-culture has been on autopilot for decades and
needs to find its next level.

~~~
sdegutis
This is a really interesting topic to me, because I've felt a pull toward
things like this myself and seen it for decades. For example, the top of the
README in my Zephyros project[1] was very playfully done, and very well
received too.

People are naturally more lighthearted and organic than the sterile software
and documentation we tend to write, and we're more social too, wanting to
connect with others even if it means through a simple in-joke or catch phrase.

[1]
[https://github.com/sdegutis/zephyros](https://github.com/sdegutis/zephyros)

~~~
kashyapc
I am not saying to not be human and take all the zest out of the writing.

FWIW, I suggest the book _On Writing Well_. William Zinsser has valuable
advice on how we can still retain humanity in our writing without being dull
and lackluster when dealing with dry and "sterile" topics like software.

------
beefhash
> _Quoting the Sentry documentation, system.secret-key is “a secret key used
> for session signing. If this becomes compromised it’s important to
> regenerate it as otherwise its much easier to hijack user sessions.“; wow,
> it looks like it’s a sort of Django SECRET-KEY override!_

One wonders why that is even there. Was Django's own session code not good
enough?

~~~
the_mitsuhiko
`system.secret-key` is a value from the options system that gets propagated to
different parts. It actually just sets the `SECRET_KEY` value in the settings
file for all intends and purposes which has different consumers.

This abstraction exists because some options can be set from the admin UIO.

------
smsm42
So, summarily we have:

1\. Enabling debug mode in production

2\. Running publicly-accessible app with publicly-accessible crash screens
without any monitoring system noticing it's happening

3\. Relying on auto-cleaning in debug facilities to sanitize security
information (never works)

4\. Using over-powered serialization protocol which allows for code execution
for storing user-accessible data

5\. Thinking that merely signing content prevents abuse of (4)

Not bad.

~~~
TheDong
Note that 5 is true though. Using a secret to sign or encrypt a cookie does
normally work, and it's a common practice. Usually the impact of the secret
leaking is that you can impersonate anyone, not that you can run arbitrary
code, but the practice of using a session secret is common and not a bad
practice nor broken inherently.

3 as well I think is unfair. That isn't something facebook implemented or is
relying on; it's just the default behavior of django's debug stack. That's
entirely on django to do that and lull people into a false sense of security
in some cases (though it also probably helps in many cases too, so it might be
okay).

The real issues are 1 (leaving debug mode on by accident), 2 (not noticing 1),
and 4 (which is a django issue I think, not a facebook one).

~~~
smsm42
> Using a secret to sign or encrypt a cookie does normally work

If the secret key is not compromised. So you have to ask yourself - why you
send to the user some info that is so sensitive that needs signing? Why not
just keep this info to yourself and send an opaque ID instead? Yes, I know
there are issues with it too, but at least _this_ issue is not there.

> 3 as well I think is unfair. That isn't something facebook implemented

I didn't say it's Facebook fault - though ultimately, of course, it is as much
as if you run certain software on your servers and do not configure it
properly, it's your fault. So there's a fail in having security key in a place
that's so easily accessible that debug mode dumps it without even asking. Not
necessarily a direct Facebook fail, but a fail.

~~~
nijave
See JWT. You can make stateless apps easier without worrying about a trip to
the database to grab session info

[https://12factor.net/disposability](https://12factor.net/disposability)

If you want disposability with the ID method you need some sort of datastore
or cache to contain session info

------
Grollicus
This is a good example why you need regular pentests in big companies.
Everyone (should) know that using pickle is insecure and everyone (should)
know that django debug should be False in production. Still, if the numbers
get large enough someone will miss something.

~~~
Polycryptus
The use of Pickle isn't uncommon for session cookies in Python apps, from what
I've seen. Pickle isn't really a problem unless you end up unserializing
untrusted data... which a sign+encrypt scheme is supposed to ensure doesn't
happen. You just can't leak the secret key or you're in trouble.

Though, there's no excuse for leaving Django debug on in production.

~~~
smsm42
I'd say it's a bad idea anyway - why you need to trust the user with anything
that needs pickle (as opposed to much more primitive format) to unserialize?
If you ever have a reason for non-opaque-id cookies at all, it should be very
simple. If you stuff very complex objects that require native serialization
into user-side storage, it's probably bad idea regardless of security
implications.

------
QuadrupleA
Great article. Big frameworks like Django are nice to get an app started
quickly, but if you use them long-term it really pays to study how they work
under the hood, read some source code, etc. Although "don't run debug mode on
production" is probably in the first page of the tutorial :)

------
rhacker
Congrats on the bounty payment! That has got to be one of the most concise
blog posts too.

------
amelius
The fact that the machine has a hostname "*.thefacebook.com" doesn't imply
that it also runs software of the "Facebook" social media software. So not
sure how much impact this exploit would have had.

~~~
bluntfang
I don't think anyone will ever know how much impact, but it implies that
Facebook is not good at security.

~~~
tetha
No one talks about the working parts of security. Like, in this case, having
the application on a separate box, and having that box separated by vlans from
important things.

~~~
bluntfang
I don't think the solution is to put things you don't care about behind vlans
and having applications on separate boxes. This box is/was an attack vector.
It was holding secrets. Secrets provide other attack vectors.

~~~
tetha
It is - it is defense in depth. The application should be secure, and there
should be steps to secure it. Especially because a secure application can
protect a less secure network setup.

But on the other hand, the server or the network should not trust the
application to be secure at all. The infrastructural setup should assume the
application to be an <exec($_POST['do_me']);>. And that's why the application
should be isolated on a system level, on a network level, and as much as
possible. That's the good part I mean - the part that worked.

------
green_on_black
He got $5k for an arbitrary remote execution bug? What a rip-off.

~~~
0x8BADF00D
He should have gone to the black market, better yet sat on it. How long did it
take Facebook to come forward with its user privacy violations?

~~~
chias
What would you guess the going rate on the black market would be for an RCE on
some error-log collection box that happens to be in use by Facebook? My guess
is "less than $5000".

~~~
brainkim
The only code they executed was "sleep(100)" If they started dumping env
variables/snooping around they would have done a lot more than $5000 worth of
damage.

~~~
throwawaymath
How are you quantifying that?

------
mandeepj
> scanning an IP range that belongs to Facebook (199.201.65.0/24)

ping -4 facebook.com

results in 157.240.18.35. Maybe, author used some other way to get those IPs.
Can anyone throw a light on this?

~~~
jelly
Facebook, like many large internet companies, buys their IP blocks outright,
so they show up under their AS number [0]. Facebook seems to have 3 AS numbers
[1,2,3] and that IP appears in [3]

[0]
[https://en.wikipedia.org/wiki/Autonomous_system_%28Internet%...](https://en.wikipedia.org/wiki/Autonomous_system_%28Internet%29)

[1] [https://bgp.he.net/AS32934](https://bgp.he.net/AS32934)

[2] [https://bgp.he.net/AS63293](https://bgp.he.net/AS63293)

[3] [https://bgp.he.net/AS54115](https://bgp.he.net/AS54115)

------
Cthulhu_
Wow, a fix in <24 hours, that's pretty impressive.

~~~
jrowley
I mean the fix is toggling a single environmental variable from True to False,
on a system that isn't normally accessed by customers, so the risk is really
small in rolling out the change.

~~~
lathiat
Sometimes you’re lucky if a company reads your report in this time but of
course I would expect and we generally see much better from the likes of
Facebook etc

~~~
jrowley
You're right, being able to read, triage and act on something in such a
massive system is quiet the accomplishment.

------
BFatts
Great article...

One suggestion: You use "However" quite a bit. Not sure if you intended to
show your thought process as it evolved, but that is the feeling I got.

------
nickthemagicman
So they were pickeling EXECUTABLE objects in the session and storing it in the
users browser cookie? Interesting. Nice find.

~~~
ebikelaw
There is no such thing as non-executable pickle. Pickle is not safe and must
not be used for anything, ever.

~~~
nickthemagicman
Wow. I have to look into it. That sounds wildy unsafe

------
Kiro
> Pickle is a Python module used to serialize data, but contrarily to JSON or
> YAML, it allows to serialize objects properties but also methods. In most of
> the cases, this is not a problem, but one can also serialize an object with
> code in the __reduce__() method, which is called when the object is
> unpickled.

Why does it run __reduce__?

~~~
detaro
if it exists, __reduce__ is run when pickling the object and returns code that
will be run when unpickling. It allows to completely customize how the object
is re-created on the other side, which might be needed e.g. when the type is
defined in a native extension (at least the docs name this use case).

------
srcmap
Other than fixing the Django source code , is there any OS level mitigation
techniques that can detect and prevent such security vulnerabilities?

I am thinking something like selinux, docker or chroot - a bit like internal
firewall for Django (or any other webapp).

Any suggestions on the links to latest best practices?

~~~
tptacek
Of course. You can lock the process down so that it can't make unexpected
system calls. If you deploy in a modern container environment, you can also
use container networking to drastically limit what the application environment
can talk to on the network. Though it's a less potent mitigation than seccomp
and container isolation (and one you get for free once you deploy in a
container), you can also limit filesystem access. If you run the application
under a non-privileged uid, these mitigations combined can raise the bar
somewhat for privilege escalation and pivoting after compromise.

Of course, in a sense, Facebook appears to have accomplished something simpler
simply by sacrificing an instance to this application and putting it on a
lonely isolated VLAN.

The other responses to your question are pretty weird, since it's obvious that
there are things you can do to mitigate the possibility of your Django program
literally calling execve or whatever.

~~~
GordonS
> You can lock the process down so that it can't make unexpected system calls

Huh, this isn't something I've ever come across before.

Off the top of my head, I guess it would be possible on Windows using a kernel
mode driver, but that's pretty hardcore, and really easy to get wrong.

I know you can easily _audit_ syscalls on Linux with auditd, but haven't seen
preventing them before.

Is this an option on both Linux and Windows, and is it commonly used?
Interested to know more!

~~~
tptacek
The Google search you're looking for is [seccomp docker].

(Docker is the most typical way this gets deployed but not the only way.)

------
mandeepj
> I found a Sentry service hosted on 199.201.65.36

I do not remember on top of my head now but I think there are few scanning
software to find all the running apps on a remote machine. If you are aware
then please share

~~~
bluntfang
[https://builtwith.com/](https://builtwith.com/)

~~~
mandeepj
Shoot. It did not strike me. I thought it was good only for web apps and not
in-background running processes like Pickle (a binary protocol )

------
protondonor
Great hunt!

------
jpmoyn
Great, concise article! But the real question is how much did they pay you for
the bounty??

~~~
lvh
Article has a timeline specifying $5000.

------
bayesian_horse
I can get remote code execution on an Amazon Server (AWS). Do I get a cookie?

------
exikyut
So, this was simply taking advantage of a crash-prone webapp running on a
debug-enabled Django instance using Pickle session serialization, and more
specifically this was only possible because __Django didn 't redact the stored
secret key used to sign serialized inputs out of the crashdump information!__

Did the author tell Django about this yet, or is this a (possibly
unintentional) 0-day?

Besides the above interestingness, the morals of this story I get are

\- Stay persistent and leave your scanners running; you never know what new
things will turn up.

\- Crashdumps _are_ interesting

\- Yay, $5,000!

\- Middleware and frameworks will always clash in useful and interesting ways?

~~~
chatmasta
Maybe I misunderstood the article, but I thought it said that Django _does_
strip this information, and the Sentry app went out of its way to store the
secret key in the SENTRY_OPTIONS payload. This custom, non-Django code
effectively circumvents Django’s protections, making the bug the
responsibility of Sentry, not Django.

~~~
exikyut
Ah. You're right, I completely got this bit wrong. Thanks.

Wait, so that means Sentry kind of has a vulnerability.

~~~
the_mitsuhiko
Sentry does secure `DEBUG` mode. It's for development only and must not be set
by customers. That said, we will try to make it less easy to set by accident:
[https://github.com/getsentry/sentry/pull/9516](https://github.com/getsentry/sentry/pull/9516)

------
Alex3917
In contrast, I submitted a bad vulnerability in Facebook’s password reset
feature yesterday that lets attacker’s send password reset PIN numbers to
email addresses the user doesn’t necessarily control. The security team said
to works as designed so they’re not going to fix it.

Basically if someone requests a password reset on your account then the PIN
number gets sent to all email addresses associated with your account, not only
the primary one. This is an issue because many people have one locked down
email address for things like registering accounts, but others they use to
talk with people, delegate to their staff, use with CRM apps, etc. (But you
still need your everyday email addresses linked to your account so that people
can find you by email, see your email on your profile, etc.)

The FB security team just says that delegating your email address isn’t secure
so it’s not their problem. Like no shit, that’s why it’s a vulnerability. But
for some reason the FB security team thinks it’s a good idea to let anyone
immediately bypass 2FA and hijack your account.

~~~
oihoaihsfoiahsf
I tend to agree w/ the FB security team here. Don't list email addresses owned
by an adversary in your account. :-/

~~~
Alex3917
> Don't list email addresses owned by an adversary in your account.

I mean if you're delegating your email address, your staff aren't adversaries.
But they shouldn't be able to, say, drain all your retirement accounts either.

Just because I want people who search for alex.krupp@gmail.com to be able to
find my Facebook account doesn't mean I want password reset requests sent
there. It wouldn't be at all unreasonable to send them there if that was
explained in the UI, but just immediately sending a password reset pin to a
non-primary email address without any warning is crazy. At least wait a few
days if the user doesn't take any action after it's sent to their primary
address.

~~~
wycy
Even if they changed it so that you could select where to send the password
reset, rather than having it go by default to all accounts, that still means
anyone with delegate access to the email address could go and request a reset
on your behalf.

~~~
Alex3917
> that still means anyone with delegate access to the email address could go
> and request a reset on your behalf.

That's how it should work. You shouldn't have to remember which email address
you signed up with in order to request a password reset. But that's fine as
long as the pin number only gets sent to an account that's locked down to a
degree that's appropriate relative to the assets under protection.

