
Update to Security Incident - rahuldottech
https://stackoverflow.blog/2019/05/17/update-to-security-incident-may-17-2019/
======
arciini
I think this is one of the best sets of responses to a security incident I've
seen:

1\. Disclose the incident ASAP, even before all facts are known. The
disclosure doesn't need to have any action items, and in this case, didn't

2\. Add more details as investigation proceeds, even before it fully finishes
to help clarify scope

The proactive communication and transparency could have downsides (causing
undue panic), but I think these posts have presented a sense that they have it
mostly under control. Of course, this is only possible because they, unlike
some other companies, probably do have a good security team who caught this
early.

I expect the next (or perhaps the 4th) post will be a fuller post-mortem from
after the incident. This series of disclosures has given me more confidence in
Stackoverflow than I had before!

~~~
mandevil
One major advantage SO has stems from its userbase: we are much less likely to
panic than the typical user of a system not focused on software development.

~~~
shay_ker
You say that, but the second highest comment on the HN thread for the initial
update was less than gracious:

[https://news.ycombinator.com/item?id=19936315](https://news.ycombinator.com/item?id=19936315)

~~~
brokenmachine
You can please some of the people some of the time.

------
akersten
Can't help but wonder - how obvious did the bug have to be that it was
deployed on May 5th and an attacker gained access the very same day? That's
pretty crazy - and there was only one attacker who poked around for 6 days
without the bug being subsequently noticed by anyone else? If I were to
speculate, it seems like the work of an insider or someone close to the
project.

Of course, this is based on nothing but wild guesswork and it could be just
coincidence. But in their shoes, I would really want to make sure no one else
gained access and did anything more to hide their tracks, since it seemed like
such a quickly found exploit.

~~~
tedunangst
It doesn't say the bug was deployed on May 5th.

~~~
akersten
I'm looking at the following from the post:

> The intrusion originated on May 5 when a build deployed to the development
> tier for stackoverflow.com contained a bug, which allowed an attacker to log
> in to our development tier as well as escalate their access on the
> production version of stackoverflow.com. > Between May 5 and May 11, the
> intruder contained their activities to exploration

To me, that reads "we deployed a bug on May 5th that allowed intrusion, and it
was exploited the same day." If that's not what they meant - I certainly
recant my criticism - but "on May 5th when a build deployed contained a bug"
is difficult for me to interpret differently.

~~~
sucrose
Exactly. If it wasn't deployed on May 5th, instead of writing, "when a build
deployed to the development tier", I'd expect something along the lines of,
"The intrusion originated on May 5 in the development tier".

------
hathawsh
I am impressed with the way Stack Overflow is handling this, but I would
appreciate details about how a bug in the development tier gave an attacker
the ability to escalate their access in production. Does the development tier
have production keys? Did the attacker learn about some other bug through
access to the development tier? Is the development tier sharing a privileged
private network with the production tier? Hopefully the security community can
learn from this incident to improve best practices.

~~~
Xylakant
This is obviously speculation, but development does not require direct access
for a pivot to production. CI-Systems might be reachable from dev and I often
see CI-Systems that are less well secured than prod. Pivoting from an owned CI
to a prod system is often comparatively easy: inject code in the next build,
the artifact is implicitly trusted (hey, it’s from a known-good source) and
bang! Prod owned. Or developers expose their ssh keys via agents to dev
systems, the same key works for prod and you might be in.

I share your curiosity :)

~~~
penguat
Which is why best practice is now to start with CI in production - it needs
limited access to dev, and the access from development environments into it
can be literally just enough to collect the latest artifact.

------
ThinkBeat
I find it concerning that access to the dev tier made it possible to escalate
into production.

Should dev not be sandboxed from production?

"The intrusion originated on May 5 when a build deployed to the development
tier for stackoverflow.com contained a bug, which allowed an attacker to log
in to our development tier as well as escalate their access on the production
version of stackoverflow.com. "

------
ddtaylor
> contained a bug, which allowed an attacker to log in to our development tier
> as well as escalate their access on the production version of
> stackoverflow.com.

Why are the credentials for production granted to development systems?

------
djhaskin987
> As part of our security procedures to protect sensitive customer data, we
> maintain separate infrastructure and networks for clients of our Teams,
> Business, and Enterprise products and we have found no evidence that those
> systems or customer data were accessed.

I sure wish more companies did this. Such peace of mind.

~~~
closeparen
More targets to defend, more chances to slip up. It’s a mixed bag.

~~~
jascii
Uh, no. You need infrastructure to handle each anyway, dividing it in separate
realms makes sense.

------
advisedwang
250 users affected? Sounds like a targetted attack rather than a spray-and-
pray. I wonder who the victims and attacker are and what the motive was.

~~~
tgsovlerkhgsel
Could also be that the attacker made a set of random API calls that returned
private data, e.g. to test that his permissions are working, and received
whatever that API returned at that time without targeting any specific users
(or caring about that specific data).

------
SoReadyToHelp
As it happens, this incident was discovered by a curious user who had a script
watching for new accounts with staff privileges, who brought the attacker's
account it to the company's attention (in chat) because it looked unusual.

Stack Overflow seem to be following a very responsible incident response
procedure, perhaps instituted by their new VP of Engineering (the author of
the OP). It is nice to see.

~~~
mkagenius
> As it happens, this incident was discovered by a curious user

Some recognition would have been nice instead of saying "we" quickly found
out.

~~~
icebraining
Maybe the user doesn't want to be named?

------
BentFranklin
I would be having a close look at whoever checked in that bug.

