
CircleCI's incident response - cr4zy
http://blog.circleci.com/mongohq-security-incident-response/
======
WestCoastJustin
Ouch -- Amazing how there is this ripple effect happening now that everone is
using the cloud and there are so many middle men or service APIs.

> _In order to protect your data and users, we strongly urge you to secure
> each of these systems:_
    
    
      > SSH keys uploaded to CircleCI
      > API tokens stored in CircleCI env vars
      > API/SSH key stored in a GitHub repo accessible from your CircleCI
    

Interesting to note that Stripe and Kickstarter are customers. This is a
little scary to think their source code could have been exposed, given the $$$
flowing through their systems. So, someone could have used these keys to have
much wider access!

~~~
sehrope
> Interesting to note that Stripe and Kickstarter are customers. This is a
> little scary to think their source code could have been exposed, given the
> $$$ flowing through their systems.

If either of them have any sense (which I think they do!) then having there
source code exposed shouldn't in and of itself pose a security risk. If it did
then that would be security via obscurity.

Now I'm not saying that having the source code for a site is useless or that
having it exposed is a good thing (it's not!). If there is a bug/hole you
could find it, but if there isn't then the source being available should not
make anything less secure.

> So, someone could have used these keys to have much wider access!

SSH keys and API tokens are another story. These need to be revoked
immediately (CircleCI's response looks great by the way) as there would be
nothing besides firewall rules (if they exist!) preventing an attacker from
using them.

This whole incident is a friendly reminder to follow the principle of least
privilege[1] with your XaaS providers. There's not much you can do for API
tokens. If you're relying on someone else to provide a service on your behalf
then they'll need their own token.

For data though it's possible to encrypt sensitive data on your end and use
the DBaaS provider as an opaque blob store. Doing it across the board gets a
bit unwieldly as it's hard to query encrypted data but it's fine sensitive
details that you don't want to necessarily expose (such as ENV vars, payment
info or equivalent tokens). If the provider doesn't need access to X, and X is
sensitive, then encrypt it before you save it with them. That way you don't
have to worry about them leaking it. One of the best examples of this approach
is tarsnap[2] which considers _everything_ sensitive and encrypts everything
client side (on your machine) before uploading to its central server (and then
S3).

EDIT: One more quick point. If you explicitly encrypt sensitive data as part
of storing it (regardless of whether you're using an XaaS provider or running
a DB yourself) there's one more big advantage: Backups!

Your backups themselves should be encrypted (ex: we use gpg prior to uploading
them to S3) but in addition to that by following the above, the sensitive
fields _in the backup_ are also encrypted. This means that they're encrypted
in any scratch/temp files that are part of the backup generation process.

Otherwise they would be stored in _plaintext_ on your server and possibly be
on your server's hard disk (or your XaaS providers disks) without you even
knowing it. Oh and remember that disk blocks don't actually get overwritten
when you "rm" a file so this is more important than you think.

[1]:
[http://en.wikipedia.org/wiki/Principle_of_least_privilege](http://en.wikipedia.org/wiki/Principle_of_least_privilege)

[2]: [http://www.tarsnap.com/](http://www.tarsnap.com/)

~~~
revelation
Stripe is in the business of fraud detection, just as any other payment
provider. Their source code likely contains lots of automated heuristics
checking for fraudulent transactions, information which is tremendously useful
for criminals looking to defraud people. No need for the "security by
obscurity" reflex, because the problem at hand is impossible to solve
completely.

------
tkiley
Now might be a good time for circleci customers to pick up a code analysis
tool such as brakeman and look for obvious security holes in their apps. If
the attackers have hundreds or thousands of web apps' source code, I'd expect
them to start trawling for vulnerabilities with automated tools.

------
damncabbage
What happened between the 19th and the 28th?
[https://news.ycombinator.com/item?id=6638004](https://news.ycombinator.com/item?id=6638004)

    
    
      Hi Justin. To clarify, from what I understand, October 28
      is the date MongoHQ detected this. They've provided us 
      with the logs of database access, and unfortunately the 
      queries leading to our spam attack on Saturday started as 
      early as October 19.

------
johnwards
So, to be clear. I need to contact Github directly to see if my source has
been downloaded using compromised deploy keys?

~~~
cr4zy
I would. Although you can see some of that info here:
[https://github.com/settings/security](https://github.com/settings/security)

Edit: There are also logs for your organization in
[https://github.com/organizations/<your_organization>/setting...](https://github.com/organizations/<your_organization>/settings/security)

------
jonny_eh
If I had a heroku API key stored with CircleCI, should I assume all of my
app's production config variables have been compromised?

~~~
1qaz2wsx3edc
Err on the side of caution and rotate your keys.

