
CircleCI security incident - markhelo
http://status.circleci.com/
======
Cogito
Is there any evidence there has been a breach of CircleCI directly, or just
that MongoHQ suspended all access to their database(s) and so the service went
down?

At this point it doesn't seem like they were 'hacked'[0], but simply involved
in a larger security incident.

[EDIT] seems the linked page doesn't mention 'hacked' anywhere (literally - I
searched for it!) Seems like the title[0] is a classic case of editorialising

[EDIT 2] As per below, would be very interested to see if their database being
'compromised' means that data was actually accessed by the attackers, or if
they are just fall out from the bigger attack

[0] original submitted title was "CircleCI has been hacked"

~~~
thejosh
>We are contacting you to inform you of an ongoing security incident affecting
CircleCI customers, as a result of the compromise of our database
([http://security.mongohq.com/notice](http://security.mongohq.com/notice)).

Seems like a 'security incident' is code for 'hacked'.

~~~
Cogito
I think I agree with you, however there is a big difference between having
your service stopped as a precaution and having data lost/accessed by a 3rd
party.

From that line it does seem more likely that they might be one of MongoHQ's
clients who MongoHQ noticed had had their data accessed as part of the bigger
breach.

In that context, saying they have been hacked is probably fair. Would love
more details.

------
morgante
Up next: CircleCI customer announces data breach due to tokens accessed via
the CircleCI breach (via the MongoHQ breach).

~~~
nikcub
This is very likely. I'd guess that the likely target was a MongoHQ customer,
and that MongoHQ getting hacked was collateral damage as part of an attempt on
one of their users.

------
pilif
This is why I'm so mindful about using third-party services that require
access to our code to perform their service. It's not that I don't trust the
third-parties not to abuse their power, but by trusting them with my code, I
also have to trust their security measures.

As such, I always very carefully weigh the benefits such a service can provide
against how much work it would be to do it myself (which will likely produce a
shittier result, but which will be fully under my control).

It's not even that I think I can do security better than $SERVICE, but I'm a
much smaller target. I'm not as afraid of a targeted attack than I'm afraid of
somebody compromising one of these focal points and then just lifting
everything.

------
entelarust
Quick theoretical senario...

User signs up to try circleci for a private project of theirs. Grants read
access to their private repos via github oauth

User also has many other private repos (company they work for, open source
projects, forks, etc)

Could they have used the stored github credentials from circleci to clone
every private repo in full the user had access to?

~~~
xentronium
Github has a feature to allow access to a singular repository via a key. It
would be logical for CircleCI to use that feature, although I'm not sure they
actually did.

[https://help.github.com/articles/managing-deploy-
keys#deploy...](https://help.github.com/articles/managing-deploy-keys#deploy-
keys)

~~~
gravitronic
They did not.

~~~
j15e
They did, Circle-CI client here.

Info have a Circle-CI deploy key per private repository (which I will revoke).

------
trimbo
I am contemplating whether MongoHQ should just post their customer list, so
their customers' customers know what to expect?

I know that's not exactly proper but... at this point, the hackers are way
ahead of the users of sites that happen to depend on MongoHQ.

~~~
ghayes
That could be irresponsible as a company which uses MongoDB for non-sensitive
data (say data in publicly available pages) would be shamed when they didn't
have a data breach of sensitive data. Or worse, a company that has signed-up
but only played around with MongoHQ without using it in production. Each
company should announce its own breach (and the scope of the data exploited)
to its customers.

~~~
trimbo
I understand all the implications you mention, and that's why I phrased it the
way I did.

Though, the assumption I'm making right now is that the hackers know exactly
what sites they've compromised. The longer they have this information in
advance of MongoHQ's customers' customers, they more damage they can do. That
sucks terribly for those people (potentially)

All this said, a serious lesson about PaaS/SaaS/DBaaS--whatever you want to
call it--has been learned today.

[edit] - I'm reminded here of the Epsilon breach. Epsilon is magnitudes larger
than MongoHQ and they apparently did decide to announce the names of affected
companies. I am not sure how they handled that internally. [1]

[1] - [http://mashable.com/2011/04/03/epsilon-
hacked/](http://mashable.com/2011/04/03/epsilon-hacked/)

~~~
benologist
In Epsilon's case those end users had direct consequences - their names and
emails were compromised.

The end users of sites/apps/services using MongoHQ are probably unaffected in
many cases since "a database" doesn't imply credit cards or even credentials
or emails. The average database is probably just holding content for a site or
app.

In probably close to all cases the end users would have no clue how they may
be affected since they won't know what's in the databases, or what a database
is.

------
outworlder
This is one of the reasons why we have shutdown our own CI solution. We didn't
have enough manpower to develop the product, audit the whole stack and monitor
against attacks.

CircleCI's response to the incident appear adequate. In the best case, their
system was shutdown before any keys were compromised. If so, that level of
monitoring is certainly better than what the average business would be able to
roll on their own.

------
thejosh
from the page:

Identified - We have notified all users and recommended appropriate action:

"We are contacting you to inform you of an ongoing security incident affecting
CircleCI customers, as a result of the compromise of our database
([http://security.mongohq.com/notice](http://security.mongohq.com/notice)).

We are taking aggressive action to protect your data and systems. At this
time, we have suspended all CircleCI account access, and all builds & workers
have been suspended. In addition we have revoked all access to Heroku and
GitHub OAuth tokens and API keys uploaded to CircleCI.

We do not yet know the scope and impact of the intrusion and are therefore
treating this event as if all data has been compromised. While we have no
evidence that these credentials have been compromised, we urge you to revoke
the following:

SSH keys that were uploaded to CircleCI API tokens added to CircleCI as
environment variables secrets stored in GitHub repositories We will be keeping
you informed at [http://status.circleci.com](http://status.circleci.com) and
will update you at regular intervals as the situation progresses.

We deeply regret that this has happened and are working around the clock to
resolve this incident and protect your data and systems." 22:25 PDT Update -
We are still investigating the issue. The full team is engaged and we are
working with upstream providers to diagnose and respond to the issue, and
protect all of our users. We will keep you informed. 21:17 PDT Update - We are
currently investigating an ongoing issue with our database service. At this
time, we have suspended all account access to our service. All builds &
workers have been suspended. We will have another update in the next 30
minutes. 20:20 PDT Investigating - "CircleCI is experiencing technical
problems. We're investigating and should have an update within 30 minutes."
19:30 PDT

~~~
dgtized
I'm curious why they mention uploaded SSH keys. I presume they mean cases
where SSH keypairs have been uploaded? The public key is _public_ , and in
fact anyone can use the github API to pull the verified public keys for any
user in the system,
[http://developer.github.com/v3/users/keys/](http://developer.github.com/v3/users/keys/).
If there is code uploaded with private keys in it, then it's likely there are
other security problems in the organization.

~~~
nthj
Because CircleCI stores private keys for the sake of deploying to your servers
when your build passes.

It's separate from your actual codebase having any private keys, which I agree
would be a Really Bad Thing.

------
yla92
I think it's more like MongoHQ is compromised and consequently, circleci is
affected than 'circle has been hacked'.

------
stephenitis
[http://security.mongohq.com/notice](http://security.mongohq.com/notice) So
this really is mongohq's fault... Demonstrates the dangerous domino effect if
you think of all the code/keys/secrets...

nightmares will be had.

------
jonny_eh
Since circleCI users upload the private heroku SSH keys to automate deploys,
how hard would it be for the intruders to get access to the environment
variables stored in heroku?

------
chacham15
Cache to their homepage:
[http://webcache.googleusercontent.com/search?q=cache:1oo0VIR...](http://webcache.googleusercontent.com/search?q=cache:1oo0VIRO6iAJ:https://circleci.com/+&cd=1&hl=en&ct=clnk&gl=us)

~~~
joshchaney
Customers: stripe, Red Bull, Kickstarter.. and MongoHQ. How meta. This hack
could get a lot more interesting depending on how much code they could have
obtained from those companies.

------
xwowsersx
I started testing out CircleCI a bit ago and this really doesn't help seal the
deal for me. What am I supposed to do now? CircleCI was connected to my github
account...are my SSH keys not safe now? If so, not clear on what I'm supposed
to do.

~~~
robertfw
Revoke the CircleCI key that you added to your repo on Github. In the worst
case scenario, that key has been compromised and could be used to access your
source code.

~~~
xwowsersx
I didn't create a new SSH key...I just used my public rsa one. I should just
delete that key and create a new one?

------
rhizome
Is there any general sense of the techniques being used in these cracks? Just
from a defensive coding perspective, I'd like to know if there are programming
errors, weak links in frameworks, or what.

~~~
Cogito
If you read MongoHQ's extremely detailed report[0], you'll see that the
original intrusion was based on shared credentials (that were cracked on
another system). This was then exploited in a number of ways to compromise
various clients of MongoHQ.

Lots more discussion over at [1].

The main takeaway is to take security seriously, and employ multiple levels of
security. The MongoHQ team are doing things like 2 factor auth, and
restricting customer service tools to a vpn. As far as I can see, no framework
or coding bugs.

[0] [http://security.mongohq.com/notice](http://security.mongohq.com/notice)

[1]
[https://news.ycombinator.com/item?id=6637426](https://news.ycombinator.com/item?id=6637426)

------
tlrobinson
Oh boy, another MongoHQ casualty. I wonder where the fallout will end...

~~~
ukd1
You think this was a casualty, or the whole reason?

~~~
tlrobinson
Well they also used Buffer's data for spamming, so it's hard to say whether
one or the other was targeted or opportunistically attacked.

CircleCI is certainly the more interesting target as they have access to
numerous companies' source code.

~~~
helpfulElf
>access to...source code

How much do you think that is worth on the black market?

~~~
richardlblair
Honestly, not much.

You might be able to find a few vulnerabilities... maybe some good ones, maybe
some hard coded keys, but about that's all.

Having a companies code doesn't mean you can make a successful business out of
it. Especially if they are already established as a major player.

------
presty
any suspicions on who might be behind these attacks?

was the buffer spamming incident just a way to deflect attention of something
more sophisticated?

I wonder if anyone used the keys stored in circleci to get to the code of some
service and if we'll see even bigger hacking news in the near future

</conspiracy>

------
brryant
if you're using stripe, make sure to recycle those keys if you have them
checked into your repo

~~~
bradhe
Better yet, if they are checked in, rotate them and _don 't_ check in the new
keys. Make them an environment variable instead.

