
CircleCI Security Incident - sciurus
https://support.circleci.com/hc/en-us/articles/360034852194-Security-Incident-on-8-31-2019-Details-and-FAQs-
======
147
Relevant post on HN 2 years ago:
[https://news.ycombinator.com/item?id=15442636](https://news.ycombinator.com/item?id=15442636)

~~~
robocat
"CircleCI trusts 8 analytics companies with your source code and API tokens,
October 9, 2017. When you navigate to your project in CircleCI's UI,
Javascript from eight different analytics companies gets loaded and executed
in your browser."

So, did they do anything about that?

Even if using iframes and a secure api over postMessage/onmessage: safe, and
security can be controlled by the external provider i.e. safer for both
parties (ideally the servers pass the security context between each other - we
even use an iframe internally for all the same reasons).

[https://kevin.burke.dev/kevin/circleci-is-hopelessly-
insecur...](https://kevin.burke.dev/kevin/circleci-is-hopelessly-insecure/)

~~~
aeonflux
> for e.g CircleCI’s account for Google Analytics

Google allows custom JS injection via Tag manager, so there are possible
vectors, even if only access to 3rd party site was stolen. Of course I have no
idea what was the deal here.

It's worth noting, that CircleCI did nothing to address issues raised in the
linked post. Their main app still loads tons of third party analytics garbage
for e.g. Google, Hotjar, Amplitude and even Facebook of all people. I do block
all those, but as someone pointed out it's not a solution and cant even be
reliable at all times.

Please bear in mind, that CircleCI not only has access to private
repositories. More often then not they do store private SSH keys to your
production servers.

------
aftbit
This is how you handle a security notification. Well done CircleCI, looking
forward to the full postmortem.

~~~
rvz
I absolutely agree. This is a very professional example of handling a serious
security incident. CircleCI immediately took action and notified all affected
accounts afterwards and as with this post, they did a detailed FAQ-style
breakdown of what happened and how they will stop another attack.

> Security is taken very seriously here at CircleCI.

Compared with the other security handlings i've seen, this statement sounds
genuine.

------
guytv
> Security is taken very seriously here at CircleCI ... > In the meantime, we
> will review our policies for enforcing 2FA on third-party accounts to the
> extent possible, and continue our transition to single sign-on (SSO) for all
> of our integrations

Well, if you had taken security seriously, the first thing you'd do is ensure
2fa on all accounts.

------
jacques_chester
> _On August 31st at 2:32 p.m. UTC, a CircleCI team member saw an email
> notification from one of our third-party analytics vendors and suspected
> that unusual activity was taking place in this particular vendor account._

A convention I like is that team members creating, updating or removing
accounts should reply to any automated emails sent to mailing lists saying
"this is me". Ideally with a link to the relevant issue or story.

It really cuts down on unnecessary nerves.

------
nisten
What's sad about this is that CircleCI actually has a great product and is one
of the nicest ways to do end to end automation for mobile
development/releases. Having their pipeline in place actually feels quite
liberating.

The sad part is that they take this for granted and liberate all your data and
security weaknesses too to unknown third parties for either a weird
ideological reason about interoperability or a small marginal profit.

I don't understand why they can't do both(interoperability and security),
their main profit is not data, it's customers paying for infrasctructure. They
don't have to share any data at all and are fully capable of providing a
competent level of security.

If anything I think if they cut out the profit line from sharing metrics etc
with third parties it would actually bring more customers in because of
increased confidence in the product.

~~~
jjeaff
I doubt they are getting paid to share any data at all. Usually, the point of
3rd party services is for improving the service or keeping it bug free and
functional.

I would imagine they have some services that are logging JavaScript errors and
probably some that are tracking how people use the service.

If you have a Freemium or tiered service especially, you have to understand
how people use your service in order to convince them to pay or upgrade.

~~~
aeonflux
> you have to understand how people use your service

Thats why they need Facebook's tracking inside the admin part of the APP (on
na paid account)?

------
traskjd
I didn't see it, but "who is the third party vendor?" is my key question,
since this could be impacting anyone working with that vendor.

~~~
antoncohen
The vendor might not have been compromised.

> we will review our policies for enforcing 2FA on third-party accounts to the
> extent possible, and continue our transition to single sign-on (SSO) for all
> of our integrations

I think it is more likely the CircleCI didn't enforce 2FA or SSO[1] with the
vendor, and a CircleCI employee reused an already compromised password.

[1] Nearly all SSO identity providers support 2FA, so using SSO is often the
easiest way to get 2FA with third-party SaaS providers.

~~~
gingerlime
Naming the vendor in this case should be fine. Shouldn’t it?

In general what I’m missing is _why_ they share my email and github account
with a 3rd party. GDPR should prevent them from doing so (at least for
European users) without a legitimate reason, or explicit consent.

~~~
PeterisP
That's not exactly how GDPR works - the criteria depends on what you mean by
"share with a 3rd party".

If _you_ as the data controller are allowed (according to GDPR) to do some
particular processing of some particular data, then GDPR also allows you to
subcontract that processing, and thus provide the data to third parties (data
processors).

There are a bunch of restrictions in place (you have to have a specific
contract mandating that they only do the thing with the data you're
contracting them to do, you're responsible if they violate that, some
restrictions on "exporting" data out of EU), and you have to _inform_ the
customer to what third party 'data processors' you (as the data controller)
are outsourcing these activities - so CircleCI would be required to give an
exhaustive list of all the data processors to which they have given your data,
but you don't need explicit consent or a very particular reason for that
outsourcing.

What _is_ prevented by GDPR is the common (at least in USA) "sharing data with
trusted third party partners" where the data is essentially sold to third
parties which are free to use that data as they wish for their own business or
re-sell it further. _That_ would require a very particular reason or explicit
consent, but this is not the common scenario of outsourcing to a GDPR-
compliant vendor.

~~~
gingerlime
My (admittedly limited) understanding of it is that there still needs to be a
legitimate reason to provide my email.

So if the email is required in order to, say, send me notifications about
CircleCI jobs that are running. That's a legitimate reason. If they share my
email with the 3rd party to, for example, optimize their onboarding flows,
then this isn't legitimate, unless I explicitly gave my consent to it.

To me, "analytics vendor" as they stated it, usually implies something similar
to the latter rather than the former. But since they didn't indicate the
provider, nor the reason, I can't really say.

I think we're not in any disagreement here by the way.

------
telecuda
Curious to hear from CircleCI how their FedRAMP authorization played into what
was (or wasn’t) compromised and if it improved their response.

[https://circleci.com/blog/modernizing-federal-devops-
circlec...](https://circleci.com/blog/modernizing-federal-devops-circleci-
becomes-first-continuous-integration-tool-with-fedramp-authorization/)

------
agustif
I recieved the email from them a few hours ago, because of the wording of
that, and because I spend last night integrating segment's analytics on my
project.

I might suspect bc of their wording, segment and not google analytics would be
such said 3rd party, and it seems it was one of the team's accounts in such
service which got compromised and created a new destination, and no one
noticed until 2 months later..

This is just pure speculation, but I would like to know!

~~~
jbattle
I just got an email from Segment reporting something very similar. I think you
are right

~~~
agustif
Yeah I just saw it on HN homepage, By the timing of it an alert about
-segment- or just a segment employee watching this comment thread might have
forced them to come clean sooner than they were planning to?

------
mr_justin
Nobody deserves this, but lordy I have lost days of dev time to Circle in the
last several years.

~~~
SkyPuncher
If I have the choice, I will not be starting any project with them in the
future after the crap they pulled with CircleCI 2.0.

"We're introducing this new, 2.0, version. It has some benefits, but is also
missing a lot of features and bug ridden. It's easier for us though, so you're
going to have to waste a couple days converting from 1.0 to 2.0."

~~~
aeonflux
My feelings as well. What we really need is a ansible script or a docker file,
which would build a Jenkins instance which everyone could use. You would of
course need different tools for various build setup and technologies, but
CircleCI is far from point and click anyway. I really which I would spend all
the time I wasted on their config to build the above. This way everyone could
run 5000$/mo of CircleCI on a 50$/mo bare metal.

~~~
nathanaldensr
You might want to try Azure Pipelines. It uses a dual model of Microsoft-
hosted build agents running several OSes including macOS, and self-hosted
build agents running on infrastructure of your choosing. The team I lead uses
for Windows-, Linux-, and macOS-based builds, Microsoft-hosted and self-hosted
build agents.

~~~
aeonflux
If I finally decide to pull the plug I will probably invest time to build my
own setup around OS tooling. Main benefits would be much lower price and
independence from lock in. And it's not like one has to build their own
solution here. There are tools available.

------
minimaul
CircleCI has been very frustrating for us recently.

We have a lot of problems with waiting for builds to start - it seems CircleCI
don’t have even remotely enough capacity at peak times. We regularly get 10min
plus delays to builds starting, sometimes it’s over 30 mins!

Mix that with this security issue... it’s not a rosy outlook for us.

Edit: oh, and the UI has terrible problems with caching and not updating
state. Constantly have to hard refresh pages without cache for build status to
update.

------
EamonnMR
Initially freaked out when I got the email, but it looks like actual repo
contents weren't compromised.

------
cgijoe
Dupe?
[https://news.ycombinator.com/item?id=20882624](https://news.ycombinator.com/item?id=20882624)

~~~
paulddraper
Yes, that was later. Comments have been moved.

------
paulcarroty
CircleCI also use wordpress, not ideal in security aspect (as Google
Analytics, too), be honest. Also found these services inside my profile page:

* cloudfront

* githubusercontent

* launchdarkly

* zendesk

* zopim

* gravatar

* optimizely

* pusher

* pusherapp

* segment

* statuspage

* wp

* zdassets (3 domains)

I use uMatrix to cut half of this crap, cause the page is too heavy to load on
my laptop. Started looking for alternatives.

------
rficcaglia
The initial reaction is to TL;DR as weak passwords and no MFA but I am
equally/more concerned that the policy granted to vendors allows them to
access more data than they really need, or the data they must have access
could be better de-identified. For example they could hash/uuid the email and
the git repo and branch info. Hacks are going to happen; no password or MFA
will prevent it. Zero trust in your policy/schema design minimizes blast
radius in these inevitable cases.

~~~
pushpop
The idea of treating email addresses as secrets is a nice idea in theory but
given the plethora of databases where it’s already stored as a user name,
including those that have already been compromised, and how email addresses
are published in the git commits anyway, I don’t think it makes much sense
hashing email addresses in practice.

Repository details are another interesting point though. They too are
published openly in any git clones but I accept that some people’s
repositories, including any local clones, would be private and kept secured
from prying eyes. That said, what matters more there is access credentials,
encryption at rest (for local clones), etc. So keeping the branch names and
repository URIs secret feels a little redundant compared to the actual
security benefit it brings. One might even say it’s security through
obscurity, though I don’t personally think it offers up even enough benefit to
fall under that category. That said, I’m happy to be convinced otherwise if
you feel I’ve overlooked some important detail.

------
mindfulplay
I would like one company to post an "incident" reveal in a more honest way:

" Due to our carelessness and relatively insecure practices, we had improperly
disclosed user accounts to a moderately savvy hacker. We realize this is our
fault.

If you'd like to help and given that we have your attention now, it would be
valuable if you can help pentest our servers: the attacker used a simple SQL
attack based on an unpatched server via CVE-3245. Are we missing anything
else? Please let us know.

Thank you."

------
symlinkk
I have a feeling this is the nail in the coffin for CircleCI. Github Actions
are the way forward.

