
CircleCI trusts 8 analytics companies with your source code and API tokens - tomwas54
https://kev.inburke.com/kevin/circleci-is-hopelessly-insecure/
======
z00b
Rob Zuber, CTO of CircleCI here. We take security very seriously and are
taking a deep look at the issues Kevin raised.

We'll save additional commenting until we have gathered more information. In
the meantime, our security policy and steps for reporting issues are here:
[https://circleci.com/security/](https://circleci.com/security/) and we'd like
to ask the community to please use our outlined methods for reporting
potential security issues so we can keep CircleCI as safe as possible for
everyone.

~~~
kevinburke
OP here. Thanks for responding promptly. To be clear, letting third party JS
run in a trusted environment like a dashboard is an industry wide problem. If
we assume CircleCI is the _only_ bad actor we're kind of missing the point of
the exercise.

CircleCI is a notable example, due to the fact it hosts source code and
secrets for so many different companies and loads so many third party scripts
in its dashboard context. But they're not unique in this regard. I hope
everyone reading this is reconsidering the scripts that have access to their
dashboard and pushing for changes at their company.

If CircleCI was immediately vulnerable to injection via the scripts above I
would have used the private disclosure route and I encourage others to as
well. But I don't know that there is a "vulnerability" here so much as a
discussion that we need to have about what and how much third party code we
let run in a trusted context. I wrote a little more in the thread below,
[https://news.ycombinator.com/item?id=15442988](https://news.ycombinator.com/item?id=15442988).

~~~
ukd1
IMHO you should have followed the standard practice of telling them first,
then posting the story afterward. Blindsiding them wasn't nice, even if
(hopefully) this makes them and others think more about what they risk with
external JS, rather than just "oh shit circle...". Just my 5c.

p.s I'm curious why you thought this route was better?

~~~
rapind
I hear this attitude a lot and it always rubs me the wrong way.

It's a corporation... Why should he care about it's _feelings_? What does he
owe CircleCI? Are they paying him? What claim do they have to his time to jump
through their hoops / process?

I'm positive in this case that they weren't intentionally messing with
people's security, but shouldn't we and their customers be able to judge that
ourselves instead of getting it swept under the carpet via private channels?

I do believe it's good practice to "not be a douche", especially at a personal
level, but I wouldn't even come close to categorizing this as douche
behaviour.

* Note: this comment is not actually directed at CircleCI, just the attitude that we all somehow owe it to companies to tell them about their goofs privately.

~~~
michaelbuckbee
It's rarely about the company and much more about those affected by a
potential issue. This issue's a little different as it's the potential for a
vulnerability more than an actual vulnerability.

A typical case for responsible disclosure is something like a bug in Apache or
Nginx that's serious and if a security issue is found that they're given time
to address it. So when they make an announcement its:

1\. Here is the issue

and

2\. Here is the fix

Instead of just spreading and publicizing a vulnerability.

~~~
rapind
I'm mostly with you when there's potential damage to their customers,
especially when those customers are individuals (like a database of social
profiles being compromised), which was not the case here. However I think
saying it's "rarely" about the company is a bit naive.

There's been some really high profile breaches where the extent of the impact
to the customer has taken an unacceptably long time to be made public... which
results in increased fallout damage.

A company's natural inclination is going to be to minimize losses
(generalizing of course), and that often means dragging their feet, or dealing
with it privately and never notifying their customers.

We should recognize these incentives and be a bit more aggressive about
holding their feet to the fire, instead of criticizing the researcher who
discovers the security flaw. Unless of course his / her behaviour is blatantly
malicious.

It anecdotally feels a bit lopsided in favour of the corporations at the
moment. Especially when someone like in this post, who clearly didn't endanger
any customers, gets criticized...

------
sytse
At GitLab we separated the infrastructure between about.gitlab.com (our
marketing site, lots of third party javascript) and gitlab.com (our
application, zero third party javascript). We recently deprecated the Piwik
instance we ran in house for gitlab.com because it slowed the page load time.
Please let me know if there is anything we can do better, these things are
complex.

~~~
eropple
_> Please let me know if there is anything we can do better_

You can not opportunistically spam nearly every thread where GitLab might be
tangentially related.

I don't use GitLab largely because of the way you post on HN, though the
software seems reasonably well-written. It's very frustrating.

~~~
orf
You refuse to use an awesome product because the CEO of a startup is actively
engaging with a community that may (or may not) use it, accepting feedback and
discussing the product with people?

Ok....

~~~
n42
This isn't engagement, it is quite clearly advertising.

~~~
orf
Sure, he's advertising how Gitlab, a related product in the same space,
handles this and the steps they took to prevent this problem from happening.

It's informative, not "gitlab doesn't suffer from this, here's a 10% off
coupon [https://gitlab.com!!"](https://gitlab.com!!").

~~~
n42
You're right, his post does contribute to a discussion on how to avoid this
situation (which is what I'd like the thread to be about). I think it is based
on a history of seeing GitLab's marketing team or CEO in any comment section
tangentially related to their product that has put a bad taste in my mouth. I
usually know before clicking into a comment thread whether or not they will be
pushing their platform inside.

~~~
jlgaddis
In general, I would tend to agree with you. In this case, though, the original
article was about CircleCI specifically but mentioned this being an issue
across the entire industry. GitLab is a part of that industry and they have
lots of users on HN so, in this case, the comment is relevant (IMO).

More generally, however, you're right although I have just come to accept that
this is how things are on HN. Choose pretty much any "Show HN" thread or any
thread about some cool new app/product/service and in the thread you'll find a
few "shameless plug" comments that would be considered spam in any other
forum: "Hey, we also make an app that does $foo, sign up for our beta at
example.com and check us out!".

(One of the worst "offenders" is Userify. Pick any thread that relates
tangentially to user authentication and chances are good you'll find a comment
from them that manages to work their name/URL into the conversation somehow.)

------
Animats
Google's policies are a big part of this problem. Ad code and tracking code
ought to be in an iframe, where the code can't snoop on the surrounding page
context. Google insists that _their_ ad code must not be in an iframe. That
gives the lesser players an opening to insist that they, too, should't be
constrained.

A few big site operators should push back against Google on this. The New York
Times and CBS, for example. They use Google Publisher Tags, but no Google ads,
so they don't really need Google on their pages.

~~~
manigandham
Most ad code does run in iframes and the industry standard is to use "friendly
iframes" which offer security but also access to the parent window. And even
if you use iframes, something has to create that iframe initially and
publisher devs are not going to do that manually, which is why there are top
level script tags. All tag managers and adservers today default to using
iframes for 3rd party tags.

~~~
Animats
Google AdSense FAQ:

 _Q: Is it violating program policy if I place ads on iframe webpages in my
software?_

 _A: Yes, it does violate our policies. Firstly, you’re not allowed to place
ads in a frame within another page. Exceptions to our policies are permitted
only with authorization from Google for the valid use of iframes._ [1]

[1]
[https://support.google.com/adsense/answer/3394713?hl=en](https://support.google.com/adsense/answer/3394713?hl=en)

~~~
manigandham
That's different. Yes, their policy is that you cannot have a page of your
site that you load in an iframe, then put ads into that child page. This is to
prevent fraud and viewability issues like loading 100 invisible child pages in
the background to rip off ad networks.

None of that has to do with how ads from their network are actually rendered
when you use their tags. Adsense/DFP tags do use iframes for almost everything
unless explicitly set to load directly on page, usually for legacy code or
rich-media. The relevant help article is here:
[https://support.google.com/dfp_premium/answer/183282](https://support.google.com/dfp_premium/answer/183282)

 _"...SafeFrame is supported in DFP and enabled by default when using Google
Publisher Tags. Reservations serve into SafeFrames by default, but you can
disable this setting...and use friendly iframes instead, if needed.... Some
creatives, such as expandable ads or creatives that access your page’s DOM
elements, might not render correctly in SafeFrames or other cross-domain
iframes. We recommend updating these creatives to make them SafeFrame-
compatible, in order to retain SafeFrame’s security benefits. If this is
absolutely not an option, there are a few things you can do to allow
reservation ads of this type to render properly..."_

~~~
lamlam
To add to this, there's also a slow transition towards safe frames [1] that
allows for better protection than friendly frames by restricting all page-
iframe communication to a specific API.

[1]
[https://www.iab.com/guidelines/safeframe/](https://www.iab.com/guidelines/safeframe/)

------
koolba
> Send an email any time an API access token is created. Add a setting to
> allow org-wide disabling of API token creation.

I wish more apps supported this.

Also on my list is a " _Don 't ever let me disable 2FA_" setting. I'm more
worried about malicoius resets than I am about ever losing my 2FA device _and_
backup codes.

> Add an option to delete old logs. If you have ever dumped env vars to the
> log file, an attacker can export these.

I surprised that secrets make it into the logs at all. I would have expected
them to filter anything that's in an env secret from the log output. Pretty
sure Travis does this.

> Enable subresource integrity, or serve JS from each of these companies from
> the CircleCI domain.

That's not much of an option for this type of thing as the third parties would
then have a PITA time dealing with upgrades (so they wouldn't support it).
Using <script src="hxxps://tracker.example.com/path/to/script.js"></script>
(rather than a fixed version) allows for live upgrades as they're in control
of the resource that is loaded.

And you can't load the script from your domain as at the end of the day it's
got to get instructions and upload data to the third party. At best you'd be
loading a shim that does the same thing as the script tag.

~~~
orginal__idear
> "Don't ever let me disable 2FA" setting.

I like this idea. I'd only want to use this though if my Google 2FA backup
codes are backed up via Authy or a similar app.

I don't trust codes I've printed out and have always lost.

~~~
dabernathy89
I tend to just stick all my 2FA backup codes into "secure notes" in LastPass.

~~~
ihattendorf
That sort of defeats the purpose of 2FA though. Now if your LastPass database
is compromised, they have access to your 2FA codes as well.

~~~
dabernathy89
True. I think it's a bit of a security compromise for convenience. Having
backup codes stuck in a folder somewhere at home is only useful if I need to
regain access to one of my accounts while I'm at home.

------
danpalmer
I feel calling all 8 of these companies "analytics" is a little unfair.

Pusher is a SaaS product for websockets and push notifications. I think it's
reasonable that a company like CircleCI might outsource this instead of
maintaining the infrastructure for it in-house.

Launch Darkly is a feature flagging service. While I feel the value is much
less than Pusher (and I'd be inclined to just build an in-house solution), I
think it's still a relatively reasonable tool to use to create a better user
experience.

Intercom when used for customer communication is also something that I feel
I'd want as a user, although this could perhaps only be loaded when requested
by the user.

~~~
ballenf
I feel like narrowing down what any of those companies _could_ do to what they
are doing today irresponsible. And analytics is one of the could-do's that
many third party tools do offer or add on later to increase monetization or
perceived value to customers.

The important point is that there are now 8 attack surfaces instead of 1.
Whether they're analytics or pad-string doesn't really matter.

~~~
danpalmer
I agree there are now 8 attack surfaces, but to call all 8 "analytics" I does
not think does justice to the reasons why CircleCI may have chosen to use some
of them.

------
epanastasi
Thinking about mitigation here... It appears that some of the included scripts
have crossorigin="anonymous" script tags. Wouldn't this prevent authenticated
access to to the circleCI domain, aka preventing the creation of api tokens or
access to the API using the logged in browser context for any script loaded
off the circleci domain?

Also, not that they do so, but would Access-Control-Allow-Origin set to
something other than * prevent 3rd party requests to the API for scripts
loaded from 3rd party domains.

Also curious if anyone has written a JS library that patches
XMLHttpRequest.prototype to audit exfiltration of data in the DOM.

~~~
noway421
usually blockers like ghostly and ublock/adblock are doing that, but modifying
xhr in runtime is interesting idea!

------
nexfitter
I use CircleCI because it allows 1 free private repo. Personally though I do
not like Circle too much, it has a lot of cool features but the UI loads so
slowly because they have so much stuff going on. They use the UI skeletons
technique to make the load time appear to be faster but it is still very
apparent how slow the load time is.

I have hit numerous bugs with their website as well where stuff doesn't load,
builds kick off into infinity, when I transferred a repo everything broke,
ack!

Furthermore, I am frustrated with their pricing, they go from free to $50/mo
for the next upgrade tier, that seems like a crazy jump to me. I would gladly
pay $10/mo or so for another container or 2x parallelization. The issue is
being a single developer I rarely would save any time and the $50/mo is just
too steep.

Finally, when I tried to build a justification case for my manager to pay the
$50/mo, I wanted to use data from their CircleCI Insights which shows how long
your builds are queued on average and some other important data points. But
you cannot access these insights from a free account. Seems like that info
should be available and prominent to help people understand the cost-savings
they might receive by upgrading. I emailed support and asked for a one-time
data point for that statistic to build the case as I was considering upgrading
and they said sorry, nothing we can do for you. Is their goal to make money
and have happy customers? If so they aren't doing a great job of it.

Overall a lot of frustrations with the platform and this just adds more fuel
to the fire.

~~~
daxorid
Serious question: why CircleCI over, say, Jenkins or custom_build_script.sh
running on a VPS that costs much less, and that you control?

~~~
bm1362
Drone is a self-hosted alternative, although you still have to do some
configuration:

[https://github.com/drone/drone](https://github.com/drone/drone)

We went from Drone -> CircleCI -> Jenkins in the last two years. Jenkins feel
opaque and hard to reason about, periodically dies for no _good_ reason and
the UI is atrocious.

Drone itself was fine, as a minimal feature set. I think the docs are offline
now, but it's fairly straightforward.

~~~
acejam
> periodically dies for no _good_ reason

You must be doing something wrong then. I run a Jenkins cluster with hundreds
of automated pipeline build/deploy jobs and the master node never dies. (It's
also the only node up and running 24/7) You should be offloading your work to
build agents.

~~~
bm1362
I’m certain there is a reason, we scaled up and hired an infra team that owns
it so it’s out of my domain now.

I was merely a poor steward in the early days out of necessity. My era of
tinkering ended once we moved to CircleCI.

------
raesene6
Loading third party JS is increasingly common for a lot of sites, and I tend
to raise it when doing security reviews, for this reason, you're trusting the
security of those 3rd parties.

There are some defenses that can be put in place. The first one is kind of
awkward in many cases which is to host the JS on your own domain. There's
still the risk of course that it will go off and get additional code from the
3rd party source to execute, but that can be reviewed for.

The other option is to use sub-resource integrity
([https://developer.mozilla.org/en-
US/docs/Web/Security/Subres...](https://developer.mozilla.org/en-
US/docs/Web/Security/Subresource_Integrity)) to ensure that only scripts
you've reviewed are used.

Of course you need then to make sure you're notified before the 3rd party
makes changes that would break the signature.

~~~
rhizome
_Loading third party JS is increasingly common for a lot of sites, and I tend
to raise it when doing security reviews_

What kind of pushback do you get and how do you handle it?

~~~
raesene6
To be honest I'm a external security assessor/pentester and I've not had much
pushback from clients on this. That said I don't always get visibility of
whether they implement our recommendations or not :)

To me, it's not really a debatable point that loading JS from a source you
don't control implies trust in that source and therefore a risk that if they
are compromised it affects your site.

Whether that risk is ok for a business depends on a number of factors like :-

\- How trustworthy are the sources they're loading from? \- What reviews have
they completed on the security of those sources? \- Do they have contracts in
place with those sources that cover the requirement for security?

------
sbr464
I noticed that Galaxy - Meteor Development Group’s hosting solution for Meteor
apps does similar. I checked the console and saw multiple analytics, even some
that seem to monitor your visual usage/screenshots etc, which would seemingly
capture screens that have portions of environment variables / settings etc. So
basically potentially a lower level marketing employee who has an infected
laptop and is reviewing UX patterns etc could expose sensitive data on their
MacBook Air that could potentially be infected. They might not expose that
data. But I had the same thoughts as this article points out.

------
alpb
I'm trying to understand why "CircleCI browser context has full access to the
CircleCI API, which is hosted on the same domain".

Doesn't API have seperate authentication credentials (API key/tokens) than the
web UI (i.e. cookies)? I understand these loaded 3rd party scripts can scrape
what's rendered on the UI, but making calls to the API??? I wonder how that's
possible in CircleCI case.

~~~
avitzurel
That's the first thing that triggered me as well.

The only thing that 3rd party JS will have access to is everything on the
page.

Now, the things on the page are sensitive (the secrets from the env variables
are dumped in the UI).

Secrets in the secrets page are not exposed at any time, even when you go to
edit. Only in the build screen, it's exposed to the output.

Those 3rd party libs DO NOT have access to your source code at any time. The
only time they will have access to your code is if they gain SSH access to the
machines that are running the build. And that's not trivial.

Even if your public key is exposed, they don't have access to checkout github
with that.

So yeah, it's 3rd party libs in a "private" context but not more than that
IMHO.

~~~
detaro
From searching for discussions around CircleCI API and cookies it seems like
at least some API endpoints might accept session cookies for authentication.

I guess we'll have to wait for a CircleCI statement to address this in detail
(or someone testing it).

------
richardknop
Wouldn't similar vulnerability be present in most SAAS platforms? All of them
include tons of analytics scripts on their website. If you are a logged in
user, your session presumably includes a token (encrypted cookie?) to use the
SAAS API.

Therefor in theory other scripts loaded on the page could grab the token and
make authenticated API calls as well? Although I guess this is mitigated by
verifying script integrity when loading scripts from CDN (e.g. integrity
attribute of script tag)?

~~~
kevinburke
There are a few approaches, but yes, one thing I hope to do by publishing this
is to draw attention to the problem of third party Javacript scripts running
in a privileged environment and not on e.g the marketing page.

------
Posibyte
> Why don't you include a POC? I have one that demonstrates this attack, but I
> don't want to show script kiddies how. If you can't figure out how to
> construct it by reading the above description and the network traffic, you
> don't deserve to know how.

I feel this is a bad way to approach it and a bad attitude in general. Showing
people how this can be exploited (after responsible disclosure) is a great
opportunity to spread awareness of potentially their own issues in the same
space and create points of discussion. If a user feels compelled to do so,
they can seek out the info. But to show disdain to the reader for their lack
of knowledge comes off as unprofessional in spirit.

~~~
kevinburke
I disagree, and we'll just have to leave it at that. Anyone who knows how to
construct an AJAX request and look at the Network tab can figure this out.

~~~
lloeki
While I understand the point you made in that last sentence, the very last
part of it sounded needlessly loaded _even though I can easily figure such
things out_. Replacing "you don't deserve to know how" by, say, "you're not
the target audience" or anything to that effect goes a long way to not
detracting from getting your _real_ point across.

BTW this last easily misunderstood sentence sits just above your "I'm
available for hire" link, which is not exactly painting you in a bright light
for recruitment.

~~~
otterley
I agree. There was no need to insult your audience, and it is a career-
limiting move in the long run. The Internet has a long memory.

~~~
kevinburke
My career's doing just fine, thanks.

------
jeffnappi
As someone who has been a web developer since the 90's, this topic brings me
back to ~2005 when Google Analytics launched. I thought it was absolutely
insane that people would just stick someone else's JavaScript in their pages
(and especially on e-commerce sites!).

Today it's perfectly normal. I don't have exact numbers but at least 50% of
all sites pull in third party code. It really is bizarre to me that this
became the norm.

------
CaliforniaKarl
To me, this seems like a vulnerability report. The following line is what
really made it stand out for me as being a vulnerability report:

>Why don't you include a POC? I have one that demonstrates this attack, but I
don't want to show script kiddies how.

That leads me to this question: Did the author reach out to CircleCI to report
this issue, before publishing their blog post?

~~~
kevinburke
No, I didn't. If one of the JavaScript sources was immediately compromisable,
I would have.

~~~
kierenj
Why post a big public dig? If there's a problem, make the world a better place
by helping to solve it maybe? Just having a bad day?

~~~
kevinburke
1) they know exactly what they are doing; 2) arbitrary 3rd party JS running in
privileged contexts (dashboards) is a wider problem I would like to draw
attention to; 3) It's not immediately exploitable; 4) It takes five seconds to
understand what the problem is; 5) I did not want to wait 90 days to be told
"we are not going to fix this;" 6) my history of reporting problems (not
security related) to the company in question is that they are not very
responsive to fixing them.

~~~
kierenj
Umm, either its a problem and worth reporting responsibly, or not a problem
and so not worth a hit piece on your blog?

~~~
icebraining
It's only worth reporting "responsibly" if waiting might prevent users from
being hurt. The way that concept has been manipulated into implying that the
companies are owed an early warning is crap. The author doesn't owe CircleCI
anything.

~~~
kevinburke
Precisely

------
nathan_f77
No one has mentioned this, but I'm really curious about the script from Quora.
Why is Circle CI loading JS from Quora?

Or is that a browser extension that the OP forgot about?

~~~
kevinburke
I don't run browser extensions

------
motdiem
I think these kind of issues will become more and more important, both from a
security standpoint, and from a privacy standpoint as well. GPDR might apply
to api access from 3rd party tools (not just analytics btw) - it’ll be up to
them to prove they don’t, and I’m curious to see how it’ll be regulated.

I’ve tried to explore the privacy aspects here [1] (like many, we completely
separated libraries for the public site and libraries for the apps)

I think we need a framework when thinking about which companies to pick when
adding 3rd party features to our products - and probably a system to trust
such 3rd party with their data processing.

I’d be interested to know how companies make these decisions today...

[1] [https://blog.getkumbu.com/posts/building-a-privacy-
respectfu...](https://blog.getkumbu.com/posts/building-a-privacy-respectful-
startup)

------
Sir_Cmpwn
The web is a disaster. Why do people insist on adding this bullshit to their
pages?

~~~
richardknop
Marketing and sales people usually ask to have tons of third party scripts
loaded everywhere and devs don't have power to turn them down. They need
analytics for their work. It's a bad idea from security standpoint for sure.

~~~
soared
Yeah, management and marketing need web analytics. If you don't want 3rd party
scripts you'd have to, what, roll your own analytics software? There is some
open source analytics tools you could self host, but they're garbage compared
to google/adobe/etc.

~~~
kevin_thibedeau
> Yeah, management and marketing ...

should get off their duff and analyze their own internal server logs.

~~~
manigandham
Server logs dont have much useful information, especially for SaaS interfaces.
You need to have the events from the actual UI tracked, which requires a lot
more work and is the reason why these analytics tools are used.

------
coding123
If npm or even a single popular npm package is compromised, almost every
website will be compromised.

------
ivanfon
Does anyone know if Travis CI does something like this.

~~~
matthewcford
pretty sure they also use pusher

------
grandalf
Also, Github and Bitbucket should allow much more granular ACLs such as single
repo, single repo shallow clone only. Ideally they would also support some
form of steganographic tagging of repo contents as well, in case of a breach.

------
kevan
At my last company we had a similar issue with third party scripts on credit
application forms. Analytics is critical for ecommerce sites but it's way too
easy to accidentally log sensitive data across dozens of third party services.

~~~
bm1362
There are self-hosted analytics (like Piwik) that you could self-host for
sensitive data to solve this generally. I'd like to think that conversion of
credit applications isn't something a marketer is trying to gamify though,
that's probably wishful thinking.

~~~
kevan
>I'd like to think that conversion of credit applications isn't something a
marketer is trying to gamify though

If it's in the overall sales conversion funnel it will be optimized. Credit
application flow was scrutinized just as much as adding to cart and checking
out.

------
jbg_
uMatrix.

~~~
fluxsauce
That treats the symptom, not the problem.

~~~
beagle3
uMatrix is defense in depth. They could solve the problem today, and tomorrow
a new CircleCI programmer/marketer/CEO would decide that, in fact, they do
care more about analytics than about user's secrecy, and revert this change.

By installing uMatrix and not approving these things, you are (a) reducing the
probability that a mistake, misaligned incentives, etc, would harm you, and
(b) get a notification that there's something you wish to block.

uMatrix IS part of the real solution. The other part is avoiding providers
whose security standards are not up to yours.

~~~
45h34jh53k4j
Its a real solution, for YOU personally. I use and love umatrix (whitelisting
ALL js/content by hand on all domains).

We need to consider the good of the many, and client side solutions like
umatrix are not a solution for the many.

