"CircleCI trusts 8 analytics companies with your source code and API tokens, October 9, 2017. When you navigate to your project in CircleCI's UI, Javascript from eight different analytics companies gets loaded and executed in your browser."
So, did they do anything about that?
Even if using iframes and a secure api over postMessage/onmessage: safe, and security can be controlled by the external provider i.e. safer for both parties (ideally the servers pass the security context between each other - we even use an iframe internally for all the same reasons).
Google allows custom JS injection via Tag manager, so there are possible vectors, even if only access to 3rd party site was stolen. Of course I have no idea what was the deal here.
It's worth noting, that CircleCI did nothing to address issues raised in the linked post. Their main app still loads tons of third party analytics garbage for e.g. Google, Hotjar, Amplitude and even Facebook of all people. I do block all those, but as someone pointed out it's not a solution and cant even be reliable at all times.
Please bear in mind, that CircleCI not only has access to private repositories. More often then not they do store private SSH keys to your production servers.
Hi, I wrote that post. I don’t think that this is quite the same issue.
The 2019 post is vague but I think what happened is an attacker stole the credentials for e.g CircleCI’s account for Google Analytics. So an attacker could see URL’s that were visited, IP addresses, but they couldn’t run JavaScript on CircleCI’s domain.
The difference is between an attacker compromising the Google Analytics JavaScript snippet CDN and an attacker compromising a Google Analytics user account. The latter only lets you read the metadata not make any of the juicy AJAX calls.
(Also, it’s probably not actually Google Analytics, just listing a tool most people are familiar with).
It’s obviously not good but it’s not the same vulnerability I wrote about nor as bad as the one I described two years ago.
However if they haven’t resolved an obvious security issue in two years, that is a strong signal that security is a low priority, which is relevant to their risk of compromise (as has occurred).
Integrating third-party JavaScript has multiple known solutions that are secure, but they require choosing compromises and are usually significant work to implement. If CircleCI haven’t fixed serious security issues after 2 years then it it shows they probably don’t care enough about security - a very bad signal.
The question was: have they mitigated the security issue you raised, or ignored it?
I absolutely agree. This is a very professional example of handling a serious security incident. CircleCI immediately took action and notified all affected accounts afterwards and as with this post, they did a detailed FAQ-style breakdown of what happened and how they will stop another attack.
> Security is taken very seriously here at CircleCI.
Compared with the other security handlings i've seen, this statement sounds genuine.
I love the fact that it took them 10 minutes to first respond to the incident, and cut-off the account in under 30 minutes from the first sign something was wrong
> Security is taken very seriously here at CircleCI
...
> In the meantime, we will review our policies for enforcing 2FA on third-party accounts to the extent possible, and continue our transition to single sign-on (SSO) for all of our integrations
Well, if you had taken security seriously, the first thing you'd do is ensure 2fa on all accounts.
> On August 31st at 2:32 p.m. UTC, a CircleCI team member saw an email notification from one of our third-party analytics vendors and suspected that unusual activity was taking place in this particular vendor account.
A convention I like is that team members creating, updating or removing accounts should reply to any automated emails sent to mailing lists saying "this is me". Ideally with a link to the relevant issue or story.
What's sad about this is that CircleCI actually has a great product and is one of the nicest ways to do end to end automation for mobile development/releases. Having their pipeline in place actually feels quite liberating.
The sad part is that they take this for granted and liberate all your data and security weaknesses too to unknown third parties for either a weird ideological reason about interoperability or a small marginal profit.
I don't understand why they can't do both(interoperability and security), their main profit is not data, it's customers paying for infrasctructure. They don't have to share any data at all and are fully capable of providing a competent level of security.
If anything I think if they cut out the profit line from sharing metrics etc with third parties it would actually bring more customers in because of increased confidence in the product.
Ex-Circler here, I was there for the second half of 2018.
The product is the source of profit, not the data.
I was never asked to work on anything relating to sharing metrics for money, and didn't hear of any such product in my time there. That's not the business model.
In fact, some users the only thing they know is the VCS username, and only then because that user has pushed a commit that has triggered a build: the user data they have for most users is literally worthless to any other party.
I doubt they are getting paid to share any data at all. Usually, the point of 3rd party services is for improving the service or keeping it bug free and functional.
I would imagine they have some services that are logging JavaScript errors and probably some that are tracking how people use the service.
If you have a Freemium or tiered service especially, you have to understand how people use your service in order to convince them to pay or upgrade.
Do you have any sources to back your claims that they sell your data? You repeat that all over your comment, but the crucial piece (the evidence) is lacking.
I ask this because as a Circle customer, if they are selling my data, I'll be out quick. But I see no indication they're doing anything of the sort.
Fanned out, or triggered builds, don' trace well. User api token env variable is wierd and docs are terrible on that point and others. Wanna see a list of projects? Go to "add project". You are required to "follow" projects in some cases. There is no search build feature. There is no real filter builds feature. The paging is terrible. No date search feature. No parameter search feature. If I go to circleci.com even thoigh I am logged in I see a big fat hero advert about how great circleci is and who their clients are. (Hey I'm your client ever thought about that?) Cant name your individual builds. Cant tag your individual builds. Build parameters even when fanned out dont show so 10-20 builds are indiscernible in a monorepo. I could really go on all day, this is not half of it. It just comes across as this big legacy software blob that have too many bad design decisions to revamp and will just go irrelevant through time and they instead of improving their product to gain new customers they just hold on and bleed their existing ones as long as they can. Suppose it makes financial sense though in their defense, their code must be in a total deadlock.
> we will review our policies for enforcing 2FA on third-party accounts to the extent possible, and continue our transition to single sign-on (SSO) for all of our integrations
I think it is more likely the CircleCI didn't enforce 2FA or SSO[1] with the vendor, and a CircleCI employee reused an already compromised password.
[1] Nearly all SSO identity providers support 2FA, so using SSO is often the easiest way to get 2FA with third-party SaaS providers.
Naming the vendor in this case should be fine. Shouldn’t it?
In general what I’m missing is why they share my email and github account with a 3rd party. GDPR should prevent them from doing so (at least for European users) without a legitimate reason, or explicit consent.
That's not exactly how GDPR works - the criteria depends on what you mean by "share with a 3rd party".
If you as the data controller are allowed (according to GDPR) to do some particular processing of some particular data, then GDPR also allows you to subcontract that processing, and thus provide the data to third parties (data processors).
There are a bunch of restrictions in place (you have to have a specific contract mandating that they only do the thing with the data you're contracting them to do, you're responsible if they violate that, some restrictions on "exporting" data out of EU), and you have to inform the customer to what third party 'data processors' you (as the data controller) are outsourcing these activities - so CircleCI would be required to give an exhaustive list of all the data processors to which they have given your data, but you don't need explicit consent or a very particular reason for that outsourcing.
What is prevented by GDPR is the common (at least in USA) "sharing data with trusted third party partners" where the data is essentially sold to third parties which are free to use that data as they wish for their own business or re-sell it further. That would require a very particular reason or explicit consent, but this is not the common scenario of outsourcing to a GDPR-compliant vendor.
My (admittedly limited) understanding of it is that there still needs to be a legitimate reason to provide my email.
So if the email is required in order to, say, send me notifications about CircleCI jobs that are running. That's a legitimate reason. If they share my email with the 3rd party to, for example, optimize their onboarding flows, then this isn't legitimate, unless I explicitly gave my consent to it.
To me, "analytics vendor" as they stated it, usually implies something similar to the latter rather than the former. But since they didn't indicate the provider, nor the reason, I can't really say.
I think we're not in any disagreement here by the way.
I'm not sure "make graphs for us" is a legitimate reason to share personal information (and email is considered personal information under GDPR).
Error logs might be a bit different I suppose, if they are used to help resolve specific issues for your users, but even then I suppose one could argue that pseudo-anonymised data would be better than sharing an email address.
I'm no GDPR expert, but that's my understanding of it.
I recieved the email from them a few hours ago, because of the wording of that, and because I spend last night integrating segment's analytics on my project.
I might suspect bc of their wording, segment and not google analytics would be such said 3rd party, and it seems it was one of the team's accounts in such service which got compromised and created a new destination, and no one noticed until 2 months later..
This is just pure speculation, but I would like to know!
Yeah I just saw it on HN homepage, By the timing of it an alert about -segment- or just a segment employee watching this comment thread might have forced them to come clean sooner than they were planning to?
If I have the choice, I will not be starting any project with them in the future after the crap they pulled with CircleCI 2.0.
"We're introducing this new, 2.0, version. It has some benefits, but is also missing a lot of features and bug ridden. It's easier for us though, so you're going to have to waste a couple days converting from 1.0 to 2.0."
My feelings as well. What we really need is a ansible script or a docker file, which would build a Jenkins instance which everyone could use. You would of course need different tools for various build setup and technologies, but CircleCI is far from point and click anyway. I really which I would spend all the time I wasted on their config to build the above. This way everyone could run 5000$/mo of CircleCI on a 50$/mo bare metal.
You might want to try Azure Pipelines. It uses a dual model of Microsoft-hosted build agents running several OSes including macOS, and self-hosted build agents running on infrastructure of your choosing. The team I lead uses for Windows-, Linux-, and macOS-based builds, Microsoft-hosted and self-hosted build agents.
If I finally decide to pull the plug I will probably invest time to build my own setup around OS tooling. Main benefits would be much lower price and independence from lock in. And it's not like one has to build their own solution here. There are tools available.
CircleCI has been very frustrating for us recently.
We have a lot of problems with waiting for builds to start - it seems CircleCI don’t have even remotely enough capacity at peak times. We regularly get 10min plus delays to builds starting, sometimes it’s over 30 mins!
Mix that with this security issue... it’s not a rosy outlook for us.
Edit: oh, and the UI has terrible problems with caching and not updating state. Constantly have to hard refresh pages without cache for build status to update.
The initial reaction is to TL;DR as weak passwords and no MFA but I am equally/more concerned that the policy granted to vendors allows them to access more data than they really need, or the data they must have access could be better de-identified. For example they could hash/uuid the email and the git repo and branch info.
Hacks are going to happen; no password or MFA will prevent it. Zero trust in your policy/schema design minimizes blast radius in these inevitable cases.
The idea of treating email addresses as secrets is a nice idea in theory but given the plethora of databases where it’s already stored as a user name, including those that have already been compromised, and how email addresses are published in the git commits anyway, I don’t think it makes much sense hashing email addresses in practice.
Repository details are another interesting point though. They too are published openly in any git clones but I accept that some people’s repositories, including any local clones, would be private and kept secured from prying eyes. That said, what matters more there is access credentials, encryption at rest (for local clones), etc. So keeping the branch names and repository URIs secret feels a little redundant compared to the actual security benefit it brings. One might even say it’s security through obscurity, though I don’t personally think it offers up even enough benefit to fall under that category. That said, I’m happy to be convinced otherwise if you feel I’ve overlooked some important detail.
I would like one company to post an "incident" reveal in a more honest way:
"
Due to our carelessness and relatively insecure practices, we had improperly disclosed user accounts to a moderately savvy hacker. We realize this is our fault.
If you'd like to help and given that we have your attention now, it would be valuable if you can help pentest our servers: the attacker used a simple SQL attack based on an unpatched server via CVE-3245. Are we missing anything else? Please let us know.
