I totally get the intention behind this and the final outcome is definitely a safer internet. It's also somewhat justified considering the author has mentioned they never controlled its domain, yet the library has been distributed through that domain, correct? This is a reflection of the extremely poor security practices in the web development world.
At the same time, there is a wrongness in Cloudflare being able to overwrite content and changing the 'truth'. I'm like that larry david gif, I just don't know how to process this.
One more thing to note, if you go to the polyfill repo, they've also mentioned they're using Cloudflare to distribute the library.
For me, this is why it makes sense: customers on Cloudflare signup (at least in part) for Cloudflare to protect them against attacks.
Cloudflare is all about changing the "truth". When your site is behind Cloudflare, they block many requests to your site. The lock in the browser can be lies. Cloudflare is decrypting that content - and if the site owner hasn't setup TLS between Cloudflare and the backend, it's being re-transmitted over the internet unencrypted. On paid plans, Cloudflare will compress images and swap in their version. Cloudflare will compress an uncompressed response before returning it. Cloudflare will take the HTML returned by your backend and obfuscate any email addresses in that HTML before sending it along to the browser.
Your server returns a page with evil-polyfill/bad.js and Cloudflare inspects the HTML and rewrites it to say good-polyfill/good.js
You might not want this behavior and you can shut it off in Cloudflare, but it seems like a reasonable default given that customers have signed up for a product meant to protect them against attacks. Cloudflare has never been about passing back the raw HTML it receives from the backend or passing along the raw requests it receives from browsers.
My question is when does cloudflare starts to inject ads into my content?
Is it “shots fired” situation?
Because you know bullies and other bad people start testing ground to see what they can get away with. That is why you slap them hard and quick on the first attempt right away so they see that they cannot fool around.
So I am asking can it be we have to say - well yeah technically they are right - but stop right there doing that and never do it again!
They could make terms that they don’t serve vulnerable things from their cache and it is up for the customer to update or fix it - but they shouldn’t overwrite stuff, period.
Who is the "we" here? If you're a Cloudflare customer, you're probably glad to have Cloudflare automatically defend your website against supply-chain attacks like this: after all, security is one of their selling points.
when does cloudflare starts to inject ads into my content
You content, as in, you the developer who is making a website, and has set up Cloudflare in front of your website? Presumably they wouldn't inject ads into your website, or else you'd stop using Cloudflare: just change the nameservers in DNS to point to someone else.
Cloudflare's primary raison d'être is messing with the response they serve to users - to perform caching, to inject CAPTCHAs when they detect a DDoS attack, etc.
If you don't trust Cloudflare to not abuse this to inject ads, stop using Cloudflare.
They can mess with response to make redirects/checks before my content is served, but my content is mine they should only cache it and that is it.
If they serve ads on their captcha or some redirect I'd say well it is not nice - but for me fundamental difference is messing with my content even in a good faith - send me a notification an email or stop serving my content if it is active malware but don't change it.
If you own a website and you really think this is a slippery slope, that it crosses some line in your sandpit, you can stop using Cloudflare.
But worrying about ad injection off the back of a legit good-guy action seems like an overreaction. They've always been able to alter the websites they serve. They do frequently to optimise and this seems like a very benign extension of that.
> At the same time, there is a wrongness in Cloudflare being able to overwrite content and changing the 'truth'.
This is one of the many features of Clouldflare though, that you can enable additional features that modify your website in various ways - whether it's image resizing or analytics or security scanning.
If you don't trust Cloudflare to deliver your website, then you shouldn't use Cloudflare.
I think GP’s point was more about the truth for the consumer of the site. As a user, I trust a site to deliver the content it intends to deliver. Of course this is just a trust by proxy with Cloudflare because you’re trusting the site’s judgment on who to trust. As a user of the site, it makes me just a little bit more uncomfortable knowing there’s another layer of indirect control over the content coming in.
Of course this is all just philosophical anyway. We’ve not even touched on external module usage.
That is what Cloudflare is doing. Using a Cloudflare security product intends on modifying the website it's proxying. Everyone involved in running the website - the original website owners and Cloudflare, intends for this to happen.
The one exception is Cloudflare turning this on by default for free 'customers', but hey, if you're not paying really what leg do you have to stand on?
Odd that github archived the repo claiming it contains malware. Was any of the malicious code even in the repo? I would guess it's only on the server they're serving the polyfill from, since it has very specific conditions for triggering.
Cloudflare can't just do this at will. This only applies to websites that have explicitly trusted Cloudflare with their TLS cert in order to protect themselves from cases exactly like this one.
Nobody tell him about the “Google Safe Browsing” censorship bloom filter list used by Firefox, Chrome, AND Safari.
Relevant today as ever given that the Supreme Court ruled that the 1A challenge to the executive branch asking big tech to censor doesn’t have standing to be ruled on.
Google has a big red button that can shut down any webpage on the internet for 99.9%+ of the web browsing public. There’s no bypass button in the UI like a TLS failure.
This is sort of exactly what cloudflare does, though. They host your content on their servers. They're serving a mirror of the original files from polyfill.io and browsers are getting the same original bytes. If it's a fact that pulling content from polyfill.io is a security issue, why wouldn't you want them to do this? The "truth" as you describe is a third party that you don't trust (who acquired a service you use from the original provider). The status quo is inherently bad. "We served your page for you, but without the injected code and with full compatibility" feels like an awfully nice thing for Cloudflare to do that they simply didn't need to do.
Cloudflare is rewriting the link on websites hosted on Cloudflare. I think it can be argued that those websites have in some sense opted in to some limited rewrites by Cloudflare? (Presumably they wouldn't be doing it unless their terms allowed it.)
"Contrary to what is stated on the polyfill.io website, Cloudflare has never recommended the polyfill.io service or authorized their use of Cloudflare’s name on their website. We have asked them to remove the false statement and they have, so far, ignored our requests. This is yet another warning sign that they cannot be trusted."
> We have taken the exceptional step of using our ability to modify HTML on the fly to replace references to the polyfill.io CDN in our customers’ websites with links to our own, safe, mirror created back in February.
I get that they want to do a good thing, but is this something you agree to when you sign up as a Cloudflare customer? If so, that’s kind of crazy.
edit: I am talking about both free and paid users. A toggle does not discount my question whatsoever, especially if this is on by default for free users. I am asking specifically about terms of service.
People are often using cloud flare to improve the security of their site. I think in this case if they failed to act when they could that would be worse.
> It does show that you have to trust cloudflare, which seems like a safe bet given their desire to keep the internet secure.
This is assuming that Cloudflare's interests will now and always align with your own interests, and that their desire to "keep the internet secure" will never lead to them actually screwing over you and your customers.
It's a lot like many classic AI Sci-Fi stories. If their mission ever comes into conflict with your real interests, a mission statement that sounds perfect in theory can suddenly become very very dangerous.
> This is assuming that Cloudflare's interests will now and always align with your own interests, and that their desire to "keep the internet secure" will never lead to them actually screwing over you and your customers.
This is literally the tradeoff of using any SaaS/PaaS/IaaS vendor. Every single one. It's not unique to cloudflare in any way.
Correct. I'm just noting that Cloudflare's mission to keep the internet secure doesn't make them a "safe bet" for any given company, no more so than any other provider.
"Safe" is subjective. Someone else with different values might consider them an extremely safe bet. After all, they're financially secure, reliable (by all accounts), trusted by lots of folks, and have a track record of not fucking up too badly.
True, but they say "we decided". That implies they could also have decided to turn it on immediately. And they did for the log4j and Shellshock issues.
I can’t think of an enterprise use case where someone would be upset they did that. can you? I’m struggling. I helped triage the log4j incident in a pretty traumatic week and was pretty happy they took the steps they did. It’s like, kind of what you are paying for.
Unfortunately, neither of these early warnings seem to have gotten much attention. I wonder if there is some news site or service that would make suspicious ownership transfers of this sort more noticeable? Or maybe this type of supply chain changes actually happen all the time and we just got used to them?
Hi Matthew, Cloudflare just charged my card $235 with no prior warning (for a service I intended to cancel when I received the renewal reminder). I did not receive any renewal reminders, no advance notice of payment, no invoice. Nothing.
Is this normal operating behaviour for Cloudflare? This seems like very very bad anti-consumer behaviour?
cloudflares business model is literally to mitm their customers. they incidentally also get access to unencrypted versions of all https traffic to cloudfare enabled sites
Setting a dangerous precedent, especially doing this by default (no opt-in) needed.
But then again, if the people who carelessly include 3rd party dependencies (i.e. playing with fire) are those who use CF... they probably won't object to it :-)
They're a US based man-in-the-middle for large parts of the internet at the mercy of the America's secretive FISA Court. There is no way they don't have a Room 641A.
They're US based now, but there's no guarantee it'll stay that way, if a buyer were to come along with enough money. I guess if your theory is correct then pressure would be applied to make sure any foreign buyout didn't happen.
In any case it's extremely unlikely to happen any time soon, but who knows what could happen 50 years down the line, especially if they were to lose market share or the internet fades from relevance.
I'm not accusing cloudflare of anything malicious, I want to be clear. But this polyfill wasn't originally malicious either, it was just eventually bought by a malicious actor.
My original comment was just commentary on the symmetry of the exploit and the mitigation, they're essentially the same vector.
While I appreciate their intention and their transparency on what they're doing, I'll echo other comments here in that I, too, feel a bit uneasy that a 3rd party is rewriting my content.
The good news is that a (strict) CSP can help with that - and they do mention it in their blog post that they don't rewrite anything if there's a CSP header.
It's also worth noting that a (strict) CSP also prevents them from injecting their analytics JS from being injected to your site.
I'm curious as to how Fastly can defend their choice to keep Jake Champion in their employ after the damage he caused by transferring polyfill.io to Funnull[1], triggering this entire incident.
The warning signs should have been obvious. There's no profit to be had in this kind of free CDN.
If you own the domain for a service used for free by many many websites and have encouraged people to use that service for free I think there is some responsibility to not transfer that to a bad actor.
If he simply made the DNS not resolve to a server anymore I'd be fine with it (and if giving people a warning a few months in advance would be great) but this is not inaction, this is selling trust. It's reasonable to have less than zero trust in anyone that would willingly sell my trust to an unknown third party for profit in this manner.
I think that people should have never used this service or used it with subresource integrity (which by design is not possible in this case), but that's not how it was pitched so now the owner has some responsibility if they want to maintain dignity and trust.
I think this touches on an interesting question. What obligation do free or open source project maintainers have?
Even if a maintainer slaps on a, “I do what I want with this project. I am not responsible for any damages. There is no support” disclaimer, I am not sure that necessarily removes some social responsibilities.
This is not an "open source project", this is a service. When I use a open source project I take it as it is now and take a risk on it not being updated, but any updates are "pull", as in that I willingly take in changes.
In this case the service is "push", which is very different. Any website that used polyfill.io can have any changes pushed to it, regardless of if the author even had known about a change being made.
If my popular project is replaced with a single poop emoji on NPM any existing user is fine (especially since NPM keeps old versions after the whole left-pad thing) and will find an alternative. If polyfill.io replaces their code with
document.documentElement.innerHTML = '💩'
that's not fine, since it affects existing users without any update step.
I think that nobody should use these public CDNs at all, including things like unpkg and cdnjs, or at the very least using subresource integrity. Either way this has been something that has been on the horizon for years and similar to the buying of popular webextensions.
I don't have an answer, but the idea that the person providing you with a free service owes you anything at all just reminded me of this Simpson's quote I think about sometimes.
---
Comic Book Guy : Last night's Itchy & Scratchy was, without a doubt, the worst episode ever. Rest assured that I was on internet within minutes registering my disgust throughout the world.
Bart Simpson : Hey, I know it wasn't great, but what right do you have to complain?
Comic Book Guy : As a loyal viewer, I feel they owe me.
Bart Simpson : What? They've given you thousands of hours of entertainment for free. What could they possibly owe you? I mean, if anything, you owe them.
I never claimed bad faith. Lack of professional competency can also cause poor decisions. As it stands, Jake hasn't even admitted to his mistake yet.
This isn't just about an open source project. It's about an online service. The owner could have simply shut it down instead of allowing it to be acquired.
Comments like this are part of why I'll never run a free service or open source project that people might grow to depend on. I'm not interested in taking on the risk that a mistake I make in my volunteer time causes an internet mob to call for the termination of my employment.
I imagine that your example scenario would involve talking to your users to keep their trust and owning up to mistakes if you were running a free service whose security ended up being affected by those mistakes.
Not in this case, for better or worse. The point of polyfill.io is they tailor the delivered Javascript based on the deficiencies of the user agent making the request, so it's not compatible with SRI.
Yeah, I agree, but also, what is the way out of it? By virtue of their value proposition, the only natural progression for a CDN is maximum scale, isn’t it?
> We have not outright blocked the domain through any of the mechanisms we have because we are concerned it could cause widespread web outages given how broadly polyfill.io is used with some estimates indicating usage on nearly 4% of all websites.
Frankly I find this to be a weak argument. "We'll continue serving malware because broken websites are worse than infected ones" is not a good argument. I'm a cloudflare customer and I think this is a very bad look, even if it does mean websites keep "working."
At the same time, there is a wrongness in Cloudflare being able to overwrite content and changing the 'truth'. I'm like that larry david gif, I just don't know how to process this.
One more thing to note, if you go to the polyfill repo, they've also mentioned they're using Cloudflare to distribute the library.
https://github.com/polyfillpolyfill/polyfill-service/commit/...
reply