Hacker News new | past | comments | ask | show | jobs | submit login
Cloudflare CDN Partial Outage (cloudflarestatus.com)
114 points by ericholscher on Oct 25, 2022 | hide | past | favorite | 60 comments



It's not DNS

There's no way it's DNS

It was DNS


This is better and more relevant with each passing year.

Beautiful calligraphy, too.


This seems to have affected many services:

https://downdetector.com/

Shopify admin pages and "some storefront images" were down for a few hours and I suspect its related to this cloudflare outtage.


Interesting Amazon shows up there who doesn't use CloudFlare.


I'd just chalk that up to DownDetector having a false positive. They aren't directly looking at those services but rely on sources such as users checking Downdetector and reporting problems, sentiment analysis of social media, etc.


Fix is rolling out.


Is this related to switching the firmware on the Minitel 2?



I'd like to read an article about how the firmware caused the outage. It wouldn't be true, but it'd be a fun read.


10 years ago we bought every Minitel in existence from eBay and connected them into a giant X.25 network spanning the globe. Unfortunately, one EPROM didn't have a cover over its window and bright sunlight in Madrid caused firmware corruption.


> 10 years ago we bought every Minitel in existence from eBay and connected them into a giant X.25 network spanning the globe. Unfortunately, one EPROM didn't have a cover over its window and bright sunlight in Madrid caused firmware corruption.

What a tease. If only.


Thank you! What happened out of curiosity?


Team is writing it up. Will get it out later today.


We were seeing some pages with cookies incorrectly getting cached, causing users to get logged in as other users. We quickly disabled cache on the dashboard, is this related to that?


Curious what you find happened with that.

On the webdev side—what can be done as extra defense-in-depth step to guard against this kind of issue? Unrelated to Cloudflare I feel like it is a common issue that crops up on even massive sites quite often. Is there some sort of secondary check / content decryption that could be required on the client-side to contain session cookie crossover?


Outside of HTTPS, typically it would be tying the session cookie to the IP address or netblock but because it's Cloudflare IPs, for regular browser navigation requests I don't think there's anything that can be done?


Please email me details (jgc).


Awesome - thanks :)


Doubtful anyone would post anything here before there's time to write a proper RCA.


Many thanks sir, from a company that uses Cloudflare Enterprise for delivery.


(Parent is CTO of CloudFlare for those unaware)


I wish my CTO even knew what HN was.


Some would be happy if their CTO knew anything about computing!


Nice. In that case I'll try to get an update on Pingora's release date. Haven't heard anything about it lately :(


Got a similar message on Digital Ocean for their App Platform/static pages, does DO use Cloudflare underneath?


Yes, DO uses Cloudflare to at least protect (and cache) their main website.

https://hostingchecker.com is a good resource to see where a website is hosted.


Looks like that site has a misleading name. It isn’t telling me who I am hosting with. It’s only telling me that I have cloudflare in front of it.


If Cloudflare is masking your origin to provide WAF capabilities, which it does for most - then I'd say this is expected results.


It is expected result yes. Not disagreeing with you on that. I am simply saying that CF is not "hosting" the site. It's the middle man. The host is hidden.


WhatsApp and Cloudflare on the same day?


iMessage as well


Nice decentralization.


Yeah because BGP never goes haywire.


Does this mean I won't get annoying captchas on some of the sites I visit since Cloudflare seems to be using their services to train some AI model.



I would have liked to read that. Instead here's what I see:

> Checking if the site connection is secure

> Enable JavaScript and cookies to continue

> blog.cloudflare.com needs to review the security of your connection before proceeding.

What does it mean to "review the security" of my connection?

Wouldn't that be my business? (Feel free to review the security of your connection by all means) :)

Why would that "need" running JavaScript here on my browser (which I don't for fairly obvious security reasons) Other websites seem to have no problem delivering basic content without that.

Also, no thank-you to cookies. We're not entering into a "session" relationship here, I merely wanted to read the document you advertised at the URL.


This is not specifically about Cloudflare’s “challenges“/etc, but —

The reality of operating a big site/service on the internet in 2022 is that it’s sometimes necessary to use methods that annoy a few people (with very non-standard browser settings) in order to protect the service as a whole from a million bots trying to attack it at any given time.


> operating a big site/service

This sounds like a very plausible argument. I've heard many of the arguments and don't dispute the threat model to something like Cloudflare.

And yet something about it still doesn't add up.

It turns power into a weakness.

How is it that much smaller sites - still able to serve something as simple as a plain-text blog to millions of users from a modest rack shack - operate perfectly well without any impediment?

Wouldn't an operation with all the power, might and money of Cloudflare be able to do a better job and still maintain the QoS (accessibility, interoperability etc) as Basement Bob with her Raspberry Pi?

Remember, all I want to do here is read a static web page of (I guess) less than 1000 words.

I'll take a punt: if "defending against millions of bots" is Cloudflare's business offering, then being able to serve a static site off a Raspberry Pi doesn't look good :)


Is your claim that Basement Bob’s raspberry pi could withstand the kind of attacks that companies like Cloudflare handle?

Eg - https://blog.cloudflare.com/26m-rps-ddos/


I think the parent comment's claim is that serving a CAPTCHA page to potential attackers may actually be more resource intensive than serving a lightweight page that has the actual content on it.


So, you posit, that Cloudflare has never thought of this before or weighed the pros/cons before building an entire business out of it? Sure.


Typically, the pages that Cloudflare protects are not especially lightweight or efficient.


No. It's that Basement Bob's Raspberry Pi doesn't need to.


I don’t think your line of reasoning was very coherent, and displayed a lack of understanding of reality.


Cool. That is certainly your choice. It is also the choice of the website operators whose sites you try to visit to block your traffic since you won't opt in to their security precautions.


But I don't invite people to my house and then slam the door in their face.


Your house isn't also accessible by anyone in the world, anywhere, at any time. People who complain about this stuff conveniently ignore that a massive amount of traffic to public websites is malicious and automated in nature. It's not crazy that website operators want to block that stuff. They would rather block the .0001% of people who choose to block javascript than expose their site to the junk. You can rage at Cloudflare all you want but if it wasn't a service people wanted then they wouldn't be offering it.


> .0001% of people who choose to block javascript

The measured number of browsers blocking JavaScript varies between about 0.7% and 4% with an estimated global average hovering around 1.5%. Here are some sources [1,2,3].

[1] https://www.cotsweb.com/blog/how-many-people-have-javascript...

[2] https://www.searchenginepeople.com/blog/stats-no-javascript....

[3] https://deliberatedigital.com/blockmetry/javascript-disabled


No, but you do have undiscussed conditions to enter...


Explains why I start getting captchas due to using uBlock to disable JavaScript. How freaking annoying!


I see a bunch of corporate talk about how they use captchas but now it is a different type that is less difficult?


Here is the status messaging, pasted below for context (since it'll ostensibly be scrubbed once the issue is resolved):

--

Increased HTTP 530 Errors

Identified

The issue has been identified and a fix is being implemented.

Posted 1 hour ago. Oct 25, 2022 - 16:29 UTC

Investigating

Cloudflare is investigating an increased level of HTTP 530 errors.

We are working to analyse and mitigate this problem. More updates to follow shortly.

Posted 2 hours ago. Oct 25, 2022 - 15:53 UTC

This incident affects: Cloudflare Sites and Services (CDN/Cache).


Also, I was unfamiliar with the 530 status code; turns out it's unofficial and somewhat arbitrary:

> 530 Site is frozen

Curious what this means in Cloudflare land.

https://en.m.wikipedia.org/wiki/List_of_HTTP_status_codes


> 530 Site is frozen

That's how Pantheon uses the code, not how Cloudflare uses it. The wikipedia article you linked has a whole section dedicated to Cloudflare's use of unofficial error codes.

Specifically, it doesn't mean much in particular for Cloudflare other than "check the other error code we returned elsewhere in the response to see what the actual issue is"


The only question I have is: how partial? I would hope they designed so it's impossible for a single change to affect the entire network.


Yesterday, I started getting HTTP 409 errors on some of the domains. It's still happening - wondering if this is related at all.


Experience shows that there are also a plethora of 403's with "cloudflare-nginx" in the message.


Is this impacting MS Teams? Been having screen sharing issues all day


That's probably just Teams impacting Teams.


One hopes that teams is not just a steaming pile of garbage, but the hopes are always dashed...




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: