
Cloudflare Network Performance Issues - drcongo
https://www.cloudflarestatus.com/incidents/tx4pgxs6zxdr
======
buildbuildbuild
Cloudflare: Your status page showed "all systems operational" for over 20
minutes while your primary domain was returning a 502 error. Please change
this to update automatically, many other engineering teams depend on you.

[https://i.imgur.com/qHBM2JW.png](https://i.imgur.com/qHBM2JW.png)

~~~
matt_oriordan
Sadly this reminds me of AWS outages too where the same applies. How is it
that hundreds of developers know there's an issue before AWS do, or Cloudflare
in this instance. See my blog post on similar AWS uptime reporting issues at
[https://www.ably.io/blog/honest-status-reporting-aws-
service](https://www.ably.io/blog/honest-status-reporting-aws-service).

At Ably, our status site had an incident update about Cloudflare issues being
worked on (by routing away from CF) before Cloudflare did:
[https://status.ably.io/incidents/647](https://status.ably.io/incidents/647)

We have machine generated incidents created automatically when error rates
increase beyond a certain point stating "Our automated systems have detected a
fault, we've been alerted and looking at it". See
[https://status.ably.io/incidents/569](https://status.ably.io/incidents/569)
for example. I think much larger companies like Cloudflare and Amazon could
certainly invest a bit in similar systems to make it easier for their
customers to know where the problem likely lies.

~~~
samstave
Heh, I am reminded of when the control plane at AWS went down... and we had a
custom autoscaling config that would query for the number of instances running
and scale appropriately... but when the AWS API died... we kept getting zero
running instances...

So our system thought none were running and so it kept launching instances....

These were SPOT instances and thus only cost like .10 per hour...

But we launched like 2500 instances which all needed to slurp down their DB
and config - so it overloaded all other control plane systems...

We had to reboot the entire system. Which took forever.

The only good things was this happened at 11am - so all team members were
online and avail... and then AWS refunded all costs.

\---

The other fun time was when a newbie dev checked in AWS creds to git - but he
created the 201th repo (we had only paid for 200) -- and as it was the next
repo which wasnt paid for, it was by default public - thus slurped up by bots
asap - which then used the AWS creds to launch bitcoin mining bots in every
single region around the globe. Like 1700 instances.

The thing that sucked about that was it happened at like 3am and we had to
rally on that one pretty fast. AWS still refunded all costs...

~~~
therein
> but he created the 201th repo (we had only paid for 200)

That's an odd choice of a failure mode.

> AWS still refunded all costs...

Yeah they should. It was their silly design choice that lead to disclosure of
secrets after all.

What kind of failure mode is that even. Failing to create the repo would have
led to a better user experience for sure.

Can you imagine if S3 charged more for private objects and once you reach your
count, it just makes them public and posts them on Reddit?

~~~
joncrane
>It was their silly design choice that lead to disclosure of secrets after
all.

Wait, was the 200 private repos issue an AWS thing or a GitHub/GitLab/whatever
thing?

What AWS product has a concept of private/public repos and limits on how many
of the former you can get for a certain price?

~~~
samstave
It was a git thing.

Never post aws secrets to git.

------
profmonocle
Once cloudflare.com came back I decided to check out their business SLA, and
it's not very encouraging:

> For any and each Outage Period during a monthly billing period the Company
> will provide as a Service Credit an amount calculated as follows: Service
> Credit = (Outage Period minutes * Affected Customer Ratio) ÷ Scheduled
> Availability minutes

\- [https://www.cloudflare.com/business-
sla/](https://www.cloudflare.com/business-sla/)

So assuming an outage affects 100% of your users (this one seems like it did,
but that's not clear), they only refund the time the service was offline?
According to pingdom this outage lasted ~25 minutes, so that's 25/(31 * 24 *
60) = .056% of our bill, roughly 11 cents.

It sounds like you just don't pay for the time the service didn't work, which
isn't much of a guarantee, that's just expected (of course you shouldn't pay
for services not provided). Most SLAs for critical services have something
like under 99.99% uptime you get 10% of your bill back, under 99.5% you get
20% back, under 99% you get 50% back. (*Numbers completely made up to
demonstrate the concept.)

Am I misreading this? Morning coffee hasn't kicked in yet so maybe I am.

~~~
btown
This doesn't surprise me at all - SLA's are widely overrated. No SLA will
cover damages incurred by lost business due to an outage. What you likely want
is some kind of third-party insurance for downtime caused by outages out of
your control - but I'm not even sure this exists.

~~~
kortilla
I guess you’ve never worked in enterprise. SLAs for critical systems very
frequently incur payback in excess of the billing on outages.

~~~
dboreham
But not consequential losses which is what the parent mentions.

~~~
profmonocle
I'm definitely not suggesting CF should cover losses. Sorry if I gave that
impression. That would effectively require them to be an insurance company
since they'd have to investigate claims, and possibly charge customers
differently based on risk. (i.e. you don't want to bill a customer $200 per
month if 10 minutes of downtime could lose $20 million in sales.)

I mean something like Amazon EC2's SLA
([https://aws.amazon.com/compute/sla/](https://aws.amazon.com/compute/sla/))
where credits are proportional to downtime, but not 1:1. i.e. they credit 100%
for >= 5% downtime. With Cloudflare's SLA, 5% downtime (1.5 days in a month)
would only give you a 5% credit.

------
grey-area
Throughout this outage,
[https://www.cloudflarestatus.com/](https://www.cloudflarestatus.com/)
continued to show green - all services operational, with almost all services
marked 'operational' and some vague cryptic message about users in this region
being affected:

 _Investigating - Cloudflare is observing network performance issues.
Customers may be experiencing 502 errors while accessing sites on Cloudflare.
We are working to mitigate impact to Internet users in this region._

It seemed very much like a global outage affecting all services. Is this
status page not automatically updated with service status, or is it just
manually updated by humans? Even if manually updated, surely when posting that
status message, the status of all the services should be set to degraded?

~~~
NikolaeVarius
This is not my experience, I have received many updates both through email and
updates to the cloudflare status page throughout the incident, except for
possibly the first 10 minutes

~~~
grey-area
You're probably talking about the yellow note at the top of the page (which is
still there, with a little more detail now). That was updated and is fine.

I'm talking about the service indicators for each service lower down the page,
which remained green throughout and appear to be just decoration, not an
actual indication of service status, they all said operational throughout the
incident (I reloaded a few times).

In particular I'm thinking of the _Cloudflare Sites and Services_ section.

------
dessant
A great example for why you shouldn't transfer your domain to Cloudflare
Registrar if you're also using their CDN. Those who have transferred their
domains cannot change DNS servers to mitigate the outage.

~~~
nullify88
You can setup your domains using their CNAME method. You do not have to
delegate your entire domain to them. [https://support.cloudflare.com/hc/en-
us/articles/36002061511...](https://support.cloudflare.com/hc/en-
us/articles/360020615111-Configuring-a-CNAME-setup)

Together with a short TTL we were able to recover without relying on their
dashboards.

~~~
chaz6
Only if you are a paid customer. The free service does not allow this, and
this is why I do not use Cloudflare for personal use.

~~~
dewey
For personal use you also probably don't need the high availability of
switching over your domain the moment they are having problems?

~~~
MrStonedOne
One thing is not every free or pro plan on cloudflare is personal use.

I'm running the web servers, official wiki, and game external resource portal
for the most active open source video game on github, through cloudflare, and
maybe we might not want our 60 million requests a month website to go down
when cloudflare does.

Because I can tell you right now our 300 a month budget (that mind you, is
capable of covering 7 game servers that can handle 100 connected players
(each)) can't take the 80 dollar hit just to make cloudflare not a single
point of failure.

~~~
dewey
If you don’t count this as a personal or hobby project it’s a community
project. They don’t have a pricing plan for that so you either have to go pro
or go to some other provider who gives you this much for free. Why should a
company give you even more pro features for free if you are already getting a
lot of things for free?

~~~
MrStonedOne
Pro($20) does not give you the cname feature, business ($100) does.

------
TomAnthony
They have now released an initial statement [1]:

 _For about 30 minutes today, visitors to Cloudflare sites received 502 errors
caused by a massive spike in CPU utilization on our network. This CPU spike
was caused by a bad software deploy that was rolled back. Once rolled back the
service returned to normal operation and all domains using Cloudflare returned
to normal traffic levels._

 _This was not an attack (as some have speculated) and we are incredibly sorry
that this incident occurred. Internal teams are meeting as I write performing
a full post-mortem to understand how this occurred and how we prevent this
from ever occurring again._

[1] [https://blog.cloudflare.com/cloudflare-
outage/](https://blog.cloudflare.com/cloudflare-outage/)

~~~
technonerd
Ahh the "we test in prod" method

~~~
jgrahamc
Yeah, except that wasn't meant to be the way things work.

------
pmlnr
Good.

I'm tired of people not learning that trusting a single gateway with 50% of
the internet is bad.

Yes, I know, free DDOS protection. There has to be another way of doing this,
some mesh based DDOS protection or so.

~~~
lkbm
This seems like a case _for_ putting more of the internet through a single
gateway. Having my downtime correlated with everyone else's means users will
be more forgiving because they'll perceive it as "the Internet's down" rather
than "lkbm's site is broken". (We saw this with CloudFlare. Some users were
pissed, and others jumped in with "it's not their fault; AWS is down". That
doesn't happen when our stuff specifically goes down.

We need to decentralize the Internet, but this occurrence is not the reason.
It's an argument to keep consolidating.

~~~
hoechst
How cares whose fault it is? In the end, you're not able to provide service to
your users and you potentially lose money.

~~~
pmlnr
Business risk cares. SLAs, SLOs, SLIs, they are all about this; to be able to
direct the blame.

------
tmlee
So it seems... Even [https://cloudflare.com/](https://cloudflare.com/) itself
is down

~~~
btown
More importantly, their admin dashboard is down. It's impossible to bypass
their "orange cloud" proxies and send traffic directly to our hosting. That
they can't flip a switch and have their nameservers send dash.cloudflare.com
to a separate piece of redundant infrastructure is mind-boggling.

~~~
ilogik
change the nameserver at the registrar to someone else

~~~
skrowl
By the time that change propagates, cloudflare will be backup

~~~
skywhopper
Yes, but then next time you will be able to control your DNS.

~~~
scaryclam
Though, only if you're using a short TTL. I'm not arguing against your
position in general though.

------
deca6cda37d0
7,5% of all websites...

[https://w3techs.com/technologies/details/cn-
cloudflare/all/a...](https://w3techs.com/technologies/details/cn-
cloudflare/all/all)

~~~
snug
9.6% of websites that have one of those CDNs

~~~
deca6cda37d0
aah i misread so 7,5% of all websites

~~~
lgats
How about a weighted percentage of the Alexa top 1m?

------
svirelka
Update - Cloudflare has implemented a fix for this issue and is currently
monitoring the results.

Description: Major outage impacted all Cloudflare services globally. We saw a
massive spike in CPU that caused primary and secondary systems to fall over.
We shut down the process that was causing the CPU spike. Service restored to
normal within ~30 minutes. We’re now investigating the root cause of what
happened.

------
auscompgeek
[https://www.cloudflarestatus.com/incidents/tx4pgxs6zxdr](https://www.cloudflarestatus.com/incidents/tx4pgxs6zxdr)

(edit: thank you mods for changing the submission URL)

~~~
bytebuster
oh wow. that's a long list.

~~~
yoran
aka the whole world.

------
danShumway
Don't want to go off topic, but if I want to prevent my website going down
because of stuff like this in the future, will having back up DNS entries
solve the problem?

I know DNS will fall back if it can't reach a service, but would a 502 trigger
that?

~~~
aclelland
Yeah, you'll want to keep your domain registered through a different registrar
and if CF goes down you can update your DNS Name Servers point from CF to
something like AWS Route 53.

This has a few drawbacks like making sure your Route53 configuration is
identical to your CF config, ensuring your origin servers can cope with the
additional load if CF caching isn't available and the DNS propagation time
required for the Name Servers to update.

During the last outage, we were able to get into the CF dashboard and simply
disable the proxy which allowed our clients to access our origin servers
directly but this time we can't even get into the Dashboard.

~~~
danShumway
Yeah, if I had access to the DNS records this would be easy, but like you
said, even the dashboard is down.

Ideally, I'd want something where if Cloudflare goes down, I don't have to
change anything, but... 502 isn't going to trigger that without some work on
my part.

Meh.

------
gadgetoid
I just changed a setting in my CloudFlare account... did I break everything?

~~~
darkcha0s
Oi! Change it back!

~~~
Gigablah
Well, they can’t now :p

~~~
gadgetoid
Touché.

On one hand I think "Maybe I should diversify my infrastructure."

And on the other I think "But one of the biggest upsells was convinience."

And it's fortunate I don't have a third hand, because I'd be thinking "Oh crap
oh crap I just migrated a client website to LightSail + CloudFlare saying how
super awesome and robust it would be."

But it's okay now because it looks like everything is back up!

------
Thaxll
After trolling Verizon curious what reasons they will come with for that
outage.

~~~
voidwtf
Care to elaborate how they were "trolling" Verizon?

~~~
EamonnMR
They came off that way in this blog post:

[https://blog.cloudflare.com/how-verizon-and-a-bgp-
optimizer-...](https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-
knocked-large-parts-of-the-internet-offline-today/)

~~~
acdha
That didn't seem like trolling – just a public call for Verizon to follow
internet best practices. Given that most large ISPs treat failures as a PR
exercise, that's probably necessary.

~~~
voidwtf
Agreed.

They reached out to Verizon privately, a Tier 1 carrier with expectations and
responsibilities as a good netizen, and got no response.

They attempted to reach out through Verizon's public forms of communication
and got a bullshit irrelevant CS response despite requesting escalation.

They then called out Verizon before the community as a whole.

They don't have the luxury of waiting for a well prepared letter from some
Verizon lawyers. Modern day customer expectations don't allow for it. You may
call it trolling, but all I saw was a company asking another company to stop
pissing in the public pool.

------
jason_zig
So what should the ideal redundancy plan be here? If you can't log into the
CDN provider and they are down do you you just have a second one ready (and
paid for) and then log into your registrar and be ready to switch to that
secondary CDN provider in this scenario? Or is there some sort of load
balancing / routing solution between CDN's that I don't know about /
understand?

~~~
morpheuskafka
If you use Cloudflare nameservers, you have to change to new nameservers, wait
for that to propogate, and then wait for clients cached records TTLs to
expire. So it will be a major disruption no matter what you do.

~~~
manigandham
If you're using them for TLS certs then it's an even bigger problem unless you
have them provisioned elsewhere.

~~~
morpheuskafka
Unless you need EV you can just pull some wildcards from Lets Encrpt (as long
as you don't use pubkey pinning). No need to automate as it's just a one off.

------
r1ch
The Cloudflare DNS-over-HTTPS resolver was serving up 502 errors as well,
though the standard port 53 UDP resolver was working. This event definitely
made me regret choosing Cloudflare as my sole DoH server.

~~~
captn3m0
Hear hear Mozilla!

------
yuchi
Either downforeveryoneorjustme.com is itself served by Cloudflare too, or has
been hugged to death.

~~~
bwb
Yep, we are 100% on CF workers :)

Here is a quick image of the peak downtime on downforeveryoneorjustme.com:

[https://ibb.co/PZ9BMRc](https://ibb.co/PZ9BMRc)

~~~
yuchi
Thanks for the fantastic work on the service. And thanks for sharing that
stats!

------
cube00
Not impressed it's serving an error page claiming the underlying host is to
blame (this one from discord)

Error 502 Bad gateway

You - Browser - Working

Sydney - Cloudflare - Working

storage.googleapis.com - Host - Error

What happened? The web server reported a bad gateway error.

------
jrwiegand
Seems fitting that Cloudflare spoke so aggressively against Verizon[0][1] last
week and then this incident happens to them. I will be interested to read the
postmortem on this situation. I really like Cloudflare but you should be
careful not to jinx yourself with blogs posts like that.

[0]: [https://blog.cloudflare.com/the-deep-dive-into-how-
verizon-a...](https://blog.cloudflare.com/the-deep-dive-into-how-verizon-and-
a-bgp-optimizer-knocked-large-parts-of-the-internet-offline-monday/) [1]:
[https://web.archive.org/web/20190628223129/https://blog.clou...](https://web.archive.org/web/20190628223129/https://blog.cloudflare.com/the-
deep-dive-into-how-verizon-and-a-bgp-optimizer-knocked-large-parts-of-the-
internet-offline-monday/)

------
sprite
When you can't update your DNS because your registrar uses CloudFlare also....

~~~
stevekemp2
You could use a dedicated dns provider, such as AWS

------
frenchman99
Any workarounds or solutions ? I'm an on-call engineer with lots of questions
coming in. I'm not sure what I can do apart from moving the domain off
Cloudflare, bug DNS propagation would take a few hours and by then Cloudflare
might be up again.

~~~
NKCSS
Outages can always happen, when they do with companies like this, at least
you'll know that some of the best people out there are working on the issue
and that it will be resolved asap.

CloudFlare has proven in the past to be a very capable party, I don't think
panicking now and try to move everything away is a smart move. Also, a few
people have been saying that even if you want to, the site to do so is not
reachable, so that would be a challenge as well.

~~~
frenchman99
Panicking was not the plan. Asking for advice was.

------
Nas808
1.1.1.1 DNS is down, and seeing a lot of 502s on cloudflare sites.

~~~
jakear
The 1.1.1.1 vpn still seems to be working

~~~
eternalny1
Not here it's not (NE US).

~~~
wilsonfromdevon
dns is working in London UK.

------
adamparsons
I thought I was going insane, googling it returned nothing, but trusty old
hacker news has my back

------
heavymark
Holy crap, cloudflare down, and seemingly all the covers. Major sites such as
Digitalocean are all down, and no way to easily disable cloudflare since their
site is down.

~~~
hu3
[https://www.digitalocean.com/](https://www.digitalocean.com/) frontpage
working for me:
[https://i.imgur.com/8yJRXVI.png](https://i.imgur.com/8yJRXVI.png)

~~~
bpicolo
The outage appears to have ended between that post and your post.

~~~
rc_kas
Yeah, not a long outage, but a big one.

------
seibelj
Single point of failure - we all shouldn’t trust CF alone anymore...

~~~
zzzcpan
Any CDN is a single point of failure and limits your availability to as low as
three nines. Although anycast-based CDNs like Cloudflare are much less
reliable than DNS-based CDNs, those can do orders of magnitude better.

~~~
cortesoft
Why do you think anycast based CDNs are worse than DNS based ones?

~~~
zzzcpan
They rely on a single network infrastructure as opposed to many independent
networks with independent edge nodes where isolating faults is rather trivial
in comparison.

------
0x123456
I’m eagerly waiting to read their blog post: "The Configuration Mistake That
Almost Broke the Internet"

------
oliverrowlands
What will we do without our favourite "Enterprise MiTM" SaaS provider?

------
tcarn
Holy shit. This impacts nearly everything.

------
jafingi
Noticed that draw.io were down. 1.1.1.1 and all other Cloudflare backed sites
as well... I'm off for today.

~~~
auscompgeek
Cleartext DNS 1.1.1.1 is still up though, thankfully.

------
johnxie
Looks like everyone is impacted. We are seeing 502 Bad Gateway across multiple
domains and regions on [https://taskade.com](https://taskade.com)

~~~
johnxie
Back up and running!

------
nnx
Damn... getting a bunch of alerts and can't even open Pingdom either... also
running on CloudFlare
[https://my.pingdom.com/newchecks/checks#](https://my.pingdom.com/newchecks/checks#)

------
tnolet
npm and yarn are not working either...there goes Javascript land...

------
lol768
Yes, seems to have impacted all CF sites. (UK here)

Status was just updated here, but it was showing everything as operational for
a while:
[https://www.cloudflarestatus.com/](https://www.cloudflarestatus.com/)

------
apple4ever
Seems like it is clearing up. Our site is backup. Digital Ocean and Patreon
are up as well.

------
jshb
Tried to go to some usual places where people discuss outages and met with
outage. Ouch.

------
daxorid
A possible downstream effect of this: Pingdom appears to be alerting VERY
late, at least for us. I'm guessing with 8% of the web affected, their
alerting systems aren't prepared for this many simultaneous alerts.

------
quicksilver03
Several sites behind CloudFlare are returning 502 errors for me as well
(France).

------
eternalny1
They have mitigated it ...

[https://twitter.com/eastdakota/status/1146057946494713858](https://twitter.com/eastdakota/status/1146057946494713858)

------
obituary_latte
This is frustrating - can't access CF to turn off CF to make sites accessible.
There should be an emergency admin/dash access to turn off protection for
cases like this.

------
jf-
Also getting this in the UK. Not completely down, bits and pieces of medium
comes through but very slowly and incomplete. I’m also unable to access npm.

------
kortilla
Can’t wait to see Verizon’s blog post about this one :)

------
elamje
It’s better that this happens now than later. I am confident CF will put
protections in place to prevent this from happening again, but also put a
switch in place to provide an instant fix the next time something like this
happens.

It does suck to have a service down for a bit, but what CF offers, at the
price point is pretty incredible.

Good luck to CF, and I wish you the best with coming up with a robust future-
proof solution.

------
bytebuster
And I'm sure that everybody hitting F5 to see if it's back is causing no
problem at all, no no. Waiting anxiously for the writeup!

------
bensmrs
Also fun to notice that you have to agree to their privacy policy to receive
updates, which is hosted on their website, which is down

------
sb057
>We are working to mitigate impact to Internet users in this region.

>This incident affects: North America (Ashburn, VA, United States - (IAD),
Atlanta, GA, United States - (ATL), Boston, MA, United States - (BOS),
Buffalo, NY, United States - (BUF), Calgary, AB, Canada - (YYC), Charlotte,
NC, United States - (CLT), Chicago, IL, United States - (ORD), Columbus, OH,
United States - (CMH), Dallas, TX, United States - (DFW), Denver, CO, United
States - (DEN), Detroit, MI, United States - (DTW), Houston, TX, United States
- (IAH), Indianapolis, IN, United States - (IND), Jacksonville, FL, United
States - (JAX), Kansas City, MO, United States - (MCI), Las Vegas, NV, United
States - (LAS), Los Angeles, CA, United States - (LAX), McAllen, TX, United
States - (MFE), Memphis, TN, United States - (MEM), Miami, FL, United States -
(MIA), Minneapolis, MN, United States - (MSP), Montgomery, AL, United States -
(MGM), Montréal, QC, Canada - (YUL), Nashville, TN, United States - (BNA),
Newark, NJ, United States - (EWR), Norfolk, VA, United States - (ORF), Omaha,
NE, United States - (OMA), Phoenix, AZ, United States - (PHX), Pittsburgh, PA,
United States - (PIT), Portland, OR, United States - (PDX), Queretaro, MX,
Mexico - (QRO), Richmond, Virginia - (RIC), Sacramento, CA, United States -
(SMF), Salt Lake City, UT, United States - (SLC), San Diego, CA, United States
- (SAN), San Jose, CA, United States - (SJC), Saskatoon, SK, Canada - (YXE),
Seattle, WA, United States - (SEA), St. Louis, MO, United States - (STL),
Tampa, FL, United States - (TPA), Toronto, ON, Canada - (YYZ), Vancouver, BC,
Canada - (YVR), Tallahassee, FL, United States - (TLH), Winnipeg, MB, Canada -
(YWG)), Middle East (Amman, Jordan - (AMM), Baghdad, Iraq - (BGW), Baku,
Azerbaijan - (GYD), Beirut, Lebanon - (BEY), Doha, Qatar - (DOH), Dubai,
United Arab Emirates - (DXB), Kuwait City, Kuwait - (KWI), Manama, Bahrain -
(BAH), Muscat, Oman - (MCT), Ramallah - (ZDM), Riyadh, Saudi Arabia - (RUH),
Tel Aviv, Israel - (TLV)), Asia (Bangkok, Thailand - (BKK), Cebu, Philippines
- (CEB), Chengdu, China - (CTU), Chennai, India - (MAA), Colombo, Sri Lanka -
(CMB), Dongguan, China - (SZX), Foshan, China - (FUO), Fuzhou, China - (FOC),
Guangzhou, China - (CAN), Hangzhou, China - (HGH), Hanoi, Vietnam - (HAN),
Hengyang, China - (HNY), Ho Chi Minh City, Vietnam - (SGN), Hong Kong - (HKG),
Hyderabad, India - (HYD), Islamabad, Pakistan - (ISB), Jinan, China - (TNA),
Karachi, Pakistan - (KHI), Kathmandu, Nepal - (KTM), Kuala Lumpur, Malaysia -
(KUL), Lahore, Pakistan - (LHE), Langfang, China - (NAY), Luoyang, China -
(LYA), Macau - (MFM), Manila, Philippines - (MNL), Mumbai, India - (BOM),
Nanning, China - (NNG), New Delhi, India - (DEL), Osaka, Japan - (KIX), Phnom
Penh, Cambodia - (PNH), Qingdao, China - (TAO), Seoul, South Korea - (ICN),
Shanghai, China - (SHA), Shenyang, China - (SHE), Shijiazhuang, China - (SJW),
Singapore, Singapore - (SIN), Suzhou, China - (SZV), Taipei - (TPE), Tianjin,
China - (TSN), Tokyo, Japan - (NRT), Ulaanbaatar, Mongolia - (ULN), Wuhan,
China - (WUH), Wuxi, China - (WUX), Xi'an, China - (XIY), Yerevan, Armenia -
(EVN), Zhengzhou, China - (CGO), Zuzhou, China - (CSX)), Africa (Cairo, Egypt
- (CAI), Casablanca, Morocco - (CMN), Cape Town, South Africa - (CPT), Dar Es
Salaam, Tanzania - (DAR), Djibouti City, Djibouti - (JIB), Durban, South
Africa - (DUR), Johannesburg, South Africa - (JNB), Lagos, Nigeria - (LOS),
Luanda, Angola - (LAD), Maputo, MZ - (MPM), Mombasa, Kenya - (MBA), Port
Louis, Mauritius - (MRU), Réunion, France - (RUN), Kigali, Rwanda - (KGL)),
Latin America & the Caribbean (Asunción, Paraguay - (ASU), Bogotá, Colombia -
(BOG), Buenos Aires, Argentina - (EZE), Curitiba, Brazil - (CWB), Fortaleza,
Brazil - (FOR), Lima, Peru - (LIM), Medellín, Colombia - (MDE), Mexico City,
Mexico - (MEX), Panama City, Panama - (PTY), Porto Alegre, Brazil - (POA),
Quito, Ecuador - (UIO), Rio de Janeiro, Brazil - (GIG), São Paulo, Brazil -
(GRU), Santiago, Chile - (SCL), Willemstad, Curaçao - (CUR)), Oceania
(Auckland, New Zealand - (AKL), Brisbane, QLD, Australia - (BNE), Melbourne,
VIC, Australia - (MEL), Perth, WA, Australia - (PER), Sydney, NSW, Australia -
(SYD)), and Europe (Amsterdam, Netherlands - (AMS), Athens, Greece - (ATH),
Barcelona, Spain - (BCN), Belgrade, Serbia - (BEG), Berlin, Germany - (TXL),
Brussels, Belgium - (BRU), Bucharest, Romania - (OTP), Budapest, Hungary -
(BUD), Chișinău, Moldova - (KIV), Copenhagen, Denmark - (CPH), Dublin, Ireland
- (DUB), Düsseldorf, Germany - (DUS), Edinburgh, United Kingdom - (EDI),
Frankfurt, Germany - (FRA), Geneva, Switzerland - (GVA), Gothenburg, Sweden -
(GOT), Hamburg, Germany - (HAM), Helsinki, Finland - (HEL), Istanbul, Turkey -
(IST), Kyiv, Ukraine - (KBP), Lisbon, Portugal - (LIS), London, United Kingdom
- (LHR), Luxembourg City, Luxembourg - (LUX), Madrid, Spain - (MAD),
Manchester, United Kingdom - (MAN), Marseille, France - (MRS), Milan, Italy -
(MXP), Moscow, Russia - (DME), Munich, Germany - (MUC), Nicosia, Cyprus -
(LCA), Oslo, Norway - (OSL), Paris, France - (CDG), Prague, Czech Republic -
(PRG), Reykjavík, Iceland - (KEF), Riga, Latvia - (RIX), Rome, Italy - (FCO),
Saint Petersburg, Russia - (LED), Sofia, Bulgaria - (SOF), Stockholm, Sweden -
(ARN), Tallinn, Estonia - (TLL), Thessaloniki, Greece - (SKG), Vienna, Austria
- (VIE), Vilnius, Lithuania - (VNO), Warsaw, Poland - (WAW), Zagreb, Croatia -
(ZAG), Zürich, Switzerland - (ZRH)).

Quite a wide region, innit?

~~~
DuskStar
I went to post the list in Slack and got a 'too many characters' error...

------
sauldcosta
Some people on Twitter are reporting that it's due to a DDoS.
[http://www.digitalattackmap.com/#anim=1&color=0&country=ALL&...](http://www.digitalattackmap.com/#anim=1&color=0&country=ALL&list=1&time=17948&view=map)
seems to indicate Iran is the source?

~~~
po1nter
The date on your link is February 21st.

~~~
sauldcosta
Right you are, heh. Not sure why it jumped back to that date. False alarm.

------
AlphaWeaver
I'm beginning to see the issues disappearing for me on the East Coast of the
US, as of 10:14 EDT.

------
sebastianconcpt
Wonder how can we use CloudFlare and have a fallback plan in place for
situations like this. What would be a good architecture for this? So far I've
read that would be good to have the registrar out of CloudFlare and use them
as CDN only. What else?

------
elamje
Updated initial postmortem: [https://blog.cloudflare.com/cloudflare-
outage/amp/?__twitter...](https://blog.cloudflare.com/cloudflare-
outage/amp/?__twitter_impression=true)

------
ratsimihah
This article entitled Now Running on Cloudflare seems down

[https://seankilleen.com/2016/12/now-running-on-
cloudflare/](https://seankilleen.com/2016/12/now-running-on-cloudflare/)

------
Raphmedia
"Unfortunately, one of these rules contained a regular expression that caused
CPU to spike to 100% on our machines worldwide. This 100% CPU spike caused the
502 errors that our customers saw."

So, it was a dirty REGEX. I can't even be mad.

------
bodono
I'm getting '502 Bad Gateway' on all Cloudflare sites here in London too.

------
hokumguru
My shopify sites are down this morning, seems to be 502-ing because of this
issue.

------
jakemcgraw
was hoping John Graham-Cumming would be in here to give us the behind scenes
:(

------
_yesmeck
From [https://www.cloudflarestatus.com's](https://www.cloudflarestatus.com's)
response header, it seems it runs on fastly, so it's not down.

------
exdsq
Can someone from Cloudflare give us a hint at what's going on?

------
song
What's ironic is that my experience of Cloudflare Sales is that they take
advantage of any downtime from their competitor to try and get people to
migrate to their services...

Beyond scummy

------
40four
And so it begins......

------
johnstonnorth
Yes - seeing lots of our sites offline as of 14:45 GMT.

------
madethemcry
The internet is designed to withstand a nuclear attack. It's so sad to see
half of the internet going down because of a single system failure.

~~~
rc_kas
It's very convenient for me to be able to tell me clients their site is down
because half the internet is down, and there is nothing I can do about it.

A smaller DNS service would not make headlines.

------
chadmckenna
Looks like it, one of my sites is cached through Cloudflare, I'm getting a 502
from them. I'm seeing other random sites down as well.

------
s_dev
Even getting this error coming up in Android Studio trying to sync Gradle.

Anyone got a good Android NFC tutorial I can read up on with working code
examples?

------
deadalus
8chan, 4chan down as well. Damn, this is big.

[https://cloudflare.com](https://cloudflare.com) itself is gone.

~~~
chasd00
that's bad news, i like 4chan being up because it keeps them all in one place
and they don't start wandering around.

------
mkr-hn
Someone needs to turn Cloudflare off and on again.

------
Raphmedia
It is. [https://www.cloudflarestatus.com/](https://www.cloudflarestatus.com/)

------
psim1
Title should read, "Cloudflare Outage."

------
oerpli
Here in Liechtenstein things are working again.

------
isatty
Pagerduty’s main website seems to be affected.

------
nodesocket
July 2nd, 2019... The day the internet broke.

~~~
triod
Yup, we could have "Days without an accident" counter for the whole
internet...

------
oxfordmale
This is resulting in build failures in our AWS CodePipeline as we can't pull
some Docker images from Dockerhub

------
maple3142
The outrage this time seems to be more serious than last time. More websites
including Cloudflare itself are down.

~~~
sauldcosta
Without question. I've never seen an affected regions list that immense.

------
anoldgangstah
9gag is having trouble loading the memes.

------
nielsole
Good reminder for us of why you want to mirror all code dependencies required
during autoscaling :/

------
thequailman
This seems to be impacting NPM for me.

------
ComputerGuru
“Network performance issues” which is to say, completely unavailable for a
huge chunk of the world.

------
rwc
Seems to be back to normal now.

------
stedaniels
Cloudflare DNS 1.1.1.1 down too

~~~
anoldgangstah
Good thing I switched to OpenDNS literally two days ago.

edit. not that it matters, I'm just scrolling 9gag.

------
jarym
Looks like a flare up (sorry my attempt at humour always gets the better of
me...).

------
makkesk8
Holy hell did everything explode at work, glad they got it resolved pretty
quick.

------
cocoflunchy
Wow... our site went down so I went to cloudflare.com -> 502 bad gateway

------
teimer2
It's what we get for centralizing this here internet, folks.

------
BuddhaSource
So many crypto sites are down. Coinbase WazirX CoinmarketCap

~~~
cableshaft
Also WorldCoinIndex.

------
Vaskerville
As long as Reddit is not down everything is OK.

------
lukevp
Our website is also down, origin near Dallas.

------
Heliosmaster
It seems to be up for me now in AMS.

------
jimaek
jsDelivr was the only free CDN that stayed online during the outage thanks to
Multi-CDN.

------
yreg
The rumor is a DDoS attack

[http://www.digitalattackmap.com](http://www.digitalattackmap.com)

------
bytebuster
digitalocean.com is down as well (seems to affect cloudflare-gated websites)

------
anoldgangstah
Is 1.1.1.1 affected aswell?

------
minikites
Wasn't the internet supposed to be decentralized to avoid this very problem?

~~~
langitbiru
That is the mission of IPFS. However, there is a big gap between their mission
and the current capability of their technology.

Be the change you want to see in this world. I guess next time I build a
personal website, I have to host it in Raspberry Pi at my home. :)

------
maz1b
Getting 502s as well.

------
the-dude
And it is up again.

------
davexunit
Yay centralization!

------
khebbie
I see it as well

------
flixic
Back online.

------
clarkmoody
Back for me.

------
akriukas
Same here.

------
damniatx
oh my god,

------
chromaton
dammit

------
max0563
Cloudflare is too big.

------
perprit
omg I was not the only one.

------
weixiyen
It's back up for us now [https://sleeper.app](https://sleeper.app)

------
cltsang
This is just my personal guess, but it's likely China flexing again after the
protest escalation yesterday in Hong Kong. A few weeks ago Telegram was
attacked by China [0], and Hong Kong protesters used Telegram to communicate.

This time when CloudFlare was down, the most popular local forum among
protesters, lihkg.com, was brought down as well.

[0]: [https://techcrunch.com/2019/06/12/telegram-faces-ddos-
attack...](https://techcrunch.com/2019/06/12/telegram-faces-ddos-attack-in-
china-again/)

