
Cloudflare was down - dewey
https://www.cloudflarestatus.com/?hn=2020
======
rococode
Discord is entirely down right now, both the website and the app itself.
Amusingly, a lot of the sites that normally track outages are also down, which
made me think it was my internet at first. Downdetector, monitortheinternet,
etc.

Lots of other big sites that are down: Patreon, npmjs, DigitalOcean, Coinbase,
Zendesk, Medium, GitLab (502), Fiverr, Upwork, Udemy

Edit: 15 min later, looks like things are starting to come back up

~~~
maxk42
Discord attempted to route me to:
everydayconsumers.com/displaydirect2?tp1=b49ed5eb-
cc44-427d-8d30-b279c92b00bb&kw=attorney&tg1=12570&tg2=216899.marlborotech.com_47.36.66.228&tg3=fbK3As-
awso

(Visit at your own risk.)

Hack?

~~~
iamtheyammer
Sure you didn't misspell discord?

~~~
maxk42
I've never even heard of the site before. Nor have I searched for "attorney"
any time recently.

~~~
biermic
We operate that site and are using Cloudflare to prevent DDOS attacks.
Probably some sort of hash collision...

~~~
rocho
Crazy stuff!

------
ashleyn
It really defies the original vision of the internet to have so many services
depend on a single company. Almost every news site I was reading dropped off
at once. I thought for a second that I lost internet in my own house.

~~~
remmargorp64
I consider DNS and the way how top level domains are handled to be one of the
weakest parts of our current Internet design.

We REALLY need a truly decentralized, distributed DNS system that is not owned
by private entities.

~~~
hpfr
[https://handshake.org](https://handshake.org) is pretty interesting.

~~~
spenczar5
The "decentralized internet" folks always talk a lot about fighting corporate
control. I think they should spend more time talking about resiliency and
blast-radius reduction.

------
karlmcguire
"All systems operational"

What's the point of a status page if it doesn't reflect the real status...

It's either the status page goes down with everything else or the status page
is wrong. Great.

EDIT: Looks like it's accurate now, 20 minutes later.

~~~
jeremyjh
Let's start a betting pool. How many upvotes do you think OP will get before
the status page acknowledges a problem? I say its going to be 600.

~~~
jedberg
You lost. ;) 476 points, status page says it's down now.

~~~
acid__
As one would expect, it says "degraded performance" instead of "down" lol

~~~
gpm
Tested with tor and it's right. Some exit nodes aren't affected.

~~~
acid__
Hm, maybe it's just the SRE in me talking, but if major chunks of the internet
being entirely inaccessible doesn't count as an "outage", what does?

~~~
gpm
I mean, I guess your entitled to look at it that way, but I don't think it's
dishonest of them to distinguish between "nothing is working" and "some
regions aren't working".

------
geerlingguy
I don't think it's just Cloudflare; I just had a fun 10 minutes seeing servers
start flipping on my Server monitoring service[1]. This has only happened once
or twice per year, and is usually due to weird global DNS issues.

[1] [https://servercheck.in/](https://servercheck.in/)

(To give an update, I'm seeing from my monitoring systems (about 15 points
around the globe) sporadic outages for Microsoft, Apple, Reddit, Bing,
Node.js, Twitter, Yahoo, and YouTube. And my own servers (not behind CF at
all) are also flipping up and down. It started around 21:14 UTC.)

~~~
cm2187
a DNS issue wouldn't cripple all of the internet at once, with all the
caching.

~~~
RL_Quine
Most sites set the absolute minimum TTL for every record, for no reason.
There’s a lot less caching than you’re thinking.

~~~
qeternity
Eh, what? There are many good reasons to have low TTL DNS...this exact outage
being one of them. Update your records to go direct to your servers, and not
through Cloudflare and bam you’re back up. Doesn’t work if your TTL is 86400

~~~
unilynx
Doesn't help as cloudflare wants you to host their name servers with them, so
you can't flip any records if the DNS itself is in trouble, like it is now

And changing DNS servers often takes many hours (or days, if .net is involved
apparently)

------
jgrahamc
This was a problem with our backbone network; wasn't caused by an attack. The
effect was regional and not global. Naturally, we'll write it all up.

~~~
EE84M3i
Was it a problem with a provider you use?

~~~
jgrahamc
Looks like problem with one of our large routers in Atlanta.

~~~
nodesocket
I am in Nashville, and lost 1.1.1.1 and 1.0.0.1 completely. Also one of my
sites went down according to statuscake, but not all of them.

------
clairegraham
We were down (downforeveryoneorjustme.com) completely, but back up now (as of
a few minutes ago). Our domain wasn't even resolving; we use Cloudflare for
frontend and DNS.

We had a surge of people checking if Discord was down on our site, then I
noticed everything went down shortly after. Discord is still the top check
right now.

I can't ever remember hitting these kind of traffic numbers before.

~~~
wcchandler
I enjoy your service. Have you ever thought about expanding your offerings? I
would love to see a recreation of "Internet Pulse"

~~~
clairegraham
Thanks! Yep, we have a lot of things on the todo. We want to add more user-
focused / location-based outage information since our site is still too
reliant on simple HTTP checks to report downtime. This is especially a problem
with a Discord outage, for example, where the frontend website is not down,
but there might be problems with the API, apps, or other components.

And I'd like to be able to have our site communicate outages like this
Cloudflare one, where more than one site might be affected by a larger
provider. Automating that is difficult.

This is still a side project, though, so I mostly work on it when I get the
urge :)

~~~
wcchandler
That would be pretty interesting. Being able to drill down into individual
pieces of a stack would be very informative, especially from a parent source.
I bet a lot of services would even have that information readily available in
their documentation for APIs.

How comfortable are you with open source? If you were willing to release your
stack on Gitlab/Github, it might be worth your while.

------
jchw
Something’s wonky, because it’s not _just_ Cloudflare. One of my personal
sites is down that uses nothing but a VPS, and I noticed my Unifi AP
disconnect from its controller a little bit ago. Fiber cut? Routing issues?

~~~
parliament32
If that VPS is on DO they're down too cause of CF. Or if you set the resolver
on your VPS to 1.1.1.1 that's also down.

~~~
jchw
Why are digital ocean VPSes down due to a Cloudflare outage? Hoping for a
clarifying post mortem...

~~~
drchiu
My Digital Ocean load balancer went down. I think there's probably some
internal routing? Would be interested to understand more.

------
rozab
We can't keep going on like this. The vulnerability of centralised internet
infrastructure is a huge problem for everyone. Somebody, somewhere, really
ought to sort it all out

~~~
fivre
10-20 minute router misconfigurations and subsequent fixes are sometimes a
fact of life. big network infrastructure is complicated, and sometimes the
best laid route tables of mice and men do go abloop and die.

Outages happen no matter what the infrastructure is. There's no solution,
they're just something you need to recognize and handle, which Cloudflare
seemingly did relatively quickly here.

~~~
lima
Yes, but other providers are not a single point of failure for a significant
percentage of the internet.

Level 3 or Telia going offline is perfectly survivable for any customer who
has multiple upstreams.

~~~
Kalium
Remember the big Dyn outage? Or when AWS-US-East was severely disrupted by a
hurricane?

It may perhaps be an exaggeration to say that there are not other providers
that are similarly critical for a significant percentage of the internet.

------
alex_young
Yesterday I noticed most of their lava lamps are out (which generate random
bits). Perhaps these are a critical component.

[https://photos.app.goo.gl/g6eR8V2PSY3EVjCLA](https://photos.app.goo.gl/g6eR8V2PSY3EVjCLA)

~~~
maxk42
I'm sure you were joking but they actually are:
[https://blog.cloudflare.com/lavarand-in-production-the-
nitty...](https://blog.cloudflare.com/lavarand-in-production-the-nitty-gritty-
technical-details/)

~~~
ATsch
Despite them actually mixing it into their entropy pools, the lava lamps are
still entirely for show. The noise of the camera sensor itself is going to
contribute orders of magnitude more entropy than the slow movement of the
lamps. It's not completely a fake stunt, but it's certainly headline-
optimized.

------
pier25
A tweet from Cloudflare's CEO:

> _Appears that the router in Atlanta announced bad routes (effectively a
> route leak). Only impacted our backbone. Not all of our PoPs are connected
> to our backbone, so some would not have seen an issue. Appears to have
> impacted about 50% of our traffic for a bit over 20 min._

[https://twitter.com/eastdakota/status/1284259895475236865](https://twitter.com/eastdakota/status/1284259895475236865)

------
julien
I am probably the only one who cares but even
[https://downforeveryoneorjustme.com/](https://downforeveryoneorjustme.com/)
is down

~~~
ATsch
I personally find sites like [https://outage.report](https://outage.report) or
[https://downdetector.com](https://downdetector.com) which tally up the
number, regions and history of people saying it isn't working for them more
conclusive.

------
1f60c
That page still shows "Cloudflare System Status: All Systems Operational" for
me, but it's _definitely_ down for me. Along with 1.1.1.1, which is... _bad_.

~~~
cryptoz
For me it says "Minor System Outage" for about 0.1s and then shows "All
Systems Operational".

~~~
ehsankia
Everything works fine for me (Canada), am I missing something or is it over
already?

~~~
cartoonfoxes
Also in Canada. Shit's fucked, yo.

------
JoshGlazebrook
Having DNS issues. Had to switch to Google's DNS (8.8.8.8/8.8.4.4) as
1.1.1.1/etc were not resolving anything.

~~~
fpgaminer
Same. Even then, a bunch of sites are down. Maybe only ones behind Cloudflare?
So far I've been trying to hit the various down detector sites and none of
them will load. Google, Reddit, Hackernews are all fine.

~~~
drrotmos
From what little I've been able to gather, anything using Cloudflare's DNSes
are down.

------
jlmorton
Monday Morning RCA: "We pushed out some routine code updates, but this really
weird thing happened causing a resource utilization spike on our DNS systems.
Because of this other really weird thing, this affected all of our global
infrastructure simultaneously. Here's a deep engineering dive into this one
weird thing that brought everything down."

------
caudamus
Cloudflare's DNS (1.1.1.1) is failing to respond to most/all queries, which
I'm observing as the root cause of a bunch of connection issues (name lookup
failure).

Interestingly the same domains don't show up on google's (8.8.8.8) DNS at all.

~~~
parliament32
8.8.8.8 is a caching resolver, it still needs to talk to CF's nameservers for
authoritative records.

------
minxomat
Lol, talk about timing. I'm currently working on a TLS library and was pulling
my hair out trying to figure out why tests against CF sites suddenly failed.
Can't even ask my cohorts on Discord because they are behind CF, too!

~~~
cknoxrun
No kidding! We had literally deployed a major page redesign and started
watching our analytics drop off on it's way to zero. My heart is racing still.
I wouldn't normally be happy for a cloudflare outage but in this case it's
better than Google deciding to remove us from their index.

~~~
akuji1993
That's unfortunate timing dude. Good for you, it's probably not your mistake
:D

------
synack
I can't change my NS records to point to a different DNS provider because my
registrar, Namecheap, also uses Cloudflare. Didn't expect that.

------
alfg
Yep, Cloudflare, DigitalOcean and 1.1.1.1 down for me. I thought it was my
internet and was so confused for a bit there.

------
wenbin
Friendly reminder (and notes to myself):

Don't use Namecheap and Cloudflare at the same time.

Namecheap is using cloudflare. So if cloudflare is down, you can't change DNS
settings on Namecheap as well!

------
rgbrenner
Reminder for firefox users: Firefox uses DNS over HTTPS and the default is
cloudflare. If you're having DNS issues, you need to disable it until
cloudflare is back up.

------
aaomidi
Oopsie Daisy half the internet goes down

~~~
parliament32
Eggs and baskets etc etc

~~~
nobleach
Good DNS practice (at least when I did system admin 10 years ago) was ALWAYS
having a secondary at some other location/network. Why do we just put some
info in Cloudflare and call it good these days?

~~~
toast0
It's hard to use Cloudflare as a reverse proxy without using them as your
delegated name servers (maybe you can use CNAMEs on paid plans?), and fancy
dynamic nameservers make it hard to run secondary servers with zone transfers.

~~~
nobleach
This is definitely the answer I needed.

------
randomstring
What luck, I chose today to install a new piece of network gear. I thought I
had managed to totally FUBAR my network. DNS was failing, "ping 1.1" (my
current goto test "Am I connected to the internet?" as it requires the fewest
keystrokes and hits the Cloudflare DNS 1.0.0.1) failed and I just assumed it
was _my_ fault. Backed out my changes, and discover in fact, the internet was
down.

~~~
BenjiWiebe
Ping 1.1... thanks for that!

------
typingmonkey
I was trying to fix my router the last 15 minutes :)

~~~
arkitaip
Same here. Only figured it out because just one of the computers uses
Cloudflare dns and the others were fine...

~~~
ghastmaster
Ditto except visa versa. My machine is set to the router which uses
cloudflare. Other machines use whatever is default for mac(I try not to touch
those). Once I realized they were working and I could access internal network
from outside, I started diagnosing DNS. Came here via 8.8.8.8.

------
sillysaurusx
Our TPU management page is also down:
[https://www.tensorfork.com/tpus](https://www.tensorfork.com/tpus)

Seems cloudflare took out a good chunk of the internet temporarily.

Doesn’t HN use cloudflare? Why did it survive? (I haven’t looked for about a
year, but I seem to remember HN being proxied behind CF at one point.)

~~~
searchableguy
Is there a status page for HN?

~~~
sillysaurusx
EDIT: Yes: [https://twitter.com/HNStatus](https://twitter.com/HNStatus)

HN is so reliable that’s it’s almost never needed one. I’m extremely curious
how HN survived this; almost positive they used cloudflare at one point.

I think the official status page is @hnstatus on Twitter, or something like
that.

~~~
toomuchtodo
They did use Cloudflare, but also haven’t for some time.

------
heliodor
Is it me, or has this been happening way too frequently for them lately?

~~~
jeremyjh
Honestly at their scale once a decade would be too frequent. Too many eggs in
this particular basket.

~~~
bithaze
Once a decade doesn't seem realistic. At some point you get diminishing
returns chasing as many mines as possible.

------
iamtheyammer
Appears to be working for me now

; <<>> DiG 9.10.6 <<>> discordapp.com ;; global options: +cmd ;; Got answer:
;; ->>HEADER<<\- opcode: QUERY, status: NOERROR, id: 8092 ;; flags: qr rd ra;
QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION
SECTION: ;discordapp.com. IN A

;; ANSWER SECTION: discordapp.com. 140 IN A 162.159.135.233 discordapp.com.
140 IN A 162.159.129.233 discordapp.com. 140 IN A 162.159.130.233
discordapp.com. 140 IN A 162.159.134.233 discordapp.com. 140 IN A
162.159.133.233

;; Query time: 69 msec ;; SERVER: 1.1.1.1#53(1.1.1.1) ;; WHEN: Fri Jul 17
14:37:40 PDT 2020 ;; MSG SIZE rcvd: 137

------
floren
1.1.1.1 is not resolving anything for me at this time.

------
Rockjodd
For people who reports this is down, which country are you in? Because all the
reported sites works flawless from Norway (Europe) :-)

~~~
Rockjodd
I put my bet on some peering fuckups, causing outrages since people are having
packet loss etc.

~~~
Rockjodd
"Update - This afternoon we saw an outage across some parts of our network. It
was not as a result of an attack. It appears a router on our global backbone
announced bad routes and caused some portions of the network to not be
available. We believe we have addressed the root cause and are monitoring
systems for stability now. Jul 17, 22:09 UTC" \-
[https://www.cloudflarestatus.com/](https://www.cloudflarestatus.com/)

------
floatingatoll
> * This afternoon we saw an outage across some parts of our network. It was
> not as a result of an attack. It appears a router on our global backbone
> announced bad routes and caused some portions of the network to not be
> available. We believe we have addressed the root cause and are monitoring
> systems for stability now.* Jul 17, 22:09 UTC

------
blisseyGo
A lot of unusual internal traffic seems to be around Thailand (you might have
to select "UNUSUAL"):

[https://www.digitalattackmap.com/#anim=1&color=0&country=ALL...](https://www.digitalattackmap.com/#anim=1&color=0&country=ALL&list=1&time=18458&view=map)

~~~
lgats
been pretty large numbers from thailand this month according to that tool

------
Exuma
Update - This afternoon we saw an outage across some parts of our network. It
was not as a result of an attack. It appears a router on our global backbone
announced bad routes and caused some potions of the network to not be
available. We believe we have addressed the root cause and monitoring systems
for stability now.

------
whoisjuan
Their status page says that everything is operational. So much for a status
page when half of the internet breaks down.

------
rv-de
What I found particularly interesting is that my MacBook Pro (work laptop)
didn't start up anymore properly and I wasn't able to start applications ...
sorry, but wtf. Now I hate Apple and their shitty, overpriced products even
more.

------
adrr
What was interesting and scary is that our monitoring system didn’t notify us.
Our email was down because we use cloudflare for DNS and our monitoring
provider’s SMS gateway was down. So we didn’t get sms messages.

------
FireBeyond
Another useless status site.

DNS is completely broken.

"All systems operational" in nice soothing green.

No, not so much.

------
heliodor
FYI, PagerDuty is not loading!

Time to go back to the drawing board, for a lot of us, to re-assess points of
failure.

Edit: many websites are failing to DNS resolve but the services they provide
continue to function fine behind the curtain.

~~~
twunde
This is likely your computer's DNS resolver (if you're using 1.1.1.1 you're
down. I'd switch to 8.8.8.8 temporarily. We've had pagerduty alerts coming in
since the start (a whole bunch of DNS errors from pingdom) and when I click on
the slack link, pagerduty works for me

------
r1ch
Interestingly this seemed to only affect resolver service. I use Cloudflare
pretty extensively on all my sites, but only in DNS mode (no CDN / proxy). The
hosts continued to resolve fine during the outage (following root DNS
resolution chain, no recursive resolver involved). I imagine their CDN
internally uses their resolver service which explains the outages, and some
unrelated 3rd parties who don't use CF on their domain at all still created a
hard dependency on CF by using their recursive DNS server.

------
bryan_w
Those poor Firefox users who enabled DoH

------
davexunit
Happy Friday, everyone!

------
athesyn
Ironically
[https://downforeveryoneorjustme.com/](https://downforeveryoneorjustme.com/)
is also down. HA

------
LeoPanthera
I'm having real problems with DNS, is this Cloudflare too? They say "All
Systems Operational", so maybe not?

Half the damn internet is not currently resolving.

~~~
zubiaur
Yes, same here. Changed DNSs to level 3, all better now.

------
beatrobot
More like the Internet is down.

~~~
saagarjha
When you depend on a single company for much of the internet, such things
happen :(

------
mxschmitt
Site that use Discord, Linode, Patreon, npmjs, DigitalOcean, Coinbase,
Zendesk, Medium, Gitlab (502), Fiverr, Upwork, Udemy and many more including
1.1.1.1 dns down. Ref:
[https://twitter.com/nixcraft/status/1284239374809395200?s=19](https://twitter.com/nixcraft/status/1284239374809395200?s=19)

~~~
icey
Seeing a lot of people mentioning DO, but it has been up for me without any
issues (small VPS in SF-2)

------
exochrono
My Pagerduty's been blowing up so I tried to go to their dashboard to pause
the notifications for now and pagerduty.com is down XD

------
Jonnax
340ms average latency to 1.1.1.1 and 47% packet loss. Many sites are down. But
I guess that's the problem with CDNs.

------
ed25519FUUU
It's unfortunate that both the primary and secondary cloudflare DNS is down. I
just switched my secondary to google.

This allows my internet to "work" during this time, but adds about 1s latency
to resolutions. Presumably that's the time it takes my internal DNS resolver
to try the secondary.

~~~
ingenium
Considering running your own full resolver like unbound. Then you don't have
to rely on a DNS provider like Google or Cloudflare. It's really nice not
having the whole internet go down when Google or Cloudflare DNS is down.

------
sascha_sl
DNS seems to be dead, if you have stuff in your cache and the site isn't low-
TTL things still kinda work

~~~
icodestuff
Yeah this is absolutely killing us right now.

------
emeraldd
Looks like Digital Ocean is reporting an issue with their upstream provider:

[https://status.digitalocean.com/incidents/6wtmldty17g1](https://status.digitalocean.com/incidents/6wtmldty17g1)

As big as this is, any chance a major hub/backbone went down?

------
jrockway
I don't use Cloudflare, but I do notice Cloudflare services being down.

Right now, I can't get to my own website (hosted on DigitalOcean, not through
Cloudflare), but Oh Dear claims it's up. So I suspect that the problem is
closer to me than it is to DigitalOcean (or Cloudflare).

~~~
rgbrenner
DO uses cloudflare for their DNS... both for digitalocean.com and for their
DNS service.

~~~
jrockway
Good to know! That makes perfect sense based on what I saw during the outage.
I had no idea.

------
Exuma
Aside from this one issue, is switching to 1.1.1.1 a good idea in your guys
experience? Right now I just realized I hvae the DNS for my ISP which is
probably how they inject bullshit 404 pages full of ads. What is the
fastest/best public DNS in your guys experience?

~~~
xur17
I've been pretty happy with 1.1.1.1 (before now). Might be worth using
something like 8.8.8.8 as a backup (Google).

~~~
vulcan01
You could also use 9.9.9.9 as your backup, if you're avoiding Google
([https://www.quad9.net](https://www.quad9.net))

------
britmob
Looks like it's back. No longer getting issues with 1.1.1.1 and domains are
being resolved!

~~~
ninkendo
Not for me, `dig @1.1.1.1 google.com` is returning SERVFAIL still. Their
anycast config may be broken in some way (ie. the backends for some regions
are down, but still advertising routes)

~~~
basch
Resolved as of a minute ago, still having an issue now?

------
ryanmccullagh
I thought my issue was with Comcast, then I realized I'm using CF's DNS
entries for my home network. I removed those 1.1.1.1 entries and some sites
are working.

Dang, I'm pretty disappointed in CF. I've never experienced this much an
effectful DNS outage.

------
buro9
From what I can see externally this looks like DNS.

I wonder if that includes the roots that Cloudflare operate.

------
byteofbits
It’s worth mentioning here that 1.1.1.1 is also affected by this outage which
initially made me think my internet was gone completely.

Changing back to an alternative (such as 8.8.8.8 from google) restored my
access to the areas of the internet not using Cloudflare.

------
nodesocket
This was especially bad because I use Cloudflare public DNS exclusively at my
house and it went down as well. I didn’t even think to check DNS, I just
assumed it was AT&T being shiest AT&T.

I should probably run a blend of 1.1.1.1 with 8.8.8.8 instead.

------
ricardo81
Last I read about 7 million hosts are behind Cloudflare. Maybe around 3% of
the web, but who knows if that counts for critical assets etc rather than
pages served.

Shameful that so much of our decentralised web is so centralised and breakable
in one place.

------
drchiu
Just got a whole bunch of alerts that my services are down. Tried logging into
Digital Ocean (who it seems uses Cloudflare) to get it fixed. Could not access
their dashboard to reroute things.

------
maxioatic
RIP someone's weekend

------
satysin
Guess that explains Discord vanishing from the net a few minutes ago.

------
tikiman
I'm surprised so many people still use them. They took my business down (along
with half the internet) a few years ago and I learned that they were to large
of a point of failure.

------
r0xsh
Ah Shit, Here We Go Again

------
brycewray
Vercel also appears to be dropping out and coming back in intermittently over
the last 30 minutes or so. Not aware they're using Cloudflare, although they
do mention using AWS.

------
awinder
NextDNS got taken out by this, id been really happy with it up until now. And
unfortunately “dns service went down” has a wide enough blast radius at home
now that it’s a real pain.

~~~
ricopags
How did you verify that? I determined the issue was with Cloudflare's DNS by
toggling on NextDNS, which worked and continues to.

~~~
awinder
Most of my devices went belly up and was trying to figure out what it might be
(I run NextDNS on my router), switched off to cell and noticed discord was
down too so started thinking about NextDNS. I toggled dns to google and
noticed it immediately work.

------
Aldqueath
this is great, i already have bad enough internet (rural area with 3 to 6
digits latency and average 4 digits, barely a few kilobytes of speed) and
having both google smearing everywhere their recaptchas that are not really
friendly toward low speed internet / non chrome users and cloudflare proxying
half the internet but lately not really doing a great job at keeping a
consistent uptime does not help much

at least i am glad hn exists, it is the only thing that loads everywhere

------
gautamcgoel
I was trying to play video games but couldn't connect. Amazing how connected
the web is now - one big hub goes down and brings the whole house of cards
down with it.

------
pgrote
Did anyone else see their ATT internet go down? The DNS issues started and
then the Pace 5268AC rebooted. I don't use cloudflare for dns. Does ATT's
backend?

~~~
MertsA
On the contrary, ATT actually squats on the CloudFlare DNS IP address. IIRC
that modem is one of the affected ones where it uses 1.1.1.0/24 internally.
You shouldn't even be able to use CloudFlare DNS normally.

~~~
pgrote
Thank you for the response and information. Unhappy coincidence.

------
usr1106
It shows "Minor system outage" when I load the page, but it switches to "All
systems operational" immediately. Same behaviour on several attempts.

------
parliament32
The status page linked shows "All Systems Operational" for me. Tested in
private browsing and on my mobile.

Looks like DNS issues, their nameservers aren't reachable.

------
VectorLock
A lot of people are saying AWS. I'm having intermittent network connectivity
issues intra-AZ, so perhaps they lost a data center or route flapped one.

------
microcolonel
What's all this about ”building a better internet”? Wide-reaching general
service outages that are invisible to your status page are really not great.

~~~
erichocean
I wonder if they'll cover why their status page is a steaming pile of garbage
in the post-mortem?

------
peterwwillis
Yesterday CloudFlare took down some of our products because they (not us)
misconfigured some DNS thing. Kind of funny to see it happen again a day
later.

------
xtracto
This is hitting my production environments as well :-(

------
BookmarkSaver
Twitch.tv channels are like 50/50 right now. Some are ok, some aren't.

Basically all Riot Games (League, Valorant, TFT) are down, dunno about LoR.

------
Kye
That would explain why Patreon is down. I was going to post a little frog I
took a picture of on Lens. Went down just as I opened the app.

------
rob-olmos
Hopefully with this outage Cloudflare will finally provide non-Enterprise
plans a CNAME record, allowing us to quickly bypass Cloudflare.

------
tomxor
Thought I was going crazy for a second.

This affects so many things it's scary, and Cloudflare status page has still
not updated. HN got there first.

------
mathattack
Interesting. They had an outage in the midst of a negotiation I was a part of.
Are they less stable than Akamai and the others?

------
fluxsauce
My modem also disconnected with signal problems, which was interesting. I'm
not sure Cloudflare could have caused that?

~~~
lgats
maybe your modem uses 1.1.1.1 dns?

------
tompccs
I was paying a bill in a bar in London about an hour ago and they couldn't
process any payments. Seems likely related.

------
semicolon_storm
Must be regional or some other factor involved. Various sites others are
reporting as offline load for me as does 1.1.1.1.

------
kebman
Oh, did hacker season just start? * Grabbing popcorn! * So this was an
accident, or is it connected to the Twitter hack?

------
ddevault
Will we take this as a much needed lesson about putting all of the internet's
eggs into one basket? Probably not.

------
heliodor
WebGazer.io managed to shoot me an email about my site being down. This in
spite of their site being down too.

~~~
th0th
Hi @heliodor, WebGazer founder Gokhan here :)

Actually the site is running but not accessible due to the issue. Glad you got
the heads up, after a while I had to pause monitoring to prevent side effects.

------
xenospn
Yup. 1.1.1.1 stopped responding as well.

------
greggyb
DNS resolution at 1.1.1.1 seems to have gone down and come back up for me in
the course of 10-15 minutes.

------
dpcx
Also breaking a bunch of deploys because npm and yarn are heavily dependent on
Cloudflare, it would seem.

------
xen2xen1
Remind me to check and see that I have 8.8.8.8 and 1.1.1.1 on my networks, not
just one or the other..

------
kords
It's back now (at least for me).

------
jorgenphi
The uptime tool I use (StatusCake) is itself down... Was wondering why I
didn't get an alert.

------
neurostimulant
Seem to be localized issue. Cloudflare is up here in my country, but down for
many people in US.

------
noobermin
People talk about a single point of failure then go on to depend on a single
point of failure.

------
elviswolcott
The status page is now showing degraded performance for the Cloudflare API and
Recursive DNS.

------
html5web
Cloudflare status site is also partially down. Some resources are not loading
properly.

------
kube-system
Of course, I read this _after_ I spend an hour debugging some strange DNS
issues.

------
mjayhn
Thank you for the early Friday.

------
zadkey
I noticed Udemy was down when I wanted to go to the next video I was watching.

~~~
zadkey
And now it's back up.

------
hugoromano
I can only see the dashboard down, all my sites with Cloudflare are up.

------
amasad
Looks like it's resolved -- we're coming back up at Repl.it.

------
gregory90
Cloudflare DNS is down too

------
DangerousPie
Is this US specific? Everything seems to work fine here in Europe.

------
formerly_proven
Centralising on a single host suddenly not a good idea any more?

------
Majora320
Seems like they managed to break half the internet for everyone.

------
mauriciogior
how can I have cloudflare plus something else as a DNS failover? We are afraid
to set a long TTL and have our IP changed for some reason. What do you guys
recommend?

------
bartwe
DNS is also (partially) down with my ISP (xs4all.nl) it seems

------
stri8ed
Ironic, isitdownrightnow.com is down.

All my DigitalOcean instances are down.

------
sbr464
Yep, the DoorDash app is affected currently, my burritos!

------
solarkraft
Finally we see how much we depend on this single company.

------
haloblue
Looks like it's starting to come back in the SE US.

------
easytiger
Boats. People. Stop putting them in one single boat.

------
tomklein
Back online for me.

------
ryanmccullagh
Dns issues for sure

------
Hanabishi
Well, sheit. This is all around the world. Press F.

------
chris_engel
Seems to work again for me in germany (Frankfurt)

------
sdenott
League of Legends down too, not sure if related.

------
solarkraft
Most pages mentioned here seem functional again.

------
nomdep
So THAT is why the Internet was acting weirdly!

------
decad
DNS seems to be resolving for me in the UK now

------
MH15
Back online just now for me in Midwestern US.

------
interator7
2:36 PM PST - status.discord.com is back up.

------
tomklein
NPM is down too.

------
bitclaw
Seems like it's starting to come back.

------
andrewnicolalde
Was wondering why my DNS wasn’t working...

------
unilynx
digitalocean.com DNS (on cloudflare) is now resolving again. looks like
several things are coming back now.

------
mindfreeze
I was having troubles with overleaf.com

------
mongol
How does Cloudflare compare to Akamai?

------
iJohnDoe
Ironically StatusCake is down as well.

------
salmaanp
who deployed on a friday afternoon?

------
ranrub
Cedexis gets another lease on life

------
ransom1538
Jesus. Does anyone know anything?

------
michael_j_ward
Having issues with gitlab myself

------
jonplackett
Seems like it’s back right?

------
johnxie
Can confirm for taskade.com

------
jpomykala
Friday

------
andrewnicolalde
1.1.1.1 is back for me now

------
dangwu
League of Legends, Valorant and Discord both down. I took today off to play
games...

------
askbill
Felt like a BGP issue.

------
devy
Looks like CF is up!

------
RL_Quine
Some POPs are fine.

------
sys_64738
Five nines uptime?

------
cartoonfoxes
Back online here.

------
bhaak
the internet was built

to withstand a nuclear war

brought down by cloud flares

------
chuckdries
lmao it even took down my local stack

------
megadethz
hn algolia search broken

~~~
redox_
yes hn.algolia.com is powered by Cloudflare /o\

------
arjun27
more like Cloudflared

------
techlaw
itch.io down

isitdownrightnow.com down

------
navinag
yes

------
logicalmonster
Given that the US is basically in a non-shooting war with China, I wonder if
this is something technical or part of some kind of attack. Something that I’d
keep in mind.

~~~
wolfgang42
There are enough ways for bits of the Internet to go kablooey on their own
that “it’s an attack!” is a pretty big jump to a conclusion. If this turns out
to be something other than Cloudflare tripping over a weird bug, my first
guess would be that someone fat-fingered a BGP table yet again.

~~~
wolfgang42
Update: Yep, BGP issue, though I was thinking it would be something on the
public Internet rather than CF’s backbone.

