
Turns out half the internet has a single-point-of-failure called “Cloudflare” - StuntPope
https://easydns.com/blog/2020/07/20/turns-out-half-the-internet-has-a-single-point-of-failure-called-cloudflare/
======
miki123211
Cloudflare is horrible for blind people.

Screen readers, the programs that use synthesized speech to tell us what's on
the screen, cannot read images. Good captchas usually have audio equivalents
(which come with their own set of problems), but this one doesn't. If you're
blind and flagged by Cloudflare for some reason, you're cut off from accessing
half the internet, potentially critical
banking/governmental/medical/communications/educational services. We rely on
the internet way more than our sighted peers, so this is very important. This
has recently happened to me on a few sites, fortunately not critical ones, but
it was not a pleasant experience nonetheless. CF engineers, please fix this
ASAP. I'm surprised there still isn't a huge lawsuit over this, as this is
clearly violating all sorts of laws.

~~~
1f60c
Thank you for sharing your experience.

Cloudflare switched to using hCaptcha a couple months ago, I think and I only
just noticed that they do not offer an audio-based captcha.

However, hCaptcha integrates with Privacy Pass[0], and you can top up your
tokens by solving a captcha at [1] and [2]. What you could do (and I realize
this is far from ideal), is getting a sighted friend to solve a couple of
captchas so you have enough tokens to last you a couple months.

[0]: [https://privacypass.github.io/](https://privacypass.github.io/)

[1]: [https://captcha.website/](https://captcha.website/)

[2]: [https://www.hcaptcha.com/privacy-pass](https://www.hcaptcha.com/privacy-
pass)

~~~
waheoo
This is such an out of touch dev response.

The problem is hcaptcha, the solution is stop using hcaptcha. Not placate and
evade with work arounds you wouldn't even suggest to non blind people.

Also this ask a friend solution is like telling someone in a wheelchair to ask
a friend to help them up those steps instead of just installing a ramp.

You don't get to take over half the internet and just ignore social
responsibility.

~~~
BTinfinity
Solutions can’t be implemented instantly; a work around is needed until that
ramp is installed. As much as the lack of audio solution sucks, you can’t
expect half the internet to just give up on a piece of technology overnight

~~~
zorpner
No. The ramp needs to be built when the thing is built. Accessibility is not
an afterthought.

~~~
mynameisvlad
I mean, yes, in an ideal world that's true. Yet, here we are.

It's not possible for a dev to go back in time to before hcaptcha was created
or when CF decided to switch to them to create said ramp, so until it's built,
workarounds are the only real thing a community member can offer someone in
the short term.

~~~
binsh
It's not like trying to find the Fountain of Youth or something. It's a
company with billions of dollars making it impossible for certain people with
disabilities to access the Internet. Let's not pretend that it just has to be
this way. A world where half the Internet can't be broken by one company
should be the baseline.

~~~
mynameisvlad
Nobody is saying it _has_ to be this way. As I said, in an ideal world,
accessibility happens when development happens and is not an afterthought. But
we don't live in an ideal world, and it's clear here that it wasn't thought
out when they did the switch, or other forces caused the switch to happen
without this piece in place.

So, with that in mind, knowing where we _currently are_ , not where we'd like
to be in an ideal world, what, _exactly_ , do you want to be done right now by
members of the community?

It's not like we can all go in and change the Cloudflare code to stop using
hCaptcha. The most we can all do is give alternatives and workarounds while
pushing on Cloudflare and hCaptcha to support this scenario. Which is all
being done in this thread already.

------
michaelbuckbee
Honest question: I've never really understood the back of the napkin math of
how Cloudflare functions economically, which I feel would go a long way
towards my understanding of why/how they were able to become such an integral
and generally positive part of the Internet.

Did they have some crazy in to get cheap bandwidth? Did they bet big on
bandwidth prices falling? Did they figure something else out that nobody saw?
Do they just to a tremendous job of migrating sites from free to paid plans?

~~~
jgrahamc
Read our S-1, it's all in there.

[https://www.sec.gov/Archives/edgar/data/1477333/000119312519...](https://www.sec.gov/Archives/edgar/data/1477333/000119312519222176/d735023ds1.htm)

~~~
morrbo
II love cloudflare. It has really helped out with several sites/projects that
I have worked on and the service is top notch. I am also an investor. I tend
to invest in stuff which I use a lot or trust/respect the employees. Weirdly
what made me really invest is the level of geekiness on the company. I
remember seeing you guys using a lava lamp wall to generate entropy and just
thought "that's awesome". I just wanted to say don't change. Your lz4
implementation, aes gcm golang optimizations have directly benefitted me.
Coupled with really high quality post mortem articles, articles on interesting
things like compression, encryption, networking, a few random articles (the
privacy focussed file system recently comes to mind, written at the same time
I was making my own distributed fs) just leave me with a lot of respect for
the culture there.

I was worried a bit when I saw the initial IPO that the free tier would leave
(despite promises it wouldn't) but that doesn't seem to be the case. Literally
the only bad thing I've seen on the site is recently you switched from
recaptcha to a new one that I had a real tough time with logging in today - it
was a bit glitchy on my pc. The only suggestion I thought of as well would be
a simple "maintenance mode" similar to the "I'm under attack mode" which would
allow those of us without super-ha to quickly toggle on something to pop up a
"sorry server"/site is down form maintenance page without having to mess with
our proxies/web servers.

Anyway I know this comes across as totally kissing-ass but I just wanted to
say thanks to someone who actually works there. Everyone fat fingers stuff
every now and again,don't sweat it.

~~~
renewiltord
It's a bit of a sad story, but maybe you'll like this read about one of the
guys who laid the foundation for their tech and his sad decline:
[https://www.wired.com/story/lee-holloway-devastating-
decline...](https://www.wired.com/story/lee-holloway-devastating-decline-
brilliant-young-coder/)

I really liked the stories of his skill when he was in his prime. Very
inspiring.

~~~
ksec
I dont know how I missed the Cloudflare IPO and this Story. So thanks for
posting it. Extremely good writing, though the story was very sad.

I wonder if there are any relationship between FTD and how clever a person or
how much brain a person uses.

~~~
thereticent
Nothing beyond the normal "cognitive reserve" findings that high education and
broad cognitive engagement makes the brain pathology take longer to reduce
your functioning to a dementia state.

------
badRNG
Interesting perspective, but it seems like this is just an ad for easyDNS and
their "Proactive Nameservers," though I couldn't imagine a better time than
the misstep of a behemoth of a competitor in this space. Not to detract from
the more important discussion about the internet's dependence on Cloudflare
overall.

~~~
sradman
It may be just an ad for easyDNS' _Proactive Nameservers_ [1] product but it
provides a roadmap for one possible solution to this type of problem. From a
quick reading of the marketing info, the solution can be summarized as
"Provision, Monitor, and Fail-Over DNS Name Servers across multiple DNS-as-a-
Service providers". The question I have is whether the following constraint is
artificially introduced or not:

> We must be your domain registrar for this to work...

IIRC, Netflix OSS published some tools quite some time ago to support multiple
DNS providers but I don't know/remember if they tackled the availability
problem. The question comes down to build vs buy and whether the solution is
general enough to warrant an Open Source Software solution.

[1] [https://easydns.com/dns/proactive-
nameservers/](https://easydns.com/dns/proactive-nameservers/)

~~~
pas
They want to be the registrar to be able to update your NS records. But ...
that's not really important nor needed (So the answer to your question yes,
it's likely artificial). Just use two anycast-ed IPs/domains. (Like
Cloudflare.)

The magic happens at BGP level.

I considered CF as a domain registrar, but they don't allow setting the NS
records. So you must use them. (They basically use sane no-nonsense domain
registration as a way to gain leads for their main product. Pretty smart
actually, because it's a great high-level add-on for their main product, but
they just went ahead and made that the bait for everyone.)

Anyway, ideally, if you add 2 separate sets of NS servers to your NS records
then you eliminated this SPoF, great. Sure, it's your job to keep them
updated, and in sync (preferably, to avoid problems like half of your users
landing on a different CNAME/IP/etc).

And recursive nameservers will handle the failover.

~~~
StuntPope
easyDNS has to be the registrar because only your registrar can change your
nameserver delegation with the registry. This is, in essence, the registrar's
job. To maintain your domain record and info, including nameserver delegation,
with the registry.

You could do it with BGP, but it is non-trivial and you need your own ASN to
do that.

~~~
pas
But you can just add multiple DNS providers yourself. I mean you can add the
namservers of both easyDNS and cloudflare. EasyDNS just automates this.

In theory they could simply create a few subsidiaries, let's call them
saferDNS1,2,3 and have them build completely different redundant DNS
architectures, and add then add the resulting nameservers.

That said, it'd be good to see an actual domain that uses this "proactive"
feature to see what easyDNS is doing.

~~~
donmcronald
I'm not a DNS expert, so... It's not really that simple is it? If you have
multiple nameservers I thought they get equal weight, don't they?

So if you have Cloudflare + (ex:) NS1, and you're using Cloudflare for
caching, you need your NS1 records to return Cloudflare proxied IPs normally,
but origin IPs under failure conditions. That's a lot of infrastructure.

It also fails completely if you're relying on Cloudflare for DDoS protection
and IP obfuscation because a failure means your origin IPs get exposed. That's
assuming Cloudflare DNS being down means Cloudflare proxying is down too. It
might not be the case, but I think you'd have to plan for it.

Then there's also Cloudflare's detection of nameservers. I haven't tried it
with more than Cloudflare's nameservers set for a domain, but if your domain
doesn't actively use their nameserver they'll drop your site from their
system. So, at the very least, you can't use Cloudflare as a secondary DNS
provider (at least the last time I checked).

~~~
pas
I was talking about just the DNS layer, but ... you can periodically check
what IP Cloudflare would return and return that. (There's also the apex-CNAME
record type, ALIAS or DNAME, I don't remember right now, but PowerDNS supports
it, and you can set up the resolver to use CF's NS.)

Of course CF doesn't support anything like this, but it works well (because
they use quasi fixed, static anycasted IPs for the HTTP(S and TCP?)
proxying/load-balancing too), even if it's hacky as hell.

If they have any active verification of nameservers, and if they disable the
proxies if they detect something bad, then ... it won't work obviously :)

But technically there's nothing amazing in being a registrar of a domain. So
both CF and easyDNS are just stubborn in the name of user experience
(consistency).

... all in all, working around any SPoF (reliably) will usually require
exponentially more resources/engineering/care.

------
johnklos
I think people are Cloudflare fans simply because so many other services suck
more.

I gave Cloudflare a fair shake. But after hearing their lies way too many
times, I'm calling them out for being deceptive and unscrupulous.

After being told that sites pretending to host Adobe Flash updaters and
pretending to be Bank of America can't be taken down by Cloudflare because of
their rights to free speech, I knew their attempts to pretend to be one of us,
attempts to pretend to care, were nothing but bullshit.

They claim they don't host. If hosting DNS is not hosting, then what do you
call DNS hosting? You literally CANNOT use their abuse web form to report
domains which use Cloudflare just for hosting DNS. They do not handle abuse
sent to their abuse email address (they simply send a form response saying to
spend ten minutes filling out their crappy form that has all sorts of
problems).

Of course, their web proxy services are also "not hosting", even though
they're protecting all sorts of scammers.

So why should we think they're not bullshitting us when they run 1.1.1.1 and
tell us they're not logging? Why should we trust them more than our ISPs by
running DoH through them?

They WANT us to be dependent on them, because the more control they have, the
more money they can make. It's dangerous, and they've shown they have no
honor.

I've genuinely tried to correspond with them on Twitter, and they excel at not
answering the question asked but instead just diverting. It's scummy,
unprofessional behavior and I encourage everyone to consider whether they
deserve anyone's data or business.

~~~
blisseyGo
For me, the biggest disappointment was when they were defending hosting
terrorist site for "free speech" reasons:

[https://www.fastcompany.com/90312063/how-cloudflare-
straddle...](https://www.fastcompany.com/90312063/how-cloudflare-straddles-
its-role-as-privacy-champion-and-hate-speech-enabler)

[https://blog.cloudflare.com/cloudflare-and-free-
speech/](https://blog.cloudflare.com/cloudflare-and-free-speech/)

> the company serves at least seven groups on the U.S. State Department’s list
> of foreign terrorist organizations, including al-Shabab, the Popular Front
> for the Liberation of Palestine (PFLP), al-Quds Brigades, the Kurdistan
> Workers’ Party, al-Aqsa Martyrs Brigade, and Hamas.

> CEP has sent letters to Cloudflare since February 13, 2017, warning about
> clients on the service, including Hamas, the Taliban, the PFLP, and the
> Nordic Resistance Movement. The latest letter, from February 15, 2019, warns
> of what CEP identified as three pro-ISIS propaganda websites.

All this while routinely censoring other sites. Their actions are very
different than their words.

~~~
tekknik
From the article:

“ This question assumes the answer. A website is speech. It is not a bomb.
There is no imminent danger it creates and no provider has an affirmative
obligation to monitor and make determinations about the theoretically harmful
nature of speech a site may contain.”

~~~
creeble
Are DDOS-for-hire sites "speech"?

Their only use is to suppress speech.

~~~
tekknik
Well that would depend on what the target of the DDOS is wouldn’t it? Either
way the point being we want companies out of politics. They have no business
in them.

------
drawkbox
That is the problem with massive centralization even if it is market level and
internally Cloudflare (or any other big fish) does decentralization/fail-over
of their own. Many of these companies should have had fail-over to competitors
at least for reliability.

The problem with near market monopolization, oligopoly, even the singularity,
the fail-case is catastrophic and may even wipe out decentralized, diffused,
dispersed, decoupled system solutions that can't make it due to so much
relative size from the big fish that it squashes them along the way. The
bigger the ship the longer it takes to turn.

This Cloudflare issue is like the recent Facebook SDK startup crashes where
everyone has a single point of failure on Facebook SDK where people should be
using or able to use the OpenGraph API directly as they need which is more
robust to the app that uses it, it won't crash on startup.

In business it is a goal to centralize to grow, in nature and robust systems
it is more differentiation and decentralization to survive. There will always
be a push and pull between these two forces.

Systems and markets are like gardens. The garden must be maintained, new seeds
planted and helped to grow from small to mid-sized, mid-sized plants the bulk
of the garden, and then the larger plants need to be culled back when they get
too big to not take the mid-sized and then all the resources from the new
seeds/small plants. The problem is we have allowed the top end to take over
the garden and when they fail they fail spectacularly. The bigger the scale
the bigger they can fail.

~~~
twblalock
There is no CDN monopoly. There are several to choose from. Cloudflare is one
of the new kids on the block.

There is no cloud monopoly either. Customers can choose from AWS, GCP, Azure,
and several others.

The problem is not market concentration. There are plenty of options. The
problem is customers choosing to put all their eggs in one basket.

~~~
david-cako
True, however at AWS at least, customers are specifically told "multi-cloud
doesn't allow you to fully leverage the benefits of AWS", whatever that means.

It makes sense that cloud companies are inclined to keep customers from giving
money to competitors, but they way they sell it and structure services,
reserved instances, and enterprise discounts is such that you basically are
putting all of your eggs in one basket.

~~~
theptip
> "multi-cloud doesn't allow you to fully leverage the benefits of AWS",
> whatever that means.

One of the selling points of cloud providers is managed services like SQS. If
you run a multi-cloud architecture, you either can't use managed services, or
have to build abstraction layers on top of them (and only use the features
that exist in both cloud providers' versions of the managed service).

If you want to use a managed service that only exists on AWS, then that's
obviously incompatible with a fully multi-cloud architecture.

~~~
AnthonyMouse
> If you run a multi-cloud architecture, you either can't use managed
> services, or have to build abstraction layers on top of them (and only use
> the features that exist in both cloud providers' versions of the managed
> service).

And this is, of course, why they do everything they can do discourage it.
Because if you do that, not only are you not reliant on them for availability,
you can switch more of your business to the other provider(s) based on current
pricing, and they do not want that big time.

------
synaesthesisx
What's funny about this outage is I'm sure many of of us (myself included)
used this window to analyze large services and determine an increase in major
Cloudflare customers and presumably, revenue. Even ISPs like T-Mobile faced
issues due to the Cloudflare outage! The situation has exposed just how
critical Cloudflare is.

I went ahead and bought calls ahead of NET earnings next month. Cloudflare is
becoming an increasingly bigger part of the internet backbone. Purely
speculating here, but I wouldn't put it past AWS or another large player
acquiring them soon.

~~~
dharmab
> an increase in major Cloudflare customers and presumably, revenue. Even ISPs
> like T-Mobile faced issues due to the Cloudflare outage!

Careful about this methodology. Some services at my org were impacted despite
not being direct CloudFlare customers. They had external dependencies that
used CloudFlare.

~~~
Fiveplus
So it's much bigger proverbial 'blast-radius' lest something happen to
CloudFlare? Can you elaborate a bit on that part? I'm interested in knowing
more.

~~~
dharmab
Simple case of dependencies failing. Not much to elaborate.

e.g. NPM.js uses CloudFlare DNS, so services which needed to talk to NPM.js
weren't able to do so.

------
valuearb
“Cloudflare apparently fat-fingered a routing update and sent all of their
global traffic to a single POP, vaporizing it almost instantly.”

Made me chuckle, as it gave me the image of a large server in some massive
server farm glowing red, then bursting in a massive burst of light as dozens
of bearded Sysadmins run out of the building screaming.

~~~
sciurus
In my experience the sysadmins would be running in the opposite direction.

~~~
Baeocystin
As usual, a relevant one: [https://xkcd.com/705/](https://xkcd.com/705/)

------
ehsankia
Didn't people also say the same thing about AWS a while back when that had a
downtime? I guess the internet has multiple "Single-Point-Of-Failure"s.

~~~
benbristow
Ironic since the internet (ARPANET) was specifically designed to not have a
single point of failure

~~~
KirinDave
Doubly ironic since in doing this, they created a system where a _protocol_ is
the SPoF; namely nonsensical or false BGP advertisements can quickly kill the
internet as a whole if done correctly.

------
JohnTHaller
Many of us don't even know it. My DNS for PortableApps.com is run through
Digital Ocean, which uses Cloudflare for DNS.

~~~
gripfx
On an unrelated note. Thank you so much for PortableApps.com! It was
invaluable at university when studying in the library. To this day, I still
use it for utilities that don't need to be installed on my desktop.

~~~
mrsalt
I am also grateful for PortableApps.com. Along with Scoop and Homebrew, they
make using a system without root or admin privileges a really nice experience,
in both Windows and Linux.

My sincere thanks to John and all the PortableApps.com contributors.

~~~
JohnTHaller
You're welcome! I'm glad it's helping you be more productive!

------
aneutron
Okay here's the thing. I'm okay with people bashing other (competing)
companies when they do wrong. However, I believe it is somewhat childish and
uncalled for to bash another company, because of a mistake.

First of all "I use easyDNS so I didn't notice it at all tbh" is not only a
childish assertion, it's borderline a falsehood. You DO NOT offer the same
services, nor the same scale. (No, VOD would not work if your VOD provider
used Cloudflare's offering.)

Second of all, as some have noted in other comments, you are very welcome to
get just as big as them if you can offer similar (excellent) service and
similar extremely competitive pricing. Otherwise, keep working on your offer
and stop going for low hanging fruit like bashing the competitor for an outage
when they literally might handle 1000x your traffic, and perhaps offer 20x the
services your offer.

Just a little rant ...

~~~
bszupnick
Honestly I didn't notice the domain name, and I actually thought the author
was being quite understanding by saying things like "This is inevitable and
unavoidable and entirely excusable. Everybody blows up, every DNS provider in
existence will experience downtime. No exceptions." and that they use
Cloudflare themselves.

This is obviously subjective, but to me it didn't come across as "they suck
use us" but rather pointing out the inherent flaws in this quite popular SPOF
and cautioning to avoid it.

~~~
csharptwdec19
I think the root cause (which, IMO, you correctly point out) is lost on many
modern developers.

For whatever reason there's this modern idea that if a company A is paying
money to company B for a service, that company B will handle all the 'hard
stuff' for them.

The end result is we have a lot of applications/infra built with SPOFs, in
some cases known, but in many, swept under the rug and abstracted away to
passing the buck in case of a large failure (i.e. major AWS/Cloudflare/Azure
outages).

You also see this at times when vendors pitch internal software solutions.
I've been at more than one shop where a vendor's 'silver bullet' turned into a
SPOF time-bomb because nobody considered this company's solution could fail.
After all, the sales presentation said it had %nines%!

~~~
TheCoelacanth
"Your app will go down when half the Internet goes down" is not that big of a
deal to most software companies, because:

1\. no one's going to blame me if my app goes down when half the Internet is
also down but they are going to blame me if my custom solution to the same
thing causes an outage,

2\. there's no way my custom solution is going to achieve the same uptime.
AWS/Cloudflare/Azure are not perfect, but whatever I roll for myself is almost
certainly going to be much less perfect.

~~~
Sephiroth87
They do blame you though, most people won't be aware of the real issue, when
cloudflare went down, the trending things on twitter were #spotifydown and
#applemusic

~~~
mercer
I suspect customers complaining on Twitter are not the ones cared about to
whoever decides to use Cloudflare.

~~~
yjftsjthsd-h
I build an app. I use CF for my app. Customers use my app. CF has an outage.
Customers don't care _why_ I'm down, they care that my app isn't working.

~~~
when_it-rains
What is your solution for never going down?

------
crazygringo
I guess this is off-topic, but the stock photo they're using is _cracking me
up_.

Guy at work... coffee cup on a _tablet he 's using for a coaster_... except
he's also _drinking whiskey_ from a beautiful crystal glass... there's a
folded paper airplane... there's just so much to unpack here, it's pretty
hilarious.

~~~
hinkley
> except he's also drinking whiskey from a beautiful crystal glass

And he has left the stopper off of the decanter like some sort of animal.

I think we are supposed to believe that the person here was just dicking
around online, fiddling with paper, finishing his morning coffee, when all of
a sudden he gets an email asking if anyone knows what's going on with the
website.

So he stops what he was (not) doing, puts down his coffee, and starts poking
around. At which point he realizes he needs something stronger than coffee. As
he is pouring his glass he is confronted with the true horror of the
situation, drops the stopper on the floor and just holds his face wondering
why the Universe hates him.

I wonder if we can get JJ Abrams to option the movie rights.

------
arkitaip
> We call it Proactive Nameservers, and we’re the only company in the world
> doing it for some reason.

Wait, why? [0]:

> Proactive Nameservers is a patent-pending system that optimizes the
> nameserver delegation for your mission critical domain names.

Oh.

[0] [https://easydns.com/dns/proactive-
nameservers/](https://easydns.com/dns/proactive-nameservers/)

------
niutech
The solution is the Decentralized Web (DWeb), such as IPFS
([https://ipfs.io](https://ipfs.io)), Freenet
([https://freenetproject.org](https://freenetproject.org)), GNUnet
([https://gnunet.org](https://gnunet.org)) or Hypercore ([https://hypercore-
protocol.org](https://hypercore-protocol.org)). We should start using them to
avoid centralization and embrace freedom.

~~~
nemothekid
Can you explain a bit more how this is a solution? Cloudflare isn't facebook.
They have plenty of competitors, they don't have a massive moat, and they are
almost exclusively used by businesses. Despite all of this, we should ask
ourselves how we even got here. Why would companies move to DWeb, when they
are already choosing to use Cloudflare instead of Fastly/Cloudfront/Akamai.

~~~
return1
No salespeople to sell dweb

------
dentemple
If more companies are willing to provide the same level of service and price
as Cloudflare, then they can get in on the game, too.

------
nickreese
This is just an advertisement for easydns.

~~~
lopis
Bingo. Still a good read, and easydns is just trying to profit over a screw-up
from a competitor, but still essentially an ad.

~~~
toomuchtodo
> but still essentially an ad.

Compared to the endless content marketing Cloudflare posts [1]? It's an ad,
but they're still right. That's just good content marketing (informative,
relevant, and perhaps you buy something because of it).

[1]
[https://news.ycombinator.com/from?site=cloudflare.com](https://news.ycombinator.com/from?site=cloudflare.com)

------
tannhaeuser
Whether this is an ad piece by a competitor or not, the problem with
monopolies is that "the market" (if there is one) gets skewed incentives.
Cloudflare has received heavy investment by FAANG (+MS) [1] before their IPO,
so rather than eg Google or others with a vested interest and capability
stepping up the game and invest into new IP control plane-level DDOS
protection standards or similar, the situation smells more like a backdoor
deal, such as an agreement to not go after a particular market segment.

Let's also not forget Cloudflare in particular have been accused to host/hide
the very bad boys that make protection from DDOS necessary in the first place.
Whether or not that is the case, a quasi-monopoly leaves customers with no
choice.

[1]: [https://petri.com/microsoft-google-and-others-invest-in-
clou...](https://petri.com/microsoft-google-and-others-invest-in-cloudflare)

------
raverbashing
If only DDOS attacks were taken seriously and their perpetrators punished
accordingly (and maybe if the network had better ways of self-defense) instead
of companies and websites having to fend for themselves (or having to resort
to solutions like Cloudflare).

~~~
ericlewis
I am not sure I understand this comment, in the context- cloudflare
misconfigured some routes and it was quickly resolved. was this a DDoS?

~~~
jaywalk
I think the point was that so many companies wouldn't have to rely on anti-
DDoS protection from Cloudflare.

------
black_puppydog
What I find really shocking is the abundant use of CF on _piracy_ websites of
all things. Not the serious ones of course, SciHub and library genesis are
mirrored differently.

But a lot of small torrent websites and such simply won't load without JS and
specifically CF code. It's pretty crazy. Luckily I don't use any of those
bEcAuSe IlLeGaL but still, I find it really depressing, especially when
webtorrent, IPFS etc are available, and frankly many of those pages will never
have to bear a load that makes CF a requirement.

~~~
jdc0589
its not about caching or handling normal traffic. its about ddos protection.
sites like that are frequent targets.

~~~
black_puppydog
and there should be enough of them that that shouldn't matter.

------
eloff
Now that Cloudflare is also a registrar, they could pretty easily implement a
nameserver failover like EasyDNS. I hope this event underscores the importance
of that to them.

It's worth noting Cloudflare also supports secondary DNS, but only for
enterprise customers: [https://blog.cloudflare.com/secondary-dns-a-faster-
more-resi...](https://blog.cloudflare.com/secondary-dns-a-faster-more-
resilient-way-to-serve-your-dns-records/)

------
actuator
Does anyone know a dashboard/list for past Akamai outages. Surprisingly, I
haven't seen a lot of news about Akamai downtimes. I searched on Google and
the last one I found in a news report is from 2011 when its customers like
Facebook, Twitter were impacted.

They were a PITA to work with when I used them in the past but if they are
really that good in service availability, you can have some justification for
their overpriced service.

~~~
EE84M3i
AFAIK Akamai only makes their service notifications available directly to
subscribers.

------
anthk
Cloudflare doesn't work without JS, even for static jsless sites. It's a pest.

------
zelphirkalt
Well, I usually block it anyway, because it is not only a single point of
failure, but also a single point of concentration of data, that can be used to
track, spy on and profile users. As such, I do not blindly trust Cloudflare. I
remain sceptical, no matter how positive their public image is. Especially I
would not set my DNS to cloudflare.

~~~
ampersandy
Who do you use for DNS then that you do trust?

~~~
yjftsjthsd-h
For lookups, it's not that hard to do your own recursive resolver.

------
citizenpaul
I love all the comments about fail-over for DDOS/DNS protection. What is your
budget? Well we are looking at around $0.00 for our maximum allowance. Ok so
single point of failure it is I guess we are done here. Companies only say
they care when there is a problem, the reality is that they dont.

~~~
divbzero
It obviously takes more than $0.00 but not that much more. It’s a matter of
adding a second DNS provider and making sure you replicate DNS records
manually or with AFXR.

What’s really puzzling is if the same companies spend money on active/passive
failover for application and database servers while overlooking DNS single
point of failure.

------
tuxninja
AWS in 2012, DynDNS after that, now Cloudflare...I wrote about this a few
years ago, the threat of the singularity of the Internet. What was distributed
will be centralized again.
[http://tuxlabs.com/?p=430](http://tuxlabs.com/?p=430)

------
aww_dang
Cloudflare is hell when you need to load scripts and css 403'd by them.

Here's my work around:

1) open developer tools 2) refresh page 3) move tab into a new window 4) find
a blocked resource in the network tab of developer tools and open it in a new
tab 5) verify humanity 6) repeat step 4 and sometimes 5 for each resource the
page requires 7) move the problematic site out of the new window and close all
of these tabs at once

It is a terrible experience. CF devs and publishers have no idea how
inconvenient their service is. I wonder if they have ever lived in a region
where every ISP is both incompetent and listed in the CBL?

I frequently skip sites that use CF. The captchas are obnoxious.

The entire concept of a webapp firewall seems a bit backwards to me.
Developers should fix their insecure applications.

------
dschuetz
When I was figuring out how DNS works back 10 years ago, I was told "Why don't
you just use 8.8.8.8 everywhere? Why do you need your own DNS server for
anyway?" every single time when I asked a very specific configuration
question. Some years later 8.8.8.8 was just replaced with 1.1.1.1 with same
critical counter questions instead of helping. So, it appears to me, instead
of understanding DNS and routing, people and tech-bro businesses take
shortcuts by using centralized infrastructure for their systems, creating
dangerous configurations. Now, Cloudflare have made a mistake, and half the
Internet crumbled down. See the irony?

------
xwdv
Turns out no one got fired for choosing Cloudflare.

------
erichocean
"Turns out over half the Internet has a single point of failure called BGP."

------
OutsmartDan
Is it realistic for a small-medium sized business to have more than one DNS
provider?

~~~
throw0101a
> _Is it realistic for a small-medium sized business to have more than one DNS
> provider?_

Yes: as the weblog post points out, you can have EasyDNS as your master with
their multiple DNS servers, and then _also_ have (e.g.) Route53 slaved to
EasyDNS and have those _in addition to_ EasyDNS in your records.

DNS servers have had replication for decades.

------
nonbirithm
Question: is it even possible to have DDoS protection without using a provider
of it which becomes a single point of failure? Or is it maybe possible to
decouple this single feature from everything else that Cloudflare provides
that could take out all the sites in the future from an unrelated
misconfiguration?

I don't see the centralization as a positive, but I'm wondering what
percentage of the websites that were taken offline see themselves as having no
choice but to use Cloudflare in order to prevent themselves from being taken
down anyway from malicious actors instead of by accident.

~~~
divbzero
I think you could use Cloudflare as your primary DNS provider and benefit from
their DDoS protection, but also specify backup DNS name servers with a
different DNS provider in case Cloudflare fails.

------
rchaud
Starting yesterday, I have been getting Cloudflare's hCaptcha image
identification quizzes on every site that uses them. Googling the issue
indicates that the IP address range may have been blacklisted, if say my
laptop (a Mac that doesn't visit any shady sites) or my router has been
compromised and is now running a botnet or something.

It seems unlikely because my phone can access sites on wifi fine, and again my
Mac doesn't appear to have any malware.

Could it be that the outage is somehow relate to Cloudflare putting these
captchas up as a precautionary measure?

------
tyingq
It's interesting to me that Cloudflare doesn't really have any competition
with a similar business model. I suppose the free plan requires quite a lot of
spending before the upgrades offset it.

~~~
nerdponx
I don't need a free tier. I just need a "basic bitch" tier for my personal
usage. Who are some Cloudflare alternatives for this?

~~~
corford
Depends on what you want but a very solid free CDN and DNS option is: host
your site on netlify and use dns.he.net for your nameservers.

Another good DNS option is dnsimple.com or, indeed, EasyDNS. For even more
redundancy, use one provider as your domain registrar and another for your
nameservers (and set short TTLs for your zones so you can re-point IPs quickly
if you need to).

For the other things Cloudflare offers on their free tier, I'm not sure what
good alternatives exist (there must be some, I'm just not familiar with them
outside of the obvious AWS alternatives).

Edit: one caveat with above advice, I have no idea if netlify use cloudflare
behind the scenes...

Edit 2: For other options, checkout
[https://www.cdnperf.com/](https://www.cdnperf.com/) and
[https://www.dnsperf.com/](https://www.dnsperf.com/)

------
Uhhrrr
> But if you want to use a preferred DNS provider, such as Cloudflare, who use
> their DNS responses to optimize your website proxy. That works best most of
> the time, so then you want to go with an active/passive model that will step
> back when things are going according to plan, and then when these periodic
> network cataclysms do occur (and they will), they step into the breach and
> update your nameservers so that you at least stay up until the crisis is
> over.

Copy editors are cheap and your reputation shouldn't be.

------
louwrentius
Who at a C-level position is going to tell anybody that the potential risk of
Cloudflare (or Amazon/Azure/GCP) going down should be protected against?

I would applaud them, but I wonder.

~~~
Nasrudith
My guess is Wall Street HFT or other financial areas with very strict
unscheduled downtime penalties where they are effectively incentivised to be
batshit paranoid as it would take decades for any penny pinching to remotely
pay off. I don't know if many of them use Cloudflare for their domains though.

------
WarOnPrivacy
This is the 3rd DNS related outage I've seen in a week. Frountier
Communications lost their DNS. Trying to recall what the 3rd one was.

My Unbound resolver round-robins DNS-over-TLS requests, between Cloudflare &
Quad9. Cloudflare's outage never impacted us that I could tell.

Nevertheless, I am reminded that I ought to add a couple of DoT providers (who
aren't Google). Not sure who else came online since I setup.

------
hinkley
High availability is an insurance game, and perhaps we need to start treating
it that way.

Rather than admitting that your customers need to maintain a business
relationship with your competitors, _you_ need to admit you need to maintain a
business relationship with your competitors. That we need a moral equivalent
of underwriting in the cloud space.

------
yokaze
I am not sure, if I would trust anyone who misuses the term "Single-Point-of-
Failure" on matters of reliability.

~~~
yjftsjthsd-h
Why not? It's a single thing, that if it fails, causes your
app/website/whatever to fail.

~~~
yokaze
Because it isn't a single thing, it is a redundant system. Redundant systems
can also fail, that doesn't make it a _single_ point of failure.

You can add another system in parallel as the vendor of the product suggests,
or you can improve the resilience in the redundant system.

To make a hyperbole: my galera cluster is failing, its a single point of
failure, so I setup a cockroach cluster in parallel.

In a way, it is right, as there are failure modes specific to the individual
systems, but I think, it is incorrect, to label that a SpoF.

~~~
yjftsjthsd-h
That is a fair point; SPoF depends on what level you're looking at. A RAID
array removes the SPoF that is a single disk, but still leaves a SPoF in the
RAID controller or CPU or power supply; a ceph cluster can withstand the loss
of a whole rack but could still fall to certain software bugs. Likewise,
"cloud" companies are internally redundant, right up to the point where they
aren't. It depends on how you scope the question.

------
surround
Instead of trusting any DNS provider with your queries, I recommend running
your _own_ recursive resolver with Unbound.

[https://nlnetlabs.nl/projects/unbound/about/](https://nlnetlabs.nl/projects/unbound/about/)

~~~
everfree
But then aren't you just trusting your VPS provider or ISP instead?

~~~
surround
Unfortunately, ISPs can see the domains you connect to, even if you use DNS-
over-HTTPS.

[https://irtf.org/anrw/2019/slides-
anrw19-final44.pdf](https://irtf.org/anrw/2019/slides-anrw19-final44.pdf)

------
solotronics
Maybe it would make sense to have multiple independent sections of backbone at
the BGP level. Instead of having one public AS/backbone, break it down into
regions at least so that it is more confederated.

------
jonplackett
Similar problems are happening with crypto - yeah it’s distributed but so many
people are using Coinbase that if they go down it’s going to cause a lot of
problems

------
phkahler
If you keep much of a website simple - plain html and css - won't that reduce
the need for a CDN in the first place?

~~~
foxfired
Yes, but cloudflare reach is much deeper then that.

I am not a cloudflare customer but my all websites failed that day. The reason
was digitalocean uses cloudflare, I use digitalocean. So apparently I depend
on cloudflare.

------
TedDoesntTalk
> Turns out half the internet has a Single-Point-of-Failure called
> “Cloudflare”

The other one is called AWS.

~~~
louwrentius
So in the end, who cares about single point of failures?

~~~
timbit42
Militaries. The internet was created so the US President could command the US
military even in the case of nuclear war knocking out many of the internet's
nodes.

------
aabbcc1241
There are some (research) work on proof-of-human protocol, interesting
direction

------
neycoda
When Cloudflare causes the problem they were built to solve.

------
knorker
The other half is on AWS?

------
tonyfader
you can warn people, but they don't listen...

------
psim1
What responsibility and reparative measures has Cloudflare taken for Friday's
incident? Was anyone fired for the mistake?

~~~
sujinge9
Why would firing someone make you feel like reparative measures have been
taken?

~~~
psim1
Reparations would show an actual sense of responsibility. Firing someone would
be appropriate if they were negligent. Other measures might be more
appropriate. Is it enough that any time there's a Cloudflare incident, all we
get are lengthy blog posts and sorries from Cloudflare?

~~~
axaxs
Firing people for making mistakes is a great way to watch productivity dive.
The easiest way to prevent mistakes is to do nothing at all.

~~~
psim1
I understand the point being made here, but what are those affected supposed
to take away? Cloudflare made a mistake that caused (x millions of dollars of
lost online commerce revenues, y number of missed telehealth sessions, etc.)
and since we do not punish mistakes, nothing was done. Sorry everyone!

~~~
axaxs
Understood. Typically, as a company, you write - 1) what went wrong 2) how did
it go wrong and 3) what you have or will put in place to prevent it from ever
happening again

It's a learning process for all involved, really.

From the affected parties point of view, well, they should diversify their
network a bit better. End users should hold those companies feet to the fire,
not Cloudflare's.

------
ivanvanderbyl
Pretty poor form calling out a competitor for something like this. Not the
first time a DNS provider has done this, and won't be the last.

