
CloudFlare was down - zhoutong
https://www.cloudflare.com
======
jgrahamc
<https://twitter.com/CloudFlareSys/status/308154786316963841>

\- There is a global problem that affects the CloudFlare proxy and DNS
services.

\- The problem appears to be due to bad routing.

\- We are working to restore correct routes in order to bring both DNS and
proxy services back online.

\- The operations and networking team are all online and treating this as an
emergency.

\- We do not have an ETA on the response time but will continue to post
updates via Twitter as we learn more.

UPDATE. Sites are being restored now. DNS is operating.

I don't know all the details as bugging the network team while they were
fixing wasn't going to help. We'll get a postmortem blog post up.

~~~
shanelja
Just discovered this problem on my company website, thanks for the update.

\--

Seems to be getting better now - intermittent 502's and 504's, the occasional
load (speed is choppy though)

\--

Back to near-instant load times, great job guys! I look forward to the blog
entry on this (P.S. It might be worth hosting your status page elsewhere in
the future, whilst this might not happen very often, it's when it does that
your site needs to be working, the lack of redundancy here is startling.)

------
rdl
I'm starting to think it's questionable to depend on CloudFlare more than
necessary, but they're still the best option for some things. (I'm a customer,
but probably going to stop being a customer this week; I was mostly curious to
test it out. Not really decided, though.)

1) The CloudFlare security model for SSL basically lets them MITM all your
traffic. Probably not a big deal for SSLizing a normal website, or even for
accepting credit cards), since they're a decent-sized US company with legal
liability, although I'd be concerned about their internal security vs. your
own internal security (since you're still fully exposed on your side, too --
it doesn't improve security, and can at best not be a source of new
vulnerability).

2) Their DNS doesn't appear particularly redundant; it's just anycast in one
big block. Using CloudFlare for DNS seems to be bad practice; you should use
something else and cname to CF. Ideally something with multiple DNS servers
either individually anycast or in at least two independent (probably anycast)
netblocks.

3) Performance of the proxy service seems adequate in my experience but for
sites with large amounts of overseas-source traffic, I've heard of people
getting lots of suspected-bad-guy path. For a free forum like 4chan that's
probably fine; for an e-commerce site, probably not.

~~~
saurik
Can you say more about "lots of suspected-bad-guy path"? (I do not use
CloudFlare currently and am not intending to do so, but I do run an e-commerce
site, and have not heard this term used before: the idea of an entire axis
along which I have not been evaluating my infrastructure intrigues me.)

~~~
benologist
I saw their 'bad guy' page many times living in Costa Rica then Nicaragua. I
didn't realize what it was until a friend started testing out CF and I got it
on his site and then we realized what was going on.

That was over a year ago though, I assume things have improved as they got
more and better data (and resources). I was in Costa Rica in January/February
then again in August/September last year and didn't come across it I don't
think.

I'm in Turkey at the moment and for the last couple weeks and haven't seen one
at all, and that's including using an EC2 proxy because of the censorship
here.

------
ultimoo
My feels go out to the ops folk at Cloudflare. Mistakes happen no matter how
many years of experience people bring in, or how much they're paid. We're all
humans after all. It must be a pressurizing task to be responsible for
potentially millions of dollars of losses during this downtime.

I hope the issue is resolved soon and if a person caused it, they're not in
too much trouble.

~~~
moot
Amen.

~~~
shanelja
I guess you're here because 4chan is down?

Something I always wondered - do you use Cloudflare simply for the CDN in
respects to photos, or does 4chan also frequently become the target of DDoS
and other external attacks?

~~~
moot
We use CloudFlare for everything -- CDN, DDoS mitigation, Railgun (see below),
et cetera. They serve ~1.5 petabytes of bandwidth on our behalf every month
and proxy billions upon billions of requests. I'm a huge fan of the product
and team, despite hiccups like this one.

More on Railgun: [http://arstechnica.com/information-
technology/2013/02/cloudf...](http://arstechnica.com/information-
technology/2013/02/cloudflare-blows-hole-in-laws-of-web-physics-with-go-and-
railgun/)

Edit -- I am officially faster than carrier pigeon:

me: was i the first human to notify you guys? | me: i caught it within the
first 30-60 seconds i think | me: because i have no life and never sleep |
CloudFlare Ops pal: yeah. you did.

If anyone wants to hire me to check their site instead of Pingdom, feel free
to ping me!

~~~
bowmessage
Even the creator of 4chan has to have his own 'first' moments once in a while
:)

~~~
moot
What's the opposite of slowpoke.jpg?

------
overshard
As someone who hosts hundreds of PAID sites with CloudFlare this is pretty
unacceptable. I'm giving them thousands of dollars so that this doesn't
happen. Will probably be moving off unless they have some very good reasoning
behind a world-wide shutdown of a geo-redundant service...

~~~
zhoutong
CloudFlare offers CNAME option for paid customers. So you can use an
enterprise DNS service and only point to CloudFlare via a CNAME record.

When disasters like this happen, a quick DNS change can be a life-saver.

~~~
foobar2k
There's no such thing as a quick DNS change.

~~~
zhoutong
My startup NameTerrific can support instantaneous DNS updates in a geo-
redundant Anycast infrastructure. As long as your TTL is sufficiently low
(<300), the impact is quite limited as propagation time is negligible at
NameTerrific.

EDIT: Sorry guys. We got some issues with a gem after installing the recently
updated ruby2.0.0p0. The unicorn workers were timing out. TerrificDNS is
completely unaffected and the site is already running again.

~~~
Uchikoma
Not the best advertising

<http://www.nameterrific.com>

"502 Bad Gateway

nginx/1.2.7"

~~~
Nux
Hopefully the back-end is not running Ruby. ;-)

~~~
rdl
The back-end is AWS Route53, it seems.

~~~
zhoutong
Well, we have already soft launched our own TerrificDNS Anycast and it has
replaced the Route 53 solution. TerrificDNS platform is running on Redis +
PowerDNS.

~~~
Uchikoma
Terrific.

------
zobzu
In my experience, cloudflare has been little more than a scam for anyone with
half decent traffic. Not really surprised. Funny how the status page shows all
green (are those just static button?) while they acknowledge there is an issue
and that they don't know what's going on.

<http://www.cloudflare.com/system-status>

------
brador
From the cloudfare business page:

"2500% guarantee This extended Service Level Agreement guarantees 100% uptime,
and adds a multiplier to owed service credits resulting from any lapse: 5
times any downtime minutes and 5 times customers affected = 2500% guarantee."

~~~
hu_me
well they have 785000 sites were down for 60mins so that adds up to... a lot
of service credits

~~~
robotkad
Most of them would be on the free tier

------
packetlss
Looks like they dropped off the internet:

<http://www.youtube.com/watch?v=wMRaKtydILI>

AS13335 = Cloudflare

------
aytekin
This is down as well: <http://www.cloudflare.com/system-status>

They should host this page on a third party provider.

~~~
MichaelApproved
In that case, it should be <http://status.cloudflare.com>

~~~
dbuxton
Except that I think they are hosting their own DNS... I'm not able to retrieve
their SOA information even at the moment.

~~~
mh-
ouch. been awhile since i've seen that failure mode.

------
andyhmltn
I use a pretty major forum that has a huge amount of traffic. The owner
migrated it to CloudFlare. For the past 5-6 weeks the site has 50% of its
request go to a 'Sorry xyz is not available right now'

~~~
earless1
You should make sure that the site is not actually returning 500's for those
request. We had some similar issues when we first started using the service

------
ksec
Somebody pointed out about the CNAME available on Cloudflare. I never knew
that and i checked out the article. The First paragraph:

"CNAME setup is a manual process generally available to paid CloudFlare plans
only. If you are interested in testing CNAME setup, please contact CloudFlare
__first __with the domain you would like to test CNAME with. Please
specifically mention CNAME Setup in the subject field for faster review.
Allowing for CNAME setup is entirely at the discretion of CloudFlare."

So NO: This isn't even a features at all. They made it as hard as possible to
set this up and will grant you the use of it as they like.

------
zhoutong
It seems that CloudFlare's DNS is down, and affecting NameTerrific as we have
a CNAME record pointing to them. I had to change the CNAME record to get our
site working again.

EDIT: Based on Twitter search, all CloudFlare sites seem to be down.

------
DigitalSea
Cloudfare may be there when your sites go down, but who is there for Cloudfare
when they go down? Nobody it would seem, ha.

~~~
mh-
including their status page

~~~
mmahemoff
That part surprised me. I thought it was common for ISPs to _not_ use their
own services for critical web presence, i.e. places users might visit when
it's down.

------
stenehall
And it's up again!
<https://twitter.com/CloudFlareSys/status/308170566760792064>

------
hahainternet
It's not just a case of DNS being down, I can't see any BGP either. Pretty
major failure.

------
endijs
Even their status page is down. And sure - all my sites too. And funny part is
that for most of my sites i have stopped Clouflare features and use just their
DNS. Never thought that I will fail because of DNS not being available.

~~~
mmahemoff
Cloudflare user here. Can I ask why you turned CloudFlare services off?

~~~
endijs
btw - sites are back. I hope they will not go down.

Why I have switched off services? Because once i enabled them sites got
slower, not faster. Sure - I doubt that most users noticed that, but for
example, if i checked Pingdom or Google Crawl Stats - 'Time spent downloading
a page' situation was very clear. With Cloudflare it took Google 2x more time
to download page than without. I had no time to investigate why thats so, but
after switching Clouflare off i was again back to 500ms.

Edit: Sites are not back. I guess that work the ones for which i have DNS
cached. Lets wait...

~~~
mmahemoff
When I briefly did that test, I didn't find much difference on Pingdom; but I
like some of their other services and I assume they can handle burst activity
at least as well as my own server. So I'm a big fan overall and keep using
them.

------
endijs
All my sites are now back online! ~ 40min downtime.

Edit: Looks like DNS is back. However if you use CloudFlare services then you
might still have problems. Like:

504 Gateway Time-out cloudflare-nginx

------
wyuenho
It took a CloudFlare total wipeout to discover how useless our browsers are
against domain name lookups that take a ridiculously long time to timeout.

News flash: CDN fallback like the one below is next to useless unless the
first request times out reasonably quickly.

[http://css-tricks.com/snippets/jquery/fallback-for-cdn-hoste...](http://css-
tricks.com/snippets/jquery/fallback-for-cdn-hosted-jquery/)

------
jelled
imgur seems to be down for me as well.

~~~
mappu
A workaround: try replacing imgur.com in the URL with filmot.com.

------
SchizoDuckie
I'm going to say this on the posibility of this being seen as a flamebait...
But You should have chosen Akamai over Cloudflare.

It's so funny how everybody jumps on top of new companies that say they can
proxy all of the interwebz for a low price. (Cloudflare, Blackberry)

And then they fail...

~~~
rdl
Akamai won't even talk to people for $0, $25, or $200/mo. I doubt they'd do so
for $3000/mo, directly, unless they thought you'd grow.

~~~
saurik
At that $3k/mo level, you can definitely talk to CDNetworks (the CDN I
currently use), which is sandwiched between two orders of magnitude of scale,
CloudFlare on one side and Akamai on the other. (That said, CDNetworks seems
to be much better positioned with regards to China than Akamai.) (That said,
I'm actually pretty certain that Akamai would talk to you at the $3k/mo level:
have you even tried calling them?)

~~~
rdl
They'd talk to you if you were a startup and $3k, but probably not so much
(directly) if there wasn't growth potential. There are hosting providers who
resell Akamai for smaller customers, though. (The only time I ever cared about
high-end CDN involved businesses Akamai wouldn't serve, though.) I haven't
gone through the normal sales route with them, but I know lots of internal
Akamai people in security/ops/etc. and small customers are not really their
market.

CDN itself is essentially a commodity; it's not too hard to keep multiple CDNs
in rotation. There are probably 20+ big CDNs worth consideration and another
bunch of resellers. (Amazon CloudFront, BitGravity, Level3, Limelight are
probably the first ones I'd think of for smaller sites; Akamai is still the
undisputed king for top performance.)

DNS is the thing which is more interesting to me.

I'd probably go with Route53 for cheap good anycast DNS right now; everyone
else seems to either be a clown or super expensive (or bundled with other
expensive DNS service). Ultimately I guess I'll end up doing internal DNS.
(non-anycast DNS is also a total commodity, but good anycast dns not as much)
DynECT also looks pretty good. Not sure what other anycast DNS providers there
are in the <$500/zone/mo range.

~~~
saurik
> There are hosting providers who resell Akamai for smaller customers, though.

There are also many other CDNs that exist in the massive territory between
CloudFlare and Akamai (such as CDNetworks, the company I had mentioned).

> CDN itself is essentially a commodity; it's not too hard to keep multiple
> CDNs in rotation.

For latency-insensitive use cases in generally centralized territory, I agree
that CDNs are "essentially a commodity". The correct strategy would seem to be
to call a number of them, and negotiate a good deal, not to assume that the
one that has a printed sticker price is somehow the right choice (as some
people here seem to have been doing ;P).

However, to make the counter-point to this: the cache hit ratio that is being
reported by CloudFlare for evasi0n.com (note: I do not have control over that
site's hosting; that's choice was due to planetbeing and pod2g) is 81% <\-
this is for a static single-page information site. How various CDNs handle
caching, whether they cache you on disk or in RAM, what they do with regards
to hot connections or pre-fetching... these all have massive performance
implications on your website.

~~~
rdl
It's a totally reasonable thing for a person who is busy to "satisfice" on
many priorities, vs. optimize. Maybe CloudFlare isn't optimal, but if I can
get a price and sign up in minutes, and it's good enough, that might be the
right choice. It's not just the time; it's that talking to a salesperson is
usually psychologically draining. You'll never be able to pick up a phone and
get a price in a few minutes; it's always "where is your business located", "x
is the rep", "x will call you back", etc. It turns into a fiasco. You end up
having CDN sales reps come to your office to meet with you to "understand your
requirements". etc.

Punishing "old-school enterprise sales tactics" which try to keep price from
being transparent is a reasonable choice. If you're a big content site, yes,
you should go through the effort, but for someone who just wants a small
service, buy from people who publish their prices.

CloudFlare isn't the only CDN which publishes pricing -- CloudFront with AWS
is very transparent. Rackspace Cloudfiles is transparent. BitGravity is fairly
transparent. Cachefly. etc.

Akamai is the worst at this, but Level3, CDNetworks, and Limelight don't
publish pricing either.

Offering a free service like CF does is the genius of the freemium model --
even if your service is more expensive or less suitable at the high end,
people who start out because it's free and easy will often stick with you as
long as you're "good enough" as they grow.

~~~
saurik
I find it interesting that you bring up CloudFront, because they are also very
expensive. As far as I can tell, because there are so many people of there who
have a mental aversion to talking to another human and negotiating, they can
charge an insane premium on an "engh" service.

Regardless, if picking up the phone and negotiating with a CDN, someone whose
opinion of you is totally irrelevant and where the worst-case outcome is "we
won't do business with you", how are you going to handle support on your own
product, or court investors of your company?

------
oellegaard
I'm so happy we didn't go on that wave. Redirecting your DNS to someone else
seems like a bad idea in any case. In any case, what do they do, that I could
not have done with Varnish?

~~~
mmahemoff
Bringing content closer to the user; throttling and protecting against certain
attacks; providing a response when your site is down (yes, ironic).

Also, randomly useful filters like adding the user's country to the request
header, tweaking outbound images, and auto-injecting google Analytics.

~~~
ryancl
Yes this and increasing load times tenfold.

------
grose
Seems like half of the internet is broken just because of this...

------
EdisonW
Was just about to post this. All of my sites are down. :(

Edit: 5:01 EST ..seems to be up and down according to mass pingdom messages.

------
Neso
They are back online, all my sites active

------
UnoriginalGuy
This took out Imgur.com (Reddit's favourite image hosting site) for several
hours. So it was definitely felt.

------
ensky
UP now!!! <https://twitter.com/CloudFlare>

------
ck2
<https://twitter.com/cloudflaresys>

------
vini
down here too and I can't change the dns of my site cause my registrar is down
too, oh boy.

~~~
Neso
You are really lucky :p

------
fiendsan
its sunday! seems like they pushed another faulty update (like last time)! yep
confirmed, all is down including their own site, thats pretty fucked up, when
they dont even have offsite status! good thing i dont use cloudflare in all my
sites...

------
sairamkunala
Its back as of 10:51 GMT . CloudFlare and my sites on CloudFlare seem to work
again.

------
nodesocket
Seems to be coming back, though intermittent http 502 bad gateways errors.

------
xPaw
Yep, any website using CloudFlare's DNS is not resolving.

~~~
Ateoto
For those of us using cdnjs.com, hopefully you have local fallbacks.

------
jtchang
Whatever happened to primary and backup DNS servers?

------
tuananh
all of my sites are down as well. Should I change name server or this issue
should be resolved quick (enough)?

------
tonyjin
Confirmed, several of my sites are down.

------
tuananh
Seems like the main website is back up

~~~
rorrr
504 Gateway Time-out for me.

~~~
tuananh
it's back up for me <http://monosnap.com/image/pYrdbObsYUhKwAWla5gDzzsg0>

------
noveltysystems
What a let down... Very disappoint.

------
ryancl
And who exactly didn't expect this?

------
Hengjie
Yeah mine is also down.

------
ippa
dns back, now I get bad gateway instead, progress :P

------
oron
Going back to go daddy :-(

------
ddaeo5
Still down. When they get back up, I will cancel my subscription and gladly
DDOS them. :D

~~~
Neso
Our fault, we should have back NS ...

~~~
ensky
we all expected cloudflare to be more stable than our server ...

~~~
dbaupp
Presumably they are.

I have no idea what is wrong, or how long they will take to fix it, but I'd
imagine that CloudFlare has significantly better network engineers than the
average company, and so they will fix it in far less time than the average
company would fix the same problem.

~~~
saurik
But, for the amount of money many paid customers are paying them (in essence,
anyone at that $3k/mo level that includes the critical 24/7 phone support),
you can actually get an account with a company like CDNetworks or Akamai (if
nothing else, with a reasonable CDN like EdgeCast) and have still-better
network engineers than CloudFlare.

Also, even if you are using them for free: they aren't replacing people you
have in house... they are an additional component that can independently fail,
in addition to any of the things that would have caused your average company's
network engineers to fail. They don't promise to cache enough content to
replace much of your infrastructure.

