
Netlify CDN Outage - ajsharp
https://www.netlifystatus.com/incidents/2hn44zlbyh6h
======
epoch_100
Netlify is describing the outage as being the result of some kind of attack:

"We’re currently suffering an attack that’s affecting our Free and Self-Serve
CDN. Enterprise is currently unaffected. We are working on mitigating the
effects and will update with more information as it becomes available."

------
chrisblackwell
I understand the growing pains Netlify must be going through. They have
created a great product, and are becoming very popular. I'd love to see the
Enterprise version (or any paid version) segregated out from the free sites so
we can get more reliable uptime.

The build pipeline was down for several hours a few days ago, now a CDN outage
makes things much worse, as it's taken down all our sites.

I'm sure they will resolve things quickly, in the meantime our clients will
start asking about other failover options.

~~~
drewda
According to their overall status page, they already do run two separate CDNs
for normal and enterprise customers:
[https://www.netlifystatus.com/](https://www.netlifystatus.com/)

------
gravypod
It'd be really cool to see a writeup of what is going on at Netlify.

~~~
nwsm
They are tweeting about it, and I'd expect a post mortem as they have been
transparent in the past.

------
sergiotapia
So I wasn't going crazy haha - just today we migrated our site to Netlify from
Heroku and the second I wanted to demo it my boss the COO, it was failing and
not loading.

It wasn't just netlify though. I also was having trouble loading random
websites like aptible, youtube, imgur.

------
swaraj
This affects us too. We were able to migrate to zeit now in ~30 mins.

~~~
ngrilly
Will you migrate back to Netlify when Zeit will suffer a similar downtime?

~~~
swaraj
We're definitely staying on Netlify after this outage is done. We pay for some
of their enterprise features, e.g. audit logs. We just need our sites up right
now and zeit was a fast, temporary alternative

~~~
ngrilly
Do your DNS records have a short TTL to enable such a quick switch from one
provider to another?

~~~
swaraj
Fortunately (or non-optimally), our DNS TTLs were 300s (Route 53 default) so
our sites were not fully accessible off Zeit for ~5 mins. This wasn't too bad
given that our sites were completely inaccessible from Netlify.

One day when we're bigger, we'll extend those TTLs

~~~
ngrilly
Short TTLs are bad, but I confess using them in some small scale projects in
anticipation of this kind of worst case scenario :)

------
nwsm
My site is back up. (1:13 EST)

EDIT: intermittent issues now. Sometimes nothing loads, sometimes everything,
sometimes SSL issue, sometimes HTML but no CSS or JS load. (1:22 EST)

------
alfredxing
Their CDN hostname, `cdn.netlify.com`, resolves to a DigitalOcean node, so it
seems like they are rolling their own instead of delegating to a third party.

------
tschellenbach
Yup, Stream's blog is down as well. Thought it was WPEngine at first, but
turns out it's netlify.

------
losvedir
Huh, loaded HN because it seems YouTube videos are down for me. I wonder if
it's related.

~~~
nwsm
Another comment claimed that Netlify rents servers from other providers, so
this could be the case.

FWIW Youtube is fine for me right now.

------
volkk
insane coincidence but literally the minute i started my first project ever on
netlify just to play with it, i deployed it and nothing was loading; i kept
googling and thinking it was my fault and then it clicked to check their
status page and alas.

~~~
jagannathtech
So it was YOU who took down the network then

------
ajsharp
netlify.com appears to be affected as well.

Curious if this is a software issue or a host issue. I believe netlify rents a
lot of its infrastructure from other providers, such as Digital Ocean.

~~~
cfors
At any sort of scale (which I have to imagine netlify is running at), this
will be a software issue since hosts are designed to fail.

Serving static content is crazy easy for computers, my guess is a bad
configuration got pushed out.

------
twobat
Bad publicity is still publicity, right?

