

CloudFlare Was Down Due To Edge Routers Crashing, Taking Down 785,000 Websites - iProject
http://techcrunch.com/2013/03/03/cloudflare-is-down-due-to-dns-outage-taking-down-785000-websites-including-4chan-wikileaks-metallica-com/

======
ghshephard
First reading of this is cloudflare doesn't pre-stage their router upgrades.
Or, if this was just a config change, they exceeded some limitation that they
weren't aware of.

As a network engineer for small startups, I've made that mistake more times
than I can count - but the excuse was always "no budget for lab gear to
replicate our production environment" - I would think at cloudflare's scale,
that wouldn't be an excuse that made much sense.

~~~
caw
When you get to a certain size, there's a limit to how much you can pre-stage
and test. Sometimes you just have to roll it into production and hope you
didn't miss anything in the planning.

That doesn't seem to be the case here, but it's possible.

------
eastdakota
Here's our post mortem on what happened:

<http://blog.cloudflare.com/todays-outage-post-mortem-82515>

------
jrs235
Looks like they are down again. I can get to their homepage but none of my
sites that are hooked up to use them.

Update: back up. Looks like intermittent issues still.

