
UltraDNS Server Problem Pulls Down Websites, Including Netflix, for 90 Minutes - rdl
http://www.nytimes.com/2015/10/16/technology/ultradns-server-problem-pulls-down-websites-including-netflix-for-90-minutes.html
======
jonah
Here are the sequence of emails we as a customer received:

1:44PM PST - The Neustar UltraDNS service is currently experiencing DDoS
traffic in the U.S. East Region. The Security Operations Team is currently
working on mitigating attack traffic and further updates will be provided as
soon as possible.

3:07PM PST - The Neustar UltraDNS service is currently experiencing a major
spike in traffic across several regions. The Security Operations Team is
conducting routing changes and working to reroute traffic at this time.
Customer will experience DNS timeouts and lack of resolution. Further updates
will be sent as soon as possible.

4:50PM PST - At 4:18 EDT time, a service outage affected Neustar’s UltraDNS
service. The impact was predominately on the East Coast of the U.S., however
some impacts could be seen in other U.S regions. Neustar UltraDNS Service was
restored at 6:34 EDT time. We are currently investigating the root cause of
the outage and will provide an update as more information is available.

~~~
lalt
The data from the outage is all here:

[https://blog.thousandeyes.com/ultradns-outage-
october-2015/](https://blog.thousandeyes.com/ultradns-outage-october-2015/)

Check it out.

------
tshtf
Serious question here: Do people really spend $90/month for 2M DNS queries?

[https://www.neustar.biz/services/dns-services/managed-dns-
pa...](https://www.neustar.biz/services/dns-services/managed-dns-packages)

~~~
bwblabs
And that's even without 'DNSSEC and IPv6 support', 'Regional Routing' and 'DNS
Load Balancing (where available)', that's only available for 'UltraDNS
Enterprise'... (I'm running my own (Power)DNS, after EveryDNS was bought by
DYN and they didn't grandfathered old plans, charging per domain, which made
it extremely pricey)

~~~
mbubb
Yeah - i looked at DYN and they are expensive. The "Internet Intelligence"
stuff looks interesting but that is an add on.

~~~
voltagex_
I need some dynamic DNS for my VMs, I'm getting away with Avahi/mDNS right now
but I need to set up an Active Directory lab. Do you have any recommendations
for dynamic DNS? Bonus points if it's BIND9 compatible.

~~~
jlgaddis
Put AD-joined hosts in a subdomain, e.g. _subdomain.example.com_. AD DCs are
authoritative for the subdomain. Your public facing / authoritative
nameservers for _example.com_ should be listed (publicly) as the only NS's for
_subdomain.example.com_ and should slave it from your DCs.

------
fletchowns
I suppose this was also the cause of the 90 minute Bitbucket outage today?

~~~
dholowiski
Surprisingly, no. Or at least they're not claiming so. They say it was a
problem with a database server.
[http://status.bitbucket.org/](http://status.bitbucket.org/)

~~~
fletchowns
That was a different "Git and Mercurial over HTTPS slow or not responding"
outage from yesterday (or "02:00 and 03:00 UTC on 15 October 2015")

The more recent one started @ Oct 15, 20:19 UTC, they don't seem to have root
cause on the status page for it yet.

------
edgan
I am surprised that Netflix uses UltraDNS when they use AWS and they could be
using Route53.

~~~
b1naryth1ef
I would imagine they don't use it for exactly that case. Imagine a case where
most of AWS is having critical problems, you would want (at the very least)
control of your DNS to attempt a partial recovery of service for some portion
of your customers that have providers who respect DNS TTL's.

~~~
hueving
Netflix is 100% dependent on AWS for infrastructure. If AWS is having
problems, there is nowhere to redirect to.

~~~
tiagobraw
of course there is, they could redirect to a page explaining the problem
instead of a generic error page...

------
Thaxll
How can unicast be DDoS?

~~~
edgan
The article says it was an internal issue, not DDoS or hacking.

------
jonah
Interesting. We use UltraDNS and didn't go down at all.

------
lorddoig
Titular hyperbole, much?

~~~
dang
We changed the title from "UltraDNS takes down the Internet for 90min" to the
article's title.

