
Rackspace DNS DDOS - treerunner
https://plus.google.com/+RackspaceHosting/posts/8yVxbLqfx6Q
======
Erwin
RAX has a 6.3 billion market cap. If an org this size -- specialising in
hosting -- cannot field a DDOS-resistant DNS server, who can?

I'd like to migrate or even perhaps add a secondary DNS but RS DNS doesn't
seem to even offer zone transfers (the best you can do, I guess, is to use the
API to get your records out).

~~~
drzaiusapelord
>cannot field a DDOS-resistant DNS server, who can?

Its tough to do. DNS is a dumb and ancient protocol from the "lets all be
friends" days of the internet. Its a bit more complex than installing mod-
evasive and calling it a day.

DDOS is currently an unsolved problem for many popular protocols and services.
Blaming just Rackspace seems unfair. Last week in was Namecheap. The week
before it was someone else, etc. DNS DDOS is just a non-trivial problem to
solve.

This should also be a reminder to have more than one DNS provider in your
domain record. A backup nameserver from a different provider saves you and
your customers a lot of heartache.

~~~
zzzcpan
Hm. I always thought DDoS protection for DNS was rather simple problem to
solve, since anycast works with DNS traffic pretty much by design, so you can
have hundreds of machines spread all over the world, handling hundreds of
gigabits of bandwidth. And filtering of DNS packets is also relatively simple
thing to do at line rate.

~~~
skuhn
It's simple except that you need to have capacity greater than what is
achievable by people controlling the DDOS.

I'm not up to date on what that number is these days, but a few years ago
100Gbit/s was the target. For an Anycast DNS service, you don't need that in
every site -- if you're willing to take small sites offline and degrade your
service's end-user performance in this scenario. You will still need it in 2-3
sites, probably located in US west / US east / EU west because of the cost of
datacenter space and the availability of cheap connectivity.

Serving 100Gbit/s requires 20 10G circuits in each site (~$15-30k/mo), 80
servers (2-3 racks, so $5-10k/mo), and a not-so-cheap router.

So your monthly opex is $20-40k per site, $60-120k worldwide. Your capex is
$450k per site, $1.35mm worldwide.

It's not an insane amount of money, but it is kind of a lot to spend on DNS.
It's particularly tough to justify at a lot of companies that you need this
kind of buildout, when normally you can operate on 1/20th that amount of
hardware.

~~~
phil21
Good post. I would argue some of your locations and how many servers are
actually needed, but that's splitting hairs.

The real point I want to make is that the only companies really putting this
amount of money into DNS infrastructure are DNS providers themselves. I know
of no hosting provider/ISP/etc. who puts nearly this much thought or effort
into DNS.

Even for myself - and I consider myself a DNS junkie - my anycast cluster was
a dozen or so servers worldwide co-located w/ a mid-market CDN provider (plus
a few locations from my own ASN). That setup was greater than 99% of DNS
installs out there - but it would have fallen over under a major attack. It
truly takes some skill and financial commitment to build out correctly.

~~~
skuhn
I agree with you, there are only two kinds of companies with the kind of DNS
infrastructure that could withstand a large scale DDOS:

1\. DNS hosting companies (Dyn, Amazon, etc.) 2\. Large Internet companies
(Yahoo, Google, etc.)

And for sure not all of them! A former employer's public zone was hosted by a
DNS hosting company that didn't use Anycast, didn't have a footprint outside
US/EU, and seemed to have little to no idea when or why traffic spikes
happened.

So to do DNS really right you need a fair amount of money, some specialized
expertise and knowledge of some non-DNSy things (Anycast, among others). I'm
really not interested in hosting my own (public) DNS these days at work, it's
just not worth the time and effort until you get gigantic. As with e-mail,
there are specialists who can do it better than (most) places can.

I do kind of worry about depending on one DNS provider, so I've tried to use
two when possible. That's a whole other world of hurt though.

------
ericcholis
For those looking to export your zone from Rackspace, the rackspace python
library (prax) will let you:

[https://github.com/rackspace/pyrax/blob/master/docs/cloud_dn...](https://github.com/rackspace/pyrax/blob/master/docs/cloud_dns.md)

You could also do it via CURL:
[https://community.rackspace.com/products/f/25/p/1743/4945#49...](https://community.rackspace.com/products/f/25/p/1743/4945#4945)

------
gtCameron
Been dealing with this all morning, finally got all of my zones migrated to
Route53, but with the amount of time it takes for DNS changes to propagate we
are going to be feeling this all day.

Any advice for the future on how to add redundancy to my DNS setup? Is it as
simple as maintaining Nameservers on two different providers and pointing to
them both on my domain?

~~~
benmorris
Some decent suggestions brought up from a few weeks ago here

[https://news.ycombinator.com/item?id=8716662](https://news.ycombinator.com/item?id=8716662)

Seems namecheap and DNsimple have suffered from these attacks lately. I had
some sites affected by namecheaps DDOS.

------
bluedino
It's easy to blame 'large hosting provider' and suggest going with
'specialized provider', but isn't stuff like this really leapfrog between
hackers and the good guys?

Sure, service X might be able to block a 50 foobit attack but what about when
the next vulnerability is found and they can launch 500 foobits of DDoS?

~~~
higherpurpose
> but isn't stuff like this really leapfrog between _hackers_ and the _good
> guys_?

Said the guy on the _Hacker News_ website.

------
jlgaddis
[https://status.rackspace.com/](https://status.rackspace.com/)

------
jimschley
We went through this recently at Codeship when our provider, DNSimple, had an
outage due to DDoS- [https://blog.codeship.com/dnsimple-ddos-
outage/](https://blog.codeship.com/dnsimple-ddos-outage/) DNS is a service
that often ends up as a single point of failure in infrastructures I've seen
as it's non-trivial to implement redundancy. Having a repository/API approach
to deploying DNS records saved us in this incident:
[http://blog.codeship.com/dnsimple-dns-history-continuous-
dep...](http://blog.codeship.com/dnsimple-dns-history-continuous-deployment/)

~~~
stevekemp
Nice to see you mention a repository - I love storing DNS details in git, and
setup a site to push that on to Amazon's route53 infrastructure ([https://dns-
api.com/](https://dns-api.com/)).

Having revision-control is wonderful for history-tracking.

~~~
colinbartlett
This is a really cool idea. I'd like it even more if I could add "remotes" for
different providers. DNSimple, Route53, etc. and push my changes to each one.
I suppose the service would need to support a number of providers that
themselves support APIs.

~~~
fmotlik
Would also be possible to use the dns_deploy tool one of my team mates wrote
([https://github.com/codeship/dns_deploy/commits/master](https://github.com/codeship/dns_deploy/commits/master))
or by extending Terraform
([https://terraform.io/docs/providers/dnsimple/index.html](https://terraform.io/docs/providers/dnsimple/index.html))

~~~
colinbartlett
Hey I wanted to thank you for showing this to me. I am using it on my personal
domain and plan to use it on some customers as well. Really cool tool, thanks!

------
riteshpatel
It's nice having cheap/free DNS from people like Rackspace and Amazon, but
situations like these make you realise that it's sensible to use a company
like Dyn ([http://dyn.com/](http://dyn.com/)) that are experts in highly-
available DNS, rather than something that's a small part of a hosting
provider's services.

It's easy to forget that you can have redundancy in your load balancers, web
servers and databases (replication, multiple data centres, etc), but DNS is
how you're found by the rest of the Internet.

No DNS resolution = no one reaches your expensive, lovingly-crafted
infrastructure.

~~~
mikegioia
I think Rackspace invests a lot of money into their DNS infrastructure. Yea
it's free but its mostly because you're paying so much extra for the servers
and support.

No DNS doesn't always mean no one reaches your expensive infrastructure. That
may be true for websites relying on lots of random traffic, but most if not
all of our customers have been to our site before so there's a strong chance
that the DNS has already been cached on their computer or router.

~~~
Khao
But that's no use if you use the default TTL of 2-3 hours that I always see
whenever I configure new dns entries. Sensible TTL should be at least 24-48
hours, maybe even more for your returning users to not be affected by those
kind of outages.

~~~
dangrossman
A 24-48 hour TTL means you're hosed for 24-48 hours if your hosting service
has an extended outage because you can't point any of those visitors at the IP
addreses at another host. With a short TTL on the other hand, the worst-case
scenario is that your DNS provider is offline and you have to change
nameservers at the registrar. With a hosting outage, you can just point the
domains at the new IPs and be back up in minutes to hours.

~~~
larrys
Agree with your points.

"you have to change nameservers at the registrar"

Would add that that this is also one of the reasons not to use your registrar
for DNS, as a generality.

------
philip1209
Here is the status page - note that it uses Rackspace DNS, though:

[https://status.rackspace.com/](https://status.rackspace.com/)

------
anthony_franco
For anyone else worried about this, the best way to mitigate this going
forward is to have secondary DNS servers.

Your primary DNS provider should allow automatic zone transfers. This makes it
so that any changes you do to your primary service gets propagated to the
secondary service within seconds.

Once setup you'll automatically have redundancy incase the primary provider
starts timing out.

------
alexbecker
Who launches these massive DDoS attacks against DNS infrastructure? It would
require a substantial botnet to pull off, so they must have some compelling
reason. Maybe there's something obvious I'm missing, but I don't see one,
except perhaps for a government or large organization which had many
competitors using the DNS service they attack.

~~~
gnopgnip
Until more orgs implement BCP 38, it only takes a handful of servers with an
open resolver to multiply the attackers bandwidth by a factor of 30 or more.
UDP does not verify the source IP is the sender. Most of the internet will
drop spoofed packets already, but there are still enough netblocks passing
spoofed traffic, and vulnerable DNS servers for this to be a concern.

------
dubcanada
All of my websites using Rackspace DNS where down for around 9 hours.

It was not a good start on a Monday. And it was even harder to explain that
your websites are up, but not up to a client lol, and there is basically
nothing I can do.

So yah this marks the point at which I will be using R53 and RS DNS servers.

~~~
colmmacc
Tip: If you're using >=2 providers to survive DDOS attacks, it's best to use
no more than two name servers from any one provider. That's because resolvers
will generally try up to 3 different name servers before giving up.

The "no more than two nameservers" rule means that those attempts will always
span at least two providers.

~~~
dubcanada
Amazon by default gives 4 nameservers, can you only use two?

~~~
rustyconover
Sure you can.

------
some-dude314
I'll make the answer simple. Market cap and size do not matter. All it takes
is _competent staff_ and management support. Actually DNS is not that hard to
mitigate. You just need optimized compute and the bandwidth to take it to the
clean up equipment.

------
king_phil
Anyone with an idea how well Google Cloud DNS handles DDoS? I would image it
just sucks it up, because they already saw any DDoS volume (would it be any or
every, I'm not a native speaker?) before.

------
codezero
I'm not savvy to this stuff so pardon the unsolicited conspiracy theory, does
this DDoS have anything to do with the NK internet outage, or is it just
coincidence?

~~~
ratsmack
It can be difficult to tell. If the traffic is coming from a botnet, only
identifying the Command And Control operators will tell you anything.

