
DNSimple DDOS Attack - dedene
http://dnsimplestatus.com/incidents/v0x4h75gxf7x
======
whafro
We're kinda tied into DNSimple since we use an ALIAS record for our
bare/naked/root domain. Amazon's Route53 supports aliases, but via a 301
redirect, which doesn't work in an SSL context (without browser warnings).

Nonetheless, we just spun up a Route53 zone, exported our zone from DNSimple,
imported to Route53, and hand-migrated our ALIAS records to static A records
in the new zone.

Not perfect or permanent, but we've gotten around the outage. Also, I just
learned that pointhq has (seemingly-undocumented) support for ALIAS records in
the same style as DNSimple, so this could be another avenue to explore.

~~~
petercooper
How are you exporting the zone with things as they are?

~~~
whafro
Oof, really good point – I suppose I got lucky in that I was at dnsimple.com
just a few hours ago for other purposes. Their DNS was still cached for me,
and it all worked flawlessly. I didn't even suspect their outage extended to
their web servers, though that makes perfect sense.

FWIW, the IP I have cached is 50.31.213.210.

~~~
Pwnguinz
How were you able to find out what IP was cached for a particular domain?

------
jameskilton
We can watch this happen live @
[http://map.ipviking.com/](http://map.ipviking.com/)

Fascinating traffic floods from various locations, but the attack is not
continuous.

~~~
diminoten
How does this link show this specific attack?

As far as I understand, ipviking simply hosts honeypots around the world and
uses those to graph "attacks" against IP blocks, etc.

I would _very much_ like someone to correct me if this assumption is
incorrect, because it'd be neat to actually watch targeted DDoS attacks, but I
don't think that's what ipviking is offering.

------
webandtech
Free solution that worked for me: Set up a free account on cloudflare.com,
duplicate all dns records (thankfully I have a simple setup)... but next time
I will keep a backup zone file!

FYI - Instead of an Alias record on DNSimple, CloudFlare will allow a CNAME
record for the root domain using "CNAME flattening".

You can now set CloudFlare's DNS service to "bypass Cloudfare" on all records
by clicking the icon so you don't get any of their magic (unless you want it).

Then add CloudFlare's 2 nameserves to your domain as your first 2 name
servers. No need to remove dnsimple's name servers.

Now you have 2 DNS providers in case one fails, just make sure the records are
the same across them both!

~~~
kyletns
Hey thanks for the post, but how do I add CloudFlare's nameservers? DNSimple
won't let me add NS records on the root domain: "You may only delegate
subdomains". So in theory, I can fix the www subdomain, but my naked redirect
to www won't happen if DNSimple is still down.

Any ideas?

~~~
webandtech
You would set your name servers with your registrar, such as godaddy. If
dnsimple is your registrar as well as your DNS provider you may be out of luck
until they come back online. The help page is here though:
[http://support.dnsimple.com/articles/setting-name-
servers/](http://support.dnsimple.com/articles/setting-name-servers/)

------
scott_karana
DNS is so straightforward, so easily distributed, and so fundamental, that I'm
always astounded when it's a single point of failure for so many operations.

I wonder how many of the affected companies _do_ have redundant appservers and
load balancers, but missed this piece of the puzzle...

~~~
mike-cardwell
I'm surprised more places don't run their own DNS. It's not that difficult to
do and it means you don't have to rely on another third party for service.

~~~
ChuckMcM
Well if this is an attack to get at one of DNSSimple's customers, running your
own DNS would be a much easier target. Which is to say that if _you_ were the
target, you would already be hard dead by now rather than struggling as
DNSSimple deploys defenses.

I agree though that it is a pretty simple service to run for a small domain.

~~~
mike-cardwell
If you're being targetted directly, then _all_ of your services need to be
DDOS proof, not just DNS. The more third parties you add, the more likely you
are to be taken out by accident. If you have your own web server, you should
dump Bind or PowerDNS on it and write a zone file. Problem solved.

------
Cantdog
Can someone help me understand what happens to email sent to a domain hosted
by DNSimple while it's down?

I'm hoping it will get queued by the sending server, and make it's way back
when DNSimple is up and running. Is that correct?

~~~
abraham
It will depend on the configuration of the sending server. Some will retry for
a while, some will return failures.

~~~
Cantdog
Thanks!

------
zuccs
I moved from Zerigo to DNSimple, and it's been awesome until now!

What can you do to prevent this in future? Can you run multiple DNS providers
simultaneously? So, ns1/ns2 go to DNSimple, and ns3/ns4 go to another
provider?

~~~
robvolk
Yes - that should work. I'm about to make that change to bring back our site.
2 on one name server, 2 on another.

~~~
zuccs
Cool. Do you need a provider that supports 'zone transfers'? Or is that only
to keep things in sync _automatically_?

~~~
toomuchtodo
> Do you need a provider that supports 'zone transfers'?

No.

> Or is that only to keep things in sync automatically?

Yes.

~~~
zuccs
Do you know of any providers that will work nicely/automatically with
DNSimple?

~~~
toomuchtodo
I don't unfortunately.

------
aberoham
If you have an active DNSimple web UI session (or API key) you can change your
root nameservers by hitting their web tier directly at 50.31.213.210.

We've successfully switched our domains over to nsone.net.

~~~
jredburn
Even without an active session, I was able to get this working by adding an
entry to my /etc/hosts file with that IP and dnsimple.com.

~~~
tortilla
Thanks for the tip! Switched to NSONE.

------
ericskiff
For anyone else who needs to mitigate this in a hurry:

Set up a new account on another host that does ALIAS records (I used pointDNS)

Create your new record without much in it

Change your nameservers on your domain now - they'll take time to propagate

Fill in the records on your domain. If you can't remember them, print out most
of your existing records with

dig yourdomain.com ANY

Add the rest of the records to pointDNS

Wait for the new Nameservers to propagate (0-24 hours - it took 15-30 min for
us on a small-medium traffic domain today during sales crunch)

~~~
scott_karana
For those with a more deterministic bent: "propagation" time has a maximum
bound of your TTL, which will show with any dig queries.

~~~
colmmacc
Propagation is bound at the DNS TTL _plus_ whatever time it takes your DNS
provider/setup to relay records to all of its authoritative servers.

~~~
scott_karana
Hmm, never considered that. Is it a slow, static process for ISPs to do that?
I just assumed that they ran more-or-less stock DNS resolvers with in-memory
caches.

------
englishm
Here's where you can request your cached SERVFAILs be flushed from Google's
public DNS (i.e. 8.8.8.8): [https://developers.google.com/speed/public-
dns/cache](https://developers.google.com/speed/public-dns/cache)

~~~
bowyakka
Thank you, I wish I could buy you a beer for that

------
dedene
"30 minute ETA from our network provider to begin scrubbing traffic in a
location with capacity."

[https://twitter.com/dnsimplestatus/status/539551209452232705](https://twitter.com/dnsimplestatus/status/539551209452232705)

~~~
chrisbolt
It's surprising that they're appear to not be multihomed...

[http://bgp.he.net/AS32771](http://bgp.he.net/AS32771)

Unlike Dyn or CloudFlare:

[http://bgp.he.net/AS33517](http://bgp.he.net/AS33517)
[http://bgp.he.net/AS13335](http://bgp.he.net/AS13335)

~~~
feld
They're in ServerCentral's datacenter and ServerCentral is very much multi
homed. They wouldn't gain anything by doing native BGP to all these peers in
the exact same datacenter when SC's backbone will handle this stuff for them.

~~~
toomuchtodo
Doing native BGP would allow them to anycast, which would increase their
reliability and allow them to sink traffic much more easily. DDOS traffic sink
starts announcing your AS and anycast IP block close to the traffic source,
sinking that traffic and allowing real traffic through.

~~~
feld
I'm pretty sure they're only in one of Server Central's datacenters. Anycast
won't help. That's why I said they don't gain anything by directly peering.

------
kjttm
Does anyone have a simple explanation or link to an article / blog that
explains the naked domain / ALIAS "problem" that DNSSimple solves? I recently
set up DNS with DNSimple (due to nudging by Heroku) and am affected by this
DDoS. I am still struggling to understand the exact nature of this issue. All
of Heroku's documentation is pretty cryptic (to me):

"Some DNS hosts provide a way to get CNAME-like functionality at the zone apex
using a custom record type. " .. and then on to suggest DNSimple as their
first suggestion.

~~~
beevek
[http://blog.cloudflare.com/introducing-cname-flattening-
rfc-...](http://blog.cloudflare.com/introducing-cname-flattening-rfc-
compliant-cnames-at-a-domains-root/) is a reasonable explanation.
fundamentally a CNAME says "when you get queries for this name, go look at
this other name instead". among other things, doing a CNAME at the zone apex
means resolvers can't then find your NS, MX, or other records at the apex,
which is problematic.

------
scott_karana
For those wondering about alternatives to ALIAS: if you use a www subdomain,
then you can simply use CNAMEs. (Though the appearance is a matter of
taste...)

Google, Facebook, etc, all use this approach.

~~~
dbrgn
Yes, but if someone visits your apex domain directly, you want to redirect
him/her to the www version... And if you use cloud based hosting where the IP
can change quite often, that's a pain to maintain manually.

------
shoxxx
Anyone switching from DNSimple? I really don't want to, but we've been down
for almost 3 hours. I've seen chatter about Cloudfare and it looks pretty
good, reviews?

~~~
hglaser
We switched periscope.io from DNSimple to Amazon Route 53. DNSimple doesn't
have an exporter so it took about an hour, including having one engineer
review the other engineer's work.

Many customers were able to resolve the domain in the minutes immediately
following the switch, and the rest seem to be trickling in.

~~~
walmartian
Are you talking about a zone file exporter? DNSimple does have one, we just
used it to migrate to Route53. [http://support.dnsimple.com/articles/zone-
export/](http://support.dnsimple.com/articles/zone-export/)

EDIT: right, must have been able to log in during a brief period where
dnsimple was not down.

~~~
hglaser
That page doesn't load for me. :)

We went by this:

[https://twitter.com/dnsimple/status/539521794802483202](https://twitter.com/dnsimple/status/539521794802483202)

[https://twitter.com/dnsimple/status/539520808599957505](https://twitter.com/dnsimple/status/539520808599957505)

------
ataco
DNSimple is my registrar and (was my only) DNS provider. Now that they're back
up I've exported the zone file and imported it to route 53 for redundancy in
case this happens again. I also I updated the name servers in DNSimple to be 2
route 53, and 2 DNSimple, in that order. Is that the right way to do it? Does
the order of the NS records matter? I set them up so that they're in the same
order in both places.

~~~
anderly
That should be right. I'm doing the same thing with CloudFlare. However, it
appears that DNSimple won't keep your secondary name servers in the order
entered. They are showing for me sorted alphabetically. Apparently, they are
working on providing this ([http://blog.dnsimple.com/2014/12/incident-report-
ddos/](http://blog.dnsimple.com/2014/12/incident-report-ddos/)) for failover
in case of an event like this again.

------
brianarmstrong
I wrote a follow-up article about what we at Canopy.co learned from this
incident. Check it out (this covers and expands on some of the ideas talked
about here):

[https://medium.com/@brianarmstrong/youre-probably-doing-
dns-...](https://medium.com/@brianarmstrong/youre-probably-doing-dns-wrong-
like-we-were-6625efaed390)

------
soci
Unfortunately, it's not the first time it happens, my app is down and
customers unhappy.

I always wonder, why is it that someone wants to attack a small company like
DNSimple ? Is it that they were blackmailed and did not surrender to the
criminals? If so, why would anyone be interested in blackmailing such a small
company?

~~~
whafro
More likely, I'd guess based on past experience, it's that someone wanted to
take down one of their customers, and decided (or found) that the weakest link
was the DNS provider.

------
anderly
You can use my cross-platform cli for dnsimple to export your zone files
easily to txt or json format: [https://www.npmjs.org/package/dnsimple-
cli](https://www.npmjs.org/package/dnsimple-cli)

dnsimple domain record list example.com > example.txt

OR

dnsimple domain record list example.com --json > example.json

------
boopadoop
DNSimple says it was not a direct attack on them but rather domains being
brought over by new customers. Does anyone know the actual target?

------
stockkid
RubyGems.org and Travis-ci are down as a result of this! Not helping with my
productivity this morning.

------
beck5
What are the recommended practices to prevent too much down time when your DNS
provider goes down?

~~~
mjibson
My site is currently offline from this attack. I am considering that providers
like DNSimple simply cannot provide the networking availability to mitigate
these kinds of attacks. This is because the solution to these kinds of
attacks, often, is that you need a larger pipe than the attackers. Very few
people are good at both <some service> and having a huge pipe. For web apps,
you can use CloudFlare, which does have a bigger pipe and is designed to
mitigate this. But DNS is not a web app, so you can't just put DNSimple behind
CloudFlare.

Hence, I'm going to try CloudFlare (assuming they take over DNS hosting, I
need to check) and Google Cloud DNS, because then all parts of my site (from
DNS to CSS hosting) will be with providers with bigger pipes than attackers
can create. Hopefully that will prevent this kind of attack from taking my
site down.

~~~
throwaway90446
Yes, CloudFlare has a full-featured DNS offering, even if you don't use their
proxying services.

~~~
stevekemp
Which has had a lot of problems this past week:

[https://news.ycombinator.com/item?id=8665367](https://news.ycombinator.com/item?id=8665367)

~~~
throwaway90446
CloudFlare != CloudFront

~~~
stevekemp
My apologies, cloudfront had issues, cloudflare did not.

------
pkfrank
Can anyone expand on what this means: "This attack is volumetric in nature."
(?)

~~~
styger
According to [http://www.arbornetworks.com/attack-
ddos](http://www.arbornetworks.com/attack-ddos):

"DDoS attacks...will generally fall into one of three broad categories:

Volumetric Attacks: Attempt to consume the bandwidth either within the target
network/service, or between the target network/service and the rest of the
Internet. These attacks are simply about causing congestion."

