

Incident Report – DDoS Attack - alainmeier
http://blog.dnsimple.com/2014/12/incident-report-ddos/

======
latch
I need to learn to let things go, but:
[https://news.ycombinator.com/item?id=4280515](https://news.ycombinator.com/item?id=4280515)

I've been a DnsMadeEasy customer for a while (they had an outage ~4 years ago
from a 50Gbps attack), but once my year is up, I'm switching to Route53. The
addition of the Geo DNS Queries was key for me. It isn't clear to me why I
shouldn't pick Route53. DnsSimple's unlimited queries seems nice, but I kinda
like having actual scaling costs forwarded to customers.

~~~
kyledrake
I've had a similar thought RE using Route53 for Neocities. Here's the problem
with Route53 though. If you get a DDoS attack using it, it's quite plausible
that you would be charged for resources used in the DDoS attack. A recent Vice
article discussed this: [http://motherboard.vice.com/read/inside-the-unending-
cyber-s...](http://motherboard.vice.com/read/inside-the-unending-cyber-siege-
of-hong-kong)

DDoS is a nasty problem. We've received a DDoS attack that shut the entire
site down for days. We can't use Cloudflare because they don't support
wildcard domains without their very expensive plan. I've also heard stories
from people using Cloudflare that have still not been able to resolve DDoS
issues (I'm not knocking Cloudflare, they're a great company that does a
really good job fighting this very hard problem, but sometimes even they have
trouble with it).

I'll be completely honest and say that I have no idea how to solve this
problem. It's really, really, really hard. Switching to different service
providers won't get you very far against the monster DDoS attacks that some
people can execute.

~~~
stevekemp
If you're going to go the Amazon route then you absolutely need to keep an eye
on billing, and set up alerts so that any DDoS which caused a spike in your
costs would be caught as soon as possible.

~~~
Intermernet
I was burnt by this in the first 48 hours of using Amazon DNS. Very unlucky I
guess... I'm amazed they still bill for DDOS traffic, or even traffic from
black-listed IPs. It seems many of their competitors don't.

------
kator
> A new customer signed up for our service and brought in multiple domains
> that were already facing a DDoS attack. The customer had already tried at
> least 2 other providers before DNSimple. Once the domains were delegated to
> us, we began receiving the traffic from the DDoS.

I'm curious did they know this in advance or discovered it after the fact?

I often wonder about business models where the core expense is "unlimited and
free". The reality is there is nothing unlimited or free for the service
provider. It seems with a business model like this you open yourself to people
abusing your service either by accident or by choice. Imagine poor Mr.
Customer here who most likely was having horrible problems thinking to
themselves "These guys can do it and for free, if I go to X service they'll
cost me a lot of money".

I'm a big believer in business models that incentivize both parties properly.
I'm sure in general this service provider is arbitraging the 99.9% of domains
that barely need any services. That said it only takes a couple of "opps"
customers to drive your operational costs through the roof.

~~~
aeden
Anthony from DNSimple here. We discovered it after the fact, via a tip from
other DNS providers.

~~~
rpug
As someone who has been down this road many times before - I can't stress this
enough: DDoS mitigation solutions don't solve the problem of an app-specific
layer7 attack and it is important to do some testing of how well your
mitigation service responds (and that it isn't a silver bullet.) Additionally,
you need to make sure your team has tested and proven procedures for engaging
the service, respond to attacks, etc. Services like NimbusDDoS
(www.nimbusddos.com) are good because you can do some real scenario testing
and make sure your team and infrastructure is prepared. There are other
services out there too that I am less familiar with, but either way really
good stuff to do.

------
stephenr
The solution here is one for customers, not providers.

Manage your DNS at one location on "master" (potentially a "private" server
with IP restricted access and zone transfer ACLs).

Setup 2+ accounts with "DNS providers" that support incoming zone transfers -
that is, they can operate as "slave" DNS servers, pulling records
automatically from your "master" (once access rules are set of course) and
returning results directly to clients making DNS queries.

Most "Secondary DNS" packages are < $50 year, so use a few, and don't worry
about individual DNS networks being burnt to the ground.

~~~
jhealy
It seems like inbound and outbound zone transfers aren't offered by a number
of providers (like AWS). Do you know of a list of DNS providers that support
either option?

~~~
mike-cardwell
I used to use these two services together do this:

    
    
      https://puck.nether.net/dns
      https://acc.rollernet.us/
    

They're both free to sign up, provide free secondary DNS, zone transfers and
fully support IPv6.

I only stopped using them because I wanted to run my own DNS service.

------
abalone
So who do you think the "well-known third-party service that provides external
DDoS protection using reverse DNS proxies" is they're going to use now?

CloudFlare?

~~~
crystaln
Hopefully not. CloudFlare is remarkably unreliable for a service that claims
to improve uptime.

~~~
ad_hominem
[citation needed]

Last I checked CloudFlare routinely handles[1] 10Gbps to 65Gbps attacks, and
has successfully handled attacks as large as 300Gbps and 400Gbps. According to
this report DNSSimple crumbled under 25Gbps.

[1]: [https://support.cloudflare.com/hc/en-
us/articles/200170216-H...](https://support.cloudflare.com/hc/en-
us/articles/200170216-How-large-of-a-DDoS-attack-can-CloudFlare-handle-)

~~~
etcet
Their last significant outage was only 2 months ago:
[https://blog.cloudflare.com/route-leak-incident-on-
october-2...](https://blog.cloudflare.com/route-leak-incident-on-
october-2-2014/)

~~~
xxdesmus
As the blog post outlines, the outage was related to an upstream network
provider leaking routes. Note exactly something we can prevent for them.

------
cm2187
Out of curiosity, what are the follow ups of an attack like that? The
perpetrators are probably using their own servers or compromised clients or
servers. Would DNS Simple follow up on this with the abuse/complaint dept of
the ISP of the attackers? Are ISP typically responsive to abuse and
complaints? If they are not is there any way to black list blocks of IPs
assigned to ISP who do not care about being the source of DDoS attacks?

Investing in anti DDoS devices is important but even more important is for the
perpetrators to face the consequences of their acts (or anyone who lets his
machine being used by pirates - terminating or suspending their contract would
be a fair response).

~~~
iancarroll
I was looking at [http://map.ipviking.com](http://map.ipviking.com) earlier
and it was apparent it was a botnet, most likely innocent home users with a
virus.

~~~
brownbat
It'd be nice if IPs involved in botnet DDoS's could go into a public registry,
then get a banner from Google saying, "Hey, you might have a virus, someone
reported you to this list."

Abuse would be tricky, you might be able to limit it by letting only a few
DDoS mitigation providers populate the list.

~~~
Xylakant
A lot of ISPs for example in Germany reuse IP addresses and force a reconnect
every 24 hours. I don't think showing me banners because the previous "owner"
of the IP had a virus is going to improve the situation.

Other people share a network behind a NATed IP which is also a problem. They'd
all receive a banner, check their computer and a test would come up negative.

~~~
cm2187
Google wouldn't know but the ISP would know who was behind a particular IP at
a specific time. They are the ones who should police their network when there
are abuses.

~~~
Xylakant
The original proposal was that google delivers the ads. So google would have
to contact my ISP who would then have to return whether or not I was using any
of the given "spammy" IPs at the time that they were spammy - or my ISP would
have to deliver the banner.

No thanks.

------
milos_cohagen
What was the overall makeup of the attack traffic? For example, 50% tcp syn,
etc.

