

Zerigo DNS services down for 6+ hours due to massive DDoS - _phred
http://zerigostatus.com/

======
cbsmith
I can totally buy DDoS flooding network capacity, but I'm befuddled these days
by statements saying the servers are "under load", which typically means "out
of CPU". It's kind of hard for me to imagine even an i5 not being able to
saturate a gigE line with DNS lookups (yes, it is a lot of packets, but it can
be done) unless DNSSec is going on. Even 10gigE, _if_ you can amortize
interrupts, seems like it'd not be hard to saturate with today's hardware.

What am I missing here?

~~~
gabriel-samfia
There are many types of DDoS. Some max out your CPU, some your network. Given
that a DDoS (Distributed Denial of Service) involves potentially thousands of
willing or unwilling systems, it's relatively easy to make a server
unresponsive.

I have a 100 Mb/s internet connection. Scale that up to 10000, and you have
saturated even the fastest of internet connections.

Mitigating a DDoS is not easy. Heck, its damn near impossible, considering the
fact that DNS DDoS attacks are done via UDP, which allow you to spoof the
source IP address. Even if you do block the IP address of al the attackers,
your upstream provider is still impacted by the packets trying to come into
your server. Most upstream ISPs will blackhole your server IP to diminish the
impact on their network.

------
aeden
I run DNSimple (<https://dnsimple.com>) and we have a full REST API and
support domain registrations, transfers and SSL certificates as well. Plus we
have an ALIAS record type that's very useful for pointing your apex to
services where they only provide a hostname.

I'll be happy to answer any questions you have regarding our service either
here or through our support channels.

~~~
latch
I know there's much love for DNSimple, but this is the first time that I can
remember when the top comment of an _X is down_ post is a competitor
essentially posting an advert with no insight on the OP.

~~~
aeden
I tried to add some insight on another comment, but it's tough to say anything
about DDoS that hasn't already been said. DDoS attacks suck, mitigating them
requires a multi-prong approach and proactive monitoring and agressive banning
and even then you can still be screwed if your bandwidth is saturated.

I feel for the operational folks at Zerigo - dealing with this type of outage
is hard. The best thing they can do at this point is get back on Twitter and
talk to their customers - the last post was 4 hours ago - that's a lifetime
when your system is critical.

~~~
genek
How much capacity does DNSimple have though. It appears as though you are
another unicast network. ns1.dnsimple.com is a server at Slicehost / Rackspace
ns2.dnsimple.com is a server at Linode ns3.dnsimple.com is a server at
prgmr.com / EGIHosting / Hurricane Electric ns4.dnsimple.com is a EC2 instance
on Amazon

How much computing power and attack traffic can those really handle?

If you are going to offer a solution to a massive DDoS I would think that you
would be careful on when to propose your solution.

Instead of adding another unicast network to the mix, why wouldn't you start
using an IP anycast network?

Please explain how much capacity you have.

------
_phred
Going on 8 hours of Zerigo's downtime I've had to move all of our Zerigo DNS
to DNSMadeEasy. It's a shame, because I really, really like Zerigo, especially
their API.

Shit happens, but 99.9% (8 hours a year of downtime) is completely
unacceptable for a DNS provider.

------
sstarr
Add these to your hosts file to access your account:

64.27.57.25 manage.zerigo.com

64.27.57.8 dns.zerigo.com

Source: <https://twitter.com/coldclimate/status/227369346891132928>

~~~
PonyGumbo
Thank you so much!

------
latch
Seems like if you are serious about mitigating this type of issue (as a
consumer), you really should be specifying name servers from different
providers. Your primary DNS server can be from dnsimple/zerigo/dnsmadeeasy and
your secondary can be route53, or you could run your own.

The only problem seems to be keeping them in sync. Seems like you'd have to
poll the primary (using whatever API it exposes) to update the secondary.

Mostly thinking out loud, surely someone more experienced could provide better
guidance?

~~~
aeden
Ideally your primary provider would support AXFR and NOTIFY which are part of
the DNS zone transfer protocol. It's something we're working on adding to
DNSimple, but we're not quite ready to launch it yet. The primary and
secondary providers also both need to report the correct authoritative name
server delegation details so the primary needs to ensure that that data is in
the zone file.

There is another challenge in that we're pushing the envelope a bit by
offering features that rely on more than just a DNS record (for example ALIAS
and POOL records). These are useful features for some people, but if you're
using these types of features then they won't be portable to secondary
providers.

------
jbarham
I run a DNS hosting service (SlickDNS, www.slickdns.com) and have seen a spike
in signups today as a direct result of the Zerigo DDOS attack.

I can't claim that SlickDNS is invulnerable to DDOS attack, but FWIW it does
run tinydns name servers which have good performance and excellent security.
So if you're impacted by the Zerigo outage, feel free to check out SlickDNS.
There's a 30-day free trial with all plans and record updates are pushed
through to all the name servers in under 5 seconds.

~~~
aeden
As you're probably aware the server that you use has little impact when the
DDoS sends enough traffic to actually saturate your allocated bandwidth.
Anycast provides a good way to handle DDoS, along with proactive monitoring
and defense mechanisms, but at the end of the day DDoS are still extremely
difficult to defend from completely. The downside is that Anycast is expensive
and thus you need the capital to build it out and run it - which often raises
the cost of systems built using it.

------
_phred
Apparently no ETA for restore as of 2 hours ago:
<https://twitter.com/zerigo/status/227322909230768128>

------
gaia
Best thing Zerigo could do for their customers at this point is export all
zone information and email it to them or make available for DL. I have a
feeling this is going to be a long outage. In the meanwhile, here is a great
list of free DNS providers (dont get caught without a secondary DNS provider):
<http://www.lowendtalk.com/wiki/free-dns-providers>

------
metalruler
I've been seeing a lot of reflector attacks in the past couple of weeks, where
the attacker sends a relatively small query for a valid domain that will
return a large reply. The trick is that they spoof the source IP, so the DNS
reply goes to the victim.

I ended up hacking something together to firewall any IPs which sent more than
1000 requests in a short period of time.

~~~
dedene
Do you mind sharing the script / code to accomplish that? (some gist
somewhere) I'm seeing a lot of these sort of things on our servers too..

~~~
metalruler
It really is a disgusting hack, and specific to FreeBSD. It does need to be a
bit more sophisticated than "block an IP if it floods me" because as it is now
someone can simply spoof the IP of an ISP's DNS server and effectively
firewall them, blocking their users from being able to resolve the domain
names I'm hosting.

I can give you one tip to get you started: if you're running named, you can
enable logging of every query, something like (hope this formats ok) :

    
    
      logging {
        channel query_logging {
             file "/var/log/named/querylog"
             versions 3 size 100M;
             print-time yes;                 // timestamp log entries
          };
    
          category queries {
              query_logging;
          };

};

------
AdamGibbins
And this is why I use Route 53, I'm a lot more confident in Amazon's abilities
to mitigate DDoS attacks.

Which really sucks, DDoS are really hard to combat and Zerigo are an awesome
company.

------
manveru
Well, that explains why my wife woke me up complaining about half the internet
not working. Our ISP is 3 (drei.at) and she was using their DNS, guess there
are issues all over Europe.

------
slig
What are the main advantages of paying for DNS hosting like Zerigo or SlickDNS
instead of using the one provided for free with web host companies (E.g.
Linode's DNS Manager)?

~~~
jbarham
FWIW the SlickDNS name servers are hosted by Linode so I'm a fan of their
server hosting. For DNS management, the Linode interface is fine if you have a
handful of domains with simple configurations, but beyond that it's unwieldy
IMHO.

I'd say the main reason to use a DNS hosting service is to consolidate your
DNS management for all of your domains regardless of registrars or server
hosting providers. E.g., I personally have domains registered with 5
registrars and use two server providers. And because they specialize in DNS,
DNS hosting providers should have superior interfaces, APIs and support for
DNS hosting compared to generalist hosting providers.

The SlickDNS interface has two features in particular that I haven't seen in
any other DNS hosting service: automatic management of "alias domains" and
mapping IP addresses to named servers. See
<https://www.slickdns.com/features/> for details.

------
sleighboy
US-Based customer here. Our DNS just started working again.

------
Uchikoma
Running with DNSMadeEasy, is there a way to integrate it with Route 53 through
AXFR to have two providers?

~~~
aeden
This might help: <http://route53d.googlecode.com/svn-history/r2/trunk/README>

Looks like they are close to getting NOTIFY and IXFR (incremental AXFR)
working. It's an interesting approach none-the-less.

~~~
Uchikoma
Could I integrate DNSimple with DNSMadeEasy via NOTIFY/IXFR/AXFR?

~~~
aeden
Not right now, we don't operate as a secondary provider (and I'm not sure we
will).

Take a look at DynDNS's secondary service, that might work for you:
<http://dyn.com/dns/secondary-dns/>

~~~
Uchikoma
Thx!

------
piggity
Days later and what do we have from them? One solitary email and a few half-
assed status page updates.

------
St-Clock
This took down services like Fogbugz on demand.

~~~
kevingessner
We've switched our DNS provider and we're waiting for it to propagate. Check
<http://fogcreekstatus.typepad.com/> for updates.

------
silverlight
Looks like this took Trello down, too...

~~~
kevingessner
We've switched our DNS provider and we're waiting for the change to propagate.
<http://fogcreekstatus.typepad.com/2012/07/index.html> has all the details.

------
PonyGumbo
Comodo's DNS.com appears to be down too.

