
Linode-hosted DNS zones were down - mwpmaybe
https://status.linode.com/incidents/ly9hx0plrzxn
======
mkorsak
I can confirm this was not a DDoS, and we are actively working to fix the
issue right now. We'll be updating our status page as soon as we have more
updates to deliver.

~~~
Mahn
Do you have any ETA? Sorry, I don't mean to be an ass, it's just our entire
business is currently down because of this.

~~~
mkorsak
I understand, and I don't have an ETA quite yet, but it appears that service
is already starting to return. I've tested my domains hosted on Linode and a
number of others that I know are and they appear to be working now. We're
still working asap to ensure everything is coming back appropriately.

~~~
Mahn
Everything seems to be back now in our case. Thanks!

------
mwpmaybe
My monitors started going berserk around 3:36 PM CT. Here's an example of the
problem (I am not hostmaster for The Onion, but they are a well-known Linode
customer):

    
    
        $ whois theonion.com | grep -i 'name server'
           Name Server: NS1.LINODE.COM
           Name Server: NS2.LINODE.COM
           Name Server: NS3.LINODE.COM
           Name Server: NS4.LINODE.COM
           Name Server: NS5.LINODE.COM
        $ dig @ns1.linode.com theonion.com ns +short
        $ dig @ns1.linode.com theonion.com a +short

~~~
cft
I prefer DJBDNS tools:

    
    
        #dnsq a theonion.com NS1.LINODE.COM
         1 theonion.com:
         temporary failure

------
methodover
This is one reason why I'm moving our startup over to AWS. :/ They seem to
experience less of these catastrophic DDoS attacks.

Edit: Err, not a DDoS attack apparently. But catastrophic nonetheless.

~~~
josephb
Route53 also:

A) Serves DNS from multiple domain names. A number of variants in .com .org
and others etc

B) Serves DNS from multiple internet networks / sources. Queries from an end
user don't all hit the same Anycast subnet.

~~~
kyledrake
Route 53 also has no IPv6 support, and no stated policy on how it deals with
DDoS attacks, or if you get charged for the attack if it's directed at you.
Good luck!

~~~
jtrtoo
It might be more accurate to say R53 has partial v6 support. It'll return v6
records (AAAA) happily. The nameservers themselves do not respond on v6 (which
is unfortunate and certainly means v6 support is lacking). Practically,
however, it's pretty unlikely, that a v6 only host isn't going to also have
access to a v6-to-v4 gateway. Not ideal, but generally transparent for most
intents and purposes ...and generally used by just about anyone with a v6
address assignment today, everyday.

For DDoS attacks, I'm not sure what you mean by a "stated policy." They do a
lot of countermeasures, some of which can be found via linked PDFs at
[https://aws.amazon.com/security/](https://aws.amazon.com/security/) Various
R53 items are included in their DDoS document there. Officially you get
charged. Unofficially, it depends on the circumstances. There's also a world
of difference between a DDoS attack against a fully managed offering (like
R53) versus, say, an EC2 instance.

Have you seen a better approach taken by other providers? I don't mean
technical approach (which is near impossible to compare other than by track
record), but from a policy perspective?

------
brajkovic
At one point, this said it was due to a Denial of Service attack. That text
has since been pulled, unfortunately I didn't get a screenshot.

~~~
slester
The email I received:

DNS Performance Issue Incident Report for Linode Investigating We are
currently experiencing denial of service attacks that are targeting our DNS
infrastructure. Aug 23, 20:48 UTC This incident affects: Hosted DNS Service.

------
mindslight
I recently migrated away from using Linode as secondary when they pushed
everyone over to Cloudflare. That Internet-destroying company already has
enough power!

~~~
viraptor
Why do you think Cloudflare is internet-destroying? Serious question.

~~~
curried_haskell
They are growing into a very dangerous position of acting as a gatekeeper to
the Internet. Try accessing any major website through Tor.

~~~
angry-hacker
And try having a webserver being constantly attacked by TOR.

They are not doing it for fun. If you think they are bad, give them
suggestions or new tech how to solve this problem.

I thank cloudflare for making my life easier.

------
therealmarv
Is not it possible to outsource that critical part? What are good and proofed
dns services out there?

~~~
josephb
They do outsource to Cloudflare.

One assumes some behind the scenes infrastructure exists to push and sync
updates to Cloudflare.

~~~
ehPReth
They probably use CloudFlare's DNS proxy service:
[https://www.cloudflare.com/virtual-dns/](https://www.cloudflare.com/virtual-
dns/)

------
gr2020
This was just the kick in the pants I needed to set up secondary DNS at Route
53!

------
FreeKill
Seems like it's coming back up! My resources are reachable again.

------
dino2k
This smells like a major fuckup

