

Network Solutions' DNS was down - mikegirouard
http://www.networksolutions.com

======
TranceMan
Their facebook page has been updated:

Network Solutions is experiencing a Distributed Denial of Service (DDOS)
attack that is impacting our customers as well as the Network Solutions site.
Our technology team is working to mitigate the situation. Please check back
for updates.

[https://www.facebook.com/networksolutions/posts/101514681058...](https://www.facebook.com/networksolutions/posts/10151468105866020)

------
pilif
This is one of the reasons why I self-host DNS. Even with tons of users, the
resources it takes to serve DNS requests pale compared to what you have to put
behind your application servers.

Of course if you are using a CDN to provide your users with better locality,
you might want to look into a service that provides localized DNS
distribution, but the inherent caching feature of the DNS protocol might make
that an unneeded additional burden (I remember a very old Stack Exchange
podcast episode where they were talking about this, coming to the conclusion
that self-hosted DNS and dns-internal caching is good enough for them).

When you self-host your DNS, you ensure that no provider can suddenly redirect
to an ad-filled parking pages if users mistype hostnames and you can make sure
that you can fix DNS when it's down. It's also much easier to transfer domains
between registrars because there's no need to export and import DNS config -
instead, your server just keeps serving the exact same data.

Finally, this allows you to keep the DNS config together with all other
configuration files in puppet/chef/git/whatever you use. which further helps
future deployments and/or configuration changes.

~~~
dholowiski
I thought that even if you self host - isn't it your registrar that points the
dns records to your dns servers? Would someone with self-hosted DNS still be
working in this situation? (doesn't the DNS query first go to network
solutions, who refers it to your name servers?).

I'm not trying to argue, I genuinely need to know the answer as I'll have to
explain it to many other people, later today...

~~~
jbert
No, they just go into the zone.

You run a server for example.net. Your server will respond to any requests
sent to it for records in that zone.

Someone attempting to reach your site will query the parent zone (.com) for NS
records for your zone. So they will query for NS records for example.com
against a server which is authoratitive for .com.

(How do they find .com server? They ask the root servers. How do they find the
root servers? That's the bootstrap info.)

------
simonsarris
Honest question from a networking beginner:

So suppose right now I've got two name servers configured, NS93.worldnic.com
and NS94.worldnic.com. These are down as they're the part of the Network
Solution's name servers that are having issues.

If I had added more, for instance if I used Amazon's Route53 and added two
name servers of theirs _in addition_ to the *.worldnic.com ones, would my site
be reachable right now?

~~~
area51org
Yes, but there may be delay, because the nameservers are tried in random
order, and so each server that is down will have to time out before users move
on to the next.

~~~
colmmacc
(I work on Route 53).

We've done a lot of experiments on this one and we've found that the most
common resolvers make 3 tries to 3 different servers by default. At first
those servers are picked at random, but over time the resolvers usually "home
in" on what the least-latent nameserver is. Once they do, and have a good
round-trip-time estimate for how long it takes to respond, they stay using it.
But they have a hair-trigger; if the nameserver doesn't respond, they'll very
quickly fall-back to trying the other nameservers.

In practice what that means is that up to two of your nameservers may be
completely unresponsive and the effects will be pretty negligible. So if
you're using multiple DNS providers and want to protect against one going off-
air; use no more than two nameservers from each provider.

That said, for all of the reasons above, and some more, the Route 53 SLA
currently only applies if you use all four Route 53 nameservers.

~~~
mmmooo
Any thoughts on where/how is this 'home in' state is saved? Other the cache
and filtering/acl state resolvers should be stateless, so would be interested
in any details, especially if this is on a per-zone, or per-set of nameservers
or etc basis. I've never seen bind9 act like this.

~~~
colmmacc
I haven't read the bind source in a long time, but my best guess is that it's
in memory.

[http://docstore.mik.ua/orelly/networking_2ndEd/dns/ch02_06.h...](http://docstore.mik.ua/orelly/networking_2ndEd/dns/ch02_06.htm)

Has some details, and I believe Bind and PowerDNS call it "Smoothed RTT" or
SRTT.

~~~
mmmooo
interesting, not a ton of details of how. off to the source I go, thanks.
sounds like it assigns a 'score' of some sort to each authoritative ns.

~~~
isb
This might help - section 4.4 has formal details on the Bind server selection
algorithms: [http://ocw.mit.edu/courses/mathematics/18-996-topics-in-
theo...](http://ocw.mit.edu/courses/mathematics/18-996-topics-in-theoretical-
computer-science-internet-research-problems-spring-2002/lecture-notes/dns.pdf)

~~~
mmmooo
yep definitely, an interesting implementation, and certainly almost impossible
to guess from passive observation. thanks.

------
mikegirouard
This[0] FB post is the best I can come up with as an explanation.

> Yesterday, some Network Solutions customer sites were compromised.

The funny thing is, they have a link in their post going back to their site,
which of course doesn't work.

[0]:
[https://www.facebook.com/networksolutions/posts/101514668014...](https://www.facebook.com/networksolutions/posts/10151466801446020)

~~~
muraiki
That explains my friend's site yesterday. Requests to his site sometimes
resulted in the display of a banner page by "Islamic Ghosts Team," but not all
the time. I noticed that Network Solutions was apparently running Apache
2.2.22, which has a few security flaws (I'm pretty sure he doesn't use a VPS).

------
Tenhundfeld
TechZone360 (ugh, so many ads) article with a little more information:

Network Solutions Experiences Hijacking of DNS Records
[http://www.techzone360.com/topics/techzone/articles/2013/07/...](http://www.techzone360.com/topics/techzone/articles/2013/07/17/345902-network-
solutions-experiences-hijacking-dns-records.htm)

~~~
muraiki
A comment in the article pointed out that this one seems very similar to a
Cisco article from a month ago: [http://blogs.cisco.com/security/hijacking-of-
dns-records-fro...](http://blogs.cisco.com/security/hijacking-of-dns-records-
from-network-solutions/)

------
supine
If your name servers are *.worldnic.com it appears you are affected.

First mention on Twitter was an hour ago
[https://twitter.com/thefrost/status/357483942082920449](https://twitter.com/thefrost/status/357483942082920449)

------
nodata
Site down? Don't link to it. Bad netiquette. Jesus christ.

------
billsinc
Seeing this as well, appears to be DNS only at this point.

~~~
mikegirouard
You're right. Title updated.

------
asr2bd
Our site has been sporadically up and down for the past few hours. Glad I know
the culprit finally. Hope this gets resolved soon

------
dholowiski
Oh crap. Today is going to be a bad day for me.

~~~
jryce
If the mail server goes down again it'll be a bad day for us too.

~~~
dholowiski
We run our own mail server, but the MX records come from network solutions :(

~~~
dholowiski
DNS for our domains seems to be back up now, for now at least.

In case anyone is interested, looking at our google analytics it appears this
started sometime around 6AM MST, and got 'fixed' around 8:45AM MST. Of course
it's impossible to say exactly when, because of DNS TTL & caching.

------
theg2
We're slowly coming back online, there are reports of a (D)DOS attack against
them but I can't find a source.

~~~
TranceMan
Some talk here [1] of a recorded message from their support number that they
are currently experiencing a DOS attack.

1\.
[http://www.webhostingtalk.com/showthread.php?t=1285835](http://www.webhostingtalk.com/showthread.php?t=1285835)

------
esmale
I've been in panic mode trying to figure out why our applications have gone
down. At least now I know why.

~~~
mikegirouard
Ditto. I just spent an hour hacking at my firewall rules before I tried the
obvious thing: going to the server's IP.

It doesn't matter how long I do this... DNS problems always get me.

~~~
dedward
Have you thought about setting up with an external DNS monitoring service?
It's cheap and really comes in handy over the years.

------
dangayle
Just what I wanted to wake up to. Client emails asking me why their sites were
down. Sigh.

------
SubZero
Why did I click on the link to a down website? What did I expect? Shame on me.

------
dholowiski
This appears to be a problem again this afternoon (1:44PM MST)

------
jryce
Their mail server appears to be online.

------
ottoflux
Back up.

