

A report on a DNS issue that was causing page redirections - amima
http://blog.qbaka.com/post/94537269389/a-report-on-a-dns-issue-that-was-causing-page

======
Decade
This is exactly the sort of issue that's supposed to be eliminated by DNSSEC
validation.

The registrar's database gets corrupted? They are not hosting my DS records,
anyway. That's in the top-level domain. You will be the least of anybody's
worries if the top-level domain dies. And as long as the DS record is live, a
validating nameserver can tell that the faulty DNS server is sending incorrect
resource records, and should be disregarded.

Side note: My DNS server is a hidden master, that publishes zones via
secondary servers, run by separate companies.

The corruption would still be disruptive, but it would not be such a disaster.

~~~
Panino
_This is exactly the sort of issue that 's supposed to be eliminated by DNSSEC
validation._

Emphasis on "supposed to be," because what DNSSEC brings to the table in
practice is headaches and outages. You won't see stories on HN about the IETF,
NASA or entire TLDs going down due to DNSSEC because it's normal, expected.

~~~
junto
I was thinking of trying to set up DNSSEC for my domains. Are you saying that
it isn't worth the hassle? What benefits / drawbacks does it bring?

~~~
Decade
It's yet another example of security technology that has too many adjustable
knobs, is poorly documented, and fails subtly.

The main benefit is that it makes it much harder to spoof DNS responses. Qbaka
just had an invalid zone published for their domain. The Great Firewall of
China has been leaking fraudulent DNS responses around the world. There's the
Kaminsky attack. This is a real-world problem. Also, some people hope that
such verified DNS records will allow them to sidestep the certificate
authority problem, but I think that's futile.

A side benefit I've been exploiting is that DNSSEC allows me to publish my
domains on free secondary servers, confident that the secondary is not able to
publish valid DNSSEC signatures on fraudulent records. I don't have a lot of
domains, though.

The main drawback is that it has so many moving parts that can break if
improperly used. Which signing algorithm and key size to use? Should you use
NSEC or NSEC3? How and when do you rotate the keys? (Default: no rotate.)
Which key rotation technique should you use? Where do you generate and store
your signing keys? If Microsoft can't consistently remember to renew their DNS
registrations, then how will they remember to update their DNSSEC keys?

Which brings me to a huge drawback: Now your DNS server and your DNS registrar
are part of your security infrastructure. Do you trust your registrar to hold
on to your DNS signing key? GoDaddy is happy to do so. If not, then you'll
have to maintain your zone separately. And even if you maintain your own zone,
both your registrar and the TLD administrator are free to publish an updated
DS record claiming that your DNSKEY is invalid and theirs is the true DNSKEY
for your domain.

Another drawback is that DNSSEC zones are huge. Dan Bernstein calls DNSSEC a
DDoS bandwidth amplifier, because adding DNSSEC signatures to DNS queries
makes the responses so much bigger. DNS servers should have response rate
limiting, anyway.

So, there are very important benefits, and very big drawbacks. The way I see
it, DNSSEC is horribly flawed, but it's the most widely deployed solution to
fix certain real-world security problems, so we're stuck with it. At least
it's designed to be backwards compatible and incrementally deployed, unlike
the IPv6 fiasco.

------
mcguire
"Any request to __.js file resulted in a valid javascript. This script was
loaded and executed instead of Qbaka tracking script. "

Who would do that?

"The TTL value of the corrupted DNS record was one week."

For a parking page? _Who_ would do that?

~~~
amima
I am almost sure that the reasoning for the redirect script was something like
this: since they have ads on that "parking" page, whoever tries to access the
server, should see the ads. That's probably why they have this terrible
redirection script in place. Which still keeps the original question "who
would do that", because this is terrible.

The TTL value has no direct connection to the parking page. It's probably by
the coincidence that DNS record had a high TTL and pointed to that IP address.
But no one can be sure here, because there is always a posibility that DNS was
hacked, although the DNS hosting provider sais that DNS error was not a result
of a hack.

------
spindritf
_we cannot call every single internet provider around the world and ask them
to drop DNS record from cache_

Google allows users to flush a record from their cache
[https://developers.google.com/speed/public-
dns/cache](https://developers.google.com/speed/public-dns/cache) which is
better than nothing, especially if you use Google's resolvers on your servers.

~~~
amima
The Google DNS was not affected, or more likely it was quickly flushed (I
guess, the guys who made this error were the first to flush Google DNS).

------
kelnos
One semi-solution to that (which would have made the problem almost invisible
to any webpage visitors) would be to only serve the script over HTTPS.
Presumably the domain parking page was not using a cert that matched
"cdn.qbaka.net", so a browser would fail to download anything due to a cert
mismatch.

~~~
hjlklhj
That was my original thought as well but since cdn.qbaka.net was just a cname
to their real cdn it wouldn't have worked. Unless they gave their private ssl
key to the cdn provider, which is really not recommended.

------
oasisbob
Anyone know who the DNS provider was? This seems like a perfectly appropriate
time to name and shame.

~~~
amima
No point in shaming, anyone can experience this kind of problem. Even Google
once marked all websites as "contains virus", and Amazon cloud was unavailable
due to a storm. It was not the well known domain name registrar's DNS.

------
PeterWhittaker
Thanks for the detailed report, good lessons learned herein.

