
HBO Now DNSSEC Misconfiguration Makes Site Unavailable from Comcast Networks - danyork
http://www.internetsociety.org/deploy360/blog/2015/03/hbo-now-dnssec-misconfiguration-makes-site-unavailable-from-comcast-networks-fixed-now/
======
tptacek
This shouldn't be possible. In a sane design, a transient policy failure
should produce a warning light or a browser popup; _something_ that
applications can use to sensibly convey what happened, in some form, to users.

Instead, because DNSSEC makes security policy decisions at a very low layer,
in a protocol that virtually all applications expect to work transparently so
long as you have connectivity, you get this: instead of a browser popup, a
connectivity outage so complete that it is indistinguishable to users from
"the site you are trying to get to doesn't exist and never has".

DNSSEC is a terrible, terrible idea. Comcast should stop validating it,
immediately. If someone from Comcast's network engineering team is reading
this: I was a network engineer, and then worked for 4 years with tier-1
engineering getting Netflow monitoring deployed and scalable. I've been
writing DNS security software since I was 19 years old, in the 1990s. You have
reasons for whatever configuration you've decided on, but I will take as much
time out of my day as is productive for you to convince you to stop working on
rolling this out.

What happened yesterday with HBO Now, which had half of Twitter screaming "NET
NEUTRALITY OHNOZ!", is going to keep happening over and over again.

See also:
[https://news.ycombinator.com/item?id=8894902](https://news.ycombinator.com/item?id=8894902)

~~~
Someone1234
While I agree DNSSEC has its limitations and problems, I haven't heard many
alternatives (in general).

DNS has two inherent issues:

\- Hijacking traffic via spoofing responses.

\- Private information leakage.

With DNSSEC combined with HTTPS it is very reasonable to say that both you're
connecting to the party you're expecting to connect to and in addition, no
third party listening on the line either knows what host name or page you're
visiting (only the IP address of the box).

A lot of people dismiss these limitations or just point wildly at certificate
pinning to solve all of our problems (while ignoring that HTTP isn't the only
type of traffic the internet was designed for).

Plus with NSA mass surveillance, DNS makes seeing what domains you're visiting
and building a "picture" of you as an individual absolutely trivial. The IP
addresses still may help them do that to some extent, but it certainly becomes
very easy if they can see your DNS packets.

~~~
secalex
> I haven't heard many alternatives (in general).

These problems don't call for a general solution. We should respect the
fundamental truth behind the end-to-end principle and solve these problems as
close to the application as possible. The likely threats, appropriate default
choices and trust model are all best understood by each application developer
and building these solutions into Layer-3/4 is bound to cause short-term pain
and reduce our long-term flexibility to meet new risks.

For example, EFF's STARTTLS Everywhere project takes into account the fact
that a huge percentage of the world's mail moves between a small enough number
of providers that human verification of announcements is possible. It also
recognizes that the MX configuration for these providers is reasonably static,
meaning that changes that propagate in minutes do not need to be accommodated.
Small specific solutions like this can be rolled out and provide a real,
tangible benefit to users much faster than we can upgrade the entire DNS
system to provide a more general solution.

I agree that we need a DNS privacy solution, one that hopefully doesn't
eliminate the existence of caching infrastructure.

~~~
danyork
> _I agree that we need a DNS privacy solution, one that hopefully doesn 't
> eliminate the existence of caching infrastructure._

Please do take a look at what the folks are doing within the DPRIVE working
group of the IETF:

[https://datatracker.ietf.org/wg/dprive/charter/](https://datatracker.ietf.org/wg/dprive/charter/)

They are working on mechanisms to bring privacy / confidentiality to the "last
mile" of DNS connections. Any input you have would be useful. (There's a link
there to a mailing list to which you can subscribe.)

------
st3fan
"As a result, the many networks around the world that perform DNSSEC
validation to ensure that customers are getting to the correct sites (versus
being redirected to bogus sites for phishing or malware) were blocking
customers from getting to the possibly bogus order.hbonow.com!"

The "versus being redirected to bogus sites for phishing or malware" part here
is funny. Because that generally happens when some scammer registeres
hb0now.com. Not when someone is intercepting your DNS.

And actually, in case of the latter, all bets are off anyways. Because client
resolvers DO NOT CHECK DNSSEC.

(Yes, I know _you_ are on your custom configured Linux box or OpenBSD firewall
and that DNSSEC works wonderful for you. But the majority of the internet
using world with OS X or Windows behind a $15 DSL router or WiFi AP does not.)

Stop. The. DNSSEC. Nonsense.

~~~
danyork
And actually...

> _And actually, in case of the latter, all bets are off anyways. Because
> client resolvers DO NOT CHECK DNSSEC._

... please visit APNIC's DNSSEC Statistics site where you will see that about
12% of all DNS queries globally ARE being validated by DNSSEC:

[http://stats.labs.apnic.net/dnssec/XA?c=XA&x=1&g=1&r=1&w=7&g...](http://stats.labs.apnic.net/dnssec/XA?c=XA&x=1&g=1&r=1&w=7&g=0)

In Sweden this is about 71%:

[http://stats.labs.apnic.net/dnssec/QM?o=cXAw7x1g1r1](http://stats.labs.apnic.net/dnssec/QM?o=cXAw7x1g1r1)

Slovenia 67%, Estonia 55%, Denmark 48% ... on down to the USA at 23%:

[http://stats.labs.apnic.net/dnssec/XQ?o=cXAw7x1g1r1](http://stats.labs.apnic.net/dnssec/XQ?o=cXAw7x1g1r1)

So DNSSEC validation very definitely * _IS_ * happening out there!

~~~
icebraining
Checks being made doesn't mean the _clients_ are checking. What's the point of
validating the communication between the servers if the last mile is
unprotected?

~~~
danyork
> _What 's the point of validating the communication between the servers if
> the last mile is unprotected?_

It's all incremental ways of reducing the attack surface. Get DNSSEC
validation happening at large public DNS services... then at ISPs... then on
network edge devices... then into operating system (in stub resolvers) ...
then perhaps into applications themselves.

Each step reduces the attack surface a bit more. I wrote about this at:

[http://www.internetsociety.org/deploy360/resources/plan-
dnss...](http://www.internetsociety.org/deploy360/resources/plan-dnssec-
validation/)

This is why some people are using libraries like the GetDNS API to build
DNSSEC validation directly into apps:

[https://getdnsapi.net/](https://getdnsapi.net/)

And solving the "last mile" problem is exactly why the DPRIVE working group
was chartered within IETF:

[https://datatracker.ietf.org/wg/dprive/charter/](https://datatracker.ietf.org/wg/dprive/charter/)

(And anyone is welcome to contribute to the group.)

------
danyork
Several people have confirmed to me that they were unable to see the site from
Google's Public DNS Servers (8.8.8.8, 8.8.4.4 and the IPv6 addresses). This
makes sense given that Google has been performing DNSSEC validation since
early 2013.

I've updated the post to more clearly note the fact that this was not just an
issue on Comcast networks. (I mentioned that it would be an issue on _any
network performing DNSSEC validation_ , but then only gave Comcast as an
example.)

------
Panino
DNSSEC causes more problems than it solves. And even if it were reliable, it's
still based on obsolete crypto and _isn 't even encrypted_. But on top of
that, it's an outage factory.

~~~
samplonius
Why would it make sense to encrypt queries and replies from a public data
source? Signing the data is all that is needed.

~~~
jessaustin
Some users might not want intermediate hosts to see a request for the IP addr
of "ginger-midget-porn.com", for example.

------
wtbob
Also unavailable from Android devices…

~~~
danyork
Which would make sense because they may be set to use Google's Public DNS
Servers (8.8.8.8 and 8.8.4.4 and the IPv6 equivalents).

Google Public DNS has been performing full DNSSEC validation since May 2013:
[http://www.internetsociety.org/deploy360/blog/2013/05/confir...](http://www.internetsociety.org/deploy360/blog/2013/05/confirmed-
googles-public-dns-now-performs-dnssec-validation-for-all-queries-by-default/)

~~~
yuhong
Yea, it is fortunate that people on Twitter quickly figured this out.

~~~
rb12345
To be fair though, it's not that hard to test for DNSSEC breakage if you have
dig available. If "dig +cdflag example.com A IN" works and "dig +adflag
example.com A IN" returns SERVFAIL, it's DNSSEC's fault. (If that latter
command returns responses without the ad flag, the domain lacks DNSSEC
signing.)

