
AWS DNS services break qname minimization - fanf2
https://forums.aws.amazon.com/thread.jspa?threadID=269116
======
colmmacc
AWS Engineer here. I haven't worked directly on Route 53 in a few years, but I
wrote most of the DNS answering logic back when we built Route 53. Route 53 is
a so-called "child centric" DNS server, in that it considers the child zone to
be fully-authoritative for its own NS records. That's why the name servers for
a parent zone reply with a referral, rather than in the answer, because from
their perspective, they don't really "know" the answer. The proper thing a
resolver should do is follow the referral and ask the child name servers.

I don't think that's in violation of what the DNS RFCs specify (though the
older DNS RFCs are not great at being specific), and I remember making the
decision based on what would work best, and be least likely to poison
resolvers, in the event of domains being moved between customers, providers,
or across accounts for some reason. Setting up the child name servers to say
"have moved here ..." can speed those processes up.

I'm sure the Route 53 team will look at this and make whatever change is best,
but I also strongly suspect that QNAME minimization may rest on under-
specified or un-specified ideas about what authoritative name servers should
do. I'd have it follow referrals if it doesn't already.

In any event, I think what's going in this case is more subtle but simpler,
basically "ws.fdmg.org." isn't occluding "prod.ws.fdmg.org" when it should,
but that takes a very convoluted edge-case set up involving two over-lapping
delegations from the same parent zone.

~~~
agwa
> Route 53 is a so-called "child centric" DNS server, in that it considers the
> child zone to be fully-authoritative for its own NS records. That's why the
> name servers for a parent zone reply with a referral, rather than in the
> answer, because from their perspective, they don't really "know" the answer.
> The proper thing a resolver should do is follow the referral and ask the
> child name servers.

> I don't think that's in violation of what the DNS RFCs specify

Agreed, and in fact it's pretty easy to see that the root and .org name
servers behave exactly like Route 53 in this regard.

> In any event, I think what's going in this case is more subtle but simpler,
> basically "ws.fdmg.org." isn't occluding "prod.ws.fdmg.org" when it should,
> but that takes a very convoluted edge-case set up involving two over-lapping
> delegations from the same parent zone.

Yes, that appears to be what is happening, and it breaks regular DNS
resolution, not just QNAME minimization. If your DNS resolver has the NS
records for ws.fdmg.org cached because you previously looked up a ws.fdmg.org
hostname, you won't be able to resolve any prod.ws.fdmg.org hostnames, because
your resolver will send the queries to the name servers for ws.fdmg.org. I
don't know if this is a Route 53 bug or if the customer misconfigured their
zones, but if the latter it would be nice if Route 53 could stop people from
shooting themselves in the foot this way. To reiterate, this breaks regular
DNS resolution, not just QNAME minimization.

That said, Route 53 improperly returns NXDOMAIN for empty non-terminals, which
does break QNAME minimization (and violates RFC8020). For a proper example,
see opsmate0._domainkey.sslmate.com and _domainkey.sslmate.com.

~~~
colmmacc
RFC8020 is an example of the DNS WG making backwards incompatible changes. The
"NXDOMAIN" for an empty non-terminal has long been the behavior in DJB's
TinyDNS.

That isn't what Route 53 is using, but I bring it up because there have been
mailing list threads (or flames?) where folks complained about what DJB's
software is doing, and he repeatedly asked "Show me where in the standards
this is is specified or forbidden?" and no-one would come up with the goods,
but folks still basically wanted "Just do what bind does".

Here's DJB's own mail about it, I had to forward it because his mails don't
get through to the mailing list for some reason (that might be the fault of
his own stupid qsecretary moderation system): [https://www.ietf.org/mail-
archive/web/dnsext/current/msg1108...](https://www.ietf.org/mail-
archive/web/dnsext/current/msg11084.html)

I generally sympathize with DJB in this case, because DNS really needs some
robust and competing implementations. The standards should be tolerant of
differences in behavior (as others are), rather than dominated by the
proponents of one implementation. It doesn't help that the WG is toxic, at
least in my opinion as someone who has been reading it (and mostly just
reading) for over a decade. Anyway, I'm not trying to start an argument, just
explain. In this case, the bind proponents did dominate, and we get RFC8020,
which for some software is a backwards incompatible change.

This isn't the first backwards-incompatible change either. The invention of
DNAME introduced a new kind of backwards-incompatible occlusion, and name
servers "before" and "after" DNAME respond in different ways, for the same
zone file.

Those kinds of changes place providers in a pickle: one of our golden rules is
never to make backward incompatible changes or break anything that was
working. A promise is a promise. So it means we have to lock things down (the
previous paragraph is the reason Route 53 does not accept arbitrary literally-
encoded RRs) and when changes come; either not make those changes, or
exhaustively audit and determine how many zones, queries it might impact and
whether that would change anything. At scale, you'd be amazed how small and
subtle behaviors can end up being embedded as assumptions in customer scripts
and so on.

------
FiloSottile
Heh, this is a fairly easy mistake to make when implementing a DNS server. I’m
the person that had to fix this very same bug (and possibly introduced it) for
a certain class of domains in the Cloudflare DNS server. “Empty non-terminal”
still has an emotional reaction attached to it. :)

Also, pretending all NXDOMAINs are actually empty non-terminals is a backwards
compatible hack to compress DNSSEC responses
[https://tools.ietf.org/html/draft-valsorda-dnsop-black-
lies-...](https://tools.ietf.org/html/draft-valsorda-dnsop-black-lies-00)

------
takeda
Just to clarify, the qname minimization is a privacy technique used during
recursion to only disclose what's absolutely necessary.

Traditionally if you're querying www.google.com, you'll:

1\. ask root name servers for www.google.com and get list of com name servers
2\. ask com name servers for www.google.com and get list of google name
servers 3\. ask google name servers for www.google.com and get the response

With qname minimization you would:

1\. ask root name servers for com TLD and get list of com name servers 2\. ask
com name servers for google.com and get list of google name servers 3\. ask
google name servers for www.google.com and get the response

The AWS DNS servers don't strictly follow rfc and don't provide correct
responses in specific cases

------
EwanToo
I had honestly never heard of the term qname minimisation before, but this
blog post gives a pretty clear explanation of the benefits and issues

[https://blog.verisign.com/security/minimum-disclosure-
what-i...](https://blog.verisign.com/security/minimum-disclosure-what-
information-does-a-name-server-need-to-do-its-job/)

~~~
jlgaddis
See also ("experimental") RFC7816 [0] for all the gory details.

[0]:
[https://tools.ietf.org/html/rfc7816](https://tools.ietf.org/html/rfc7816)

~~~
merb
well I'm not sure if it's sane to implement this rfc. it's experimental since
march 2016. and I do not think that privacy should be a concern when
resolving... I mean what is private about the dns system itself?!

------
feelin_googley
Try counting number of DNS queries for all lookups every day. The software I
wrote for myself does this and I can attest that sites using AWS nameservers
in almost all cases require more queries than others, including those using
similar managed DNS services. The number, which does _not_ include queries to
the root can be quite high, e.g., eight. No matter how fast the response
times, that delay adds up.

root.zone data is reasonably static. Having a local copy of this data,
resolution can be done in two queries: 1. tld ns 2. auth ns, which is the case
for many, many www domainnames (the majority, perhaps), or even one query (I
think some .mil domains used to be a well-known example). As a user, I view
every added query as an additional cost.

It is possible to minimize queries via other means that are under _user_
control and do not rely on third parties.

Running a root locally. Custom stub resolvers with tld ns IPs compiled in.
Storing RRs for common nameservers in local zone. Storing DNS data in bulk for
important sites that will be revisited, copying to HOSTS and updating the data
periodically. I have been using techniques like these for many years. And, it
is getting easier every year as more bulk DNS data is available, e.g.,
internet scans of DNS, TCP DNS services that accept pipelined queries, etc.
(If I am not mistaken, the author of the "qname minimisation" draft also was
the one with the "DNS looking glass" idea.)

[https://tools.ietf.org/html/draft-ietf-dnsop-qname-
minimisat...](https://tools.ietf.org/html/draft-ietf-dnsop-qname-minimisation)

Despite usual mailing list/forum comment/blog memes and mantras, if one
surveys _all_ websites (not just popular ones), DNS data is mostly static.
That provides great opportunities for speedups by avoiding querying ICANN DNS
for the same data 24/7/365\. Cue Schopenhauer quote about truth. Someday maybe
we will see an internet draft about "DNS minimization" for _users_.

Did AWS forums start requiring an account for read access?

~~~
becauseiam
> Did AWS forums start requiring an account for read access?

No but if you timed out from a session with your account through the console,
you might see something about not being signed in. Clicking the "log out"
button (or hosing cookies) should rectify it.

~~~
feelin_googley
I do not have an AWS account.

The reason I asked is because forums.aws.amazon.com redirects to
signin.aws.amazon.com, and generates some faked up OAuth credentials.

------
ttul
AWS DNS is all kinds of fucked. They have implemented a range of undocumented
rate limits both to their own DNS server and to external (!) DNS, presumably
to reduce amplification attacks. These things have hobbled our services many
times, which rely on making thousands of DNS queries per second.

~~~
otterley
I'm curious about your architecture. What requires you to make thousands of
DNS queries per second against Route 53?

~~~
merb
if you have thousands of services and use route53 as your internal dns service
and use it for service discovery, this could be a problem.

------
nvarsj
Tangentially related, DNS quirks show up in lots of places.

For example, FB’s graph API doesn’t work with FQDN:

    
    
      * https://graph.facebook.com./v2.8 - returns 500
      * https://graph.facebook.com/v2.8 - works
    

(I filed a bug for this, but it was closed as “working as designed”).

