Hacker News new | past | comments | ask | show | jobs | submit login
Web Developer's Guide to DNS (rjzaworski.com)
324 points by rjzaworski 12 days ago | hide | past | web | favorite | 51 comments





A few issues:

• The description is a bit vague, but the article seems to describe a model of a stack of resolvers, each resolver asking the next one in turn, the last resolver getting an answer, and the answer being then passed down the chain again. But this is not how it works. For a normal DNS query, there is ever only one resolver involved. This one resolver is the single thing asking many authoritative servers the same question, each time getting names of servers who might know the answer, and ultimately getting an actual answer. There is no stack of resolvers.

• The article completely conflates the process of a cached record expiring due to TTL expiration, with the process of an updated zone with a new serial number being transferred to slave servers. These two things have virtually nothing in common.

I also found the illustrations too detailed (i.e. confusing for non-experts). There are a lot of options to "dig" which could have been used to cut down on the extraneous output when illustrating a particular point.

The zone file was especially confusing, in that none of it was explained, and the nature of the DNS data model would have been much better illustrated with an object diagram of some kind.


"the article seems to describe a model of a stack of resolvers, each resolver asking the next one in turn, the last resolver getting an answer, and the answer being then passed down the chain again. But this is not how it works."

There may be some semantics that aren't right, but say I have a Linux box, a home router, pointed at 8.8.8.8, and ask it for something not in anyone's cache. There is a "stack" of sorts when the answer isn't cached anywhere. Various caches in either resolver libraries and real DNS servers...forwarding ones, root ones, authoritative ones, etc.

And the depth of the stack is different depending on my setup.


So where’s the stack?

The Linux box presumably uses the resolver as announced by the DHCP server on the router, which I would guess is 8.8.8.8. Therefore, the chain goes linux-box → 8.8.8.8 → auth servers, 8.8.8.8 being the resolver.

Or perhaps the router announces itself as the resolver, and proxies all requests to 8.8.8.8. This would not be necessary, but I guess it would be useful for the router to do proxying instead of NAT whenever the protocol (DNS in this case) makes it possible. There is still no reason to call the router a “resolver”, or even assume that the DNS proxy software on the router even has a cache.

Describing DNS as a stack of resolvers is like describing HTTP as always having a chain of caching proxies. It is not normally true, and even when it is close to true, it’s made to not look that way – i.e. it’s not useful to think of it that way, especially when explaining it to someone else.


The Linux box could have a caching forwarder, as could the router. There are many routers with a local cache that put their own IP in the DCHP nameserver spot.

And 8.8.8.8 may not know the answer, and may have to query for it, that's a very common "stack". A non recursive client letting a recursive server do the trawling is probably the most common home use case.

You could put a recursive resolver on your Linux desktop, and that would be different. Not common for a home PC though.


Everything is possible in theory, of course, but all of these theoretical possibilities are relatively uncommon, and, more importantly, made to not look like the aforementioned chain-of-resolvers. The normal DNS procedure is client→resolver→authoritative_server, and any deviation from this is deliberately hidden by the components which alter it.

They aren't uncommon though. The most typical home setup has a non recursive client ask their ISP to resolve a name. The ISP, if it hasn't cached the answer prior, queries someone else on the client's behalf.

Separately, Windows, Linux (most popular distros), and MacOS all have a caching/forwarding local resolver.


You’re still not describing a stack; you are describing the normal three-step client→resolver→auth_server procedure. Your “non recursive client” is the client, the ISP has the resolver, and that resolver performs many queries to many authoritative servers, which is the final step. There is still only one resolver, not a stack of them.

A common variation, which might have been what you meant to describe, is that a home router is announcing itself as a resolver to the local network, but is, in reality, merely acting as a proxy to the actual resolver, either at the ISP or some other resolver provider. But a DNS proxy is not a resolver, and might not even have a cache. A DNS proxy is made to be transparent, i.e. invisible to both sides; the local network sees the proxy as the resolver, and the real resolver sees the proxy as the client. There are even more complex setups possible on the authoritative server side, but none of these proxy possibilities alter the fundamental DNS model of client→resolver→auth_servers.

The possibility of transparent proxies in any client-server protocol should not alter the description of that protocol from “client-server” to “client-proxy-proxy-proxy-…-server”, especially not when the protocol does not itself account for proxies. The DNS protocol has, inherently, one component between client and server: the resolver. There is no chain-of-resolvers model in the DNS protocol. The possibility of any number of transparent proxies at any point in the chain should not alter the description or explanation of any protocol.


Gross over simplification. You can have multiple authoritative resolvers, such as in the case of stub delegation.

Someone spends a shit ton of time explaining a new and complex concept.

Top comment is someone else ripping it apart. Yep. Welcome to Hacker News.

Can we please try to spend a week not using the following terms:

A few things, a few issues, actually..., I agree but, or anything else that makes you feel superior by shitting on someone else


No? If the article has issues, and you have domain-specific knowledge to correct it, please share to the benefit of all of us. Just because there are a few issues doesn't mean the whole thing was a waste, especially if it motivated an expert to share.

I don't see any tone in this reply which makes any personal attacks or even any major value judgements on the article itself such as "this is crap not even worth reading" as such I, the reader of both, am informed of an alternative explanation.

The author is then also invited to revise their article if they agree with the feedback.

Perhaps the grandparent could have thanked the author or opened with some positive feedback first, but given the grandparent took the effort to write detailed answers in a noncondescending tone, I see it as a quality informational post. I hope others do too.


Is the GP wrong? Of not then I don't see why you're bothered.

i'd say it is constructive criticise more than "ripping someone apart". yes constructive criticism can make you feel bad, it doesn't mean it was intended to be hurtful. is this a class that just isn't taught nowadays

A confused explanation gets halfway round the world before a valid explanation gets its pants on.

I've also found this guide to be really helpful: https://howdns.works/

What background does one need to understand the "4.4.8.8.in-addr.arpa" paragraphs? I'm a career web developer who took a networking course many years ago but "timely inversion," "DNS zone," what "public" means as an adjective for a DNS server, "hop," and "block of IP addresses" are all presented like I'm supposed to intimately know them already...maybe I'm just missing something

I commend the author for attempting to make this subject more approachable to people less familiar with networking, but it’s always tough to present this and understand how much you need to explain.

- zone: you can think of as an SOA, start of authority, for a set of records. For example: there is an SOA record for example.com. In addition there are NS records that express what nameservers are responsible for being the authority, trusted server, of the zone. Often zones are expressed in a single file, and loaded into the authoritative server, and it contains all the additional records for the zone, like A address records.

- public: in this context probably is referring to the fact that it’s exposed and reachable from the greater internet. Often people create internal private zones that only work inside a particular network.

- hop: a network hop is a router on the network, and internet. Each represents a hop, this is important for things like the TTL on a UDP packet (distinct from the TTL on a DNS record), each hop decrements the packets TTL by 1. This is really important for multicast packets.

- block of IPs: generally refers to the CIDR block, which designated a routable network. 10/8 means all addresses that match 10.x.x.x, where as 10.10.1/24 is the 10.10.1.x network.


The article is not actually explaining it, merely obliquely hinting at it.

If you want to look up an IP address and get a server name (i.e. the reverse of a normal DNS lookup where you look up a name and get a DNS record containing an IP address), you do a “reverse” DNS lookup. Since the DNS protocol does not support this operation directly, this is enabled by a somewhat ugly hack, namely the transformation of the IP address into a name, which you can then look up and get a DNS record containing a name.

For example:

foo.example.com. A 192.0.2.3

is a name with an attached DNS record containing an IP address. To do a reverse lookup, we transform the address into the name “3.2.0.192.in-addr.arpa”, and look up not the A record (containing an address) for that name, but the PTR record, containing a name:

3.2.0.192.in-addr.arpa. PTR foo.example.com.

You can think of DNS record types (A/PTR/MX/etc.) as types or classes in a programming language. What data a DNS record can contain, and what that data should mean, is controlled by the record type.

This is the essence of it. There are always various minutiae one could get into, like the fact that IPv6 has a separate procedure to transform addresses into names, and, just as a name can have more than one address, the lookup of one address can result in more than one name.

To directly answer your questions:

• “Timely inversion” is not used as a technical term, it’s just a cute way of referencing the transformation procedure for an IP address into a name able to being looked up in the DNS.

• A “zone” is technically a collection of DNS records, restricted to a particular domain or subdomain. For example, the zone for “example.com” would normally contain all DNS records for not only “example.com”, but also the records for the names “www.example.com” and “foo.example.com”. The purpose of zones is that a zone is the mechanism by which the responsibility to respond to queries about DNS data is delegated to DNS servers. If you want questions about some names to be answered by some special DNS servers, you have to put those names in a separate zone. This is unusual but not unheard of – the other day¹ Cloudflare had problems related to the fact that they have actually delegated “www.cloudflare.com” to separate name servers from the name servers which are authoritative for the “cloudflare.com” zone. I.e. “www.cloudflare.com” is its own zone. Note however that this is unusual. You can tell if something is its own zone by the presence of an SOA record (and NS records) on the name; www.example.com does not have an SOA record, but www.cloudflare.com does.

• “Public” in this context simply means “for use by the public” or “publicly accessible”. And it’s strictly speaking a DNS resolver, not an authoritative DNS server for any specific zone or zones.

• The use of the word “hop” is, I believe, a further indication of the belief by the author in the stack-of-resolvers model, but this is a false belief. What actually happens is that a resolver asks one authoritative server (starting with the root servers), receives as a reply a list of better servers to ask, and the resolver asks those authoritative servers in turn. This is the actual “hop” referred to in reality.

• A “block” of IP addresses is a contiguous span of IP addresses. IP addresses are not allocated to organizations or ISPs around the world randomly or individually, but are instead delegated into as large and as contiguous spans as is possible. This is because the routing tables in routers are composed of lists of spans of IP addresses, and it is vital to minimize these lists, because if the core routers run out of memory, no more addresses can be routed.

1. https://news.ycombinator.com/item?id=19641155


For the cloudflare case, I found this interesting:

   dig www.cloudflare.com

  ; <<>> DiG 9.10.6 <<>> www.cloudflare.com
  ;; global options: +cmd
  ;; Got answer:
  ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38923
  ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

  ;; OPT PSEUDOSECTION:
  ; EDNS: version: 0, flags:; udp: 512
  ;; QUESTION SECTION:
  ;www.cloudflare.com.  IN A

  ;; ANSWER SECTION:
  www.cloudflare.com. 16 IN A 104.17.210.9
  www.cloudflare.com. 16 IN A 104.17.209.9

  ;; Query time: 14 msec
  ;; SERVER: 75.75.75.75#53(75.75.75.75)
  ;; WHEN: Sat Apr 13 14:46:25 PDT 2019
  ;; MSG SIZE  rcvd: 79


   dig cloudflare.com

  ; <<>> DiG 9.10.6 <<>> cloudflare.com
  ;; global options: +cmd
  ;; Got answer:
  ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16630
  ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

  ;; OPT PSEUDOSECTION:
  ; EDNS: version: 0, flags:; udp: 512
  ;; QUESTION SECTION:
  ;cloudflare.com.   IN A

  ;; ANSWER SECTION:
  cloudflare.com.  536 IN A 198.41.215.162
  cloudflare.com.  536 IN A 198.41.214.162

  ;; Query time: 73 msec
  ;; SERVER: 75.75.75.75#53(75.75.75.75)
  ;; WHEN: Sat Apr 13 14:46:47 PDT 2019
  ;; MSG SIZE  rcvd: 75

Why does cloudflare.com resolve to a different IP address than www.cloudflare.com?

Based on the outage people noticed the other day with www.cloudflare.com, we actually know that www is delegated to other nameservers.

Notice that there is an SOA and NS records for www.cloudflare.com if you dig for them.

Given this, they can't just CNAME from one zone to the other (if they want to include additionals from the subzone). So my guess is that the resolution of cloudflare.com is relying on rules setup for an HTTP 301, as opposed to the relying on DNS aliasing.


In a sort-of-related curio, IBM still keeps around their bunch of “www2.ibm.com” subdomains or something like that—likely because some links around the web are pointing to them. Afaict these addresses were used as ‘poor man's’ load balancing before multi-tier backends became the norm.

One amusing form of poor-man's load balancing was in the original Netscape 1, which had hard-coded rules for certain netscape web servers where it would automatically generate www1/www2/etc subdomains randomly

> Why does cloudflare.com resolve to a different IP address than www.cloudflare.com?

Probably because the IP addresses on cloudflare.com leads to a simpler web server which seems to only reply with redirects to www.cloudflare.com, and might not have any actual web pages.


Assume that example.com and foo.example.com have different zone files which in-turn have different ns entries day ns1 and ns2. So if I query for the A record of foo.example.com will it be answered by ns2? If so can I assume that the .com tld server has entries for both example.com and foo.example.com?

So, what are the practical application of zones? I can think of load balancing DNS queries as one.


> So if I query for the A record of foo.example.com will it be answered by ns2?

Yes, if I understand your example correctly.

> If so can I assume that the .com tld server has entries for both example.com and foo.example.com?

No. The .com zone only contains NS records for example.com. The NS records for foo.example.com exist in the example.com zone (and, for consistency, in the foo.example.com zone), but not in the parent .com zone, which knows nothing about any sub-zones.


Thanks, but where are zones used in practice?

Zones are used to split DNS data into zones of responsibility. I.e. when you want different administrative entities to have control over different sets of names, you put the names in different zones, served by different name servers controlled by those different administrative entities. If a company would split, say, the HQ and research department into separate subdomains "hq.example.com" and "research.example.com", and if they made those two subdomains into separate zones, each department could have their own name servers and each department’s name server administration team could add and delete server host names in their own zone all day long without risking breaking anything for the other department, and without even involving anyone else.

More practically, the "com" and "net" zones are managed by a different company than, say, the zone for "ibm.com", therefore they are in different zones. Also, the root zone (".") and all the country-specific zones like "se" or "de" are certainly managed by different entities (governments), therefore the root zone is separate from those individual country-specific zones.


Thanks to you and bluejekyll! I am understanding it a bit better with the breakdown

> When you hit “save”, the serial number (se) incremented to reflect the change. As clients everywhere evicted their last-retrieved cached copy of the pets.com zone, your new change (with its new serial) bubbled out across the internet, and some indeterminate time later it finished going “live”.

This sorta implies the serial number has something to do with the caching mechanism, but it doesn’t. Record caching is based on the record TTLs only (which the author later mentions).

The serial number was originally used by the BIND master/slave replication mechanism for zone files (so called “zone transfers”[0]), which occur out of band of normal DNS lookups. Other name servers could use it for similar purposes, since the serial number is required in the SOA record, but I don’t know if an so. (In fact, the formatting shown for zone files is specific to BIND. Other name server implementations have their own ways of storing the zone records.)

This entire piece is written in a way that gives me a feeling that the author doesn’t fully understand DNS.

That said, I do somewhat blame the original DNS RFCs. For a long time BIND was the only name server implementation, and the RFCs themselves reflect this with RFC 883 and its updates being as much about BIND itself as about DNS. The DNS RFCs are kinda unusual in that way. It would be like there being an RFC for a specific httpd implementation as part of the http RFCs.

[0] https://cr.yp.to/djbdns/axfr-notes.html


> [The article] sorta implies the serial number has something to do with the caching mechanism, but it doesn’t.

I agree, as I said yesterday in my sibling comment.

> In fact, the formatting shown for zone files is specific to BIND.

No, the format is a standardized (defined by RFC 1035 section 5) serialization format for zone data, called Master File format. It is used as an interchange format for zone file data.

It may (or may not) have originated with BIND, but that is beside the point. It is not, and should not be seen as, being specific to Bind in any way. Is is analogous to JSON for DNS data.


As I wrote, RFC 883 and its updates (such as 1035) are an odd mix of DNS specifications and BIND implementation details. Which servers besides BIND use that master file format?

Wikipedia says¹:

“[Master file format] has been widely adopted by other DNS server software – though some of them (e.g. NSD, PowerDNS) are using the zone files only as a starting point […]”

1. https://en.wikipedia.org/w/index.php?title=Zone_file&oldid=8...


In addition to the content, I really appreciated the site itself. Very simple, easy to read and fast to load. Good work!

I've recently found that DNS is surprisingly simple protocol, you can implement simple query/response with very few lines of code. So if you want to better understand it, you might want to code a simple client, may be even recursive one.

It seems simple on the surface, then you get into the various record types, and edge cases to support.

You just need a decent library to parse DNS packets. [1] Yes, it's in C, but it should be easy to interface to other languages given it only has two functions.

[1] https://github.com/spc476/SPCDNS


I chose to write my own: https://github.com/bluejekyll/trust-dns

And, it in theory can be used in other languages via C FFI as well, since it’s in Rust.


suggestion, add section explain OS hosts file. When, why, how to use and how bypasses dns lookup.

nice!

A special shout out to Lennart Poettering for recently breaking dig +trace!

https://github.com/systemd/systemd/issues/5897


It sounds like the underlying issue is that people think DNS resolution linux works like:

1) Read resolv.conf 2) Send DNS packets directly to one of the named servers.

But that is never how it worked. Linux has always had the wacky NSS plugins that are involved, and the underlying model ends up being:

1) glibc or similar reads resolv.conf 2) glibc or similar does random stuff depending on a variety of other files 3) good luck if you aren't using glibc (go implements its own DNS stack, for example)

This would be a Linux rant, but OS X is also crazy and I imagine all other OSes are too.

The underlying issue is that "name resolution" and "DNS" are not exactly the same thing. There are all sorts of other things people expect to be able to use for name resolution, like mDNS. (I recall using LDAP and yp at some points in the past.) Thus, expecting a DNS trace to work for all "name resolution" is folly.

If you want to dig +trace to work, just point it at a DNS server with the @ argument. dig is a DNS client, not a name resolution client.


The fact that linux (and OSX and Windows) support mechanisms of name resolution other than resolv.conf has zero bearing on dig's previous utility as a resolv.conf diagnostic tool, zero bearing on the fact that systemd changed the behavior of a nonrecursive query for root zone NS records, and zero bearing on the fact that Poettering insisted that this wasn't a problem instead of fixing it.

dig is not a resolv.conf diagnostic tool, it's a DNS lookup utility. I don't think it even parses the full syntax of resolv.conf. Heck, it doesn't even consider multiple nameserver statements.

It worked before, they broke it, and as usual the response is an egregious "fuck you" and a queue of apologists castigating the aggrieved that they were Doing It Wrong All Those Years And They Must In Future Conform To Only The Approved Way.

This is why people loathe systemd.


Something which is wrong but has mostly worked by accident is still wrong. A tool which parses a config file it does not own and whose semantics do not necessarily match what the tool expects is wrong. The blame for any breakage which results is clearly on the tool in question.

Lennart has done more to improve Linux than all of his detractors on HN combined in no small part because he is willing to break broken things and force them to be fixed.

If the people whining about Lennart/systemd spent half as much time fixing their shit as they did writing angry comments on the internet the Linux ecosystem would be in much better shape.


It wasn’t broken, didn’t need fixing, and the config files existed before systemd did, so claiming they “belong to a tool” really just exemplifies the arrogance implicit in the whole project.

As for contributing, no, you don’t get to piss in the well and then demand everyone drinks from it.


I never claimed systemd owns resolv.conf so the fact that the latter predates the former is irrelevant. resolv.conf is to be used to implement the resolver API. Any tool which uses resolv.conf for any other purpose does so at its own peril.

yep this is the only way we're going to be able to drag linux into future, where maybe one day it actually will be year of the linux desktop

That’s beside my point. Dig isn’t a tool for diagnosing resolv.conf. It’s a tool for querying and diagnosing name server responses. I have no opinion on systemd nor its maintainer.

Meta: this entire thread is even relevant to the story. Some folks just wanted to complain about systemd. It doesn’t make for interesting reading and lowers the discussion quality here.


Fortunately for the rest of us, no-one actually gets to own a thread and gatekeeper the discussion.

"poettering commented...This is really by design."

Oy. It worked before. I get his notion of not wanting to make it a full dns resolver, but breaking something that worked seems clear enough.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: