
Systemd-resolved does not keep the order of the DNS servers - beefhash
https://github.com/systemd/systemd/issues/5755
======
stonogo
The behavior these people are asking for is an implementation detail specific
to glibc. It's also incorrect: all DNS resolvers listed in resolv.conf are
assumed by relevant standards to return the same results as each other. I feel
bad for the systemd developers, getting bug reports from people who don't
understand their own computers.

~~~
king_phil
Can you point out the relevant RFC, please? Didn't know about that.

~~~
stonogo
From RFC 1034:

"The strategy is to cycle around all of the addresses for all of the servers
with a timeout between each transmission."

Note that it does not specify an order -- the resolver determines that, and
RFC 1035 even recommends keeping a weighted list of nameservers prioritized by
response time:

"To complete initialization of SLIST, the resolver attaches whatever history
information it has to the each address in SLIST. This will usually consist of
some sort of weighted averages for the response time of the address, and the
batting average of the address (i.e., how often the address responded at all
to the request)."

...which is exactly what systemd is doing. The glibc behavior is a naive
simplest-implementation approach, and if your infrastructure depends on an
implementation detail of a specific libc resolver, you should install proper
configurable resolvers and use those instead.

The fact that all nameservers are expected to return identical results is
inherent in the design of DNS. It's one namespace, and if you need to split
it, the client resolver is the wrong place to do that too.

------
ealexhudson
I would have to side with systemd project here. I had never heard of the
ordering of the DNS servers being meaningful, and I'm amazed people are
relying on that behaviour described in the ticket.

systemd appears to be the only project where getting rid of legacy cruft is
seen as a bad thing. Having a DNS setup that works in a different way when
stuff fails should be seen as a bug; DNS is supposed to be reliable.

~~~
pdkl95
> I had never heard of the ordering of the DNS servers being meaningful

Now you have. This is a common configuration in mixed-use situations.

> I'm amazed people are relying on that behaviour described in the ticket

Why? Using each server in order is the expected behavior. From resolv.conf(5)

    
    
        nameserver Name server IP address
            Internet address of a name server that the resolver
            should query [...]. If there are multiple servers,
            the resolver library queries them in the order listed.
            [...] (The algorithm used is to try a name server, and
            if the query times out, try the next, until out of
            name servers, then repeat trying all the name servers
            until a maximum number of retries are made.)
    

> Having a DNS setup that works in a different way when stuff fails should be
> seen as a bug;

You can call it a bug in _your_ use cases, but having an imperfect fallback is
important in some situations.

> DNS is supposed to be reliable.

Software is _supposed to be_ lots of things, but good engineering understands
that sometimes the real world deviates from these ideal.

~~~
ealexhudson
Y'see, I do get all that. And I could have phrased the "never heard of
ordering" thing better: of course I have.

What I've never heard of is people relying on that to get DNS results correct.
If your first server doesn't work, you _get the wrong results_. I don't
understand on what planet that is acceptable sysadmin setup.

Split DNS zones have been a thing for years. This isn't a good substitute.

~~~
ChuckMcM
I see it as implementation being perceived as architecture. The instance of
the library that is quoted does this, it is not part of the architecture. That
library has been used in a lot of places, so people start relying on the side
effect (a common one is to stand up a local DNS server, have it 'shadow' or
give alternative results to some queries, like ad network host names, and if
the host name isn't in it then to fail so that the next DNS server is queried
which is more likely a 'legit' server.)

One of the reasons the IETF likes multiple running implementations of an RFC
is they often have different side effects and so they call out things which
might hang people up.

The "correct" way to shadow, and to achieve user required semantics, is to
create a DNS proxy that answers queries that it is supposed to and recursively
getting results for ones that it doesn't.

Correct here is in scare quotes because it doesn't mean it is right but
instead just insures side effects of the libraries will not be a problem.

------
wrs
These people have quite simply configured their DNS wrong--all DNS servers are
expected to return the same results, so it can't matter which server the
resolver uses. But their misconfiguration happens to work most of the time for
their use case. The solution is either to implement a "bug-compatible"
priority flag in resolved, or to make that a first-class use case with its own
flag ("emergency backup nameserver"?).

It's a classic case...should the software follow the spec, or the real-world
usage? And people complain either way.

~~~
joombaga
> all DNS servers are expected to return the same result

Really? My local DNS server should return the same results as 8.8.8.8? Should
one never use unregistered suffixes?

~~~
jasomill
No, one should probably never use unregistered suffixes. But that’s not to say
that all DNS names should be public, nor that NAT is practically avoidable, so
this is beside the point.

What one _should_ do is only choose DNS servers capable of properly resolving
all addresses, as it shouldn’t be the resolver’s responsibility to “shop
around” for answers.

Fixing VPN configurations that complicate this should be the responsibility of
VPN client software — by providing a forwarding DNS server that falls back to
the originally configured servers, say — not the resolver, and similarly for
any other cases where standard resolver behavior does not suffice.

Application software, for its part, shouldn’t generally rely on
implementation-defined resolver behavior, as doing so is obviously not
portable, but that’s less sinful than a VPN relying on client library-level
support to work around the fact that it’s exposing a broken network
configuration.

Note that I have no axe to grind one way or another in the systemd debate:
Linux isn’t me primary OS, but I regularly use and abuse both systemd and non-
systemd Linux systems, and all the Linux init systems seem to work at least as
well as any other system I regularly use (OS X, FreeBSD, Windows) for most
practical purposes.

~~~
JdeBP
Aside: You are using "resolver" to describe the wrong thing. RFC 1034 employs
"resolver" to describe the entire subsystem that user programs use to access
what it calls "name servers". It theoretically allows, because it isn't
intended to be Unix-centric, systems where all of the query resolution
mechanism is potentially in some form of shared subsystem that runs inside
each applications program.

Strictly speaking, what you are talking about here, in RFC 1034 terminology,
is a "stub resolver", the Unix model where there's a fairly dumb DNS client
library running in the applications, talking to an external program that
actually does the grunt work of query resolution by making back-end
transactions and building up the front-end response from them.

A "resolver" actually very much _does_ "shop around for answers", amongst
content DNS servers. There's even a (slightly faulty) description of its
shopping around algorithm, in RFC 1034 section 5.3.3.

"resolver" is such a confusing term, and so often used contrary to the RFCs in
the way that you have here, that years ago I took to explaining the DNS to
people using terminology borrowed from HTTP: proxy servers, content servers,
and client libraries linked into applications.

* [http://jdebp.eu./FGA/dns-server-roles.html](http://jdebp.eu./FGA/dns-server-roles.html)

------
bahjoite
> The "Switching to fallback DNS server 8.8.8.8." message indicates that you
> have no DNS servers configured at all, in which case resolved will use
> compiled-in fallback servers because it tries hard to just work also if you
> have a locally misconfigured system

It's good to have the knowledge that Google servers are compiled-in to
resolved.

~~~
geofft
They're "compiled in" in the sense that they're the upstream default - people
are generally expected to use systemd from a Linux distro, and distributors
are expected to patch this to some appropriate default (or patch out the
fallback entirely, if your distro would prefer your users get no DNS to Google
DNS, which seems like a defensible decision to make). There are no upstream
binary releases. 8.8.8.8 is as reasonable as anything else for someone running
`make` on the upstream sources and hacking on things; it's probably not a
great default for a downstream distributor.

~~~
bahjoite
The relevant option to change is "dns-servers" at
[https://github.com/systemd/systemd/blob/master/meson_options...](https://github.com/systemd/systemd/blob/master/meson_options.txt#L160)

~~~
hedora
Heh. The line above the one you highlight makes dnssec vulnerable to MITM
attacks by default. Nice.

(Also, the one below points at more Google infrastructure...)

------
js2
systemd's behavior is completely reasonable. The folks arguing otherwise on
the issue would probably be better served by installing dnsmasq locally to do
what they want and pointing systemd at that.

It may be worth pointing out that dnsmasq has a similar behavior by default
and points out the correct way to override specific hostnames/zones:

 _Q: My company 's nameserver knows about some names which aren't in the
public DNS. Even though I put it first in /etc/resolv.conf, it doesn't work:
dnsmasq seems not to use the nameservers in the order given. What am I doing
wrong?

A: By default, dnsmasq treats all the nameservers it knows about as equal: it
picks the one to use using an algorithm designed to avoid nameservers which
aren't responding. To make dnsmasq use the servers in order, give it the -o
flag. If you want some queries sent to a special server, think about using the
-S flag to give the IP address of that server, and telling dnsmasq exactly
which domains to use the server for._

------
binaryapparatus
This is also very likely to cause DNS leaks while using VPN. Which kind of
makes whole idea of VPN unreliable and/or dangerous to rely on. Not because
VPN design has anything to do with this security hole but systemd tries hard
to do stuff it doesn't really understand.

~~~
geofft
Quite the opposite. The system of expecting resolver order to be respected
(the glibc behavior) is the one that is likely cause DNS leaks while using a
VPN: if your VPN connection _or_ its DNS server is intermittently flaky, glibc
will gladly go try your next, public resolver with the internal domain name.

Meanwhile, systemd has actual functionality for scoping your VPN domain names
to your VPN's DNS server only:
[https://www.freedesktop.org/software/systemd/man/systemd.net...](https://www.freedesktop.org/software/systemd/man/systemd.network.html#Domains=)

~~~
binaryapparatus
Regardless of marketing pitch, resolver is no job for systemd. Or any other
init/pid 1 process.

Systemd is known to but in into areas that are not / should not be of concern
for it. Also known to do stuff half baked or changing things because you can.

So I'd very much love to see systemd fingers completely outside of dns
resolver, that way we wouldn't have this conversation at all.

~~~
geofft
> _Regardless of marketing pitch, resolver is no job for systemd. Or any other
> init /pid 1 process._

Correct, which is why it's not part of /bin/systemd or any other init/pid 1
process.

~~~
binaryapparatus
You know, first thing that raised some flags with systemd for me, couple of
years ago, is constant marketing push, like this one.

Having one monolith executable or ten executables that increasingly __demand
__each other is the same thing. Oh wait its name also suggests it is part of
the package, being named Systemd-resolved.

There is number of reality distortions like this one that systemd people tried
to perform, during the several past years.

~~~
geofft
I don't know what you mean by "marketing". I am mentioning technical facts;
you are saying things like "systemd tries hard to do stuff it doesn't really
understand" and "reality distortions." Can we have a technical conversation?

~~~
SomeStupidPoint
Because being technically correct while strawmanning the other person's
position for your comment to even be relevant is the worst, not best, kind of
correct. Being technically correct is only the best kind of correct if your
technical point is actually relevant.

Having a "technical conversation" would require you engaged with the person
you were replying to, not engaging in what can only be described as "technical
trolling".

For the record, the vocal "technical trolls" are why I don't like systemd,
despite having little opinion on the technical merits.

~~~
geofft
That's a fair objection, thanks. I don't _believe_ I am trolling or taking a
strawman interpretation of 'binaryapparatus's arguments, but if I am I'd
appreciate correction!

I think it is relevant that the current state of affairs (with glibc) isn't
actually better at preventing DNS leakage, and that systemd-networkd has
relevant functionality specifically to address the underlying use case in a
way that reliably blocks DNS leakage.

I also think it is relevant that the functionality is _not_ in pid 1 and
(although I didn't say it, and maybe I should have) there is no particular
obligation or desire on the part of systemd's pid 1 binary to use systemd-
resolved. They're developed in the same project, and there are some nice
things you get by using them together, but you don't have to, and you
experience no loss of functionality as compared to the status quo ante systemd
if you choose to use systemd as init plus the glibc stub resolver. (In
particular, I think that's the standard way most distros configure systemd.)

I was probably rude in my reply about how this functionality is not in pid 1,
and for that I apologize, but I don't think I was irrelevant. Maybe I don't
understand the objection to having systemd-resolved developed alongside an
init system? (Developing everything together and using common routines where
possible _is_ the traditional UNIX approach, epitomized by the current BSDs.)

~~~
SomeStupidPoint
> there is no particular obligation or desire on the part of systemd's pid 1
> binary to use systemd-resolved

> there are some nice things you get by using them together

These two statements contradict each other, and you said them one after the
other.

The accusation of trolling stems from this behavior: if you're not even being
consistent for the span of two sentences, why should I believe you're
advancing a thoughtful, genuine argument? The evidence you present (by so
flippantly contradicting yourself) is against that.

> I was probably rude in my reply about how this functionality is not in pid
> 1, and for that I apologize, but I don't think I was irrelevant.

I believe you misunderstood the objection to the two pieces of software being
coupled in your rush to correct a technical point -- it's completely
irrelevant if the two processes both run under the same pid as long as it's
the case that "there are some nice things you get by using them together".

The concern is that a process which shouldn't be coupled to the process of DNS
resolution _at all_ is injecting these "nice things" through coupling, and
that systemd is making a whole suite of such software around their init
system. Your reply, well, doesn't address that concern, and doesn't seem to
even be aware _that_ is the concern.

> Maybe I don't understand the objection to having systemd-resolved developed
> alongside an init system? (Developing everything together and using common
> routines where possible is the traditional UNIX approach, epitomized by the
> current BSDs.)

The concern is inappropriate coupling via these "nice things" that gives
systemd inappropriate sway over what non-init portions of the system are,
while degrading the overall resilience and versatility of components, and that
over time, the process to replace systemd will require replacing substantial
portions of the OS, rather than just the init system.

~~~
geofft
> _These two statements contradict each other, and you said them one after the
> other._

OK, thanks, I understand the technical disagreement now. I don't believe they
contradict. (And I'd appreciate your help in understanding this, since I
apparently am causing people to think I'm being disingenuous!)

If you use systemd the pid 1 binary by itself, with the usual glibc stub
resolver, you get a pretty good init system.

If you use systemd pid 1 along with systemd-networkd and systemd-resolved, you
get some particularly fancy features relating to tying service management to
network state (for instance, I think this combination is required to let you
reliably implement the feature where VPN DNS queries are sent to the VPN only,
by correlating the DNS server config, the VPN interface, and the VPN process).

However, if you _don 't_ want these features, you can totally use systemd pid
1 by itself. And in fact my understanding is Debian/Ubuntu-based distros
configure systemd, maybe halfheartedly configure networkd but use ifupdown and
/etc/network/interfaces, and don't configure resolved.

Therefore, I said that there is no need (obligation) for systemd, the init
system, to use systemd-resolved, nor is there any desire, in the sense that it
works better as an init system. It gives you new features _related to DNS
resolution_ , but if you're not interested in tying your init to DNS
resolution, it doesn't work any worse. (This is different from how, e.g.,
systemd works slightly worse without libnss-myhostname if your current
hostname doesn't resolve, or systemd works I think a good bit worse without
udev, or systemd doesn't work at all without cgroups, which is a design
decision that I really dislike.)

> _The concern is inappropriate coupling via these "nice things" that gives
> systemd inappropriate sway over what non-init portions of the system are_

I am somewhat sympathetic to this argument, in that it seems to be not super
straightforward to write an API-compatible drop-in replacement for systemd
components, if you happen to like systemd's design but not its implementation
(as I do).

However, the alternative is that these "nice things" simply cannot exist at
all. I don't think these are inappropriately coupled; if you want to reliably
correlate long-running processes and virtual network devices, your service
management tool has to keep track of them together. (Admittedly, this tool
doesn't need to be pid 1, which is another point of disagreement I currently
have, and one I've changed my mind on a few times. It can easily be pid 2.)
And given that the nice thing is, in my understanding, the exact thing that
was requested at the top of this thread (reliably preventing VPN DNS leakage),
it seems like it would be making an incomplete analysis to completely
disregard this tradeoff.

(Also, I _think_ that systemd's APIs are actually documented enough that you
can write the drop-in replacement if you really wanted to, and that systemd
upstream is open to documenting these APIs where they're undocumented.)

Furthermore, while I might be wrong that this is the only way for the "nice
things" to exist, I don't think that I am so wrong that this argument is in
bad faith / trolling. (Maybe I am, maybe there's something obvious I'm
missing?)

------
cesarb
Putting servers with different views of the DNS in resolv.conf seems
unreliable to me, even with glibc's behavior. If the first server "glitches"
for any reason, the process will get the response from the second server,
which might not be what you wanted. And worse, some processes (like browsers)
have their own cache of DNS responses, so the incorrect response might get
cached.

That is: it was already broken, systemd-resolvd just made the breakage more
visible. (If I understood it correctly, it only switches after the first
"glitch", so the only difference is that it doesn't switch back until the
second server also "glitches".)

------
thehardsphere
I stopped using Linux for personal before this systemd thing was an issue, but
I was considering it again now that I might get a new computer and because
I've got an app running on a pre-systemd Linux VPN. Since I have very little
first hand experience with systemd, I have some questions.

1\. Is it actually that bad for a desktop?

2\. Is it actually that bad for a server?

3\. If yes to either, how suitable are the alternatives (e.g. Gentoo, Slack,
or BSD)?

~~~
rhn_mk1
My answer may not be satisfying, as I was never much of an init guru - never
bothered much with configuration.

But here are my systemd observations:

\- log files now are binary and accessed by `journalctl` - it's a bit less
convenient than looking for text files in `/var/log`

\- unit files are a miracle. For the 7 or so years before systemd, I wrote one
unportable init script, and I hated it. Now I churn them out whenver I feel
like it - they are standardized and the docs are incredible.

Apart from that, nothing really changed from my POV, and I use systemd on
desktop as well as in small servers.

~~~
dylan-m
For the log files, one miraculous thing you can do is `sudo journalctl -f -u
httpd -u mariadb`, which watches the log for both of those services (and
should output messages in the correct order). On a desktop, the way it's
designed should make the system log viewer (if you need it) quite a bit faster
and cleverer, since the stuff we used to do with grep or with extra tooling is
just kind of built in to the format.

~~~
duskwuff
Another handy set of tools are the time-based queries, like "journalctl
--boot" for logs since the last reboot, or "journalctl --since 16:30" to see
logs since that time. (Which can also be specified in relative terms, like
"-30m" for the last half hour.)

------
mikelward
dnsmasq does something similar by default.

[http://lists.thekelleys.org.uk/pipermail/dnsmasq-
discuss/200...](http://lists.thekelleys.org.uk/pipermail/dnsmasq-
discuss/2009q3/003295.html)

~~~
mjw1007
Right, and (many years ago) they added automatic "failback", as well as the
"failover" behaviour that systemd-resolved has.

So this isn't a case of systemd-resolved deciding to arbitrarily do things in
a new and broken way; rather it's systemd-resolved still being rather immature
(but for some reason already ending up as the default on lots of people's
systems).

------
upofadown
FWIW, the OpenBSD resolve.conf(5) man page says:

>If there are multiple servers, the resolver library queries them in the order
listed.

~~~
chronid
That's the same in FreeBSD:

> (The algorithm used is to try a name server, and if the query times out,
> trythe next, until out of name servers, then repeat trying allthe name
> servers until a maximum number of retries are made).

So it seems there is a _de facto_ standard for the default libraries in some
*NIX OSes?

~~~
JdeBP
No. All that it is is common heritage in the BIND DNS client library whose
doco this is. It is a widely used DNS client library that has made it into
numerous operating system's C libraries, from Solaris through GNU libc to
OS/2\. The manual pages for that library have spread along with it.

* [https://www.isc.org/downloads/libbind/](https://www.isc.org/downloads/libbind/)

* [http://wiki.netbsd.org/individual-software-releases/netresol...](http://wiki.netbsd.org/individual-software-releases/netresolv/)

* [https://sourceware.org/git/?p=glibc.git;a=blob;f=resolv/READ...](https://sourceware.org/git/?p=glibc.git;a=blob;f=resolv/README;hb=4e8286acfa4224ac9ccfb07e90b8fd70fab1467e)

One can even trace its spread from, say, 386BSD through NetBSD to OpenBSD.

* [https://github.com/NetBSD/src/blob/516da8a02fc8bc10555ed9acd...](https://github.com/NetBSD/src/blob/516da8a02fc8bc10555ed9acde277d4ff9dcfd3b/share/man/man5/resolv.conf.5)

* [https://github.com/openbsd/src/blob/df930be708d50e9715f173ca...](https://github.com/openbsd/src/blob/df930be708d50e9715f173caa26ffe1b7599b157/share/man/man5/resolv.conf.5)

Manual pages are not standards, not specifications, but implementation
documentation. As Chuck McManis said in this very discussion, to misuse them
as standards or specifications is to mistakenly treat implementation as
architecture.

The BIND DNS client library is far from the only DNS client library in
existence. Other DNS client libraries read /etc/resolv.conf, as part of a BIND
DNS client compatibility shim or otherwise. They do not have the same
implementation details as the BIND DNS client; they do not all have the same
access patterns as the BIND DNS client. Indeed, as I have mentioned elsewhere
in this very discussion, even other tools from the same origin, namely
nslookup, read /etc/resolv.conf and have different access patterns.

* [https://cr.yp.to/djbdns/blurb/library.html](https://cr.yp.to/djbdns/blurb/library.html)

* [http://www.chiark.greenend.org.uk/~ian/adns/](http://www.chiark.greenend.org.uk/~ian/adns/)

* [http://25thandclement.com/~william/projects/dns.c.html](http://25thandclement.com/~william/projects/dns.c.html)

* [https://groups.google.com/d/msg/comp.protocols.dns.bind/B3tg...](https://groups.google.com/d/msg/comp.protocols.dns.bind/B3tgs8nb4Rg/zLkQhY7O_zkJ)

 _There is not_ a single common conventional, let alone standard, behaviour to
rely upon.

~~~
upofadown
The complaint isn't that the new behaviour is not compliant with some
standard. It is that the change is arbitrary and pointless.

None of your links contain the text "resolve.conf".

Nslookup is an end user diagnostic tool and would not be expected to conform
to any standard, even if there was one.

------
5ilv3r
I run split horizon DNS and have my local DNS server listed before my ISP's in
my DHCP server configuration. The order (to me) is important. I want to hit MY
dns server always (which does cache), unless it is unavailable, and then I
want to hit my ISPs as a fallback only. Is there a suggested way to do this?

~~~
yjftsjthsd-h
Just from this thread, several:

With dnsmasq:
[https://news.ycombinator.com/item?id=15231319](https://news.ycombinator.com/item?id=15231319)

With systemd:
[https://news.ycombinator.com/item?id=15232027](https://news.ycombinator.com/item?id=15232027)

~~~
5ilv3r
Thanks for the reply, but can you see how the very simple configuration
possible on the DHCP server suddenly becomes much much more complicated with
the above solutions? You're basically suggesting I run a chef server (or some
other central workstation management) in my home to allow talking to machines
on my local domain by hostnames.

That's tough to swallow.

~~~
comex
Is your setup actually likely to run into situations where the DNS server is
unavailable but Internet access is otherwise working? Basic home setups
usually have DNS running (if at all) on the same device (router) that’s also
acting as a gateway, so such a selective outage would be unlikely except in
the event of misconfiguration - and in that case I’d want my setup to fail
hard, personally, so I’d know to fix it, rather than having it silently slow
down DNS lookups. Thus I wouldn’t intentionally set a public DNS server as an
alternative to the primary one on the router. But maybe your setup is more
complex, or you have a use case I’m not thinking of...

~~~
5ilv3r
I run DNS on my file server since it's on all the time anyway. It must fail
gracefully or the wife approval factor goes down.

------
king_phil
I really, _really_ like Linus Torvalds idea for the kernel: "no regressions".
NO. REGRESSIONS. I have no idea what the problem of Lennart is.

~~~
gaius
The problem is the supposed adult supervision at Red Hat the corporate entity,
that is funding all of this

~~~
isostatic
Doesn't explain why Debian and derivatives followed suit.

~~~
5ilv3r
They follow freedesktop.

------
yarrel
Why does an init system do DNS lookup?

And in such a breaking way?

~~~
tetromino_
> Why does an init system do DNS lookup?

For caching and DNSSEC validation at the system level instead of needing to be
handled ad hoc by each individual framework or application.

> And in such a breaking way?

See
[https://github.com/systemd/systemd/issues/5755#issuecomment-...](https://github.com/systemd/systemd/issues/5755#issuecomment-315810471)

Systemd designers optimized their new resolver's behavior for one common use
case, but apparently never considered that it breaks another (undocumented)
use case. That seems like an example of a lack of real user testing for an
important feature before cutting a new release.

~~~
protomyth
How is it undocumented? The resolv.conf man page on Red Hat says "If there are
multiple servers, the resolver library queries them in the order listed."

------
totony
Shipping incompatible software is somehow an issue, but the bigger issue seems
to be that canonical shipped the software without testing or ensuring it didnt
break compatibility

The job of ensuring compatibility of softwares before replacing them is a
distribution's job

------
jimjimjim
remove it.

i'm more and more becoming convinced that the systemd ecosystem should be
considered a regression.

