
How to take back control of /etc/resolv.conf on Linux (2018) - kiyanwang
https://www.ctrl.blog/entry/resolvconf-tutorial.html
======
zwischenzug
Oh man, this nightmare situation drove me completely nuts when I was trying to
get DNS + VMs working with K8s test scripts.

I ended up writing this series to exorcise the demons:

[https://zwischenzugs.com/2018/06/08/anatomy-of-a-linux-
dns-l...](https://zwischenzugs.com/2018/06/08/anatomy-of-a-linux-dns-lookup-
part-i/)

[https://zwischenzugs.com/2018/06/18/anatomy-of-a-linux-
dns-l...](https://zwischenzugs.com/2018/06/18/anatomy-of-a-linux-dns-lookup-
part-ii/)

[https://zwischenzugs.com/2018/07/06/anatomy-of-a-linux-
dns-l...](https://zwischenzugs.com/2018/07/06/anatomy-of-a-linux-dns-lookup-
part-iii/)

[https://zwischenzugs.com/2018/08/06/anatomy-of-a-linux-
dns-l...](https://zwischenzugs.com/2018/08/06/anatomy-of-a-linux-dns-lookup-
part-iv/)

~~~
3np
I'm more and more leaning towards "run dnsmasq on every single machine and be
done with it", it feels like the sweet sweet spot to get flexibility and the
predictability. Otherwise, systemd-resolved is not as unpredictable and
frustrating as resolvconf or networkmanager.

~~~
vetinari
> Otherwise, systemd-resolved is not as unpredictable and frustrating as
> resolvconf or networkmanager.

NetworkManager uses dnsmasq or systemd-resolved as it's backend...

~~~
AlphaSite
Isn’t systemd-resolvd using dnsmasq under the hood?

~~~
vetinari
No, systemd-resolved does its own thing independently.

------
leephillips
Inaccurate information in the man pages doesn’t help. The one for
resolved.conf claims that, if the DNS entry is not present in its
configuration, systemd resolved will use the DNS servers in /etc/resolv.conf.
But using strace showed me that, in fact, it is ignoring that file and using
the resolver provided by my ISP (as resolved’s status output confirms). After
copying the entries (for 8.8.8.8 and a few backups) into resolved’s
configuration, it’s doing what I want it to, now.

I’ve suspected for a while, and am now pretty sure, that this system has
become so byzantine that the (incredibly talented) people assembling distros,
writing the software, and maintaining the man pages, don’t know how it works.

~~~
fao_
> I’ve suspected for a while, and am now pretty sure, that this system has
> become so byzantine that the (incredibly talented) people assembling
> distros, writing the software, and maintaining the man pages, don’t know how
> it works.

Which is... precisely what a lot of people were worried about re: systemd :(

~~~
LukeShu
As one of the people assembling distros:

If you think things are byzantine with systemd, you should have seen how
things were before!

~~~
salawat
Deterministic, and not as prone inexplicable flakiness?

~~~
LukeShu
I don't know about deterministic, but there's something to the flakiness.

The old stuff was byzantine, but it was decades old, which meant that it had
decades of people beating on it, figuring out all the ways that it could
flake, and fixing those flakes. And that's not something to sneeze at.

That doesn't make the old stuff any less byzantine. And systemd really is a
lot more coherent than most of what it replaced. But systemd doesn't have
decades of people finding all the ways it can fall over.

~~~
leephillips
I agree. My comment wasn’t a complaint about systemd per se. The old init
system was nothing to fall in love with. I think this is an improvement. The
problem is that is hasn’t “replaced” all the older mechanisms, but is one more
layer that adds to the confusing mess, with sedimentary layers of conflicting
systems, mysterious files all over the filesystem, and obsolete and
conflicting documentation.

What I’ve enjoyed in particular: systemd timers are an improvement over cron;
and I’ve been making great use of systemd-nspawn, which is very convenient and
powerful.

~~~
narwally
I agree that it is another layer of complexity, but when it works well it's a
really good layer of abstraction for user space programs to interact with the
kernel. It's only when issues arise that the extra layer brings in more
complexity. And like GP stated, that's mostly an issue of time needed to
eliminate the most serious bugs and for edge cases to be well understood and
documented.

there were plenty of growing pains at first, but the transition period for any
project looking to replace the old init system was going to be a headache no
matter what. It does take some time to grok, and it feels monolithic at first,
but I actually think that's a good thing. It's composed of many separate
binaries that individually conform to the unix philosophy, but they're all
distributed, maintained, and released together. You still have tons of
configurations options, but you can be confident that standard systemd
components are always going to play nicely with each other.

The only issue I've really had are the binary logs. If the indexes get messed
up it's a pain to access the logs even if though the actual files rarely have
issues with corruption. But it's fairly easy to forward messages to syslog if
you need.

~~~
leephillips
The problem that I can see is not that systemd and other new systems are
complex. They deal with complex problems. The problem is that the old stuff
stays around, so a distribution like Ubuntu keeps growing in complexity and
obscurity. My Ubuntu laptop and the couple of Debian servers that I take care
of have systemd, which is fine, and I’m happily using it. But they also still
have an init.d directory, are still running cron, etc., etc. It’s not
coherent. When I have to use strace to spy on my system to I can figure out
how to configure it, something’s wrong. And it’s not just the init system. My
laptop’s sound is much more reliable after I removed PulseAudio, because it
seemed to be fighting with alsa, or something else.

------
idoubtit
This post is simply wrong. Even on a plain old Debian without any of those
programs (networkmanager, ...), the file /etc/resolv.conf will probably be
written over at boot time. I don't want to draw a political comparison to the
Brexit, but before engaging in "taking back control", one should learn how the
current situation works and what the change will bring.

The main missing concept in the article is _DHCP_. Most computers use this to
get their IP and the DNS servers. For example, for as long as I can remember
the default Debian way to handle the network was _ifupdown_ , where a hook
called `dhcp` after an interface is connected, and by doing so the file
resolv.conf is updated. If resolv.conf is manually controlled, there is no
DHCP to update it, and then you have to declare your DNS servers. But the DNS
that you can query from home may not be reachable from a public wifi. And at
work, you won't have access to the intranet unless you add some private DNS
servers at the top of your resolv.conf. So a totally manual handling of
resolv.conf is a bad idea, at least for a laptop.

The article also never wonder why do all these programs try to "take control"
of resolv.conf. Some of them just try to update it when they receive DHCP data
about DNS. Other, like resolvconf, try to separate the manual part from the
dhcp-generated part (which is what "manual control" should aim to). Other
programs may add features, like a local cache or automatic knowledge of the
local network (containers and such).

~~~
asveikau
Sometimes you don't want to blindly trust dhcp. My isp is set up so that the
internet-facing host gets an IP through dhcp. I don't want to use the ISP's
dns however.

I use OpenBSD on that machine. I think it was "ignore domain-name-servers" in
dhclient.conf that fixed that for me.

~~~
Skunkleton
> I don't want to use the ISP's dns however.

Careful. Even if you point at a different dns resolver, your ISP still gets to
see this traffic, and it can still MITM it. This is not theoretical, it is
trivial. Many ISPs do it.

~~~
zamadatix
Confidentiality isn't the only form of trust, many are simply annoyed with
"features" such as wildcard catch-alls that direct you to ad/search pages
instead of saying the domain doesn't resolve.

~~~
philsnow
I use non-ISP dns servers for a variety of reasons. I've tested and found
1.1.1.1 faster than my ISP dns in the past.

Some people use e.g. opendns because they want to use that for keeping track
of what dns names are being queried from inside their networks.

I actually do use my ISP's dns for some names (particularly netflix), because
only they know the names of their internal cache boxes. I have a dnsmasq
configuration that sends queries that end in `netflix.com` to the ISP dns and
others to 8.8.8.8/1.1.1.1 or whatever.

~~~
fomine3
> I actually do use my ISP's dns for some names (particularly netflix)

Oh it looks like good setup, I'll try it.

------
NelsonMinar
This article is a great condemnation of how badly Unix networking has gone in
the last 10 years. For many, many, many years we used /etc/resolv.conf to
configure name service. Easy and simple, but not very flexible.

All the tools this article talks about turning off are trying to solve
problems. They're trying to let resolv.conf be configured with DHCP, or handle
VPNs correctly, or roam across networks. You need software to do that, not
just a static config file. The problem is every Linux distro has a _different_
tool. And those tools behave opaquely or do bizarre things like use 127.0.0.53
as the loopback address. No one built the One True Tool that got it right. And
so we have a mess that has the sysadmin throwing up their hands, disabling all
the tools, and making immutable config files just to get back to basic static
file functionality.

~~~
floatingatoll
> _And so we have a mess that has the sysadmin throwing up their hands,
> disabling all the tools, and making immutable config files just to get back
> to basic static file functionality._

I’m not a big fan of sysadmins coasting on their existing knowledge, and this
sentiment sums up what I object to precisely. Complexity has increased
somewhat since the Slackware days, and it doesn’t always seem to be for the
best, but that’s no excuse for disabling it.

Expressing this viewpoint in an interview would materially harm a sysadmin
candidate’s chances: it suggests that they are inflexible, unable to adapt to
change, and prefer to simply sweep issues under a rug rather than document
them and resolve them.

I empathize with the desire to set it all aside and go back to bare metal, but
it’s not a position I would take as much pride in.

~~~
thaumasiotes
> it doesn’t always seem to be for the best, but that’s no excuse for
> disabling it.

Seriously, this is the best possible reason to disable something. You don't
just submit to downgrades because progress is progress.

------
iforgotpassword
Or, if you're a lazy f, just recreate the file with your desired contents and
then do

    
    
        chattr +i /etc/resolv.conf
    

To make it immutable.

~~~
noja
If you're lazy, never do this. It's not very visible and you or someone else
will lose hours troubleshooting it later.

~~~
ubercow13
Troubleshooting what?

~~~
viraptor
For example why some daemon in the future takes lots of CPU spinning and
logging errors about not being able to write to a file it's configured to
manage.

~~~
ubercow13
Sound like exactly the kind of badly designed daemon I'd be trying to protect
against by making the change in the first place

~~~
viraptor
Let's be honest: Next to no service is designed to expect immutable files. At
best they're handling a situation of read-only filesystems. If they document
how configure not writing to a file and instead of doing that, the user is
marking the file immutable, I wouldn't blame the app design.

------
maccam94
I'm pretty sure that just replacing the /etc/resolv.conf symlink with a
regular file will prevent all of the major network configuration tools from
modifying it. I've verified this for resolvconf and systemd-resolved at least:

> To make the resolver use this dynamically generated resolver configuration
> file the administrator should ensure that /etc/resolv.conf is a symbolic
> link to /run/resolvconf/resolv.conf. This link is normally created on
> installation of the resolvconf package. The link is never modified by the
> resolvconf program itself.[0]

> To improve compatibility, /etc/resolv.conf is read in order to discover
> configured system DNS servers, but only if it is not a symlink... [1]

[0]:
[http://manpages.ubuntu.com/manpages/zesty/man8/resolvconf.8....](http://manpages.ubuntu.com/manpages/zesty/man8/resolvconf.8.html#contenttoc4)

[1]: [https://www.freedesktop.org/software/systemd/man/systemd-
res...](https://www.freedesktop.org/software/systemd/man/systemd-
resolved.service.html)

~~~
d2wa
No, it won't. Some of the managers for this file will overwrite it.

~~~
maccam94
NetworkManager is the one exception I'm aware of here, but I don't think any
server distros use it. Using NM is also the best option for configuring
desktop environments, so I wouldn't bother disabling its resolv.conf
management functionality.

~~~
CameronNemo
Doesn't RHEL ship with NM by default? And test on its usage?

------
jeroenhd
Resolve.conf is a blessing and a curse. On the one hand, you've got a file
that nearly every flavour of Linux will use for DNS resolution with a
standardised format. On the other hand, you've got four hundred network
managing programs that all need to be told separately that you don't want to
use them to manage the file for you because you want to set it up yourself.

Makes me wonder if the file options should be extended contain a config option
to indicate what program manages the file so that others know to keep their
hands off. Right now there's comments in there, but those comments are often
hardly machine readable.

~~~
pravus
The handling for the resolv.conf file is embedded in the libc and is larger
than Linux. This whole issue is a failure of the Linux distribution community
to come up with coherent standards for managing a simple text file on a system
using DHCP. The solutions today are horribly complicated and appear to
optimize for laptop users on simple, naive networks. I have no idea why I have
to mess with any of this garbage on my servers.

No, we need to get back to software like it was in the early 2000s when people
cared about consistent, long-term behavior instead of re-inventing everything
because laptops.

~~~
viraptor
It makes sense to me. It solves the problem for simple networks and roaming
users - so basically situations where things are expected to "just work" with
minimal knowledge. It's our (sysadmins) job to configure the system correctly
in other places. Especially in case of weird networks.

~~~
aj3
But simple networks and roaming users are perfectly fine with regular static
file. I have no idea what user base this overengineered crap is catering but
it seems equally useless both on (my) servers and laptops.

~~~
viraptor
A significant number of hotels do the login portals using their own DNS
servers, do if you ignore those, you can't use that network. Pretty much every
corp environment forces local DNS servers.

~~~
Lammy
The web portal isn't the only way on to a hotel network, just the most
convenient. The Nintendo Switch console doesn't support captive portals
either, so I've gotten in the habit of dialing the front desk any time I stay
at a hotel and asking to be transferred to the network team. You tell them you
need a device whitelisted and provide your room number, duration of stay, and
the MAC of any devices.

~~~
viraptor
This may work in a large hotel chain, but not in a b&b which licenses their it
solution from a 3rd party that never wants to speak with you.

------
vinay_ys
I recently installed Ubuntu 20.04 and configured a home gateway. I had not
done something like that in a while (3+ years). My experience was not much
different this time than any other time before as far as I remember.

It took me an hour of reading the ubuntu official documentation and a few man
pages on systemd, interfaces, networkd, resolvd, netplan etc to figure out how
stuff works on this distro.

After that I was able to simply do 'systemctl status' and look at all the
running processes, then stop and disable all the ones I didn't want.

I wanted to run unbound dns as my dnssec verifying, tls-forwarding,
blocklist/filtering resolver. I was able to do that without any difficulty –
stopped resolvd, enabled and started unbound and changed the entry in
resolv.conf. I also brought up dhcpd for my network and was able to serve that
dns server to the devices on my network.

Then I had to spend a bit of time figuring out pppd, policy routing and
nftables to do a bit of fancy dual wan routing and traffic filtering. I found
netplan to be quite good at what it does (still incomplete for more advanced
scenarios - but then so was all the previous tools similar to this).

In all this, I actually found systemd quite helpful. I think those who
complain don't take the few minutes it takes to read the man pages and
familiarize themselves with the tools on the OS.

Ofcourse things change, and you will have to learn new things. Expecting
things to stay constant is unrealistic. As long as I'm able to depend on man-
pages and official distro documentation, I'm not objecting to change.

~~~
ed25519FUUU
Then you restart, and /etc/resolv.conf is different again? Despite resolvd
being disabled(??)

------
sixhobbits
Haha I had a pi for toy projects and somehow dns kept breaking. I didn't have
time to find the root cause so I made a cronjob to overwrite the file every
minute. Maybe top 10 dirtiest things I've ever done but it worked fine for
months

------
ulzeraj
Sometimes I’m baffled that setting a static value for DNS servers is harder
than updating resolv conf itself.

Not to mention the fact that the information containing the proper steps to do
this simple task of often buried among other network configuration
information.

I think SUSE had the best approach: if file doesn’t have a certain header (a
comment stating something like “automatically created by x”) it means
management scripts shouldn’t touch it. Not sure if it’s still like that.

------
pinebox
Pretty sure I remember finding this ctrl.blog page when fighting with this
systemd-related Ubuntu bug that changed resolv.conf to use "127.0.0.53":

[https://bugs.launchpad.net/bugs/1624320](https://bugs.launchpad.net/bugs/1624320)

My hate for this whole situation (but especially the fact that it broke
resolution of short names with search domains) is bone-deep.

~~~
jrnichols
It also adds to my hate for Google Chrome..

"The primary purpose of adding 127.0.0.53 to resolv.conf is for client
software that wants to do DNS resolution by itself instead of using NSS --
most notable example is Google Chrome, and third-party software which is
statically linked (e. g. Go)."

All this mess just to work around Chrome? That's annoying.

------
duncan_bayne
To be honest, it's this sort of thing that makes me happy to be back on
FreeBSD for everything but gaming and DRM media (hi Spotify).

Sure there's a little more work to get set up (which I've scripted and put in
Gitlab, so it's easy the second and subsequent times).

But the documentation is first class (in particular, the FreeBSD Handbook is
great), and generally, when you set something up it stays that way.

Reminds me of late 90s / early 2000s GNU/Linux in that regard.

~~~
nulbyte
I love FreeBSD. Are you using it in a laptop? If so, which one? I've been
looking for a friendly laptop to check out the latest FreeBSD. It's been a
while.

~~~
blackhaz
I've been using it on X1 as a desktop driver since 9.0. Lovely setup:
[http://trafyx.com/?p=2551](http://trafyx.com/?p=2551)

OpenBSD would be even cleaner and leaner, but I needed wine, keras, theano and
a few other things it didn't offer.

------
roadst4r
Don't tools like network manager have config options to manage resolv.conf -
so you can have autoconfig with your own customisations if you wish...

This seems similar to the "don't disable selinux, learn how to use it"
shenanigans..

------
gorgoiler
What is the real world impact of every host doing their own recursive lookups?

The popular TLD nameservers would receive a lot of traffic but the amount
depends on average TTLs _in the wild_ — metrics which hopefully someone here
will be knowledgeable about.

Presumably handling the traffic is nothing compared to handling
www.google.com, but the TLDs can’t afford the same scale of infra that FAANG
can?

Several billion people resolving their favourite websites every 86400 seconds
— it’s either a lot, or one of those things the C10k folks love to show can be
handled by a Pentiun III.

~~~
zeeZ
There's been several articles and HN posts about chromium's nxdomain/intranet
detection feature and its impact on root DNS traffic that may give a hint.

[https://news.ycombinator.com/item?id=24231857](https://news.ycombinator.com/item?id=24231857)
[https://news.ycombinator.com/item?id=24270369](https://news.ycombinator.com/item?id=24270369)

------
DavideNL
I've spent hours on figuring out Dns related issues in Ubuntu and always
wonder how such a basic part of the OS can become such a huge mess...
Impossible to figure out for average (/non techies) people.

~~~
JdeBP
[https://news.ycombinator.com/item?id=23718997](https://news.ycombinator.com/item?id=23718997)
might help you to understand some of the reasons why.

------
z3t4
I like to use Linux network namespaces for fun and profit. You can for example
run a program with it's own IP/network. Network namespaces doesn't however
work with network managers, systemd-resolv et.al. because network namespaces
create separate unique mounts for /etc/resolv.conf

------
andrewshadura
This article offers wrong advice of disabling resolvconf. You shouldn't be
doing that, since it's resolvconf that acts as an arbiter of resolvconf
writes. Rdnssd in Debian, unfortunately, bypasses it, and it's a bug in the
package.

------
pbhjpbhj
Oh, I had that fun once, when Ubuntu had moved to a systemd resolver, I'd
managed to get two/three programs conflicting - think I used `lsof` to track
which programs were opening resolv.conf.

------
known
I've saved these commands for Ubuntu;

    
    
       sudo apt install resolvconf; 
       sudo vi /etc/resolvconf/resolv.conf.d/head; 
       sudo service resolvconf restart

------
jaimex2
Great article.

I installed pihole one a server once instead of running the image. It was
amazing to see the 3 way fist fight that broke out on resolv.conf

------
Angeo34
Just point resolv.conf to your pihole and then make it read only. Problem
solved

------
person_of_color
Does anyone actually like NetworkManager

~~~
moonbug
nope.

~~~
maest
It's funny that the only technically correct answer got downvoted.

You can never answer that question with 100% certainty with "yes", only with
"no".

