Hacker News new | past | comments | ask | show | jobs | submit login
How to take back control of /etc/resolv.conf on Linux (2018) (ctrl.blog)
251 points by ausjke 34 days ago | hide | past | web | favorite | 178 comments



The article and the previous discussion here do not cover any modern distribution using glibc and systemd-resolved. /etc/resolv.conf has become legacy and is not even used by most programs anymore. "Taking back control" of something that is not used will not change much...

Modern programs should use getaddrinfo(3). The glibc implementation of that call will first consult the "hosts:" line of /etc/nsswitch.conf. That file will contain a list to determine how host names are resolved:

The first entry is typically "files". That means consult /etc/hosts file.

Next there could be something to consult avahi/autoip, Ubuntu systems typically do that.

There could also be entries "myhostname" and "mymachines". That's systemd's container support, not discussed here.

If the system uses systemd-resolved there will be an entry "resolve". Than means consult systemd-resolved

Typically the last one is "dns". That's the traditional implementation using /etc/resolv.conf. But if systemd-resolved is configured and working we never get here.

Only if you don't have "resolve" before "dns" in your /etc/nsswitch.conf then /etc/resolv.conf is still relevant.

It appears to me that musl does not use /etc/nsswitch.conf at all, so there /etc/resolv.conf is still relevant.


Jesus, I can hardly blame anyone who is confused by the naming conventions there!

"files", oh so all the files get checked first, including "/etc/hosts" and "/etc/resolv.conf", right?

No, only "/etc/hosts".

Well why don't we just write it as "/etc/hosts"? Ok, whatever - next up is "myhostname"... wait, my hostname is already in the "/etc/hosts" file, why is it spelled out again?

No no, that refers to systemd's container support (don't ask).

OK... ah, here we go - "resolve", surely that refers to "/etc/resolv.conf".

Nope, that refers to "systemd-resolved".

Well then why didn't they call it that!?!


Had some fun with "it's not quite (g)libc" recently related to:

https://github.com/golang/go/issues/22846

Static binaries are fun, but bringing a libc replacement... Can be surprising.

My trouble was with split-horizon networking, and gitlab-runner (which is a nice self-contained go program). But not so nice if your firewall, dns and internal/external ips disagree. Thankfully we could remap some fw rules, so avoided the need for split-horizon host resolution via /etc/hosts (the other alternative being full split-horizon dns).


This is what annoys me to no end with Go.


Any modern distro using a systemd resolver is one meant for the kind of people that don't care about taking back control, because they will be layman users. You are of course correct that this then does not apply to them, but I would argue this is kind of an obvious point.


Debian, Ubuntu and the vast majority of their derivatives use systemd.

Fedora and Red Hat and the vast majority of their derivatives use systemd.

OpenSUSE and SUSE and the vast majority of their derivatives use systemd.

In most production environments bar custom distributions used by Google & co. (and those are a minority), systemd is used, presumably with the systemd resolver.

I'm not sure I'd call all users of commercial Linux distributions "laymen" :)


> Debian, Ubuntu and the vast majority of their derivatives use systemd.

Debian doesn't use nss-resolve(8) by default though. I'd be very surprised if RHEL or any other other distros suitable for a "production environment" were to take that step given systemd-resolved's current level of maturity.

(This is not a knock at systemd-resolved; I use it myself on most of my machines, but unless you want to spend a lot of time debugging weird resolver issues I would not recommend doing so until it has matured a bit more.


I don't want to participate in the discussion whether systemd and systemd-resolved are good or bad or for laymen or not.

My only point was if you want "to take control" of your system you first have to look /etc/nsswitch.conf. If "resolve" is before "dns" your /etc/resolv.conf is mostly unused. Any article discussing /etc/resolv.conf without mentioning /etc/nsswitch.conf is very misleading for many users who happen to have systemd-resolved (probably because the distro decided to install and configure it)


I guess the discussion is a bit heated because distros keep breaking stuff that worked 10 years ago.

It would be fine if overall reliability increased, but it does not. I could write more reliable shell scripts for making WiFi or PPP connections than NetworkManager, which sometimes required a reboot before graciously allowing to connect again. Never happens on the command line.

Thank you for describing the status quo, I'll probably look into Slackware or OpenBSD now.


My suggestion for Linux right now is Void Linux. It's clean and minimalist and eschews systemd for runit. It feels a lot like OpenBSD without ports (although the manpages are sadly Linux-like).


Actually another daemon that seems to be deeply hardwired into glibc is nscd(8). All applications I have traced recently try to open the nscd socket.

Whether nscd is still installed by any distro by default I don't know. I haven't seen it running for years.


I don’t see any point of leaving resolved behind once both NM and nsswitch are configured to ignore resolved. Disabling resolved pretty much does a half of the trick without modifying a major system config file anyway.


in recent ubuntu releases(18.04,14.04, I don't have 16.04 to check) I failed to see any 'resolve' in nsswitch.conf, so it still uses 'resolv.conf', strange.

And it is probably a bug because that missing setting seems breaking vpn at least:

https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1769016


Interesting, I didn't remember this. It's the same on my 18.04 machine. And according to etckeeper it has been like this since the installation. But systemd-resolved is running, that's not a very consistent system configuration is it? /etc/resolv.conf is a symbolic link to something generated by systemd-resolved. So in this case we can indeed say that systemd-resolved controls /etc/resolv.conf and the latter is really used by the rest of the system. No idea why anybody would intentionally use systemd-resolved this way or whether it is really just a bug. The bug report has been open for nearly a year and it looks like no Ubuntu core developer has commented. Unfortunately all too common in launchpad :(

The root cause for "resolve" not being in /etc/nsswitch.conf is libnss-resolve not being installed. It seems to be completely optional with nothing pulling it in.

In my 16.04 systemd-resolved is not even running. That's what I would have expected.


I always disable and deinstall systemd-resolved. It's unnecessarily convoluted and just always gets in the way. Adding an 127.0.0.53 alias? Why in the world is that even necessary? Dnsmasq is fine, is used by NetworkManager and can be easily set up to query a DNS cache of your choice. On servers, an unbound local resolver is a better alternative and it's used on FreeBSD.


> If the system uses systemd-resolved there will be an entry "resolve". Than means consult systemd-resolved

I gather your saying systemd-resolved uses DNS? How do I specify the DNS servers used by systemd-resolved? Thanks.


man systemd-resolved :)

Systemd-resolved has a configuration compiled in. If thats not what you want you can edit /etc/systemd/resolved.conf. It shows what has been compiled in as comments. Actually really editing any systemd configuration or unit file is typically a bad idea. Configuration changes are done preferrably by drop-ins. These are little files to be stored under /etc/systemd/resolved.conf.d/{foo,bar}.conf (Pathnames from memory, typing on my phone, but you should be able to see the naming scheme used. It's quite systematic all over systemd.)

Roughly it works like that (not sure whether distros vary):

primary DNS server address is received from DHCP at runtime

fallback DNS servers are hardwired in the config. I remember at least seeing google 8.8.8.8 there

You can change the primary DNS server not to be set by DHCP, that's probably the first thing may people want to do.


Thanks for documenting here all of this. If this would be Reddit, I'd gold you.



Just to some people know, systemd-resolved does not override /etc/resolv.conf by default. I know because neither Arch not NixOS does this, you need to explicitly override it using `ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf`.

The culprit is actually NetworkManager. When it detects that systemd-resolved is running it sets the default resolver to systemd-resolved, and automatically makes the override for the user. I think it also does the same for the other supported resolvers (unbound, dnsmasq, etc.).


To fix it, following just the first part of the article is sufficient (setting `dns=none` in NetworkManager configuration).

There is no need to remove or disable systemd-resolved, set chattr +i or anything else (of course, if you're not using you can remove it without problems).


I thought this was common knowledge. I've been doing this since the upstart days...


OT: Recently, I had to learn systemd-resolved likes to take control of port 53 too. For me, this was a problem when I wanted to run a DNS server [1] on that port, to be able to request wildcard certificates from Letsencrypt without using the API of my Domain Provider.

[1] https://github.com/joohoi/acme-dns


Doesn't it only bind to 127.0.0.53? That wouldn't get in the way of another nameserver unless you also wanted it to bind to the same address...


Yes, that is right. Binding the DNS server to the public IP address only was the workaround I choose at the time, but I had to configure it explicitly (default was 0.0.0.0).


I hear a lot of chatter about nix. What do you use it for and how is it working out for you?


Personally I really like having the whole system configuration declared and checked in to vcs.

The way packages are like functions in the nix code repo gives you at least fair confidence that things will actually work as intended.

Or at the very least it gives you insight into why it doesn’t. More so compared to a traditional deb/rpm distro in my experience.

Not saying it’s perfect by any means, just that it has worked brilliant for a few small projects of mine involving some pi’s.

At the workplace though, resistance is high amongst my traditionalist RH/centos co-workers. :)


Pretty much what @jordanbeiber described.

I recently had to change my work's notebook. Since my NixOS configuration was commited in my dotfiles, I simply restored my NixOS configuration and $HOME and everything was working like the old notebook.

Also, bugfix for strange behaviors are written in my NixOS files, so it is like NixOS is my own personal distro.


One of my pain points with Linux is that every newly installed software thinks its okay to add new fonts that "provide" for traditional ones like Arial. I hope I will find a way to change that in nix but I'm wondering if I go change that in /etc/ and repeat for all machines or I can somehow do it in nix conf


After running into this a while ago when I was rebuilding my system and getting annoyed by all the different things that try to modify it, I ended up giving up and just setting the file to immutable. It has been working fine for the past couple of months as nothing seems to autodetect the flag and try to remove it.

  chattr +i /etc/resolv.conf


I started doing this exact same thing in Debian, I think around Wheezy. Lately I’m back onSlackware and a little *BSD. Thankfully these systems still go the traditional route so it’s not much of an issue anymore.


Same.

I'm just a Linux "amateur", but what a huge mess to configure something so trivial/simple... (in Debian/Ubuntu compared to Mac/Windows.)


I've been doing this for months as well. Works beautifully, I thought I was the only one.


I have used this in the past. There are only a few places where this bites you, for instance wifi authentication portals. Some of these wifi access points change your resolv.conf so that you can load their internal page for terms and conditions.


I solved the captive portal issue by using captive browser

https://github.com/FiloSottile/captive-browser


Very cool, thanks. I will probably be using this in the future


Is there a tool that can log processes trying to modify a file?


  auditd(8)


Thank you!


In addition, if you're using dhcp, you can add an enter hook to override make_resolv_conf so /etc doesn't get cluttered with a ton of extra files.


This is usually the easiest way especially if you use a caching resolver like unbound and you expect resolv.conf to always point to localhost.


I hate systemd-resolved because it uses glib's resolver.

The problem is that this resolver doesn't understand root queries. That means that `dig +trace` doesn't work anymore, and you have to use an external server for the root zone.

This is such a trivial use case not taken into account by systemd (they don't even want to fix it), that I'm saddened by it..


1. I think you mean the "glibc" resolver, not the "glib" resolver (I do not believe that GLib contains a resolver).

2. systemd-resolved does not use glibc's resolver.

It is true that systemd-resolved does not understand root queries, and it is unfortunate that the systemd maintainers seem to be unwilling to add support https://github.com/systemd/systemd/issues/5897 . However, this has nothing to do with it using glibc's resolver.

However, as clearly documented in the systemd-resolved(8) man-page: There are 4 supported modes of operation with regard to /etc/resolv.conf, which each have different trade-offs, and are configured by changing /etc/resolve.conf itself:

(1) symlink to /run/systemd/resolve/stub-resolv.conf

(2) symlink to /usr/lib/systemd/resolv.conf

(3) symlink to /run/systemd/resolve/resolv.conf

(4) regular file

`dig +trace` should work properly in modes (3) and (4).

The big difference between those modes is "how do you want programs that don't use the system APIs (and instead do DNS themselves) to behave?". Modes (1) and (2) say "have them send their traffic through systemd-resolved", which as you note has some deficiencies; modes (3) and (4) mean "allow them to talk to the remote DNS servers directly".


Just to add, since I didn't speak about the differences between (1) and (2) or (3) and (4):

You have to use (4) if (and only if) you're going to use a resolvconf(8) implementation other than systemd-resolvconf. For example, I use dhcpcd; for kinda silly reasons dhcpcd and systemd-resolvconf don't work together, so I'm stuck using openresolv as my resolvconf(8) implementation. So I have to use (4). If you're fine using systemd-resolvconf, you should use one of the other 3 modes.

I'm honestly not sure why you would ever use (2) instead of (1). It's like (1), except search paths won't work right for programs that don't use the system APIs. (Maybe it existed before (1) did, and still exists for historical compatibility?)


Thanks for the info; I'll fix that on my machines.

Though I don't know why they would "break userspace" like that.


Probably because the main dev "knows better". Looking from the outside at common complaints across systemd, there seem to be plenty of weird defaults that hurt power users, with that being the primary reason behind not changing it to something more standard.


It's a situation where you can't do the right thing in all places.

Some programs, like statically linked Go programs, don't use the system APIs because they don't want to link against external libraries. So, if the system is configured to use mDNS for .local hostnames (either via systemd-resolved or via nss_mdns), those programs will mysteriously do the wrong thing, because they _only_ speak plain DNS. You can hack around that, and get them to do the right thing, by having a stub DNS server that they talk to, and have that stub server fire off mDNS requests instead of DNS requests as appropriate. Similar story for things like DNSSEC or DNS-over-TLS. That's what modes (1) and (2) do, they make statically-linked-programs that would otherwise use the system APIs do the right thing.

Other programs, like `dig`, don't use the system APIs because the really do care about speaking DNS directly. They just need the list of DNS servers to talk to and search suffixes to use. Giving them that is what modes (3) and (4) do.

It's literally impossible to keep both sets of programs "doing the right thing" at the same time. So the question is: which group of programs do you want to do the right thing? The systemd devs don't know (much less "know better"), so they made it configurable. Which one is default is up to the distro, not systemd.


Thank you for expressing the use cases so clearly. It is not easy to tell from the documentation.


I tried giving resolved a go, but kept seeing issues where it would suddenly start returning an error response (I wanna say servfail), would log a dnssec validation error, and refuse to work until the process was restarted.

I've since switched back to unbound with dnssec-triggerd and life is again glorious.


I had to give it up because it was crashing on every query when I had a local DNS server also configured.

coldacid 33 days ago [flagged]

systemd is such complete garbage that I immediately write off any distros that run on it as unusable. For all the fail of SysV init, at least it sticks to its own job and little else.


SysV init was so full of fail (race conditions, no dependency management, slow as hell, stupid metaphors -- run levels? really?) that people wrote Systemd. Anyone can expand on systemd to have it do stupid shit it shouldn't. But you can do that with any technology that is composable. It does not mean that the core of systemd isn't anything short of incredibly useful.


And yet Linux distros like Ubuntu around 9.0 were much more stable than these days.

With 16.04 I get core lockups (Intel microcode) and cannot switch out of X with Ctrl-Alt-F1.

NetworkManager no longer connects the mobile surfstick, so I do it manually now with pppd.

I can't claim that I had a specific problem with systemd, but I never had an issue with SysVinit either.

The overall trend however is that Linux distros accumulate crapware by RedHat's NIH developers. I'm not surprised that NetworkManager is also by RedHat.


On current Ubuntu releases gdm (login screen) now runs on VT1, which is what you are trying to switch to with Ctrl-Alt-F1. Your current session is likely on VT2 (Ctrl-Alt-F2)

You could try Ctrl-Alt-F3, F4, etc to get a login terminal.


I assumed the new network layer in ubuntu 18.04 (which replaced /etc/network/interfaces) was some pan-distro/redhat inspired solution.

Apparently not -- netplan is ubuntu originated. They don't do themselves any favours.


I was really disappointed by this myself. Netplan seems slick, but I don’t want yet another new, distributor-specific way to manage this stuff. Red Hat seems to be quite good at wrangling such tooling, but I don’t know of a solution by them/others as slick as the netplan config.


Ctrl+Alt+F1 switching is not possible unless you expose virtual terminals on the other virtual displays. Arch Linux at least does this by default, except that your first X session will start on the same display as your virtual terminal (which is good for security relative to the earlier practice of moving Xorg to VT7 while leaving an active shell on VT1).

For networking I suggest looking at ConnMan. It has a somewhat unintuitive UI, but it is lightweight, fast, and I have yet to find a wireless network that it will not let me connect to.


> and cannot switch out of X with Ctrl-Alt-F1.

You can't switch users within x either, it's been broken since at least 14.04 (according to an old close bug report I found). It works the first time but on subsequent tries you just get the lock screen.

The bug report was closed after several years of nothing.


Since some time my main Linux box won't continue booting to the graphical login manager unless I hit some random keys (any will do) on the keyboard. I guess there is some service asking interactively for something, but I can't see any prompt (maybe swallowed by systemd, who knows).

Since some time switching VTs resets all hardware accelerated applications (i.e. crashes them).

Since some time unlocking the graphical session is randomly broken and requires loginctl session-unlock from another VT (which then promptly triggers bug 2).

These things keep coming and going... it's not even Linux-specific. Stuff just breaks all the time and you shrug and move along, hoping that maybe someday it's fixed. For most of these I wouldn't even know who should receive a bug report. Maybe it's the graphical login, maybe the compositor, maybe mesa, maybe a kernel driver, maybe systemd, maybe maybe...


I have encountered this since 16.04.


I hated working with SysvInit but to be fair Debian's version of SysvInit had dependency support and parallelism and was therefore reasonably fast. The only real issue was how hard it was to write a correct script resulting in almost all of them having bugs in edge cases.

Not all SysvInit implementations were equal, which was an issue in itself.


I take it you don't work with linux professionally? I'd enjoy being a fly on the wall during the meeting where you convince your boss to run the company critical infrastructure on void linux.


Systemd is still optional in Debian, right?


Kinda. There are increasing numbers of hard dependencies on systemd components. And systemd components pull in systemd itself more often than not. So while in theory it’s optional, in practice it seems decreasingly so.

I tend to avoid systemd so I’ve taken to running Debian machines without it. But if you’re doing something like ephemeral cloud installations then it’s not “worth” the overhead of swapping out your init (and having the consequential reboot and potential instability) so the default is a required target for automation anyway. And if you’ve already set a target then why have another one?

This is why defaults can be harmful if widely deployed and considered non-optional (as systemd is on Debian). They become enforced by convention because they are the only valid common target without wrangling.


Not really. But you can always install Devuan over it at least, which gives you a choice between sysvinit and openrc (and others like runit or sinit).

https://devuan.org/os/init-freedom/

The runit init Void Linux is quite slick. I also installed an Artix runit image in a virtual machine that works nicely. We are using it as a supervisor on openSuSE piggybacked onto systemd to run critical infrastructure. It works flawlessly, it's also included in busybox but almost always never enabled.


https://lists.debian.org/debian-devel-announce/2014/08/msg00...

  For the record, the TC expects maintainers to continue to support
  the multiple available init systems in Debian.  That includes
  merging reasonable contributions, and not reverting existing
  support without a compelling reason.


no


CentOS 6 (and increasingly Oracle Unbreakable Linux 6) are alive and well in enterprise.


Systemd has serious issues that mean personally I don't ever use it, but I feel like most common distros pick it just so gnome behaves.

I'm still not sure about the complete garbage thing but sysV init definitely wasn't good enough.

busybox init if all you want are some ttys and fs and openRC are probably what you want.


Systemd means I never have to write or debug another init script never again. It gets multiple passes from me just on that.


I don’t follow this logic. I’ve written unit files and they can be very easy, but when you start looking at something like postgres on Debian and want to move it to another directory, then suddenly you’ll see all the hacks that have been employed to make it work.

Stuff like running /bin/true for the non-versioned unit, so that it can coerce the activation to take it sequentially when it actually executes the versioned unit.

There’s also the fact that “no boot is exactly the same” which means reproducing a race condition at boot is basically impossible if it’s anything other than the most trivial.

And anyway. The truth here is that you haven’t actually removed the shell script. You’ve just added boilerplate C-code around it.

If your biggest complaint is/was the composability of shell scripts for the common use-case then this is very much a solved problem in the BSDs (where an init script is usually around 5 lines).

EDIT: please come back with some concrete examples of why I'm wrong. because not only would I like to resolve my own issues; it is also the case that you shouldn't be using downvotes as a "I love systemd so I disagree" button.


And Debian has had start-stop-daemon for over 20 years and it deals with most init script faff.


I never had an issue with sysv, but there are several modern init systems such as runit and openrc that don't require selling your soul. Systemd extends tentacles throughout a system in ways that make me question how anyone ever thought it was a good idea.


So every distro except for Void and Duvian?


MX Linux does a nice job with SysV, and Artix uses OpenRC. I've tried both and liked them, though Artix broke pretty hard once during an upgrade. But yeah, it was hard to find a Linux distro that runs well without systemd.


Gentoo!

It’s a great distro, gives you loads of choice (even lets you go with systemd) and the developers actually care about open source and the act of giving a shit (unlike what I see from the systemd bug tracker).


I installed gentoo in 2012.

Still compiling it, but I'll let you know how it goes


For anyone wondering, when you install Gentoo, you untar a stage3.tar.gz into a / that you chrooted into. That tar file contains a precompiled environment and toolchain. From there most things get compiled.

For example "emerge x11-base/xorg-x11" would install X and related utilities, but you wouldn't do that. X is a dependency, what you really want is whatever desktop environment you like. So "emerge qtile" and the rest is handled. The compilation time thing isn't a big issue on any reasonable solid state hardware, and there are precompiled packages available for large popular projects (like office suites).

Install instructions: https://wiki.gentoo.org/wiki/Handbook:AMD64#


Did you make sure to triple compile the compiler for optimal speed?


If you only care about how fast the system will be, you only need to compile the compiler twice, not thrice. Probably that is his problem and why it's taken him 7 years so far. He should have compiled the compiler thrice...


I wrote scripts that build from scratch to my custom desktop configuration in < 8 hours using Packer on a single i7. I'm also toying with the idea of a buildserver that creates precompiled binaries :)


10-Year Gentoo veteran here. Every time I move away from Gentoo I come back purely because I have some headache caused by systemd or pulseaudio (which seems to depend on / be depended on by systemd). If they didn't get in the way I wouldn't care, because not needing to compile packages would be nice.

Recently I purchased some bluetooth headphones, they don't work well without PulseAudio and so far I haven't gotten anything but raw audio files to playback directly.

Same goes for Firefox. I think there's an intermediate plugin to play directly to ALSA, but for now I'm just playing spotify||netflix via chrome.

Point is, it's slowly becoming harder to avoid Systemd and PulseAudio. Because all the major distros use them, the support, development and documentation for !systemd has died off radically :(


12 years here. I finally let pulse back on my system a year or two ago when after enough upgrades (notably Firefox) my sound config went from perfectly fine for years to unreliable. At least pulse is better now. And more stable than the version I have to use on work machines (ubuntu). No dependencies on systemd required, though, that project will never be installed. OpenRC is great, and it's simple enough that it doesn't need any team of paid devs figuring out how to cram more features into it.


There is apulse this which I used successfully in the past, it’s included in the portage tree. Since I happen to like systemd itself, I don’t have much of an issue with that. I used to avoid pulseaudio and networkmanager, but on my latest installation on a laptop I just included them and they are working fine for now. We’ll see how long they survive this time, esp NetworkManager is starting to bother me again ;)


I used to have a poetteringware mask, but I had to let pulseaudio slide (but without udev and systemd (eudev works well)) because I couldn't find support for my non-standard setup anywhere for alsa.

Your point is super valid and it makes me sad, I donate/try to support as much as I can but the amount of developers that are apatheic to niche cases is increasing day by day.


I'm surprised to hear of problems with Firefox and ALSA. On all my Gentoo machines I simple compile Firefox with USE="-pulseaudio" which sets --enable-alsa within Firefox's build. Still working fine up to and including Firefox 66 directly to ALSA.


For Firefox, I recommend using sndio.


Yep.

It's never been a better time to go BSD instead.


Alpine is great, too.


Yup, it's nice but slow to boot for some reason (dhcp?). Which isn't that great when running VMs.


My go to was PCLinuxOS, though I ended up going back to kubuntu later for other reasons. Really liked it, might go back someday.


To name some big ones: Alpine, Gentoo, Slackware.


There is Devuan for Debian based setups. For everyone claiming that SysV is bad, look at the Solaris SMF for something superior to systemd.


>For everyone, look at Solaris for something superior.

I'm still sad that such a good system fell in the hands of Oracle :(


Also see "systemd-resolved does not keep the order of the DNS servers" for another irritation with systemd-resolved:

https://github.com/systemd/systemd/issues/5755

And..."systemd-resolved does not use DNS for local resolution"

https://github.com/systemd/systemd/issues/6224



I just recently came across this in Ubuntu and was astounded that it's allowed to be this way. I think this kind of behavior will become more and more prevalent once one util is allowed to do it.

It leads us on a path where daemon authors don't allow us to override the settings to suit us, rather to kept them controlled to suit them.

The arrogance sets a very bad precedent.


The arrogance has been there since the start of systemd. It's not just a comment on a particular personality, either, but rather on the general attitude as demonstrated by design decisions.

I think it helps to think of it as making Linux more like Windows, with all of the plumbing hidden from the user and difficult to tweak by the administrator. "Why would you need to, anyway? We've already taken care of that for you."


That's rich considering that I regularly use systemd override files with no issue instead of having to hack garbage init scripts filled with bugs.


hah. i've never had an easier time tweaking linux than with systemd. My first linux was I think Debian Potato (or whatever came on a cd in 1998).


You really want to do this if you use Spectrum Cable. You are locked out of configuring the DNS settings via their cable modems (even if you supply your own). These force a DNS search suffix that leaks all DNS requests to their server, even if you are using another public DNS. I noticed network manager kept forcing the DNS search suffix, even after I manually disabled it. I did the config change to disable it messing with the resolv.conf


You can easily configure Network Manager to ignore anything from the DHCP, including DNS (properties ipv4.ignore-auto-dns and ipv4.dns* on your connection).


Sound like a good reason for DoH


DoH cuts both ways, so be sure you know what are wishing for.

Yes, it allows you to prevent your ISP manipulating your DNS. Your ISP has no way to know when you are resolving, because it is masked in other HTTPS traffic.

But it also allows the apps to prevent you from manipulating their DNS. You don't know when an app is ignoring the resolver you configured system-wide, because it is masked in its HTTPS traffic.

There is a worrying trend that apps (browsers especially) are ignoring whatever you configured in your system, and are becoming basically a blackbox outside your control with a wide open connectivity to the Internet. No explanation needed, what that means for any privacy left.


You should put your own router between their modem and your network, and not rely on DHCP from their modem for any of your network configuration.


You usually need to use DHCP to get your IP address, but any decent router will let you pick your preferred DNS server.


Worst case scenario is your first hop downstream from the cable modem does port 53 interception and redirects.

However if the only ISP you can get is so hostile, the solution is to tunnel (IPSEC/SSTP/whatever works) all your traffic to a non-hostile network.


Use a firewall with nat to redirect all dns traffic to a DoT or DoH dns proxy in your network.

That way you dont have to tunnel all your traffic. (Though technically you could also use the tunnel for only DNS, but its not much easier than the solution above if you want this to apply to all your devices)


If they’re intercepting and changing your dns packets, what else are they doing? At the very least you can assume port 80 is unsafe, and should be tunnelled. SNI as a privacy problem too, so forward 443.


You can tell your dhcp client to ignore the DNS setting sent by the server.


My previous ISP actually just blocked all DNS traffic that wasn't to/from their servers. I had to tunnel my DNS traffic (ridiculous!)...


That's a little ridiculous. I've read[0] that Chromecasts, which are fairly common, won't work if Google DNS is unreachable (also by itself an issue), so I'm surprised they didn't get a lot of complaints.

0. https://news.ycombinator.com/item?id=19170671


I use a custom resolv.conf. Really, the biggest problem is airports: some want you to go through a captive gateway, but the captive gateway's domain is not publicly registered, so if you're using 8.8.8.8 or similar, it fails to resolve. Leads to an annoying trip to resolv.conf (well, resolvconf.conf in my case) to tell it to use the DHCP settings, run through the portal, and then flip back.

(And like, is it really that hard to register your captive portal's name globally, and just let me use a global resolver to get to it? IDK if they're concerned about DNS tunneling, but this happens even on "free" WiFi (where free might mean "watch this ad", which honestly is fine w/ me.))


> Really, the biggest problem is airports: some want you to go through a captive gateway, but the captive gateway's domain is not publicly registered, so if you're using 8.8.8.8 or similar, it fails to resolve.

Ideally, the software detecting captive portals on a network and presenting their login page should also use the network-provided DNS server solely to resolve the captive portal domain if it doesn't resolve with the preferred resolver. That would solve this case.

(Also, ideally, captive portals would serve up a URL using DHCP options and not capture traffic.)


The people that run these network shave no idea what global resolving and DNS tunneling are.

The people that build the systems that provide these networks likely don't know what DNS tunneling is.


I saw another user here in this comments section reference a project which might be useful to you:

https://news.ycombinator.com/item?id=19436783


On my domain controllers (where I absolutely cannot have anyone touching the conf), I just chattr +i /etc/resolv.conf.

That prevents anything, even root, from changing the file, unless chattr -i is called first.


Wow, I wasn't aware of the chattr command at all. Here is a list of available: https://en.wikipedia.org/wiki/Chattr

Apparently, they are very file system specific. That's why they are not widely used. I guess a chmod 0444 would not do it, as system tools tend to "fix" wrong permissions automatically (but not permissive attributes).


I use `chattr +i` with NFS mountpoints for backups, it prevents the local harddrive from filling up when a network share fails to mount on boot.


Totally. If something is meant to be a mount point, a chattr +i on the underlying directory is a godsend, because it gives you an immediate error in case the mount should fail for whatever reason.


You might be able to do it in a more system-agnostic manner with SELinux.


That would be less system-agnostic because you have to be running with SELinux as the LSM. The immutable attribute is supported on almost all filesystems.


I miss the days when I didn't have to worry about outside programs changing my configs without my knowledge.


Isn't it great to have the choice wether some tool manages the system or one does it manually? Thanks GNU/Linux, thanks all the Linux distributions maintained by hundreds of volunteers!


I got the opposite out of this post; not only do RH/Cent use hellish network middlemen, but other distros have their own fresh hells. Is there really a choice that doesn't commandeer your configs?

Standard practice for me was to rip out every last shred of NetworkManager on every fresh build, but I didn't realize how many other utilities broke the resolver config as well.


What's wrong with Network Manager? I've yet to find a better tool for managing wired, wireless, vpn, and mobile broadband anywhere as well as NM. Sticking with the theme of the original post, NM combined with unbound and dnssec-triggerd is downright amazing. Your vpn connection passes down a nameserver and search zone? Bam, NM pushes that into unbound, and now queries for your vpn domain go to the vpn resolvers, and your other queries go out to whatever you set for your default resolver.

To be fair, in that particular config, resolv.conf never changes as it always points to loopback, with your preferred nameservers only existing in memory in unbound, and in network manager's config.


>What's wrong with Network Manager?

It's opaque and hard to debug. It may have utility on laptops, but on servers, it is absolutely counterproductive. I could not for the life of me figure out what it was doing w.r.t ipv6 prefix delegation and how it was dealing with dhclient6 internally. My leases would expire but not renew. Eventually, I had to rip it out and create simple configs by hand that work well.


I'll grant you that it's extremely different from static network configs, but it's far from opaque. If anything having all the logging under NM can make troubleshooting a pleasure `journalctl -f -u NetworkManager` is a godsend.

I'll also agree that the benefits on a server are next to non-existant, but at the same time it's just a default that's trivial to turn off, and I could probably count on one hand the number of times it's bitten me while building and deploying tens of thousands of hosts over the last 15 years.

It's a tool I've learned to love on my workstations, and rarely even notice on production systems.


You don't notice until it bites you. And trust me, I spent a lot of time debugging the ipv6 issues to no avail. It may be in general the whole ipv6 ecosystem is just not as mature, but the end result is that user experience suffers.


Disagree. I've "disabled" it in policy countless times, and been bitten in the ass by it re-enabling itself countless times. The only thing that works is deleting it entirely. After the umpteenth troubleshooting session, only to find out "oh, it's that thing again? That thing we keep trying to make go away?" I'm not going to spend one more minute trying to figure out how I could keep from hurting its feelings.


It sounds like you don't understand the tools you use, and have no interest in learning them. NetwokManager (just like any other daemon) cannot "re-enable" itself. It's possible if you were on an Debian derived host that package post-install scripts enable the service (as is the case with all services on Debian derived distros due to packaging policies. The right way forward here is to indeed remove the package if you aren't using NM. Or better yet, get a better grip on the packages you install in the first place, and just don't install NM to begin with.

If you're on a RH or Arch derived distro, policy is just the opposite, and if the service is ever magically enabled (aside from Anaconda enabling it after the package was selected at install time), it's a massive bug (I can't find any such bug report in Arch or Fedora).


No, it's just that, for the purpose of an internet forum comment, I don't wish to revisit years of regular and maddening problems across fleets of thousands of servers. Over the years they may be subject to different initial build and deployment regimes, to say nothing of different OS versions that may have been released over that time, and various Puppet manifests that may have been changed over that time.

When something is regularly found to be the problem, and has never made my life any easier, it gets yanked. Sometimes I may have had time to analyze further, other times it may have been an emergency. Regardless, it's been at least 1.5 years since I worked in that environment, and my memory has faded. In the real world, we don't have infinite time to analyze infinite failures.

Here's what I remember:

NetworkManager = problems.

No NetworkManager = no problems.

You may care to look down on people who don't always have the time to analyze every single occurrence of every single problem to the nth degree, and subsequently catalogue the exact cause and fix for reference when posting on forums in the future, and that's fine.

My opinion is, it sucks, and I don't want it.

If you like it, and enjoy its benefits, please do. I'm not denigrating you for your choice.


systemd's dependency logic about what is enabled or disabled is not straightforward. For example, I disable buetooth on general principle, and yet it gets enabled (as in activated) in some scenarios due to other stuff depending on it (i think something in gnome does). "systemctl is-enabled foo.service" is not a guarantee about anything. Something else can still start the service without the user's authorization.


> enabled (as in activated)

Not understanding the difference between these is likely part of the problem here.


It's not really the user's fault though. If you really want to make sure that something else doesn't start it, you need to mask the service. But then, you can't start it either. What you really need is "disabled unless I start it". I don't think there is a state like that.


Opensuse uses wicked for servers. It's nice and more suitable.


> What's wrong with Network Manager? I've yet to find a better tool for managing wired, wireless, vpn, and mobile broadband anywhere as well as NM.

Lovely. I have between one and two machines that need this. I have 1289 machines that don't.


I don't use these things. I run servers. It does nothing but break stuff, and it keeps butting in like that annoying acquaintance at parties. It's very difficult to make go away short of uninstalling every last vestige of it.


I'm on Debian stable and there's no systemd-resolved or NetworkManager running.

When I'm on a DHCP network dhclient runs and manages my resolv.conf as expected.

We're still entirely in control in GNU/Linux, if you assert control. It turns out however that most people actually want the distros to do all the magic and for things to Just Work.


Choice is "do you want to <x> or <y>" not "if you don't want <x> figure out how to implement <y> on your own". That's simply known as "doing the work yourself" and is largely possible on even closed sourced systems assuming you have rights on the box.

There is nothing wrong with that but it shouldn't be conflated as choice and paraded as success of such.


The GP is referring to all the other Linux distributions who really give you <y> (or <z> or <Þ>), not to implementing <y> on your own.

There are distributions (e.g. Linux from Scratch) for when the option you want is the NULL option—i.e., doing things yourself. But there are N other managed options, not just one managed option and NULL.


Ah yes, the strawman disto which matches exactly what <user> needs and is the reason <problem> isn't actually an issue since they can just switch to strawman distro!

Unfortunately strawman distro is never real and the problems still exist after congratulating the ability to switch to it.


You mean the choice in a way like in Windows when I need to uninstall/disable a bunch of crap to get a usable system? Totally.


Well, you chose to install it, you can configure it not to do this. For the vast majority of people, myself included, NetworkManager should have control of DNS resolvers (at least, when I have WiFi on a system. I usually don't use NetworkManager on Ethernet-only systems, since there's systemd-networkd).


Absolutely. My laptop travels across a lot of networks, wifi and ethernet, with and without OpenVPN and Mobile IPSEC. I tether to my phone. I have some IPv6 networks to worry about. Firewalld is expected to do the right thing as well. I have loads of NM profiles, one of which has several RFC1918 addresses for covering most bases when starting out with routers and switches.


Along these lines, netplan in Ubuntu is seriously confusing. That's the set of scripts that rewrites most of your network config in /etc with stuff generated from some new set of YAML config files. I'm willing to believe netplan is a Better Way to configure networking, but the kludge of having it rewrite long-understood config files in /etc is really awful.


I tried to get on baord with netplan, I really did. I thought it was a redhat inspired systemd-esque invasion.

Turns out it's an 'upstart' type ubuntu crapfest that's unlikely to survive as long as 20.04.

I wouldn't mind, but the very first machine I tried to configure with it was an ntop machine, I wanted 2 interfaces to come up with no IPs attached. Turns out it's not possible.


I ran into Netplan with an 18.04 server install and the whole time I was asking myself why this was better than the old system.


This seems backwards.

I don't feel strongly about networkd or NetworkManager either way, but I am curious on the meta-debate. I am personally a fan of the modernization of the linux userspace in the last 10 years (systemd, wayland, networkmanager), but I also understand that the puzzle pieces don't always fit together. However the argument against new things always seems to come down to "not what I am used to".

Other than tradition what is wrong with explicitly configuring NetworkManager to configure /etc/resolv.conf "my way" instead of going behind it's back. Does it not have a "DNS Server: manual" option?


I'm only a bystander in this fight but this is dishonest:

> However the argument against new things always seems to come down to "not what I am used to".

For me it's always: I can't debug this. I remember installing RHEL 7.0 just after release and nmtui crashing on me with some weird problem in glib when setting a hostname using it. I also remember trying to debug openvpn in network manager (https://i.imgur.com/zbaRcST.mp4) and both of these are beaten paths...

I'm able to somehow get around most shell scripts but when something crashes in C land on a customer box - what a PITA! It's not impossible but the amount of work I have to invest is just sooo huge. I remember yelling at reddit about some gnome stuff - and I've got a link to several powerpoint presentations... give me technical documentation not powerpoints and give me tools to troubleshoot that shit. No strace should be a last resort and not your goto tool: https://blog.uberspace.de/systemd-error-getting-authority-ko... (link in german, but google translate should work)


Having used Unix since 1989 and basically, everything related to Unix admin changes every 10 years. I'm tired of it. And I relish the fact that in twenty years, this year's set of hipster devop admins calling me a neckbeard will be screaming "get off my lawn" at the constant arbitrary changes this industry seems love.


Good point.

> this is dishonest

Of course it is, I was trying to trigger a response :)


Removing `systemd-resolved` is often enough, but like many, `chattr +i /etc/resolv.conf` is also the next trick I do.


I think NetworkManager only exists because wpa supplicant is difficult to use. I loathe networkmanager.


It exists because it's nice to have a unified place to define all your network connections and their behavior, instead of having them spread across wpa_supplicant, pppd, mbim-network, dhcpcd, openvpn, vpnc, freeswan, iptables, and ifupdown.


Try iwd which replaces wpa supplicant. It makes wifi connections a breeze.

https://git.kernel.org/pub/scm/network/wireless/iwd.git/


NetworkManager is convenient, especially for bringing up the kinds of bridged connections, virtual NATs, etc. that ordinary power users often want but don't believe they can achieve. And of course, you have every option not to use it, or to improve it, or to propose your own replacement.


at least for me, configuring vpns on network manager is a LOT simpler than doing by config files.


Once you understand it wpa_supplicant is way more user friendly than other tools.

Most of the other ones just say "stuff is broken too bad :(" when a connection fails. wpa_supplicant tells you everything that's wrong and not only that, usually gives you an easy way to fix it.


How is designing for laptop networking for a system predominantly used as a server with marginal desktop market share justified. Why not roll out something discrete for the negligible percent of laptop users instead of imposing configuration overhead on all users?

Debian prides itself on stability but voted for an 'init' that is rapidly changing with unlimited scope outside the purview of Debian. Thus we have a lot of extremely questionable networking functionality like 'predictable network names' and more just rolled out and 'imposed' on everyone without discussion, debate or technical scrutiny by a single group.

Alpine Linux servers and VMs boot in seconds and are easy to configure and use. Void is equally fast for desktop users. The real problem seems to be a lot of developers are unfamiliar with the shell and its extensive use in everything from the system to the apps ecosystem. And we are left with a situation where the shell is demonized by people who are not familiar with it.


It miss just having to "know" about /etc/resolv.conf for DNS. Now (years later) I feel I always have to google magic-recipes to get my DNS to work :(


I always edited this file than used

  chattr +i /etc/resolv.conf 
to prevent further modifications.


Does this need to be updated, or is what's in here relevant for the latest round of Ubuntu changes?


It’s up to date.


We had a great plan somewhere around Ubuntu intrepid or jaunty that rather than have apps rewrite /etc/resolve.conf, we'd leave it as a static file and have a "nameserver dynamic" type config in there.

ie. sensible default for many people, but super easy to override


Hands down the most annoying defaults on Linux.

Being an OS for nerds, I'm surprised NetworkManager doesn't handle for manual updates to resolv.conf


NetworkManager generates resolv.conf from the information associated with an arbitrary number of network connections. Which, or how many, of these N network connections should manual updates be mapped to?

Edit: now I notice a sibling comment already asked almost the same question, but it was greyed out.


If you manually change something, how is it supposed to handle it? Notify the user and ask them to accept the change? But NetworkManager changes the file based on each network. So it'd have to ask you every time you connected to any network. And what do you do until the user accepts the change? Default to nothing? That would break any headless box where resolv gets accidentally overwritten.

The simple solution is, make your manual overrides through NetworkManager, or don't use it.


NM doesn't always change resolv.conf. This entirely depends on the dns backend. If it's set to use dnsmasq, systemd-resolved or unbound as a dns backend, it never touches the file, instead passing nameservers in to the stub resolver dynamically.

https://wiki.gnome.org/Projects/NetworkManager/DNS


I laughed at the title. I've been surprised every time I open resolv.conf lately it says "do not edit!".


Using the immutable flag, of course.


it's only recently that the updates on ubuntu and other systemd'd linux distributions have been frequently writing resolv.conf.

the suspicion that a user is so incapable of having already set DNS servers is so strong that whatever-is-doing-this doesn't even bother to ask if or check whether the resolv.conf file is different from default.

this seems a little too critical of, possibly insulting to the intelligence of, people who use these versions of linux.


Alpine for the rescue. Seriously, there must be something super broken with Linux if this article exists.


It's not Linux that's broken, it's arrogant developers of system daemons that are broken.


Alpine is definitely the most sane distro at the moment. I just can't recommend it to my friends because I find myself having to compile anything that isn't super common, and of course nothing that needs glibc works (which, is acceptable to me because I do prefer the smaller libc but some people might really dislike it.)


> I find myself having to compile anything that isn't super common

Yes well, that's what happens when package managers, with their dependence on maintainers and whatnot, are your culturally lauded software distribution method.


Not Linux, mostly systemd.


systemd :')




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: