> My main gripe is that the files under /etc/rc.d/ are immutable scripts.
Are they though? A lot of, well at least old software, lacked configuration files, their command line flags and unit scripts were for all intensive purposes, their config files.
Whether this actually makes upgrade harder (the next point made), is kind of debatable no?
Arch does fine with pacsave and pacnew files.
Making changes or tracking changes to these files is going to be required regardless from a systems upgrade point of view - because people want control over their system units.
And for what its worth, I'd rather find out about a breaking update via pacsave/new than figure it out months later when my copied but now outdated and untracked unit starts failing.
Geoffrey Pullum, who came up with the term "eggcorn", also came up with "snowclone" [0]. He has had a lot of very interesting posts on Language Log [1].
Language Log is brilliant. Perfect morning reading with my coffee. “Orca emits speech-like sound; reporters go insane“ says more about our culture and media than anything I’ve read for years. Thanks for posting the link.
"Intents and purposes" has a meaning on its own word-by-word. Not so much with the eggcorn. What is "intensive" about the "purpose" in "intensive purposes"?
> Are they though? A lot of, well at least old software, lacked configuration files, their command line flags and unit scripts were for all intensive purposes, their config files.
Most of the article is dedicated to explaining how modern BSDs changed from these origin and how system owners are very much expected not to edit these scripts so… yes, they are?
> Whether this actually makes upgrade harder (the next point made), is kind of debatable no?
Not it’s not? The article explains exactly how: BSD upgrade process diffs and merged every config file, which includes core rc scripts which the system owner would or should not have edited. So said system owner finds themselves spammed with every upstream update to scripts they never touched, none of which would be relevant to them.
If you argue that this is a good thing, then surely they should also be spammed with upstream updates to every binary, utility or script these use?
> Arch does fine with pacsave and pacnew files.
The article is not about arch?
> Making changes or tracking changes to these files is going to be required
No.
> And for what its worth, I'd rather find out about a breaking update via pacsave/new than figure it out months later when my copied but now outdated and untracked unit starts failing.
Does pacsave/pacnew force you to review every built-in systemd unit?
The claim is that the Arch pacman mechanism (and the dpkg one, and the rpm one, and ...) only prompts you when there is an actual merge, and when it does that, it's actually fine and preferable for the package manager to leave old and new versions on disk and ask the sysadmin to resolve the merge by hand, and this works in practice. It does not ask you to review every change to /etc. This is why Arch is mentioned, as an example of a system where this situation works fine. (Debian and Fedora are also good examples.)
If the BSDs' upgrade procedure cannot distinguish between changes to /etc where the local sysadmin has also modified the file and changes to /etc where the file is unmodified, and they prompt you to review every single change to /etc, that seems like an easily fixable missing feature.
Do the BSDs really prompt you to accept every change to /etc on upgrade for files you didn't modify, such as sshd.conf or services?
I would argue that /etc/rc.d should be moved to /lib/exec and then use /etc/defaults files to load command line flags from environment variables. This separates the core of the service file you are unlikely to touch ever from the parts you want to touch and makes upgrades easier. It would also avoid or atleast minimize pacnew/pacsave files.
The headlined article is not about Arch, and already pointed out the existence of /etc/rc.conf (albeit that there are actually a couple of other files) containing the variable settings that configure the scripts, as documented on the rc.conf manual page.
Okay, to elaborate, a solution like the one proposed in the article, if implemented on ArchLinux, would prevent or minimize pacnew/pacsave. That doesn't make the rest of the comment irrelevant because pacnew/pacsave are specific to arch, hence that very sentence being only specific to arch. I mentioned them mostly because the top comment in this thread also mentioned them briefly.
All files that the distributor ships as configured ought to be put into /lib or /usr. Any system specific configuration belongs to /etc, but ONLY that configuration.
On a fresh install of BSD/Linux I'd love to see an empty /etc directory sans any necessary overrides to make it boot.
> On a fresh install of BSD/Linux I'd love to see an empty /etc directory sans any necessary overrides to make it boot.
And I rather have a full set of configuration files that specifies the defaults at the time of the installation so my system even boots after an update in which some program changed its hardcoded defaults.
> Makes backing up that directory a lot easier too.
I don't really understand what the problem with backing up 0.1 or 1.0MB is.
Well, the specified defaults are in a folder in /lib or /usr. You can find them there and copy them to /etc or just write what you change.
The issue is not how much is backed up but what.
If /etc only represents changes to system defaults, then the backup represents a list of my changes and it's always easily visible from each full snapshot what changes were active in the system at that time.
If /etc contains all configuration, then from a single backup snapshot, you cannot determine what changes where made to the system.
This would be an issue if you're trying to recover the backup to a freshly installed system, for example, because you cannot tell what changes you have made vs what changes occured in the backup due to updates or upstream due to updates since the backup.
It's a semantic difference that can make restoring backups easier and replicable across different systems.
a) Not all programs can read multiple configuration files. It's either in the one configuration file or a hardcoded default.
b) Not all settings done in defaults can be simply undone later on (e.g.: apache's httpd.conf, IIRC the same for nginx - good luck getting this default listener down again)
c) If you want to infer which changes were made by you or by the system's update procedures, explicitly creating snapshots before every manual or automatic modifications would be the way to go. Preferably one after each as well, so you have something to verify the integrity of a system against in case of security issues.
a) and b) are just issues with common software being idiotic about configuration and insisting one a "one truth" central config file.
c) As mentioned above, inferring changes requires two snapshots, if /etc only contains changes a single snapshot allows me to infer changes. And I can verify the integrity just as well using an "after" snapshot. Just means one less snapshot is involved, which is reduced complexity.
I'd note that Tru64 unix kept its rc.d in /sbin instead of /etc.
I'd also note that the first time I had to debug a Tru64 networking problem it took me quite some time to find the bloody things.
I mean, once I did my first thought was "honestly if anything that makes more sense than /etc would".
But argh.
Then in the end it turned out the network configuration had been fine all along and one of the developers had stolen the switch port it was using for something else.
(being a sysadmin at a software house tends to involve the developers getting to do wtf they want since their work is revenue critical and yours isn't perceived to be so much, though I still mostly enjoyed the job because my director was understanding and let me yell at developers for doing stupid shit so long as he didn't get too many complaints ;)
I'm never a fan of mixing runscripts that come with the package and runscripts written by the user. If rc.d is a libexec, it enables the perfect solution - /usr/local. It even allows one to potentially override a runscript in the most natural way, too bad it never gained the traction.
I'm a little confused why the author is getting merge conflicts in /etc/rc.d one paragraph after asserting that these files don't need to be edited by the local system administrator.
Plenty of files that properly belong in /etc get edited rarely by the local sysadmin, perhaps more rarely than rc.d. (When was the last time you edited /etc/pam.d/ or /etc/gai.conf or /etc/protocols?) Yet they can be edited, and so they properly belong in /etc. If the local sysadmin doesn't actually touch them, an upgrade procedure should just upgrade them without prompting. (I assume the BSDs are clever enough for this.)
So, if you're getting prompted, it's not just because there's a change to review, it's because there's a merge to review, and therefore it's because there was a local modification, and therefore clearly either you or some other sysadmin did in fact edit that file. It is of course worth making it no longer necessary to edit it, such as by moving configuration to /etc/rc.conf for /etc/default/ or a service-specific /etc/foobard.conf, but that change needs to actually happen before you can call the rc script immutable, and once you do make that change, you no longer care if the rc script moves to libexec because you're no longer getting merge prompts.
> So, if you're getting prompted, it's not just because there's a change to review, it's because there's a merge to review, and therefore it's because there was a local modification, and therefore clearly either you or some other sysadmin did in fact edit that file.
That's the way it should be, but while upgrading Debian I have very often seen prompts for files that were never touched locally.
You may have seen them for files that were automatically edited in some fashion (e.g., by another package's install script). IMO needing to do that is a low-grade bug.
When you run etcupdate on NetBSD, the tool will diff your current /etc contents against the new versions shipped in the source tree or in the etc.txz distribution file -- and it will ask you to manually review every difference.
It doesn't matter if you have edited a file /etc by hand or not. If the currently-installed file differs from the new version in any way, you'll be asked about it.
>My main gripe is that the files under /etc/rc.d/ are immutable scripts.
If the files/config (I know the author argues they aren't config files) are truely immutable why would the system upgrade command modify them? and how would it as if they are immutable in the same way that running chattr +i somefile on Linux will make a file immutable even to root then I don't understand how the upgrade command is modifying them.
Systemd-d is growing on me, partly because it is being forced upon us but also because once you get over the initial carpet pull from beneath you, you don't need to be exposed to the extra bullshit it brings.
If you want to experience pain with System-d, install a fresh copy of Ubuntu and try to setup /etc/resolv.conf, for an extra challenge try setup /etc/resolv.conf using unbound as a stub resolver.
I was amazed at how much effort I had to spend fighting the OS to just say "Stop managing this file, do not load this file, I am going to manage it"
I spend most of my time with Linux systems so I admire the simplicity of the System V approach. It is interesting to see other people discuss this from a BSD perspective as I have very little exposure to BSD like systems.
BSD seems great, I love reading the security.html page of the OpenBSD project, it has so many great ideas + implementations.
They are not immutable in the 'chflags uchg,schg ' sense.
Updating these files is not in any way shape or form a hassle anymore if you do not update them. etcupdate has that solved. Even mergemaster has specific options to handle unchanged /etc that only diffs in svn-id tags and similar.
But in my opinion the basic premise of the article is false. mergemaster and etcupdate with their diff and 3-way merge capabilities are there because these files are assumed to have local edits.
The startup procedure, scripts, options are expected to be customized; thus the update procedure has handling to preserve local edits.
At least with Ubuntu server it is pretty easy. The system generated /etc/resolv.conf file even tells you how to disable system control in the comments of the file.
> If the files/config (I know the author argues they aren't config files) are truely immutable why would the system upgrade command modify them?
You are confusing the two different scenarios that you need to distinguish.
The first scenario is when a config file is simply a part of a package, managed by the package manager. In any reasonable package manager, if you ever edited a config file manually under /etc, the package manager will never attempt to update it automatically. For example, apt-get will ask you whether you need to preserve the old file, use the new file, or diff-and-edit it manually. This is the reason why the author suggests to move rc.d from /etc to /libexec, because every time you upgrade your system, the package manager will ask you to diff-and-update some random runscripts that you've never written.
The author's point here is that the rc files do not behave like config files, they are closer to programs or executables, the package manager should be able to update them when a new version in installed without annoying the user, just like any other binaries.
---
The second scenario is when a config file is managed by the init system, a script, or a daemon in the system. For example, /boot/grub.cfg is managed by update-grub & os-prober, and ideally one should never edit the file manually.
In this scenario, it's completely off-topic and irrelevant to the original topic of whether rc.d should be treated as libexec, but here's my opinion anyway...
> I was amazed at how much effort I had to spend fighting the OS to just say "Stop managing this file, do not load this file, I am going to manage it"
Never attempt to fight with the system in a brute force manner - edit this file, later the file is edited back, edit again, edit back, and finally "chattr +i" the file, I never understand this mentality. Using "chattr +i" is almost never the correct solution.
All you need to do is politely tell your intention to the daemon in charge, spending 30 minutes (or each time after a system upgrade) to fight with it, can easily save your 5 minutes to read the documentation.
> for an extra challenge try setup /etc/resolv.conf using unbound as a stub resolver.
If you read the documentation, you'll sometimes find your needs are already supported, and managing it manually is not needed in the first place. For example, NetworkManager already comes with builtin, automatic support of a stub resolver. Not only that, it supports both dnsmasq and unbound, and even has extra logic to help you updating the DNS when you're connected to a VPN. All of these features are here to make one's work easier.
Yet one still chooses to fight with the system in the most brute force way such as "chattr +i", at the very least you can turn it off smoothly and it'll never bother you again.
For example,
$ man NetworkManager.conf
> dns
> Set the DNS processing mode.
> default: NetworkManager will update /etc/resolv.conf to reflect the nameservers provided by currently active connections.
> dnsmasq: NetworkManager will run dnsmasq as a local caching nameserver, using "Conditional Forwarding" if you are connected to a VPN, and then update resolv.conf to point to the local nameserver. It is possible to pass custom options to the dnsmasq instance by adding them to files in the "/etc/NetworkManager/dnsmasq.d/" directory. Note that when multiple upstream servers are available, dnsmasq will initially contact them in parallel and then use the fastest to respond, probing again other servers after some time. This behavior can be modified passing the 'all-servers' or 'strict-order' options to dnsmasq (see the manual page for more details).
> systemd-resolved: NetworkManager will push the DNS configuration to systemd-resolved
> unbound: NetworkManager will talk to unbound and dnssec-triggerd, using "Conditional Forwarding" with DNSSEC support. /etc/resolv.conf will be managed by dnssec-trigger daemon.
> none: NetworkManager will not modify resolv.conf. This implies rc-manager unmanaged.
$ man systemd-networked
> by one of the .network files will be ignored. It is also possible to
> explicitly tell systemd-networkd to ignore a link by using
> Unmanaged=yes option, see systemd.network(5).
I guess one reason that everyone's fighting with the system is not realizing the official way of telling one's intention to the system. Perhaps everyone should learn the way GRUB warns its user.
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub2-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#
Perhaps someone should contribute a patch to NetworkManager to add a similar warning.
# DO NOT EDIT THIS FILE. This file is currently under the management
# of NetworkManager. It's automatically generated according to the
# DNS configuration of the current network connection in NetworkManager.
# Any change to this file will be overwritten.
#
# If you are trying to change your DNS resolver per connection, use "nmtui", "nmcli",
# or alternatively, edit "/etc/NetworkManager/system-connections/".
#
# To stop NetworkManager from managing it altogether, edit
# /etc/NetworkManager.conf, set the following and
# restart NetworkManager, before you edit /etc/resolve.conf.
#
# [main]
# dns=none
#
# If you are trying to set up a stub DNS resolver, it's recommended to
# NetworkManager's builtin support. Set "dns=dnsmasq" or "dns=unbound"
#
# See "man 5 NetworkManager.conf" for more details.
The answer is maybe to auto merge what has not changed locally? To me, changes to the boot process is just as interesting as changes to the default shell settings.
The distinction between configuration and code isn't what's important here. The purpose of these files are to describe a system wide default state. It is not to mark them as devoid of programmable logic.
The latter doesn't even make sense if you look at what else is there. /etc/profile for example is clearly code.
IIRC plan9 has some crazy thing like this. I have gotten used to libexec but part of me groans at the desire to make things ideal. We going to rename creat next?
The entire premise of this article seems to be that code and configuration are somehow mutually exclusive, despite decades' worth of examples to the contrary (including init/RC scripts). Just because this configuration is typically managed and "owned" by a package manager doesn't mean it's somehow not configuration (nor is this even universally true; see also: Slackware, which uses the same sort of setup but does treat the contents of /etc/rc.d/ as user-editable config files).
One of the few things that I actually like about systemd configuration is how they split it up in a clear way. I can't recally where it was documented but they make the distinction between site (lib), system (etc), user (home) and runtime (run). I don't recall them making a distinction between executable or not, but since it doesn't for the most part use rc scripts, that shouldn't matter quite as much in the general case. If it we wanted that, it'd be nice to nest it on top of current tree.
This distinction is nonsensical. They are files that you can change to suit your needs. I often comment a few lines on them. Code is a particular case of data.
But the distinction is useful because it paves the way for something everyone wants, which is seamless upgrades without user intervention. The fewer places the user touches something, the fewer decision points the upgrade process has to deal with.
Indeed. I was thinking about this just yesterday, when I upgraded my KDE Neon install. I was asked about keeping or merging a conf file I was unsure what was for, and that I can't recall having edited.
It's possible I did modify the conf file, in a late-night attempt to fix some issue long ago, but that was all forgotten by now.
The changes to the conf file were extensive, and I had no idea what were _my_ changes, assuming I had indeed done any changes, vs what was just changes due to the new version.
Why couldn't my change in etc just be a one-liner? Then the OS could upgrade the base conf file safely, and simply inform me that I need to review my conf override in etc to see if it was still valid or needed.
This way the upgrade tool could potentially be even smarter as it can clearly distinguish what has been changed, so it can warn even more loudly when a configuration variable has been removed or drastically changed for example.
My point exactly. These scripts can be easily edited to suit their behaviour to my system. They are then legit configuration files. Whether they are executable or plain data is inconsequential.
/libexec is already on the root partition, at least on NetBSD, and that's why I wrote the paths that way in the article.
And that's why I also mentioned en passing that sourcing local files from a potential /usr/pkg/libexec/rc.d/ directory would be problematic.
In fact, even today, making rc.d files automatically work from installed packages is a problem precisely because of this: those files currently end up in /usr/pkg/etc/rc.d/, and /etc/rc cannot use them by default because /usr/pkg/ might be in a different partition. There have been endless and repeated discussions about this issue on the NetBSD mailing lists.
libexec exists on Red Hat lineage OSes too, and it's standardized in LSB. (I think I'd basically say that it's Debian lineage that doesn't have it; the default of e.g. anything that follows GNU's 'make install' standards is to use it unless instructed not to.)
LSB is worth reading, because it had a precise definition of /etc that is not "important system stuff."
Debian also has libexec. I can see 80 packages¹ in Debian which has files in either /usr/libexec or /usr/$ARCH/libexec. The Debian Policy² does not mention libexec, but does refer to the FHS, which does allow libexec³.
Debian generally favours individual subdirectories of /usr/lib instead. /usr/lib/policykit-1/polkitd rather than /usr/local/libexec/polkitd (in older FreeBSD packages), for example.
libexec is not really BSD-specific, but yes, this article is about BSD.
However, I also mentioned that the same problem applies to pre-systemd Linux systems because of their /etc/init.d/ directory. systemd has actually done the right thing here (which is also mentioned in the article) by putting unit logic under /lib/.
Which is not relevant because the article is about BSDs.
If your point is that rc should not be modified for <whatever> because of BSDs… rc is not a centralised and unified tool, how BSDs handle their initialisation process has no bearing on any other system unless those others decide to follow suit.
My resolv.confs tend to be written once and then go untouched for months if not years.
The problem there is that distributions, be it linux or bsd, lack a proper way to distinguish between a "static" resolv.conf and a (always-)generated one.
An "easy" fix would be to mandate that if /etc/resolv.conf contains a "dynamic" key then /run/resolve.conf (or /var/run/resolv.conf or whatver) is read as well (or instead).
However, tons of stuff tries to parse resolv.conf, so it would be better to have a user-editable /etc/resolv.conf.conf and a /run/dynamic-resolve.conf and dump the result or into /etc/resolv.conf to keep compatible.
The reality is not quite as the article paints it.
First of all, there was a system before Mewburn rc. FreeBSD had an /etc/rc.local.d/ mechanism for drop-in scripts in the middle 1990s, added by Jordan Hubbard and Garrett Wollman.
Second, /etc/rc.conf is not the only service configuration file in FreeBSD's version of Mewburn rc. The primary configuration file is in fact /etc/defaults/rc.conf , and that file lists, in a variable, the name of other secondary configuration files, including /etc/rc.conf and /etc/rc.conf.local . On PC-BSD of old, and then TrueOS, it also named things like /etc/rc.conf.pcbsd for example.
Third, if you go back to Usenet of the 1980s, and indeed contemporary books about system administration, you'll find that the distinction between /etc/rc and /etc/rc.local was problematic and not what people tell you it is supposed to be. People then too thought that /etc/rc was for the operating system manufacturer, and /etc/rc.local was for the system administrator. But in practice operating system resellers ended up putting their stuff in /etc/rc.local making it also something that system administrators touched at their peril, lest they break operating system upgrades.
Fourth, the article says "You might even want to support a separate location for user-created services". That already exists. It's /usr/local/etc/rc.d/ . And to highlight the point about resellers invading "local" stuff, note that PC-BSD and TrueOS fill it with additional PC-BSD services like pcdm and pc-samba and the problem of "reseller"-provided add-ons being "local" continues to this day.
Fifth, systemd unit files are not scripts. This is one of their defining features, and one of the things widely touted about them, in fact. As the systemd developers will themselves tell you, and have, they are configuration files. Hence the systemd developers thinking that it was a mistake to provide an EnvironmentFile setting. systemd isn't dividing things up into "configuration files" and "code", as this article talks about. It has multiple sets of things all of which are configuration files.
Sixth, the systemd search order for unit files isn't that simple, and has a history of being improperly documented. It searches 17 directories nowadays, including subdirectories of /usr/local .
Unexpected to say the least.
Good read though, mostly agree.
> My main gripe is that the files under /etc/rc.d/ are immutable scripts.
Are they though? A lot of, well at least old software, lacked configuration files, their command line flags and unit scripts were for all intensive purposes, their config files.
Whether this actually makes upgrade harder (the next point made), is kind of debatable no?
Arch does fine with pacsave and pacnew files.
Making changes or tracking changes to these files is going to be required regardless from a systems upgrade point of view - because people want control over their system units.
And for what its worth, I'd rather find out about a breaking update via pacsave/new than figure it out months later when my copied but now outdated and untracked unit starts failing.