I have my whole system running like a livecd using overlayfs, it's great. It's a real peace of mind knowing nothing will persist. No logs or information is stored without your knowledge, any configuration mistakes will solve themselves on reboot, it also enforces the separation between your data and the system install.
Nice. Reminds me of longtime security tactic for web servers of shipping logs and mounting the entire file system as read-only so that intruders have no means of installing their root kit.
If you have root, what prevents you from remounting as read-write? That seems like an advantage for the actual live-cd over the "like a live-cd". It's physically read only.
One could argue it’s fairly hard to use an exploit to get root on a ro fs. One could also argue that protecting a system from an attacker with root privileges is a futile effort.
That's a good point, I was responding to the "install rootkit"-part implying they do have root and want persistence, but yeah it does protect against other things.
I believe this was to mitigate certain PHP code execution and file upload exploits. It used to be fairly commonplace that file upload code was buggy, allowing you to upload arbitrary files into the web root. So people would upload “PHP shells”, which are backdoor scripts, somewhere on the web root and then navigate to that URL to execute the shell.
I think this technique still remains common to escalate getting admin on a WordPress blog to taking over the host.
A lot of the PHP specific issues came from poor input validation and setting the uploaded file permissions to 0777. This was of course in those days before StackOverflow where you’d copy code snippets from old blogs or forum posts, same as all those snippets containing SQL injection vulnerabilities.
> Oh, and in case you wonder, all of this only works with distributions that have completed the /usr/ merge. On legacy distributions that didn't do that (...)
The Debian situation is kind of weird with their tooling maintainers opposing the merge and the distro maintainers trying to push the merge through. In true Debian style it started as a technical issue, but now it's a political one. I don't think we'll see the merge actually making it for all packages without issues for a while.
That said, there is a clear trend in distros to move to merge these directories and I don't see why not. There are some niche use cases for different partitions containing different binaries and such, but like all niche use cases you're going to need to configure those yourself.
The opposition's argument seems to revolve around "we always did it this way so we can't do it any other way", really. This is the stuff that led us to a world where the most popular distro now packages the standard browser in a Snap package.
> The opposition's argument seems to revolve around "we always did it this way so we can't do it any other way", really.
My understanding is that the technical objection is that Debian packaging tooling isn't up to snuff to deal with a merged /usr. There are various corner cases and technical debt that cause headaches.
And that the people promoting /usr are not doing the necessary work to get Debian to that point. That is they are not submitting useful patches.
So some patches and fixes have been submitted to correct the issue, but those have been blocked because they are not technically sufficient or high enough quality (??? not sure).
Something like that. I don't follow the drama. But that is my understanding last time I checked.
....
And then there is the issue of dealing with the conversion to merged /usr in existing systems.
For distributions like Fedora this is solved by putting the upgrade logic into the installer; anaconda. This way major changes, like init systems or merged /usr, is handled when you upgrade from one version to another.
Debian doesn't work that way. They want to be able to handle upgrades 100% through the package management system and debconf.
Debconf is essentially "install wizards" to allow people to choose configurations when packages are installed/upgraded and doesn't have equivalent functionality in the RPM world. One of the very few meaningful technical differences between deb and rpm.
Beyond all that there was past goes at attempting merged /usr, which depended on user selected preferences.
So this means that there are live systems that are a mixture of possible "/usr states". So those use cases need to be handled as well.
Which means that...
Yet again Debian politics has caused the creation of a vast amount of technical debt that must be overcome for it to move forward. Technical hurdles that are not experienced by other distributions. And because of that currently existing Debian installs can be in a sort of limbo half way between combined /usr and not, which is pretty much the worst possible outcome from a technical point of view. Ideally the change should be atomic in nature.
This is not terrible as many people place politics as a higher priority than technical excellence. So this state of affairs is actually desirable for many. Which is fine. Putting in extra work to help ensure that everybody is happy as possible with the outcome is not the worst possible thing you can do with your life.
For Debian it's more than just that the code isn't ready, the debconf guys and the design guys disagree and both consider the merge something that can decide about within the project. And to a certain extend, they're both right, which makes it even worse.
My understanding is that the Debian people tried to get a merged /usr going without consulting the packaging people. The packaging people not only ran into technical issues, they also disagreed with the decision itself. Eventually some patches were made to make it sort of work, but also with a warning that merged /usr is basically unsupported and can lead to breakage (which is technically correct). Then someone else tried to remove the warning because it made the entire project look bad with all those warnings everywhere, and at this point it's just egos and politics from all sides of the argument.
I don't think there are that many actually maintained packages that will break in a merged environment. It's more about the theoretical issue and about principles than anything.
It started out as a technical issue (the invalidation of some core assumptions about the file system in critical packaging infrastructure), but it's so much more than that now. I think the conflict will blow over, but just like people are still advocating for the death of systemd it'll take many years.
I don't think this has caused as much tech debt as it all seems from the aggressive discussions. It's a problem with some packages and some very exotic system configurations (embedded stuff is great at creating those "technically correct and supported but we really shouldn't have encouraged this" edge cases) but for the vast majority of the users and maintainers the whole thing is in the past now.
In the Linux space there are so many things that actually matter, like the 5 second Firefox startup delay caused by Snap and the fact that GTK still lacks a proper thumb grid view after it's been a meme for so many years. This whole issue is kind of a useless thing to put so much energy into.
Qt has the same problem and, at this point, I doubt this will change at all in the next decade. Very few applications actually have any reason to use the default file chooser, and there are basically no developers who have any reason to modify it. If there were, they would have made the changes already sometime in the last "many years". The Snap issue is at least fixable with some caching.
The issue is the problematic usage of the word "legacy" by the author. It is disparaging, almost like trolling.
Being "legacy" or not has nothing to do with the number of users. There are many cutting-edge distributions that (gasp!) do not even use systemd. Would you call those, "legacy"?
Look at the definition of the word, in the context of computing, according to the Oxford dictionary: "denoting or relating to software or hardware that has been superseded but is difficult to replace because of its wide use".
If anything, the most "legacy" distributions should be redhat and debian, and their respective derivatives.
No he is correct, the usage is not problematic or disparaging. It fits your definition exactly. The only practical reason to still have the /usr split is to support an obsolete configuration on very small hard drives, which is no longer actually in use by any of the modern distributions. And the real reason any modern distribution still has it is because it's hard to change it, not because they actually want to. So that very much fits the definition that you just posted.
PREFIX support is different for every second buildsystem upstream offers. You'd need to patch that, its a ton of work. Manpower that needs to be invested without having a commercially visible return - thats why commercial distros rarely do it and possibly never will if upstream does not start doing it out-of-the-box.
Rejecting the /usr split is worth it just for the purpose of not having to clean up after Lennarts ideas. He isn't the guy who carries the fallout, even when he is the one causing it.
I'm sorry I don't understand what you're talking about at all. PREFIX support makes it even easier to support the usr merge. You don't need to patch it if it's already there.
>Rejecting the /usr split is worth it just for the purpose of not having to clean up after Lennarts ideas.
This makes zero sense, the main reason to do this is for compatibility with container tools and other things that benefit from having a simplified mount structure. It was not an idea that came from systemd and it also has nothing to do with "cleaning up after" anyone's ideas.
> I'm sorry I don't understand what you're talking about at all.
Have you ever written or maintained recipes for building software? PREFIX= support exists, but its not consistent and many packages need individual workarounds. Adjusting it is quite some work.
Which distro do you use? Made by whom? Be grateful that these guys do this work for you.
Yes I have, and build systems without working PREFIX support are going to be really broken and need workarounds in a lot of scenarios, not just this one. It's always going to be quite some work. I actually recoil in pain now whenever I see someone ship something with a giant makefile or a big folder full of system-specific shell scripts.
For obscure packages with a weird build system they often aren't doing the work for me, specifically because those build systems are hard to manage and nobody wants to deal with them. If you want distros to not package your software that's a great way to do it. If you want to discourage contributors from sending you patches because they can't figure out your build system then that's also a great way to do it.
If you're a big popular package like Chromium or Firefox then you can get away with having a weird build system but even then packagers don't want to deal with it past a certain point, hence browsers packaged in Snap and Flatpak...
>but a neat, simple and portable makefile is a beautiful sight to behold and a pleasure to run.
As someone who has had to hack those makefiles a lot, I can't really agree. There's no such thing as a "simple and portable makefile". The real way to generate "portable makefiles" is to use autotools. Every handwritten "portable makefile" I've seen actually has a ton of system-specific and compiler-specific rules. And when porting from Linux you're lucky if some package shipping a handwritten makefile doesn't have an undocumented dependency on GNU make.
Even something simple like PREFIX support has to be manually implemented and tested if you write your own makefile, and often the upstream project doesn't bother testing it and wants the distro to do it. If you want distros to package your stuff without any troubles and without making them a guinea pig for some weird ad-hoc build system then it would be easiest to just use one of the standard buildsystems, those are mostly just cmake, meson, autotools. A small cmake or meson script is significantly shorter than a makefile and is a lot more portable.
I'm not sure I fully understand what Lennart means by "/usr merge". Is it assuming all paths under /usr belong to the same mount point, or something else?
It means that binaries are in one place, not spread out across /sbin, /bin and /usr. The author is specifically referring to the variant where all binaries are in /usr, but i've seen distros were it was done in other ways.
Thank you. I thought it was referring to an older issue which I'm not sure if it's been addressed, where systemd had problems booting systems which needed /usr to be mounted at boot time from another partition.
It's pretty cool actually that this is possible and I also can't help but think that this is a solution to a problem that need not exist in the first place. Systemd has this problem and the solution is to use a feature in systemd, right?
IMHO systemd has amassed too many features, both documented and undocumented. Having worked with GNU+Linux for over 2 decades, I still find Devuan and Void easier to manage than Fedora and Ubuntu. The latter is fine for basic desktop needs though.
It seems like it is a solution for anything that lives in /usr and could break your system / cannot directly be put there because it's read only when you are developing it. Not systemd-specific though obviously the part that manages the system boot and system services is going to be particularly concerned by this.
> IMHO systemd has amassed too many features
But systemd is not some monolith that does a lot of things. It's a set of tools that do one thing, well, quite like GNU and that happen to be called systemd-something. Often, thin wrappers around kernel features, like nspawn. I find them very cleanly designed and easy to use. To each there own but I find the possibility to create services declaratively with systemd unit files particularly useful, instead of having to write custom shell scripts and copy paste boilerplate. Upstart was a step in the right direction in this respect. It's not just "fine for basic desktop needs", it's a great fit for desktops and also particularly helpful on servers as well (not speaking about Ubuntu, just systemd).
Having a single large overall project that manages a diverse number of individual software products is a common pattern and certainly something that isn't remotely unique to Systemd. There is no reason to denigrate systemd for that.
You're making an Apple to Orange comparison here. Your examples are not released to the outside world as a single product "take-it-or-leave-it" and painting it so is borderline bad faith.
Painting systemd in that way is also bad faith, because it's not a single product "take-it-or-leave-it" either. If you think it doesn't have enough review for your purposes then it's time for you to step up and start reviewing, either in systemd or in another project.
Part of the reason why people seem to (falsely) think systemd is monolithic is because distros have been historically bad at packaging it and haven't split up the components, but that's been improving over time.
I share your opinion, but IMO it also goes one step further: systemd uses its gravity to try and shape linux distributions and dictate how things "ought" to work. Consider this quote from the linked article:
> Oh, and in case you wonder, all of this only works with distributions that have completed the /usr/ merge. On legacy distributions that didn't do that and still place parts of /usr/ all over the hierarchy the above won't work, since merging /usr/ trees via overlayfs is pretty pointess if the OS is not hermetic in /usr/.)
IOW, if I build a system with yocto for an embedded system, which is also a very useful candidate for a read-only system image installation, I cannot use this feature for this use-case because ... yeah because what. Because lennart says so. This got old so fast.
We can't blame the developers to not support any oddity and particularity that exists… and then blame them because the distribution makers adopt their tools because they work well (as a result?).
I'm happy they focus on a standardized base, this results in less development effort focused on more useful stuff, cleaner and more robust tools, it also makes it easy to switch between distributions and to target this standardized base when developing.
It also can be seen as a assumed limitation. Any tool has limitations. But the /usr merge has been conducted with reasons, and makes this kind of stuff actually possible. It's not like keeping a split /usr is very useful today anyway.
Systemd is not something distributions have to bend to. It's something they willingly adopted because it solves well problems they have.
And it's not just Lennart. Hate he gets gets old fast.
> We can't blame the developers to not support any oddity and particularity that exists
It isn't as innocent as you paint it.
systemd developers have other agendas when making decisions , you can hear it from horses mouth:
"Well, it is definitely our intention to gently push the distributions in
the same direction so that they stop supporting deviating solutions for
these things where there's really no point at all in doing so." [1]
And after watching systemd development for years I might argue they have a very liberal interpretation of what constitutes "senseless configuration differences
between the distros for the really basic stuff"
What's a "major problem" to you is the best and unique aspect of having different linux distributions to others.
The actual major problem is that systemd developers wasn't being contempt with just building an opinionated system and let it rise on its merits, instead they choose to use its status to "gently push" their opinions on everyone else.
If your distribution is already content to ignore any efforts at standardization in favor of "unique aspects" then I don't see what the problem is. You can go about your business and ignore systemd, or choose to adopt it on its merits. It's entirely up to you, nobody is using any status to push opinions on anyone.
Also you're ignoring the last part of that sentence: "where there's really no point at all in doing so". That's the most important part. If a distro has a real reason to deviate then that's fine. If a distro is deviating for no reason then that's doing a disservice to its users, and you can expect anyone who ships other things for that distro to be upset with them for pointlessly making extra work for everyone else. It's not just the systemd developers who will be inconvenienced.
A distribution isn't a single individual. It consists of different people, developers, maintainers, contributors etc., who are at different levels of acceptance or appreciation of various features of systemd. It isn't as simple as being content with it, or adopting it. In a diverse distribution like Debian there are constant debates about various features of systemd, and that's not only because they haven't decided on systemd yet, but because systemd project isn't also making their job easier by constantly adding new things and gently pushing distributions to use them.
There's no need to be politic about this from the POV of systemd. "We support this feature on merged /usr" does not have to mean "we don't allow to run it on legacy systems because it doesn't make sense". Look, your users have more fantasy than you do how to use your software. Doesn't mean you have to support it. But it also doesn't mean you have to explicitly enable it only in the setup you envisioned this being used.
It's not an _assumed_ limitation. It's an _enforced_ limitation based on a limited world-view that also uses the gravity of the project to enforce the world-view on any user.
I'm not arguing pro or against split /usr. Reality though is that for various reasons there are legitimate use-cases for a non-unified directory tree, e.g. embedded ramdisks that contain only the barest of /bin in a crunchgen'd binary that don't want to duplicate stuff that's later used in /usr/bin. Linux is not only desktop and server.
systemd is being adopted because it solves problems well the distributors have, but THEN the distributor ALSO has to bend their distribution to the single enforced world-view the systemd folks have if they want it to integrate. They, the distributors, software integrators per se, are just forced to hack out limitations that are imposed by systemd folks for no benefit of either; systemd dev or user. No unified-/usr-user will benefit from this enforced restriction. A split-/usr-user will have to patch systemd and maintain their tree to use this feature if they have a good use-case for it because .. "systemd says so" (if you prefer that over the "B"DFL lennart).
Lastly, you are correct in that systemd is a pool of like-minded developers - not only lennart; and this is where I add: - who like to shape the reality out there instead of allowing distributors and users to easily take on their own responsibility easily. Sure, I can fork systemd (it's free software after all), but it's unnecessarily "opinionated" for its broad area of application. And that's where the "politic" dimension of systemd development comes into play ...
> There's no need to be politic about this from the POV of systemd
I don't think anybody is. It's their technical approach to develop stuff. But sure, their approach has a huge impact on the ecosystem.
> we don't allow to run it on legacy systems because it doesn't make sense
If I'm to design some piece of software that requires something, that this something is still relatively commonly not there and that I know it won't work without, yes, I'll go ahead, make a check and prevent the program to run in this configuration.
Think ./configure which will not write the Makefile if some critical dependency is missing. It's the same thing for me.
Anyway, the author did say "it won't work", he didn't say "We went out of our way to prevent the users from trying to do this if we detect an unmerged /usr". I think you read too much into the whole thing.
> But sure, their approach has a huge impact on the ecosystem.
And this is the political dimension of systemd. One they keep using to shape the ecosystem to their preference.
> If I'm to design some piece of software that requires something,
See that's the point. It doesn't _require_ a merged /usr to work. The benefit of the merged /usr here is that this guarantees to potentially overwrite/extend "all" binaries, not "only" those in a split-/usr. The latter doesn't mean it doesn't work. It just means that they only overwrite one place, not all potential places something exists.
> > But sure, their approach has a huge impact on the ecosystem.
> And this is the political dimension of systemd. One they keep using to shape the ecosystem to their preference.
Any highly successful project is political then. Linux developers in particular force their kernel and their views on how a kernel should behave and be designed down the throats of their users.
... Obviously you have your say when people choose to use your stuff. But then that's fine, you are the expert in your domain. Who else is better placed than you? If people use your stuff, probably preferences are shared.
Or they are locked-in (through network effect for instance). But the systemd folks are not out there having an evil agenda they force the ecosystem into. They solve their problems and people do happen to adopt their stuff. They do seem to have a very "if it's broken, fix it" approach but that seems to work out quite well. Sure that's not always comfortable but I find it somewhat refreshing.
> systemd is being adopted because it solves problems well the distributors have, but THEN the distributor ALSO has to bend their distribution to the single enforced world-view the systemd folks have if they want it to integrate.
It is like saying that "It is great that engineers are working to provide a unified rail line across Europe, but they are assholes because they want everybody to use the same rail gauge as well as implementing these changes that will allow the rails to handle traffic going 200 mph. The unified rail folks have an agenda!!"
The problem they are trying to solve is solved by "bending of the will", so to say.
> IOW, if I build a system with yocto for an embedded system, which is also a very useful candidate for a read-only system image installation, I cannot use this feature for this use-case because ...
I agree that this is precisely a nice use-case, but why is this a problem? I agree with Lennarts reasoning that it makes little sense if the files are arbitrarily split between / and /usr. In any case, Yocto has had support for usrmerge for a while now[1].
>IOW, if I build a system with yocto for an embedded system, which is also a very useful candidate for a read-only system image installation, I cannot use this feature for this use-case because ... yeah because what. Because lennart says so.
No this is wrong. You can still use this feature but you get to manually type out all the mount paths yourself every time and work out all the bugs in it when you try to mount other things on top of it, because that's what you signed up for when you decided to have multiple paths. I'm really baffled as to how you could even remotely say this is a systemd problem. This will be a problem if you try to do any kind of bind mount or overlayfs stuff on that system.
I really suggest that you avoid making comments like "systemd uses its gravity to try and shape linux distributions and dictate how things 'ought' to work". That stuff is conspiracy theory territory at this point, it's so disconnected from reality that it's just pointless to say it.
How does systemd have a problem? Systemd doesn't care about having a read-only rootfs at all, except that it supports it and now ships a little tool that's useful if you happen to use them. Fedora and Ubuntu doesn't ship a read-only rootfs(1).
(1) Well, Fedora Silverblue is an experimental Fedora-variant that uses a read only rootfs. But that point still stands.
I am not sure whether or not you are aware that some might intepret your comment as survivor's bias. Not everybody is a sysadmin, and definitely not everybody wants to do those tedious tasks in /etc.
There are so many potential pitfalls of everything around init/initd, especially if downstream integration scripts have to keep up with upstream changes.
And I would argue that those should not be the burden of endusers, and also not be the burden of distro maintainers. Providing a settings file that can autolaunch your application in a failsafe manner for all distributions is where systemd shines at, like it or not.
While I agree that it isn't perfect and the debugging critiques are kind of valid, I also would argue that this can be solved with better (graphical) tools for the systemd ecosystem.
Also: if you assume that debugging a broken shell script (running as root) is easier than systemd, then your perspective is already biased, cause it's a skill set that requires lots of experience.
I mean, most shell scripts in /etc/init.d don't even set -e, let alone -u ... and should be considered a risk for system integrity.
>Systemd has this problem and the solution is to use a feature in systemd, right?
More like systems with an immutable /usr have this problem, and systemd now has a way to circumvent those restrictions when developing. Fedora has been prototyping such systems for quite some time, not exactly with brilliant success so far, but it's good that they're trying.
