it's funny that many people mention boot time as a metric when for most servers, VMs and laptops the devices are either always on or sleeping. startup times do nothing in those cases.
personally I use a custom alpine live-usb that is immutable and I fully shutdown and start-up all the time. openrc doesn't give me any grief, the bulk of my startup time is copying my 500mb rootfs into a tmpfs
systemd for me is solidly "at work" software, and not stuff I'm a big fan of either (journald, networkd and the silly dns resolver all have serious issues). so please, keep away from my alpine or I'll have to fork -again- :)
> it's funny that many people mention boot time as a metric
For the majority of deployments of systemd, that's probably true. Note that this is a musl port, and that the existing musl port is from OpenEmbedded. It's embedded devices that are often using cut-down userspaces (e.g. using MUSL), and embedded devices often do care about startup time because they're not Always-On.
E.g. the VTech baby monitor I've just bought is generally pretty good, but spends >10 seconds at a splash screen when you turn it on. No idea if it's Linux under the hood, but an optimised boot time would be great.
In a previous work project, we need the Linux based software to be ready from low power state quickly (battery powered, only doing work occasionally). By getting the boot time down <1s, I could also avoid a load of sleep-state shenanigans - when we wanted to power save, we just turned off completely (there was already a supervisory micro to decide when to turn back on).
Good chance that baby monitor uses something like FreeRTOS, not Linux for doing it's stuff. If you are unlucky, it's Android and they did the bare minimum to have a functional product.
E: Also, Android uses the Linux kernel, but similarities with Linux based operating systems pretty much stop there.
Android is a Linux based operating system so saying it stops there doesn’t make any sense. There is a wild variety of Linux OS out there in embedded encompassing a lot of various patchsets, libc and userland. You just have a bad understanding of what are Linux OS if all you think about is the usual distros.
No sarcasm, Bionic libc is a subset of POSIX, Musl strictly follows POSIX 2008 and Glibc is superset of POSIX. 99% of Glibc->Musl porting is dealing with non-POSIX features (a.k.a. GNU extensions)[1]. So in the end we could find that Android apps are more POSIX compliant than software from generic glibc-based distros.
It's pretty trivial for an embedded device to trim down the series of startup tasks to only what's required, and then just write a C program that does those steps. I certainly wouldn't pick up such a gigantic, complex and potentially insecure package as systemd for an embedded project. I want less moving parts, not more
It depends on what you mean by embedded, I suppose. Having a service monitor to restart your jobs is pretty useful too, and it can be tricky enough to write a good one. Something like a baby monitor (to use an example mentioned elsewhere) might run almost as many services as a desktop Linux installation.
Can you provide any specifics of the serious issues you’ve encountered with journald and networkd? There were some performance issues with journald early on, but I know a lot of work has gone into it over the years (a coworker of mine, for example, improved the performance of log retrieval by 10x). And networkd has always fulfilled my needs nicely, so I’m curious what sort of trouble you ran into.
Only issue I have with networkd is lack of adoption: network manager is pushed hard (Red Hat still clinging to nm for servers, ew) or want to add another abstraction around it (Canonical with netplan). It's an awesome, declarative way to manage networking, embrace it.
Man, we fought hard at Red Hat to get them to continue using networkd with their latest container-focused OS. Ultimately, the organization just had too much institutional momentum behind NM. Frankly, I’m disappointed that they lacked the imagination and drive to push the industry forward.
Networkd is fine if you don't need vlans, bonding, and a bunch of semi-esoteric stuff. /e/n/i does fine there.
Journald is too monolithic IMO. Direct access to the files in a text format is standard Unix stuff. Forcing everyone through the journalctl tool is rough, performance being just one issue. The binary format has a habit of changing too, having written a journald log tailer.
Overall, like I said, systemd is work software for me. It's fine for the complexity in a company (where you have to balance time and business value), but for my own use I wouldn't choose it.
Hence, please don't touch Alpine ;). I was on KISS previously but got tired of compiling my own packages (and Firefox is nigh impossible to compile in a 'ragged' version package system)
What more semi-esoteric stuff would you be missing?
> Hence, please don't touch Alpine
I'm with you here though: I too appreciate Alpine being on OpenRC rather than systemd. And hope a musl port does not change the calculation there. You may also appreciate https://chimera-linux.org/ if you haven't checked it out already.
I recently found that systemd-networkd isn’t generating “up” events (or something similar, not posting from work) for bonded interfaces set up by cloud init which causes the network wait job to spin until it times out, adding about 60s to boot time for any debian-based box we run in our medium sized deploy, so maybe a few 1000. I noticed because my dev box took forever to come back after a kernel update and I was sweating that I’d broken something.
Unfortunately I was unable to allocate the time to figure out systemd at the layer where the event is expected but not generated to file a coherent bug. I ended up working around it several layers up by hard coding “bond0” as an argument in some config and moved on.
I don’t think openrc is perfect and I find broken shit in Alpine constantly but when I do it’s a whole lot easier to figure out what broke when you don’t have to dig through several layers of dbus event generators and consumers before figuring out what even happened.
I think I may be facing the exact same issue with bonded interfaces not being brought up properly (unrelated to cloud-init) and was hesitating if I should mention it as a caveat in the previous comment but maybe to your point I'm not 100% confident we're not missing anything in the configuration (:
For now an `ip link set up` hook has been a passable workaround here.
Yeah, the cloud init bit was from memory and might not even be right. What I found was that the dbus event doesn’t get generated for the bonded interface when it comes up.
journald is still very, very slow. As in, getting the entries for a given process can take a minute or more, whereas just grepping a normal text file is nearly instant. It doesn't help that the default rotation settings are insane (like, why would using a certain percentage of my total space be fine, what if all programs thought that?).
Generally I don't really understand what journald brings to the table. I'm sure there are esoteric setups where the hash chaining thing makes sense, but meh? A lot of things needed fixing and systemd fixed many of them, but everything journals is a terrible user experience for me.
This... is just not true. I use journald extensively on a 600Mhz Cortex-A9 daily and it's not slow. I don't know what kind of log load you have, but even when I have processes crashlooping and am sending logs over SSH over a VPN it's fine.
> like, why would using a certain percentage of my total space be fine, what if all programs thought that?
Because the journal’s role in a system is inherently different than most programs. It’s collecting logs for everything; not just itself. It seems reasonable to me that a system-wide service would use a percentage of the system’s resources.
Plus, if you don’t like it, reconfigure it. It’s only the default, after all.
> Plus, if you don’t like it, reconfigure it. It’s only the default, after all.
Or -- and I know this is crazy but hear me out here -- if you don't like it, and it brings nothing but problems to the table, just don't use it at all.
First, i'm a vocal systems appreciater. But. The one thing that frustrates me about the journal is that there is only one journal. If I have a chatty app, it can entirely take over the journal. Set a 1GB hour al, come back in a couple days, and nothing else is left in the journal other than it logging. I hate this so much.
If I could assign some apps to a secondary journal, or better quota individual services, that would make sure I can keep tabs on my entire system, even when one app is logging heavily.
Bro no. Logrotate shit was gross and gnarly. Every service has its own way of doing with logs. None of it worked the same. It never was simple it always had been crap.
I have my complaints but not replying on each service to do something special conpatible with each other log rotation system was not a complaint. Having this managed reasonably has been great.
Tangentially I now have to remove systemd-resolved on each machine. It gets too complicated because I use dnsmasq on family machines and resolved is always trying to open ports to the internet (opensnitch knows).
Already run DNS on my router, so why have a second or third cache on each machine? Network manager is involved also.
The biggest issue I ran into with OpenRC was the lack of metadata related to hot-plugging events.
There currently is no way to fire specific events based on vendor/device id like you would with systemd.
This forced me to write a shim script to enable USB passthrough with arbitrary k8s nodes. The shim was short but is very hacky and relies on 3p packages because the info needed isn't available in stock alpine.
Follow up post in the works based on additional improvements up streamed to OpenRC. The biggest issue was getting literally anyone from the project to comment on the PR to progress it forward to acceptance.
it's funny that many people mention boot time as a metric when for most servers, VMs and laptops the devices are either always on or sleeping. startup times do nothing in those cases.
People bring it up because it's usually the first thing they notice. Of course the faster boot times are not what makes systemd great, rather it's what makes those fast boot times possible.
(I’m guessing this is supposed to be a joke in a similar vein as “Emacs is a great operating system that desperately needs a text editor”.)
You probably are looking for a different word, rather than “indifferent”. Indifference is a property of the observer/subject, not the observed/object. For example, if I don’t care about sports, I might say that I’m indifferent to the outcome of the game my friends are watching.
There exists a world of devices that are off and need to power on within a few seconds. Think of your media console in your car.
Have you actually profiles that copying the 500mb rootfs is the bulk of your boot time? At around 150MiB/s, which is common for USB these days, I would expect that to only take ~4s, and copying from ram location to ram location is far faster. I would expect your boot time is a lot longer than that.
IDK; on my laptop, the boot time is dominated by opening the encrypted disk. From there it's well under 3 seconds to the Xfce login screen. (A T14, m.2 storage, runit as the init system.)
Yes. But a cloud VM instance can boot very quickly, because it has fixed hardware and does not have to probe for physical devices. Waiting for enough random events to happen to init the encryption used to be a major drain of startup time; likely not a problem any more.
I don't think there are any devices where "withing a few seconds" is a useful target for the startup sequence. Most devices should really be instant, which means hibernation / full memory snapshots and your real boot system becomes mostly irrelevent. Everything else is really in "meh" territory. That said, OpenRC is already quite fast and both my laptop and desktop boot in a few seconds using it - the EFI takes much longer to do whatever it does and hand control off to the kernel than it takes from kernel to desktop login screen.
nowadays serverless environments are microvms. that is, a standard machine type that is explicitly virtual (so device drivers have no wait time for hw to init). however, they don't either need userspace to boot because the function/application/whatever is likely init itself or not far off.
so they need to boot, but only the kernel needs to boot.
With the musl/busybox/OpenRC stack, Alpine is top notch when it comes to battery use. Just by switching from Debian to Alpine i doubled the battery lifetime on my Acer A114 from 8 to 20 hours. This includes moderate use.
My Debian 12 VM has been up for 4 days, and Xorg clocked 5 hours of CPU activity in that time, but systemd clocked only 9.5 seconds, and its subsystems each less than that.
So I doubt if systemd and its subsystems are using much energy.
It doesn't seem likely to explain 20 vs 8 hours battery.
Could probably get similar improvements by using any distro with less "batteries included" than Debian. Though I'm not aware of any except Alpine which offer stable versions.
I run a Void Linux Musl machine at home and cold boot time vs wake up is so trivial that it makes sense for me to shut down when I am finished. It takes about 10 seconds to boot to login, most of which is waiting for the POST operation to complete. In fact, I've has issues with sleep sometimes not waking up and the system hangs with a blank screen forcing a reset obliterating my session. In addition to that, sleep takes 20+ seconds to power down whereas a shutdown takes about 3 seconds.
systemd to me isn't useful at all for the desktop and is a big contraption for firing off VM's and containers in a highly automated server environment. I dont miss it which means I have zero need for it.
"personally I use a custom alpine live-usb that is immutable and I fully shutodwn and start-up all the time... the bulk of my startup time is copying my 500mb rootfs into a tmpfs"
If rootfs is immutable why copy to tmpfs.^1 Genuine question. Not Alpine or Linux-specific. For example, maybe it feels faster running binaries from tmpfs.
Immutable directories can be mounted from compressed files e.g., squashfs, etc., on the USB media. Alternatively one can copy only the binaries that one acually needs at a given time. Then delete them from tmpfs when not in use to free up memory.
1. I do this is so I can pull out the USB stick. But I use a much smaller rootfs that fits in the kernel ramdisk.
Nothing. But many devices today do not support it anymore, instead they support S2Idle. S2Idle is more power-efficient than running-idle but less efficient than even Standby. Its huge advantage is waking up virtually immediately.
Linux has no problems running S2Idle but due to how S2Idle works, more components can wake up your system, therefore more components can misbehave, incl. devices attached by USB. Ugh.
S2idle is also effectively how all smart phones and watches work. Note that not every wake up results in the same results though. Your phone wakes up all the time, but it doesn't necessarily do a lot of work when it wakes up. The same can be true for your laptop. It would be unlikely for it to wake up to the point where it needs active cooling. If it does the OS is doing something wrong.
> personally I use a custom alpine live-usb that is immutable and I fully shutdown and start-up all the time. openrc doesn't give me any grief, the bulk of my startup time is copying my 500mb rootfs into a tmpfs
I'm assuming this is for a personal use computer. Can you talk more about the setup here? Sounds interesting. If the OS is immutable, where do you store personal data and how do you perform updates?
I was interested in a little more detail. Is the home directory on another USB, or maybe a laptop's built-in SSD? To add new software, is he creating a new custom image every time? Is this practical as a daily driver?
I don't want my fridge to take 5 minutes to boot. Seriously.
I'm always super annoyed at the current state of smart appliances that take ages to power on because they run some half-assed linux on an underpowered processor that would give the apollo guidance computer a run for its money. And that's why I only have dumb appliances.
> boots in 1/3rd the time of OpenRC on the same system
Wow, that's a suspiciously impressive difference. I was under the impression that OpenRC and systemd support similar parallel service startup features and usually boot systems in approximately the same amount of time, with maybe a slight edge to systemd. 3x speed makes me wonder if there's something else going on, like the current early systemd port having some bugs that cause it to incorrectly skip some important work.
Either way, super cool. What's the likelihood that this ends up getting upstreamed?
> I was under the impression that OpenRC and systemd support similar parallel service startup features
It's not enabled by default in OpenRC because it's not stable.
> WARNING: whilst we have improved parallel, it can still potentially lock the boot process. Don't file bugs about this unless you can supply patches that fix it without breaking other things!
It is a simple little startup system with some dedicated users. Open source projects don’t need huge user-counts to justify their existence. It isn’t a profit center, people just do this stuff for fun, because they enjoy tinkering, or maybe they coded it up and decided to just share it out of altruism.
A lot of people care about BSDs and embedded Linux systems. Such as: Netflix (FreeBSD), anyone with a Linux-based router or cable/DSL modem, and so on.
That said, parallel startup is possible on those systems too. There was an out-of-tree patch for parallel rc(8) on FreeBSD some time ago, and dinit is parallel, etc. I think OpenRC should move towards making parallel startup stable and supported so that those systems can do that as well.
OpenRC doesn’t have huge consumer market share, at least in the context where this sort of thing matters. For regular Linux desktops, most of the distros use SystemD.
It couldn't be massively supported until it took over, though? It's more than a little platform effect going on, it's not all by virtue of systemd being just all over great. It has some great ideas but it's an entire world unto itself which feels very NIH-ish.
Edit: I think it serves as both a warning and an inspiration. Don't let a duct tape solution (such as for instance boot via rc shell files) live for too long, or it might be replaced by something baroque.
But systemd was already widely supported not even 2 or 3 years after its original release. The reason it built its network effect is because it was better. Sure, a lot of it is also due to the classic Redhat NIH and their marketing push for their own solutions regardless of if other stuff already exists, but in this specific case systemd was also very very useful from the get go.
Having dealt with it for some time now, the amount of "good idea, bad implementation" I hit is too damn high.
And it's nigh impossible to replace for various reasons, including at one point critical interfaces that various user-important programs depended on being a constant treadmill of updates, lack of documentation, and hidden dependencies on internals.
It's true. systemd has a lot of good ideas with questionable implementations. I'd love networkd if it didn't have silly decisions made.
And it has some questionable ideas with good implementations. For example, I hate the journal, but admit it is implemented well.
At the end of the day, it's about tradeoffs and using what suits the system at hand. For some people that will be systemd, and bringing systemd to Adélie and other musl libc distributions means that it can be used by those people.
Call me old-fashioned, but I also hold out hope that at least some of those questionable implementations could be fixed if only someone with the desire would write a patch and send it upstream. Bringing systemd to musl means the people in musl land that aren't knee-jerk anti-systemd maybe might be more enthused to do just that, improving it for everyone using it.
Can't specific examples be rehashes of past grievances? It hasn't changed much. The log is annoying and sometimes freezes while it compresses or something, for instance. That's old.
Various, but don't have time to rehash all of them - the following are just things that I was either bitten by fresh or constantly suffered from in last year:
- The utterly broken in places interface (one of the worst offenders is systemctl show non-existing-unit dumping you a screenful of systemd unit parameters instead of "this unit does not exist")
- Bonkers defaults the moment you step outside of the minimum case for new unit - spent ridiculous amount of time dealing with that trying to make a self-restarting service with sane values for restart timer etc.
- Lack of documentation resulting in having to read the source code (which is horrible code, IMO), which I can only figure as being result of built-in assumption that any resource consumed by systemd is exclusively for systemd use. Which causes problems sometimes (my specific case was totally undocumented values used by systemd secrets mechanism wrt to TPM2, which caused hard to diagnose errors)
> Folks who evangelise these "just as good as systemd" solutions always hide a key factor.
No one in these comments was "evangelising" and no one was hiding anything. If you want to have a technical discussion about the merits of various init systems you can do it without inflammatory language and without reviving debates that almost everybody is sick of
The inflection point was GNOME 3.8 not working properly with anything other than systemd and it being night impossible to implement compatible alternatives for some of the interfaces involved.
Yet systemd doesn't have an equivalent to runit's sv once. Also, everything's so opaque that I can't figure out why nftables ruleset isn't loaded on boot, but systemctl restart works correctly (all on Ubuntu 24 server without any kind of frills added).
> Yet systemd doesn't have an equivalent to runit's sv once.
It's a design difference - systemd assumes services know whether they are intended to run once or restart under various conditions. If you want that behaviour it's just a matter of a tweak to the .service file.
> Also, everything's so opaque that I can't figure out why nftables ruleset isn't loaded on boot
Has it been disabled? ("systemctl enable [service name]" will enable it)
Yes, but, as the name implies, only once, and selectively at runtime. If you have a service and issue sv once on it, it will not be restarted, but its configuration is not changed so it will run normally after system restart.
RemainAfterExit=yes is what achieves the effect (i.e. if the process terminates it is regarded as still active for system-dependency purposes / status unless you purposefully mark it otherwise).
> There is a reason systemd took over, its massively supported and widely tested.
Yes. The people behind it held key positions at Red Hat, freedesktop.org, etc. I'm not calling them some shadowy cabal pulling strings behind the scenes, but it would be silly to discount the political sway the people that started the project have.
Being widely-tested also hasn't keep it from producing system bricking issues, such as this debacle just a few months ago:
Or the time back around 2012ish when an update caused the dhcp lease renewal portion to get wedged and took many thousands of servers offline until the service could be manually restarted or the machine rebooted.
Then there are all the weird choices that are inconsistent with the rest of the ecosystem, like showing asterisks when typing in passwords instead of nothing:
https://github.com/systemd/systemd/issues/8495
Even formerly simple things like log collection for systems not hooked up to Splunk or similar is now a chore. Most distros not longer include rsyslog by default, and pulling log files out of the journal is a painful process. Before, if I wanted to rip out the past week's logs for analysis, it'd usually mean just grabbing some gzip'ed files that were relatively small in size. Journalctl files are significantly larger in size for similar timeframes. OK, so how about I just interact with the journal directly and tell it to dump just the parts I care about? This is quite resource intensive on smaller VMs running in cloud providers and can severely degrade the overall system performance, so now I have to artificially slow things down and get it piecemeal or schedule it to run during off hours when the performance hit won't matter.
I work with systemd daily. It is enough to make me wish I had to work with systemd never. I'm not some sysvinit purist, either - I like SMF on solaris quite a bit!
8: logind being repeatedly and completely busted (in 2022-2024) got me to try Debian again. 9: Debian taking 10’s of seconds to boot (just like manjaro), when devuan does it faster than a monitor sync got me to switch to devuan.
(And, yes, since logind regularly wedges, I spend lots of time rebooting).
Hotplug worked fine for me way before systemd existed, so that’s not a benefit either.
Systemd always is ahead with new features. Which is nice if its a feature you need, and double worse if it crashes your machine, like logind sending your machine into a bugged suspend. Which makes systemd linuxes unusable for newbies on some hardware.
I used OpenRC because, not knowing about logind.conf, i had no other choice.
Yes, it's massively sponsored as well. So it went heavily imposed almost everywhere even when a quite big chunk of the Linux community deeply disagreed about the imposition.
People seem to not appreciate that Lennart Poettering managed to get systemd from concept to shipping in RHEL 7 in under four years. At that point I'm not even sure anyone at Red Hat had even asked for it, they had already transitioned to Upstart for RHEL 6 after all. Whatever your opinion of Poettering may be, you have to admit that he has tenacity.
So it went heavily imposed almost everywhere even when a quite big chunk of the Linux community deeply disagreed about the imposition.
There is a substantial selection bias in the complaints towards systemd. At the very least it seems that Red Hat's customers didn't have a problem with it. For most Linux users, it was a minor change, but for distro maintainers it was a massive relief[1].
Again, Lennart Poettering actually took the initiative to develop systemd and get it adopted. By comparison, detractors of systemd did not develop a competitive alternative. Thus, it's hardly a surprise that systemd ended up steamrolling the competition. Ultimately systemd is what users deserve because no person or company bothered to make something better.
As it happens way too commonly, the loudest in OSS communities are ones that have zero stake in building or maintaining the projects they criticize. It's easy to just rant if everyone else has to pick up the extrenalities.
The reality of my situation is my job frequently involves meeting people where they are - and most people are on systemd. Issues with systemd have caused and continue to cause me significant headaches - sometimes they are one-off but major events, sometimes they are recurring frustrations because of specific design decisions. I've included a list with some examples of both elsewhere in these comments.
I don't want to build a competitor to systemd because I fundamentally disagree with the design philosophy that has led to the proliferation of a huge number of systemd-* replacements for other things that already worked fine for all of my use cases. For situations where I have complete control over the infrastructure and don't need to worry about junior engineers that only have experience in the systemd ecosystem, I don't run it, and my life is lower stress because of it. I don't even particularly care much about the init portion of it, though a lot of the improvements are things that I don't really care about, like boot time - if I'm in a situation where a server's boot time has some impact on our overall uptime, etc., then things have gone wrong in a very big way - but if it was just a new init system, I'd have minimal complaints.
But in general, it's one of the reasons that I have been glad to shift my professional focus away from being very specific to the Linux and compute side of things to broader cloud platforms, etc.
So far most of the complaints about systems are really all about the fact that there is no real competitor to systemd. The argument that systemd is difficult to replace says more about the state of the rest of Linux user land, than it says about systemd, since most of the benefits of systemd are easy to replicate.
Writing another init system isn't exactly something most people would want to spend years of their life on. At best, it'd take a few years to reach some limited parity with where systemd is today. Furthermore, most of the systemd detractors fundamentally object the philosophy of systemd. So how can they meaningfully innovate over upstart and/or runit? It's either developing systemd-lite or a more elaborate ball of scripts. Even then, there is the issue of convincing distributions to adapt your new init system.
The reality is that systemd arrived at a time when Linux distributions were dying to move off of sysvinit. Poettering struck when the metal was hot and he delivered a comprehensive tool that greatly reduced the burden on maintainers. Sure, nothing lasts forever, but it will be many years (if not decades) before systemd is slated for replacement.
If you are trying to convince me that systemd happened out of the creativity of Mr. Poettering and Red Hat just followed him... well I have to say I don't believe that. But anyway, for me it really doesn't matter. It's the beauty of free software.
As strange as it might sound, that’s basically what happened.
I mean, let’s assume that Red Hat leadership initiated the systemd project, then we are immediately confronted by a number of problems. First of all, if Red Hat planned to ship systemd with the launch of RHEL 7 in 2014, then why did development on systemd only begin in early 2010? It seems rather irresponsible to intentionally delay development of the new init system for your flagship product, particularly when you’ll be supporting it for the next 15+ years.
Second of all, why would Red Hat hand the systemd project to Lennart Poettering, a developer who was primarily responsible for PulseAudio? Moreover, Poettering was already notorious for being an outspoken member of the Linux community. So why would Red Hat management go so far out of their way to choose such a divisive figure lead their project?
Also, why would Red Hat management seek to imitate launchd of all things? Why not SMF or some other “enterprise” solution instead?
Last of all, upstart development was primarily funded by Canonical and it had proven itself on RHEL 6. Therefore, replacing upstart with systemd meant shifting the maintenance burden back onto Red Hat in one of the very few areas it could actually save on it.
As an intermediate level linux user (never seriously daily driven a distro, but I do a lot of ops work and am comfortable getting in the weeds)... I really have come to prefer systemd over e.g. upstart/sysinit.
I also recently moved to networkd and resolved when setting up some new nixos boxes and greatly preferred the way those worked when doing vlans and split-DNS respectively.
I don't think Lennart changed his opinion, if you look further in this thread[1] Lenart wasn't ever against musl per se, it is more like "we use non-POSIX features from glibc that we found useful, and we are not going to reinvent them, so either convince musl to implement those features or carry patches fixing the issues". That I kind concur with him, instead of workaround missing functions it may be better to just write a compatibility library that complements musl with the missing functions. This way, any program that needs those functions could just import this library if they're compiling in a non-glibc system.
I am seeing that lots of the fixes proposed by the author are things like "missing import because glibc exposes some symbols by mistake and musl does not". And this is definitely the kind of fix that upstream would accept[2].
> This library is designed to be used for binaries that are already compiled
against glibc. It does not contain any headers, and cannot be used to build
software that requires glibc. It is instead recommended that any software that
requires glibc APIs be modified to become more portable.
This is exactly the opposite that I was talking about. What I was talking is a library that expose glibc compatible headers to compile software that uses Glibc-isms with minor or zero modifications, but this instead is a compatibility layer for binaries and doesn't expose headers (probably for proprietary software that can't be recompiled).
I recently removed most of openrc and started using only a custom inittab. The boot time was shortened by at least 10s and I have more control over what starts and when. I'm going to absolutely hate the day when firefox will depend on systemd (or systemd replaces firefox, whatever happens first).
As a user of both, I don't really see anything missing from OpenRC. Maybe parallel startup? Hasn't been a bottleneck for me, ever. I prefer the "syslog" way over journald but that's not anything like a deal breaker. What's features I'm missing?
It might have changed since I last used openrc but I remember there was no user services (I find this quite useful with systemd) and there was no socket activated services neither which is quite nice.
> It might have changed since I last used openrc but I remember there was no user services
Was looking into this exact thing this week. Amusingly, the suggestion I read on the gentoo wiki is to run a user instance of runit, which... I actually like that, but it does start to beg the question of why not use that everywhere.
All the declarative security sandboxing stuff. Last I checked openrc doesn't let you easily run a service in a process or network namespace. Or filter syscalls etc.
I also make heavy use of user services, TPM encrypted secrets, systemd tmpfile directories, etc.
Another thing: systemd-networkd is the only Linux networking stack I've managed to get working reliably with tailscale's dns and a private dns service on my workstation.
Popular linux based embedded project called yocto will benefit a lot when systemd and musl works well together, using systemd-glibc it brings a few MegaBytes to the build, which is sometimes too large. I'm sure musl can bring down the size a lot.
Except for that LinkedIn one (which feels like it might be a bug), it seems like there isn't really much in it, which is what I'd expect tbh. Kind of like Clang vs GCC. Sometimes one is faster, but probably not by much.
To ask a very stupid question: why don't they just replace it with a better one? Aren't allocator functions fairly well studied and by definition have source code already written? What's stopping them being able to just drop-in someething else to provide malloc()?
Yeah, I suppose that's fair. Still, the musl malloc implementation is just the most infamous example of its subpar performance. Obviously glibc comes with an enormous amount of functionality and performance optimizations.
Somewhat unrelated, but I'm looking forward to llvm-libc becoming a practical replacement for musl. I'd love a modern, non-gnu libc implementation that isn't so dogmatic.
This was a really well written blog post. I felt like I was able to follow the author's debugging journey very clearly.
Separately I've not heard of Adelie Linux and from a quick glance at their site I can't really tell what niche they're trying to fill. I gather they use musl and (before this post) openRC, and they aim for POSIX compatibility for legacy systems.
I skimmed it. TL;DR: Musl developers don't want to make it 100% compatible with glibc because a lot of glibc is retarded; Systemd developers don't want to avoid glibc functionality in favour of pure and holy POSIX because a lot of POSIX is retarded, and nobody actually runs pure POSIX Linux systems anyway.
Seems to me they are both right. Probably the best solution is a compromise where Musl implements some of the most commonly used / difficult to emulate functions, and systemd avoids using the rest. Won't hold my breath though.
The systemd people are absolutely not right here. Just look at some of the interfaces they won't give up. malloc_info writes XML to a FILE* with state of the allocator. That is a comically bad interface.
Why you'd even want the state it describes in the first place I can't tell. malloc, realloc and free are the complete primitives for 50 years but somehow systemd can't function until it can spit out a few numbers into XML on top of this, for reasons? In the patchset for TFA, I was surprised to see the comment for the commit that makes this functionality conditional is marked as "this can't be upstreamed." Makes upstream sound like a bunch of babies.
I imagine it was added for debugging purposes. It looks like systemd uses it for that purpose too. Not unreasonable, though I agree, XML is a stupid output format. Supposedly so the output format can change.... and silently break clients.
Really systemd should work with glibc to make their APIs saner, and then musl can copy them.
That would be amazing, but I don't imagine that malloc internal probes would be readily accepted to musl even if they were improved. One of the things musl admits is the ability to swap allocators around; you can just LD_PRELOAD a new one. Or even, with Chimera, replace the stock one too! Having internal debug probes would mean every allocator you want to use would need to provide those same APIs, and I don't find that particularly reasonable.
How is musl special when it comes to interposing malloc? As far as I’m aware, (dynamically) swapping out malloc implementations is a messy business no matter what. Most libc implementations only “support” the behavior in the sense that they won’t stop users from “swimming in the deep end”, so to speak.
malloc_info() is not consumed internally but it is only wired up as a debugging utility via systemd-analyze, which is sorely needed when things go bad in strange and weird ways. It would be entirely fine for a libc to just implement a stub that returns a fixed string:
Honestly thank goodness. Choice is great and all that but I don't want system to infect Alpine or it's ecosystem at all. I never, ever want to see an apk that requires systemd.
I don't know where this widespread hallucination that systemd is "forced" came from.
You're allowed to use another init system, and the distros CHOSE systemd. Nothing is stopping you, it's just that nobody does it because:
1. Pretty much nothing is as good as systemd as a whole project. Stringing together 30 random packages is a huge pain.
2. Other init systems are much slower, and turns out people care about performance.
3. Other init systems are less functional, with less features which some users want.
4. Writing a competing init system is very hard, while complaining is both easy and fun.
In the hypothetical case of Alpine switching to systemd, you're 100% free to maintain your own Alpine distro or even maintain your own init system or just switch to a different one.
I disagree with most of your points, which I will say are very subjective at a minimum, but specifically
> In the hypothetical case of Alpine switching to systemd, you're 100% free to maintain your own Alpine distro
This is the problem. I don't want to fork and maintain a distro. I'm just happy and thankful to find a minimal one which doesn't have systemd. It can really seem sometimes like a damn virus.
Right, so the problem isn't "choice" it's you being lazy.
Alpine HAS the opportunity to make a choice and you're arguing "no don't make a choice that makes it harder for me!"
The only anti-choice position here is YOU. Objectively. Now, I think your position is 100% fine and justified. But don't try to bullshit me with some "oh choice!" argument if that's the case. It makes it very hard to take these anti-systemd arguments seriously when people seem hell-bent on covering them in some ideological bullshit.
There's lots of good arguments against systemd in something like alpine. Use one of those, drop the political, made-up stuff.
> Right, so the problem isn't "choice" it's you being lazy.
I never said the problem was choice, and no, it's not just me being lazy. Be reasonable and debate in good faith, please.
The problem is having an ecosystem 'infected' with something that none, or at least most of the users do not want.
> Alpine HAS the opportunity to make a choice and you're arguing "no don't make a choice that makes it harder for me!"
I'd argue most of the userbase don't want that choice made either.
You're right of course, if Alpine does adopt, a fork will happen and that fork will be maintained, but I'd very much rather that choice not be made. That's all I expressed.
If the people who prefer to use a worse init system out of convenience wish to do so, they have no shortage of options. This constant splintering and forking of distros, is, I would argue, not a good thing in the shortrun. In the longrun it has no effect but time wasted.
Besides, honestly, if the systemd fans want Alpine with systemd, they should fork it, since they are a minority.
> The only anti-choice position here is YOU. Objectively.
Objectively, not a single thing I've said has been anti-choice. If you feel otherwise, could you quote the excerpt that you think can support your claim?
> Now, I think your position is 100% fine and justified.
Then it would seem most of your comment here is just to hear yourself talk.
> But don't try to bullshit me with some "oh choice!" argument if that's the case.
My only stance has been that I don't want systemd to infect alpine. That's it. You already said you think my position is justified, everything else you've written is arguin against men made of straw. Surely that's not the best use of your time?
> It makes it very hard to take these anti-systemd arguments seriously when people seem hell-bent on covering them in some ideological bullshit.
I see what it is now - you're just making a ton of assumptions, and you come from a place where you prefer and believe systemd is superior, so you don't have a ton of empathy for people that disagree.
So, imagine there is an objectively slow, bad init system, the worst of the bunch. We'll call it dunnit. SO now your distro of choice, against your and most users objections has decided to switch to dunnit.
You don't like that. You have a ton of infrastructure built around this distro, you know it inside and out, and now it's just a disruption. So, sure, you have the choice to fork it, or work with others and fork it, and start maintaining it.
But before your distro adopts dunnit, you could also express that you hope they don't and express dismay that they might. There's nothing wrong with that, it's not an anti-choice sentiment, and it's literally all I did in this thread.
If you go back and read my very first comment. The very first line!
> I don't know where this widespread hallucination that systemd is "forced" came from
This is what I'm addressing. I am not addressing systemd itself, but rather the ideological bullshit that's made up around it. You know, the stuff you subscribe to.
As I've already said, there's plenty of arguments you could make against systemd in Alpine. And I'd probably agree. If you're paying attention, I've already said systemd makes no sense in a distro like Alpine.
But don't try to convince me or anyone else systemd was, or is, "forced". That's just not in line with reality, which would make you delusional. Do you want to be delusional? No, right? Okay, we're on the same page then.
> I don't know where this widespread hallucination that systemd is "forced" came from.
Well, it's quite simple; users who didn't want systemd had it added to their systems, and were told things like
> you're 100% free to maintain your own Alpine distro
by people who pretended that that was practical, or
> just switch to a different one
except then that one also switched to systemd.
> Other init systems are less functional, with less features which some users want.
Yes, that's much of why systemd's EEE has been so effective: Nobody ever comes along and says, "we want a nicer service manager, so we'll adopt runit". They inevitably say, "we want this extensive laundry list of features that happens to exactly match what systemd provides, and nothing else provides this exact list, so obviously we should just adopt systemd". Somehow the fact that users want features systemd lacks never comes up.
> Somehow the fact that users want features systemd lacks never comes up
These? Don't exist.
Systemd is the most feature complete and it's not even close. Runit has nothing to offer that systemd cannot provide, but the inverse isn't even close to being true.
You have to realize these distros are usually general purpose who serve A LOT of use cases. They target desktops, home servers, professional servers, enterprise. That's a lot of different users with a lot of different needs.
Now you may think some of those needs/features are stupid, unnecessary, or (everyone's favorite word) "bloat". But the reality is some people rely on them so they push for them to be added.
Now you have a problem where you need all these features. Distros can choose to maintain 2 init systems (hard) or just go with the one that provides features. The choice is, and was, obvious.
> EEE
Oh, drop the delusion. Systemd didn't do anything, the distros chose this and they chose it because it's the best. If that makes you uncomfortable I don't care - in fact nobody cares.
It must be something in the air. I find flatpak, appimage, snap etc underwhelming. The go reimplementation¹ of appimage looks promising, but it would be nicer if one could more easily link Linux applications statically.
Another idea would be to build a "meta-libc", MIT licensed, which exports a glibc compatible interface but called musl or whatever else libc under the hood for implementation. This is slightly complicated by the fact that musl does not export any version define or symbol.
Or just fork musl and start adding glibc-isms to it...
I'm with you, if what you're saying is that containerized applications are a cop-out. On the other hand, they have their uses. For example Valve seems to have a hard time packaging Steam (being a weird 32bit web browser with DRM), but the flatpak (produced independently) works nicely, and because it's closed source it's your best option.
(Also I think it's too kind to include Snap with the others, I find it uniquely objectionable since it's neither convenient nor functional.)
edit: clarify that the Steam flatpak is not made by Valve
Steam flatpak was not created by and is not supported by Valve. It's an unofficial project, and flathub says as much (I'm assuming you mean the flathub package):
A user-friendly wrapper around nix would be much more functional (in all meanings of the word) and interesting than any of those solutions. Unfortunately, the world is moving in the opposite direction.
nix answers the question of how to ship an app and have it run on different systems that aren't all identical to the developers, by shipping a thorough recipe for how to construct the app and everything the app depends on in it's environment, and uses essentially a bunch of symlinks so that that app sees the environment it needs.
The rest of the world is adopting docker, flatpak, snap, appimage, etc which all just ship a whole os along with the app.
Nix is reinventing one aspect of SCO Unix but better, and flatpak/appimage/snap are reinventing static binaries but grossly worse.
Sco didn't have anything like the nix dsl and config files that really makes nix what it is, nor containers to provide virtual environments, just that all the os files were really just symlinks.
But now I would say, separately, that Nix probably is better than all-apps-are-containers, but anything is better than that so it's not remarkable and not what I meant.
One way nix's package configs are better than container packages is that even though each app gets to declare it's own custom versions or builds of all of it's requirements which might differ from their neighbors resulting in multiple copies:
1: anything an app doesn't declare is still shared.
2: even declared special versions are shared. The app only contains a config file that says it needs libfoo 1.2, or libbar built with non-default option Y enabled etc, not a copy of the library.
The os only needs to contain one copy of each unique version of something, not one copy per installed package.
It's better in several ways and the space consumption is probably the least important.
For one thing it puts the ultimate power of configuration back in the end user and distro maintaners hands instead of every random package developer putting whatever haphazard crap they want into your system just as lazy support for their app. Instead of getting a container with it's own sshd inside, you get a config file that someone better wrote that describes how to satisfy the apps needs in some acceptable way, not necessarily the same exact way the app developer did in their docker or flatpak.
The config also serves to expose everything which makes it easier to find and improve things.
A container that just includes the world and all it's 700 little unknown forgotten hacks is hard to audit and replicate. It's the laziest of lazy shitty solutions. It's just copying the developers laptop instead of figuring out and documenting how to produce something. "I have no idea how I got here, so here's just a copy of everything as it currently is."
But the equivalent config is like a diff that shows every single detail, and gives the os maintainer, the package maintainer, and the end user the chance to see, accept, or alter all of those details.
And I'm not actually a fan of Nix. It's just that package containers are such a lazy shitty gross solution that all other solutions are better.
I guess I just don't really think that Nix and Docker are opposite directions. This very much seems like a 'narcissism of small differences' situation to me.
In particular, what I find so misleading about saying its "moving in the opposite direction" is that it makes it sound like the success of Docker and whatnot is coming at the expense of approaches like Nix which I just don't think is true.
Almost nobody thinks docker-style containerization is the perfect solution to reproducibility, rather its a usable stopgap solution to just get something working without requiring the user to change their OS. In fact, I think a lot of the interest in, and adoption of Docker is making more and more people aware of the downsides and hitches with containerization, and is driving at least a subset of users towards Nix-style approaches. I think in the long run, this will benefit Nix.
> I find flatpak, appimage, snap etc underwhelming
Same here, in fact at home I avoid these like the plague. Work was a different story on RHEL :(
But if people what to see a great sandbox type setup, look at OpenBSD's pledge(2) and unveil(2). Easy to implement plus you do not eat up disk space with duplicate copies of everything.
We're talking about musl systems, yes? That's routers, ip cameras, ...
That C you're talking about still needs to read the config file (or files) from slow storage and then parse and apply them. Bash/Busybox ash does the exact same thing and it's also written in C.
In fact, the fastest, smallest boot process is busybox init + inittab + busybox again (ip, udhcpc, klogd, sysctl, mdev, mount, etc.). There are only 2 files to read from storage: bb and inittab.
But then again, dev don't care about efficiency. They only care about how easy it is for them to write the stuff. So, systemd ftw!!!
> Bash/Busybox ash does the exact same thing and it's also written in C
Yes, with the added benefit of an interpreter and shell implementation in between. Yippee!
> dev don't care about efficiency
systemd IS efficient. The hard truth is that it outperforms every other init system and it's not even close.
You could argue "well you don't need a fancy shmancy init system at all!!" - but even embedded applications grow and new requirements are added. Then what? Your "efficient" solution sucks, and isn't fast. Whoops!
personally I use a custom alpine live-usb that is immutable and I fully shutdown and start-up all the time. openrc doesn't give me any grief, the bulk of my startup time is copying my 500mb rootfs into a tmpfs
systemd for me is solidly "at work" software, and not stuff I'm a big fan of either (journald, networkd and the silly dns resolver all have serious issues). so please, keep away from my alpine or I'll have to fork -again- :)
reply