
Systemd as tragedy - wyldfire
https://lwn.net/SubscriberLink/777595/a71362cc65b1c271/
======
jimrandomh
Back in 2016, systemd started killing user processes on logout (rather than
send them the SIGHUP signal, as POSIX says should happen). This caused
problems for programs like nohup, screen and tmux, which deliberately keep
running. Systemd's response was to say that they should incorporate systemd's
library, and use systemd's new daemonization API. As far as I know, none of
them did.

Two years later, you can find hundreds of support requests across the
internet, from frustrated users who are having their sessions killed by
systemd.

Bugs are annoying, but that's life. On the other hand, when you're an impacted
user who's lost work, and researching the bug leads you to a years-old
discussion in which someone is actively denying that the bug exists and
refusing to fix it, that's _infuriating_. I don't think systemd's developers
deserve the trust that maintaining a core piece of infrastructure requires;
they don't seem to care enough about whether they've broken things.

~~~
poettering
You know, because we knew this would be controversial we made sure it was both
a compile-time option and a runtime option. Yes the upstream default of both
defaults to on, but that's just _upstream_. We made it very easy and supported
for downstream distros to switch between opt-out and opt-in of this option for
their users. We have encouraged distributions to leave it on, but we were
fully aware that for compatibility reasons this is something downstreams
likely wanted to turn off, and most compat-minded distros did, as we expected.

Now I am used to taking blame for apparently everything that every went wrong
on Linux, but you might as well blame your downstream distros for this as much
you want to blame us upstream about this, as it's up to them to pick the right
compile-time options matching their userbase and requirements in
compatibility, and if they didn't do that to your liking, then maybe you
should complain to them first.

(And yes, I still consider it a weakness of UNIX that "logout" doesn't really
mean "logout", but just "maybe, please, if you'd be so kind, i'd like to exit,
but not quite". I mean, that's not how you build a secure system. We fixed
that really, fully knowing it would depart from UNIX tradition, but that's why
we made it both compile-time and runtime configurable)

(Also, nobody has to "incorporate" systemd's library to avoid the automatic
clean-up. In fact, there's no library we provide that could do that. What was
requested though is to either run things as child of systemd --user or just
register a separate PAM session, neither of which requires any systemd-
specific library.)

Lennart

~~~
inferiorhuman
> Now I am used to taking blame for apparently everything that every went
> wrong on Linux, but you might as well blame your downstream distros for this
> as much you want to blame us upstream about this, as it's up to them to pick
> the right compile-time options matching their userbase and requirements in
> compatibility, and if they didn't do that to your liking, then maybe you
> should complain to them first.

It's up to you as a systemd developer to pick sane defaults. Claiming that
it's okay to introduce opt-out breaking changes upstream and then abdicate
responsibility is a quite bit like walking around while waving your hands and
arms around and then blaming whoever you hit for walking into you.

~~~
youdontknowtho
Wait a minute. Why isn't it the distro's responsibility to choose the most
compatible defaults?

~~~
inferiorhuman
If you smell shit while walking down the street, maybe someone dropped a deuce
on the sidewalk. If you smell shit everywhere you go, maybe it's you, maybe
you shat your pants. When you violate the principle of least astonishment
you're creating a huge stink.

That you can configure systemd to behave in a less obnoxious manner is well
beside the point. Systemd should be unobtrusive and predictable without any
extra action on the part of the distribution folks or end users.

That the suggestion is to simply read the code or documentation is the height
of arrogance considering how sloppy and insecure the systemd code is (parse
error equals root privileges? come on…).

~~~
SahAssar
Your argument assumes that systemd is simply meant to be a in-place compatible
drop-in for what it replaces, which I don't think is something anyone
would/should expect. If systemd was meant to behave the exact same way as
systems it is replacing then there wouldn't be much point of it. For those
cases it sometimes will break things, and will sometimes have settings to
follow previous behavior.

~~~
inferiorhuman
There's plenty of room within the POSIX specs to address service management
without requiring kernel integration, breaking userland tools, etc. When your
init replacement manages to interfere with the kernel you've done something
very, very wrong.

~~~
SahAssar
Not sure if I missed something here but how has it interfered with the kernel?
AFAIK it has broken some userland tools (which is bad in itself in most
cases), but actually breaking kernelspace is not something I've heard of.

~~~
inferiorhuman
[https://igurublog.wordpress.com/2014/04/03/tso-and-linus-
and...](https://igurublog.wordpress.com/2014/04/03/tso-and-linus-and-the-
impotent-rage-against-systemd/)

 _Yet just two days ago, we see Linus Torvalds (the creator of Linux and
maintainer of the Linux kernel), launching into a tirade against – yes, you
guessed it – systemd developers because of their atrocious response to a bug
in systemd that is crashing the kernel and preventing it from being debugged.
Linus is so upset with systemd developer Kay Sievers (gee, where I have heard
that name before – oh, that’s right, he’s the moron who refused to fix udev
problems) that Linus is threatening to refuse any further contributions from
this Red Hat developer, not just because of this bug, but because of a pattern
of this behavior – a problem for Kay because Red Hat is also foaming at the
mouth to have their kernel-based, no doubt bug- and security-flaw-ridden D-Bus
implementation included in our kernels. Other developers were so peeved that
they suggested simply triggering a kernel panic and halting the system when
systemd is so much as detected in use._

The key phrase there is:

a bug in systemd that is crashing the kernel and preventing it from being
debugged

Honestly though when you get Linus flaming your behavior you're doing
something really wrong.

~~~
Per_Bothner
_Honestly though when you get Linus flaming your behavior you're doing
something really wrong._

Haven't been around here long, have you? :-)

~~~
yellowapple
Likewise, of course, or you'd know that the tirades were more often than not
in response to things that were indeed "really wrong" (at least by his
standards).

------
pdkl95
> This argument, he said, seems to be predicated on the notion that systemd is
> a single, monolithic binary.

Can we _please_ stop misrepresenting the complaints against systemd? The only
time I ever hear this "monolithic binary" argument is from _systemd
advocates_. The actual complaint is about tightly coupling important features
together. Not only does this make it difficult (often impossible) to replace
individual components, when tight coupling happens at the (internal) protocol
level, any replacement component ne4cessarily hast to implement a bunch of
(sometimes unwanted) systemd baggage.

Busybox implements all of its features in single monolithic binary, but it
isn't a monolithic _design_ that tightly couples those specific components
together. Replacing one of busybox's components is often as simple as removing
busybox's symlink and installing the replacement. This isn't even a "Unix
philosophy" issue. Even inexperienced designers shouldn't have as hard time
Understanding why systemd is a monolithic design but busybox isn't.

~~~
chris_wot
What baggage are you specifically referring to?

~~~
lmm
It runs its own logging system with non-standard interfaces and formats. It
runs its own DNS resolver with non-standard behaviour. It maintains
compatibility only with a narrow range of udev versions, which in turn
maintain compatibility only with a narrow range of kernel versions. And all
the d-bus interfaces between these pieces may change at any point without
notice. So you can't replace any piece of it, because even if you provide your
own component that implements one of the systemd d-bus interfaces, you've got
no forward compatibility.

~~~
pas
If there was a serious effort to replace/port parts of it, the needed internal
APIs can be stabilized (
[https://www.freedesktop.org/wiki/Software/systemd/InterfaceP...](https://www.freedesktop.org/wiki/Software/systemd/InterfacePortabilityAndStabilityChart/)
).

------
ordu
_> "It's software" so of course it's buggy, he said. The notion that systemd
has to be perfect, unlike any other system, raises the bar too high._

systemd is a PID 1 program, it means it have to raise bar higher. When
troubles begin, you would need tools to fix them, and if PID1 is crashed, you
are out of luck. If system cannot boot into shell, you'd need to fix it from
initrd shell. Or to boot other system, to fix this one. It sucks.

Linux kernel chases very high standards of reliability, because when kernel
panics it is even worse than PID1 crash. Init system should follow the same
standards as linux.

~~~
geofft
Have you ever had pid 1 (systemd or any other init) crash? For the last ~three
years I've been paid to maintain high reliability algorithmic trading systems
that ran systemd and a whole lot of other stuff, and systemd has never crashed
on me. Lots of other stuff, including the kernel itself, has crashed.

The bar is higher for pid 1 - if I were designing systemd I would have made a
tiny pid 1 that just did message-passing to a more complex secondary process
that could be restarted, or something, just to be safe - but I think systemd
has empirically cleared the bar.

~~~
65a
There was that time it was bricking computers by erasing UEFI variables, but
I'll allocate equal blame between systemd and UEFI

~~~
chris_wot
Actually, that was nothing to do with systemd. That was definitely a UEFI
implementation issue. And systemd didn't delete anything, the user did - they
ran:

    
    
      rm -rf --no-preserve-root /
    

[https://lwn.net/Articles/674940/](https://lwn.net/Articles/674940/)

~~~
majewsky
The bug was in the kernel (it should not have allowed userspace to write
arbitrary UEFI variables), but AFAIR it was _exposed_ by systemd because it
eagerly mounted the UEFI variable filesystem provided by the kernel into
/sys/something/efivars.

~~~
chris_wot
Indeed, but again that was a firmware issue. systemd didn't delete the
variables. And systemd was setting EFI variables, so consequently it needed it
to be mounted as read/write.

The configuration files should have set that to read only after boot.

The kernel patch where this was fixed can be found here:

[https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0389075ecfb6231818de9b0225d3a5a21a661171)

------
shittyadmin
This article is a bit of a joke "It's software so of course it's buggy" isn't
a great argument when you're replacing something that didn't suffer the same
issues.

I just count my blessings that runit is widely packaged in every major distro
because it can just happily sit on top of sysvinit, systemd, upstart, pretty
much any init system and does things in a very simple shell script style, I
really wasn't a fan of the weird ini-like format for systemd or several
different tools I'm expected to learn just to read my (now binary) log files
competently.

If you're sick of switching init systems constantly or don't want to have to
write separate scripts for your linux box and your freebsd box even, I highly
recommend checking runit out.

I'm sure I'll give it a serious shot eventually... in about 3 years once they
work all the Poettering kinks out, just like PulseAudio. They're doing some
cool things with cgroups and stuff, so I hope it gets there eventually.

~~~
cetra3
I second using runit. We use runit to be able to use the same service
definitions inside docker, on a VM or bare metal.

If you've ever tried to use systemd inside docker to bring up a couple of
services, you would know the hoops you have to jump through to get it working.

(I understand that docker wasn't invented to run multiple services in the one
container, but sometimes it can't be avoided and simplifies app deployment
vastly I.e, using CI to test your service actually starts up as per its
definition: just run up a quick docker image with runit and a service
definition file)

~~~
pestaa
I've only seen supervisord as the root process in multi-purpose containers. Is
there any significant gain to using systemd instead?

~~~
cetra3
If you use systemd, you can use standard packages from your distro to run up
services inside a container. That's basically the only reason I considered it.

------
djsumdog
I hate how he talks about knee-jerk reactions to change. because I don't think
that's what's going on here. I remember I first saw systemd on a release of
opensuse. I didn't think anything of it at first, except that I didn't really
like the command line interface (systemctl) and I found the flags and options
cumbersome. I often see new software in new releases (including the old
HAL/DBUS layer) and didn't have the same reactions (although HAL had a lot of
issues and was later removed or merged into dbus).

I've seen the BSD talk on this and I agree, having a system layer is helpful.
It'd be nice if it was plugable, NetworkManager (or others that have some
standard messages you can send/get via dbus), consolekit OR logind, etc.

systemd does make it nice that I only have to write startup/shutdown scripts
once for each distro, but I'm not happy with the layout of target files, the
way mounts are handled, some of the weird race conditions I've found between
systemd mount targets and fstab, etc.

systemd is modular, but the modules are still all part of the whole and are
not easily replaceable. The same can be said when Docker went to a modeler
refactor, but there are alternative implementations of the entire docker
engine. Every attempt to create alternative implementations of systemd have
eventually gone unmaintained because systemd keeps getting more and more
complex and engulfing more systems.

If it wasn't for distros like Void, Gentoo, Alpine, Slackware, et. al, we'd no
longer have a choice at all. There would be some things that simply couldn't
be deployed on embedded systems because all of the dbus shims just wouldn't
exist.

It's not that people are opposed to change, it's that there are legit concerns
about some of the ways systemd works and is implemented, and the way it's been
ham-fisted as a political move in a lot of ways.

Honestly, I don't think it will matter in a few years. I think the way things
are going, eventually all services will be hosted via docker containers and it
will be much easier to make Linux distros that have a tiny init layer that
just launches a docker daemon and services. RacherOS already does this, with
the init process being a container, which can be uses to start up shell
environment containers and other service containers.

~~~
lsc
>It's not that people are opposed to change, it's that there are legit
concerns about some of the ways systemd works and is implemented, and the way
it's been ham-fisted as a political move in a lot of ways.

I personally think the industry needs a lot more resistance to change when it
comes to interfaces and other things humans have to understand.

I mean, I'm not talking about systemd in particular; I'm talking about in
general about how interfaces change over time and people don't seem to take
into account the cognitive costs of that change. Sure, ss is better than
netstat and IP is better than ifconfig... but how much of that 'better' could
you have done in a way that didn't toss away the historical knowledge so many
people have of those tools?

And really, sysadmin tools are the least of it; I mean, they are operated by
professionals, so if you want to pay for retraining (or pay the costs
associated with there being fewer of us)

People change customer facing interfaces to no benefit all the time, forcing
people who are trying to do other things to put effort into re-learning their
interface.

I mean, my point is that interface changes are expensive, and should not be
undertaken without a really good argument that they bring more benefit than
the cost of retraining.

~~~
pas
First we need sane (secure, semantic, programmatic) interfaces that slowly
become standards.

netstat? /proc files? ss? parsing text? wtf?

I mean, sure why not, but at least don't call them interfaces. they are
userland apps people like to script, because they are lazy to use libnetlink
(or libwhatever thay uses the right kernel interface, if it exists at all).

That said, the recent gmail ui change made me reconsider Thunderbird again.
And android looks different every year. sometimes it's better, sometimes it's
worse. iptables, nftables. http1, http2 (and now 3 over UDP). change is the
only constant.

~~~
lsc
You need to figure out what is wrong with a machine before you write a program
to fix it; this is why it's important to be able to log in to something broken
and nose around.

Text processing is not harder than figuring out what library to use this
month.

These things change a lot... but they don't have to, and running things on
computers would be easier/cheaper if they didn't.

------
Lazare
> When Lennart Poettering started thinking about the problem, he first looked
> at Upstart, which was an event-based system that was still based around
> scripts, but he concluded that he could do a better job.

I think I'd like systemd more if I had more confidence that he _had_ done a
better job.

I never really understood why Upstart didn't get more traction; was it just
the Canonical backing made other distros avoid it, or did it have drawbacks I
never ran across?

~~~
djsumdog
The problem is full process management, and in that regard, upstart was pretty
much still just scripts. I will say this, systemd tries to solve the problem
of tracking all children/grandchildren which is difficult to do with sysvinit
or scripts by themselves.

Systemd also solves the problem of unified ways to start services across many
distributions. Packaging things is easier. I don't like the systemd ini target
format and wish something else had won out instead. With something like
upstart, in theory you could rewrite the upstart daemon and use the same
scripts, since the format/standard was much simpler.

I feel like there must have been a lot of bullying to get systemd across so
many systems. It just seems weird to see so many major distros all decide to
accept something so complex if it wasn't for Redhat cramming it into
everything they maintained and helped fund.

~~~
JoshTriplett
> systemd tries to solve the problem of tracking all children/grandchildren
> which is difficult to do with sysvinit or scripts by themselves.

> Systemd also solves the problem of unified ways to start services across
> many distributions. Packaging things is easier.

> It just seems weird to see so many major distros all decide to accept
> something so complex

You answered your own question. Distributions adopted it because it solved
many problems for them.

~~~
MrTonyD
You know, he didn't say that. He said Red Hat adopted a complex system that
solves some problems - implying that they may have some other motivations. If
anything, the point seems to be that it is a very complex and unwelcome way to
solve some problems - and perhaps unworthy of Linux.

~~~
Tor3
I run Devuan on some boxes. Essentially Debian without systemd. No issues
whatsoever. Completely problem free.

------
aidenn0
One of my "favorite" corner cases caused by systemd was that if an NIS user
tried to login to the system, it would cause my X session to restart:

1\. They changed the defaults for systemd-logind to disallow network access

2\. When systemd-logind attempts to send a message to the NIS server, it
instead hangs (or has a timeout greater than the systemd watchdog anyways)

3\. The watchdog timer causes _all_ systemd-logind services to be restart,
which then tears down all of the child processes. My X session is a child
process.

------
m463
I believe systemd solves a problem or two.

However, I believe the implementation is not great.

A couple specific observations:

* lots of binaries. /lib/systemd, the logs. This more than anything else is a tragedy. Binaries get in the way of viewing, diagnosing, understanding and changing things. It seems like premature optimization to me - I don't think speed requires it. I think the majority are completely unnecessary.

* copying from launchd? Commercial software vendors ship binaries. They are aligned with them. They resist disclosure, which protects from reverse engineering, lawsuits and user modifications. Does that align with linux?

* the config files are a disorganized mess. It is like /etc/init.d but poorly understood and then /etc/rc _.d was lumped into the same directory and organization became chaos. I looked at one system here and in /etc/systemd there are 16 directories, 23 symbolic links and just 11 files. Try unraveling the interdependencies.

_ additionally the config files (which mimic windows config files) lack depth.
Simple config files can have benefits, say if a GUI read and wrote them to
change settings, but I don't think that applies here. What I noticed is that
they cannot do anything, so additional logic requires an intermediate script
or compiled binary elsewhere.

* it is responsible for too many functions. I bet it didn't solve so many in its first iteration, but sucked in more over time. It reminds me of the accounting program that grew to rule everything in Tron.

I wonder if these problems could be resolved. It's hard to remove complexity
and add elegance later.

~~~
poettering
I kinda like how a good chunk of the community complains we are "too
monolithic", and the other side of the community complains we have "too many
binaries". We can't win with you guys, can we?

Lennart

~~~
rauhl
Multiple binaries can still be part of a monolithic system (just like multiple
services can still be part of a monolithic system in the web services world).
A well-archetected system has strong separation of concerns — and well-
defined, reasonable interfaces — between components.

If I’m writing a piece of code and I start wanting to rewrite the world around
it, _that’s a sign that I’m probably doing the wrong thing_.

In the case of systemd, you actually have a decent case that a lot of things
in Linux _and the rest of the POSIX world_ could stand to be improved. A good,
clean, portable, systems-focused approach could tackle all those things head
on.

------
squirrelicus
I hate how the Unix philosophy argument is boiled down to "one vs many"
binaries. It's about independence to the degree of reusability. There's
nothing in systemd suite that's useful unless you're two feet in. You can say
things about streams and text and binary formats, but those are distractions.
Systemd tooling is tightly coupled to each other. It's a huge codebase, a huge
system that shoddily replaces that which was historically done with less. And
it's the biggest reason I don't run a Linux workstation anymore.

~~~
progman
Basically said, systemd follows quite the opposite of the KISS principle which
made Unix and Linux so great.

------
andrewstuart
I really like systemd.

It's got tons of really great functionality.

systemd feels like it is well thought out and modern, like it's an integrated
rethink and rebuild of lots of various software utilities that have evolved
over many years.

I'm investing all my learning in systemd where appropriate instead of things I
used previously like cron and supervisor.

~~~
CaliforniaKarl
I agree with that. But at the same time, systemd makes assumptions that I
don’t think it should.

Case in point: I deal with several clusters that use Stanford Central LDAP for
account info, and our UID numbers have gotten pretty high. So much so that it
overlaps the range systemd uses for dynamic service UIDs.

The biggest annoyance is that the UID range is hard-coded, so I can’t provide
an alternate range.

More details:
[https://github.com/systemd/systemd/issues/9843](https://github.com/systemd/systemd/issues/9843)
(but please don’t spam the GitHub issue!)

~~~
vinay_ys
For an open source Free software project, I find it really strange pottering
says this: "We don't recommend people to recompile their RPMs, the same way as
we don't recommend changing the UID ranges..."

[https://github.com/systemd/systemd/issues/9843#issuecomment-...](https://github.com/systemd/systemd/issues/9843#issuecomment-420384602)

~~~
CaliforniaKarl
From the perspective of a software author, I am not at all surprised.

If you are a user reporting a bug, and you say "I am using the RPM package of
systemd from CentOS 7.4", then I know which RPM to download to get exactly the
software you are using.

But, if you say "I am using the RPM package of systemd from CentOS 7.4, plus
my own patch", then things will be harder for me. Not only would I need to get
your patch, to be completely safe, I would need to get your build environment.
For example, which compiler you are using, and which version of the compiler.

Working off of a common set of pre-built packages makes problem reproduction
alot easier.

------
Kivutar
Some years ago, I was developing a Linux distro for embedded systems. Some of
these embedded systems come with old kernels, and you will never get a recent
kernel for them.

Systemd is written in C and relies on kernel headers. Which means that they
need to have a layer that deals with different versions of the kernel headers.
I remember it was headers about network.

At the time I was working on this, I had to contribute to systemd to add
support for my kernel version. Got a few patches accepted in their master. And
things were working fine for a while.

Until they suddenly decided to drop <= 3.10 kernel support. Yep. Afterwhat I
had to maintain patch sets for systemd, which is a great amount of job, given
that my upstream was updating systemd quite often.

Now my good people, explain me how such a mess could have happen with a
previous init based on shell?

~~~
CaliforniaKarl
I'm curious: Did systemd drop 3.10 support before the kernel developers
stopped supporting it (in 2017, see
[http://lkml.iu.edu/hypermail/linux/kernel/1711.0/03167.html](http://lkml.iu.edu/hypermail/linux/kernel/1711.0/03167.html)),
or before that? If it was around the same time, then I could understand
dropping support, since even the kernel developers had decided to move on.

------
chroma
It is my opinion that Lennart Poettering is the software equivalent of Thomas
Midgley Jr., the inventor of both leaded gasoline and chlorofluorocarbons.[1]

This is the person behind PulseAudio, Avahi, _and_ systemd. In any sane world,
these software projects would have been stillborn. Sensible alternatives would
have outcompeted them. Instead, Red Hat did their best to foist them upon the
community. They promised that after the initial adoption, all the wrinkles
would be ironed out. All three times this was not true.

I hate to appear so cynical, and I'm certain that Red Hat didn't intentionally
pursue this strategy, but it sure is convenient for them that the harder Linux
is to use, the more they can sell support contracts. Not so convenient for
people who actually want to use Linux.

1\.
[https://en.wikipedia.org/wiki/Thomas_Midgley_Jr](https://en.wikipedia.org/wiki/Thomas_Midgley_Jr).

~~~
ernst_klim
>Sensible alternatives would have outcompeted them.

Of course, and they didn't, because there were no sensible alternatives on
linux. All these projects were clones of decently engineered macos projects,
before they were created, linux people used some truly silly staff like
sysvinit.

I would use something better than pulse and systemd, but there is nothing
better available.

~~~
antt
Jack and runit.

They manage better performance in real world use cases with a fraction of the
manpower invested in them. I wonder what will happen to the suite of projects
supported by redhat now that they are owned by IBM however.

~~~
ernst_klim
>Jack and runit.

Jack as a regular sound daemon? are you joking? Have you tried to run jack
with several audio sources and audio outputs (including bt headset) without
pre-configuration? It's not a pulse alternative, it's a low-latency daemon for
professional stuff. Consumer oriented daemon should just work.

>runit

Does it do anything beyond service managing? Login, timers, bootloader,
containers, logging facilities?

>runit starts /etc/runit/1

Wait, does it simply run sh scripts? No, thank you.

------
myrandomcomment
I hate systemd with a passion, however I understand the reason for it. I think
that systemd is a bit Franklinstien by the nature of having to pull udev and
d-bus into it. So I guess it is a victim of having to deal with the system it
is on. Oh well.

I think the only Unix system to get this right was likely Solaris via SMF.

[https://en.m.wikipedia.org/wiki/Service_Management_Facility](https://en.m.wikipedia.org/wiki/Service_Management_Facility)

WindowsNT also did it correctly.

------
voltagex_
I'd recommend everyone watch the video:
[https://www.youtube.com/watch?v=o_AIw9bGogo](https://www.youtube.com/watch?v=o_AIw9bGogo)

------
gerbilly
There are parallels with city planning.

systemd reminds me of the efforts in some cities at clearing 'slums' to
replace them with 'modern' apartment blocks.[1]

sytemd seems like a similar technocratic, modernist effort aimed at UNIX.

Operating systems are kind of like cities. One the surface they are often
inefficient and messy, and people often wish they could come in and make them
logical and rational, but people live in those cities and neighbourhoods, and
sometimes the postbox is where it is for a reason.

If Jane Jacobs[2] were around, she could write a book called: "The Death and
Life of Great American Operating Systems"

[1] See:
[https://en.wikipedia.org/wiki/Pruitt%E2%80%93Igoe](https://en.wikipedia.org/wiki/Pruitt%E2%80%93Igoe),
[https://en.wikipedia.org/wiki/Quarry_Hill,_Leeds](https://en.wikipedia.org/wiki/Quarry_Hill,_Leeds),
[https://www.youtube.com/watch?time_continue=862&v=Dxr2tEGlWU...](https://www.youtube.com/watch?time_continue=862&v=Dxr2tEGlWUE)

[2] [https://www.citymetric.com/fabric/against-modernist-
nightmar...](https://www.citymetric.com/fabric/against-modernist-nightmare-
legacy-urbanist-jane-jacobs-2185)

------
_pmf_
If the current state is the tragedy, I'm terrified of the farce.

My favorite systemdisms:

* (already mentioned several times above; sorry) even detached and nohup'd processes will be killed if the systemd context that transitively led to the creation of the process is killed: [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394)

* mounts being in a separate "mount" namespace that is not accessible normally: [http://www.volkerschatz.com/unix/advmount.html](http://www.volkerschatz.com/unix/advmount.html)

------
mirimir
Damn. So I'm just a simple Debian user. Systemd showed up, so I learned as
much as I needed. And in some ways, I found it easier to use than init
scripts.

Now I've been using Docker. And there is no systemd in containers. But of
course, why would you want that? But the problem is that many packages now
depend on systemd. Or at least, useful features do. So I'm back to fighting
with scripts. It's funny, no?

~~~
tomjakubowski
Maybe you wouldn't want systemd, but a basic init system as PID 1 is wise for
a Docker container. See [https://github.com/Yelp/dumb-
init](https://github.com/Yelp/dumb-init)

~~~
mirimir
Wow. Thanks.

I've been wondering about all those zombies.

~~~
cyphar
I would suggest
[https://github.com/openSUSE/catatonit](https://github.com/openSUSE/catatonit)
(self-plug -- I wrote it). dumb-init (and tini which is what "docker run
--init" uses) don't handle zombies in the most ideal fashion possible,
catatonit does. And it does so in less code.

~~~
mirimir
Thanks. I use "docker run --init", and there are indeed zombies :(

I play a lot with shell scripts, and ps gets littered with grep and awk.

------
ComputerGuru
For all those lamenting the death of standards, change for the sake of change,
system breakage, bloat, and lack of user control: give FreeBSD a try (or any
BSD, really).

------
pgtan
systemd broke UNIX. Everything that worked before, now is a mine field. Last
week I had a SLES virtual machine refusing to boot, no network, just emergency
shell. The reason was a forgotten line in fstab (without nofail option of
course), so systemd couldn't mount some file system. Coming from Gentoo, this
is just unbelievable.

~~~
the_why_of_y
Systemd correctly implements the "noauto" and "nofail" keywords in fstab(5)
that were added in 2008 in the way they have been documented since 2010: if
neither of these is specified for a filesystem, then that file system will be
mounted at boot time and a failed attempt to mount that filesystem implies a
boot failure.

    
    
      Util-linux-ng 2.14 Release Notes (09-Jun-2008)
      ==============================================
      mount(8) supports new "nofail" mount option.
    

[https://github.com/karelzak/util-
linux/commit/abe3d704b6aeb6...](https://github.com/karelzak/util-
linux/commit/abe3d704b6aeb6b82ff32e8599edb56525cdfd72)

The fact that previous init scripts gladly ignored these failures does not
reflect negatively on systemd but on those init scripts.

Since 2011, FreeBSD has the same options with the same behavior, except that
"nofail" is called "failok".

[https://svnweb.freebsd.org/base?view=revision&revision=22283...](https://svnweb.freebsd.org/base?view=revision&revision=222832)

    
    
      Add a special mount option "failok" to indicate that the administrator wants
      the system to proceed to boot without bailing out into single user mode,
      even when the file system can not be successfully mounted.
    

By your argument, FreeBSD broke UNIX.

------
j16sdiz
I hate parallel service start up on server.

Yes, it is faster. But when it breaks because bad dependency, it is hard to
reason the breakage on a initrd shell on a small remote console.

I value debuggability, reliablity and manual fallback options on server.

~~~
kakwa_
I hate writing systemd .service files.

I'm always confused with the Type (should it be simple? forking?) and my
program ends-up starting but not stopping or things like that.

I know I lack the knowledge to write a good .service file in one go, as I
write one every other month at best and forget all the details in between. It
would be the same with a traditional init scripts. However, with init scripts,
the way to debug is obvious: just add a '-x' somewhere and here we go, you
have a detailed execution. I don't know the equivalent with systemd and I end-
up bashing my head against the wall trying everything blindly.

------
arminiusreturns
I particularly like how it ripped out the old logging system but completely
failed to replace it properly. Local might be ok, but shipping to a central
log server? Just save yourself the headache and use syslog or some other
shipper like filebeat, etc. How can something that can't even do logging
properly suddenly become the defacto standard?

~~~
newnewpdro
> I still think NSA used RedHat to push systemd for potentially nefarious
> reasons.

Wow, just wow. Do you have any idea how insulting this is to the folks
dedicating their lives to writing this GPL software?

You should try having an actual conversation with the authors some time. These
are real people and long-time GNU/Linux users and advocates you're implicitly
accusing of being NSA collaborators.

~~~
arminiusreturns
RedHats collaboration with NSA is well documented. I think given what we know
of past actions it's at best extremely naive to not at least entertain it as a
possibility.

Linus himself semi-openly admits he's been approached...

[https://www.youtube.com/watch?v=7gRsgkdfYJ8](https://www.youtube.com/watch?v=7gRsgkdfYJ8)

------
progman
I like to share the following links to those who also don't like systemd.

[http://without-systemd.org](http://without-systemd.org)

Devuan (Debian without systemd):

[http://devuan.org](http://devuan.org)

More Linux Distros without systemd:

[http://without-
systemd.org/wiki/index.php/Linux_distribution...](http://without-
systemd.org/wiki/index.php/Linux_distributions_without_systemd)

~~~
fanf2
Debian Stretch works nicely for me without systemd

------
CarlFK
Notable quotes

    
    
      22:10 "Unix is dead."
      25:28 "Change is awesome when we are the ones doing it.
      27:20 "No one needed to send death threats over a piece of software."
      27:30 "Contempt isn't cool."

------
ChuckMcM
From the article -- _Windows NT had a strong service model from the beginning,
he said, and Mac OS has one now in the form of launchd. Other systems had to
play a catch-up game to get there._

Reminds me of the saying, "and if your friends all jumped off a cliff, would
you jump off too?" which exasperated parents use to council their children
that emulating others needs to be done responsibly.

~~~
darkpuma
There is a big difference here. To continue with the cliff analogy: Launchd
ran fast enough to clear the cliff and was rewarded by a refreshing dive into
the cool water; it was implemented in a competent manner and consequently
didn't cause people to become upset. Systemd on the other-hand can't run
without stumbling, it hobbled off the cliff and bounced off every rock on the
way down. It made lots of people pissed off because it interfered with their
routines.

If systemd didn't cause problems for people, virtually no user would even know
what it was, and consequently, virtually no user would hate it. If you ask
your average OSX user what launchd is, they'll come up with a blank because
launchd isn't causing trouble for them. If you ask your average linux desktop
user what systemd is, a _large_ portion of them will recognize the name, and
many of them will chew your ear off out of frustration.

At the end of the day it's not about "adherence to philosophies" or any
similarly dubious concepts. Whether or not software is hated is ultimately
determined by whether or not it causes problems for people.

------
Karrot_Kream
We specifically use runit because we've had systemd issues in the past.
Systemd may have matured now but at this point we're moving over to containers
anyway, so it doesn't matter.

------
thekingofh
Can I easily replace systemd on a distro that came with it? I wouldn't
consider myself a Linux expert, but I got to a point where I could get around
and started to understand the different init runlevels and how to set them,
then everything was ripped out from under me and I no longer understand how to
work everything anymore. Perhaps Linux boots faster now, I don't know, but it
was really very frustrating I couldn't find a major user-friendly distro that
worked the way I was used to.

~~~
JdeBP
The writing was on the wall for run-levels for a _long_ time. Longer than
Linux has existed, in fact.

* [http://jdebp.eu./FGA/run-levels-are-history.html](http://jdebp.eu./FGA/run-levels-are-history.html)

------
throw2016
The author goes into 'contempt culture' which is valid in any group of people
ie 'PHP haters' but then tries to slip through the old trope of 'resistance to
change' which is a clever bait and switch that positions change as 'always
good' and refashions dissenters as anti-change which is a time tested PR
strategy to push through unpopular changes that benefit a few by positioning
any 'dissent' as 'anti change' and thus avoiding the meat of the discussion -
the change in question.

We can agree on the problem and then on the solution, but any proposed
solution can't just pass through 'without scrutiny' because a problem exists.

Systemd proponents since the beginning have positioned any criticism as
'motivated', 'greybeards', 'haters' or anti-change which tells you that there
is no space for discussion here, only 'acceptance'. Can there be any
legitimate criticism of systemd without this kind of politics,
hypersensitivity and now posing as victims?

~~~
aidenn0
Also the whole "One solution for people who don't like it is to create their
own alternative; that is a good way to find out just how much fun that task
is."

This seems to imply that those criticizing it didn't have alternatives, which
is just not true. However it is so monolithic that as soon as a tiny part is
depended on by something else (e.g. gnome) alternatives quickly become a non-
starter.

systemd seems to be getting rewarded by _not_ playing well with others which
is something that plays strongly into a lot of people's dislike with it.

------
dekhn
I've used linux systems since '93\. I waited until the last minute to adopt
systemd. So far, for me at least, it's been great. It's a lot easier to
express service dependencies, find problems when they occur. I hear a lot of
hate, but in all, systemd seems to solve a ton of problems in a single place,
which is something I think linux needed for a while.

------
stevefan1999
I reckon there's a SystemD bug that didn't mount LVM volumes correctly, some
has lost data that's real tragedy

~~~
JoshTriplett
Surely if there's an issue that serious, you have a bug report number?

~~~
shakna
Here you go: [0], from 2017.

[0]
[https://github.com/systemd/systemd/issues/6066](https://github.com/systemd/systemd/issues/6066)

~~~
JoshTriplett
So, fstab says to mount a filesystem, and systemd mounts the filesystem. The
reporter explicitly says they want the filesystem mounted at boot time. The
response is correct: there's no such thing as "mounted at boot time but not
'after system boot process completed'", because there's no magic "after system
boot process completed" point at which that behavior could change. When the
device becomes available it gets mounted.

sysvinit and typical distribution init scripts couldn't implement the behavior
seemingly desired by the user, either.

~~~
shakna
... All I did was find the issue number.

The ticket follows the sensible approach: Document it, and close it as
expected behaviour, and it did lead eventually to an LVM issue that did
require a fix [0].

systemd did good here, above board, but there was a faff about it, because the
expected behaviour of the audience differed from the reality of the situation.

[0]
[https://github.com/systemd/systemd/pull/6174](https://github.com/systemd/systemd/pull/6174)

------
darkpuma
> _" I'm not going to defend Lennart's approach to community interaction, but
> what I will say is you have to admire the willpower and determination of
> somebody who at work one day says 'I'm gonna write my own init system-
> actually no it's going to be a full on system layer for linux' [...] and
> then he gets it put into pretty much every major linux distribution."_

Yeah, I suppose I also admire the willpower and determination of Genghis Khan.

Corbet is _almost_ as dismissive as Lennart himself of the life-disruptive
bugs real people encounter due to Lennart's approach to software engineering
and project management. _" It has bugs? It's software"_ This attitude is
positively correlated with the amount of hate systemd (and pulseaudio) have
gotten over the years.

~~~
HarryHirsch
If planes fell out of the sky due to buggy software as often as systemd
crashes due to yet another bug, air travel would get much hate, too.

~~~
freddie_mercury
If systemd cost as much as a plane, it would have fewer bugs.

We can go back and forth with dumb comparisons all day but I don't really
think we're advancing the conversation.

~~~
HarryHirsch
I disagree about the value of the conversation. Airplane software is
meticulously tested because of the safety implications. If we can't have
proper testing for a widely used, sprawling piece of software that runs with
root privileges and PID 1 we may be better off without that piece of software.

Also this: the equally widespread SQLite seems to be programmed with a much
better level of discipline and comes with much better testing. It isn't nearly
as bug-ridden.

~~~
djsumdog
It's a good comparison because sqlite is used in many embedded aviation
applications.

I do get frustrated with sqlite a lot because it's missing so much of what I
expect in SQL (no ALTER tables, having to rewrite a lot of right joins into
other types of joins, etc.) but I have never had a situation where I got
frustrated with sqlite because it lost my data.

~~~
adontz
And still there are bugs in SQLite, despite unprecedented testing, best in
industry by opinions of many.

~~~
shakna
And SQLite respond to bugs with a lot more professional attitude. They
apologise, they fix. They try and ensure it won't happen in future. They do
not make excuses.

They don't dismiss concerns - they validate them.

~~~
adontz
I really don't think that often hated "Not my problem" kind of answers
Pottering gives are that unprofessional, because in 99% of cases, he points to
exact party who's problem it is and mostly they are distribution maintainers,
which makes sense.

~~~
Karunamon
Except, defaults matter. If you ship something that violates decades of well-
known behavior, it is at least _very_ unfair (and probably a lot of other
unpleasant adjectives) to expect downstream to enable the —suck-less flag to
restore the expected behavior.

The friendly and not bull-headed thing to have done was to ship the new
behavior, but enable the compatibility flag by default, rather than expect
other people to clean up after systemd's messes.

This mentality is the main reason bag on systemd all the time. If systemd does
something to annoy you, it is automatically your fault for doing something
Wrong or not enabling some obscure config file option. It's never systemd's
fault when systemd unilaterally violates the principle of least surprise.

~~~
adontz
>> Except, defaults matter.

Do they?

Debian maintains hella lot of patches for almost every package because of
reasons I can hardly justify. Overriding a few systemd config files is
nothing. But let's say systemd sets some sane defaults and everyone happy. Or
are they? Like, for instance default NTP is "centos.pool.ntp.org". Oh, crap,
Debian still can't use it.

------
jiqiren
My impression of systemd: shoddy version of launchd.

------
fmajid
Farce, not tragedy

------
merb
a lot of people shit on systemd because of the "SIGHUP"-"bug" however what
they don't realize is that there are more use cases where logging out a user
SHOULD kill all users processes.

I mean without that behavior linux is not a multi user system. The default is
sane, however the lack of "permissions" in this kind is also a little bit bad,
however since it is runtime configurable this should be enforced by the
distribution, i.e. when installing the distri there should be a switch "this
system is only used by a single user". it's basically the same as windows uac.
while it looks bad for a single user, it's not that bad when you see computers
as a tool which can have more than one user, with more than one "privilege".

system just shows that linux is by far not ready for the masses cause a lot of
behavior is undefined, like the uid/username stuff. usernames can be created
as you wish, but most programs will behave broken if you create usernames that
start with a number, it's a security nightmare.

and the problem that there is no real process to actually get ALL people
together who can solve this, it will never be fixed. there is only the posix
standard but not everything applies to linux and not everything is a good idea
in the sense of linux.

systemd actually makes a lot of things in the right way, but of course it's
not perfect and probably at a certain point in time people would come up with
something better.

------
jiveturkey
more like abomination

------
dingo_bat
My only issue is the binary log files. When you're writing out the logs to
disk, just make sure it is in text! How hard can it be?? It creates
unnecessary friction when dealing with logs from a crashed system.

~~~
XorNot
Does this ever actually come up for all the people complaining about it? Where
are these people working that they're not willing to install rsyslog which
reads journald logs directly and will write plaintext, but who have this
problem enough to warrant all the complaining?

Like, previously to have any logs at all you had to have a syslog daemon. This
is not a new situation.

~~~
inferiorhuman
> Like, previously to have any logs at all you had to have a syslog daemon.
> This is not a new situation.

What's new is that now more and more-difficult-to-debug pieces are required.
For what benefit?

------
RemarkableMan
I just don’t have time to beta test an init system with more LoC than the 2.6
kernel plus GCC 4. For now Devuan is where we landed but moving entirely away
from Linux as Redhat moves system d into the kernel is the overall plan.

------
yarrel
_First_ as tragedy...

------
anfilt
Systemd just makes things worse.

I remeber seeing someone post one of the simplest init process one could
write. It was only about 100 lines of C.

~~~
geofft
It's much less than that, but it's sort of an uninteresting demo I think - you
could pretty easily write an init process that just spawns systemd in a
container, for instance, and run the whole system in a container. Or write an
init process that spawns systemd as pid 2, runs prctl(PR_SET_CHILD_SUBREAPER)
to make pid 2 responsible for reaping instead of pid 1, and then sits around
carefully doing nothing. If systemd crashes, whatever, hopefully sshd etc. are
still up.

Of course, if you do that then the question is why even bother with an init
process. Patch the kernel to not treat pid 1 specially. If a process gets
orphaned, don't reparent it to init, just reparent it to nonexistent pid 0.
Have the kernel deal with the thing that the 10 lines of C would be doing
(waiting on processes to terminate).

I think there are meaningful complaints about systemd but the fact that it's a
complicated pid 1 is not one of those. If it were actually a problem someone
would have written and productionalized one of the above two approaches.

~~~
anfilt
It was code that even let one make simple files to start up one shot process
or daemons. If a long running daemon died it would restart it. Plus it handled
the getting the status of orphaned processes, and few other things.

Really the biggest issue with systemd is is scope creep scope creep means more
and more packages will have systemd as a hard dependency. Like there some
aspects of gnome that will not work without systemd, but at least it's not
entirely a hard dependency just means you lose features unless you patch
gnome.

~~~
geggam
Yet OpenBSD has Gnome without systemd. So it is possible.

~~~
anfilt
So does gentoo, but if you look at there docs if want some things you may need
to patch it.

------
makz
As a conspiracy theorist, I'm going to say that systemd is aimed towards
destroying Linux and everything it represents.

They want to push the cloud, to better control us, take away power from us,
leave us at their mercy.

The next step is, when everything that matters is on the cloud, they are going
to replace Linux with something else that only they know how it works on the
inside, and that's going to be the end of it, we'll be left only with
paywalled APIs and services, but they'll say it's better for everyone and all
of that. It's sad.

~~~
newnewpdro
You're confusing systemd with docker and kubernetes.

~~~
busterarm
I hate how accurate this is. The more I work on Kubernetes implementations,
the more I feel like it's a tool that 99% of companies can't effectively
operate. The attitude is very reminiscent of the mainframe era where ops
didn't want mere users touching their precious systems.

I feel like as an industry we're wasting tons of cycles solving the wrong
problem.

------
mantlepro
A well-written, thoughtful article, worth the read.

> Nerds have a complicated relationship to change; it's awesome when we are
> the ones creating the change, but it's untrustworthy when it comes from
> outside.

~~~
mantlepro
Yikes! How in the world did I lose 2 of my 3 karma points on this?! Somebody
please explain and upvote? Linux user since the 90s but totally new to Hacker
News. I'm good with constructive criticism, opinions, etc. but not a fan of
downvotes without explanation.

