
Why Did It Take Devuan Two Years to Replace Systemd? - buttcake
http://troubleshooters.com/linux/systemd/bikebrakes.htm
======
KirinDave
I keep waiting for the reasoned, technical arguments against systemd instead
of veiled metaphors, complaints about its creator, or protestations that it's
different.

The initd-inspired community has resisted the push for declarative and
component-based core system infrastructure for so long, and it's causing very
strange distortions to the linux ecosystem as solutions get sought for the
problems it introduces.

Systemd, for all its flaws, fixes some very important problems that init and
runit and similar systems left unsolved. It's complicated, but it's hard to
argue it's unnecessarily so. We always end up at these tortured bike metaphors
or some other garbage. There's a whole website devoted to criticisms and it
feels like at least half of the site is devoted to this kind of baseless
rhetoric.

As far as I can tell, systemd is a holy war. It's the new editor debate. It's
vi vs emacs vs sublime vs atom vs ... down throughout time, with people
arguing where the tradeoff between simplicity of execution is more important
than delivering a system people asked for.

~~~
pmoriarty
One of the main problems was the speed with which systemd was rolled out on
all the major Linux distros, despite it being relatively immature and very
controversial at that.

The second major issue is it growing way out of the bounds of being just an
init replacement, thereby violating the so-called Unix Philosophy of doing
one-thing well. For a user app to do that is one thing, but for a core OS
component to do that just smells way too much like forcing Linux to be more
like Windows, which a lot of hard-core Linux users are ideologically opposed
to.

The third major issue is that on most major Linux distros, users are not given
an easy way to avoid using systemd, no matter how much they hate it or oppose
it.

The fourth issue is the apparent arrogance of the people responsible for
systemd, and their off-hand dismissal of mature and widely respected unix
conventions.

All of this adds up to a storm of controversy that the systemd people mostly
brought upon themselves. Had they just been more humble about their creation
and waited until it was mature and well-tested and did all the wonderful
things they claim it could and should do instead of stuffing what was widely
seen as a broken-by-design piece of garbage down everybody's throat, maybe
much of the Linux community wouldn't have been nearly so outraged by it.

~~~
happypants23
_The second major issue is it growing way out of the bounds of being just an
init replacement, thereby violating the so-called Unix Philosophy of doing
one-thing well._

I keep on hearing this overly worn "Unix Philosophy" argument against systemd,
which in the absence of any non-religious technical justification, basically
boils down to "its different".

 _All of this adds up to a storm of controversy that the systemd people mostly
brought upon themselves. Had they just been more humble about their creation
and waited until it was mature and well-tested and did all the wonderful
things they claim it could and should do instead of stuffing what was widely
seen as a broken-by-design piece of garbage down everybody 's throat, maybe
much of the Linux community wouldn't have been nearly so outraged by it._

Nobody shoved anything down anyone's throat, least not the SystemD developers.
And the same goes for PulseAudio, DBUS, and all the other projects on
freedesktop.org.

Anyway, in reality, software does not become mature and well-tested without
early exposure to real users and systems. Chicken-and-egg problem there.

I feel absolutely no sympathy for those who are outraged. There are many
distributions out there (including Devuan) which align with their particular
vision of an OS should be.

~~~
sliken
Why does an init system need a DNS implementation? Why not use named, unbound,
or any of the other implementations?

Instead they implement it poorly, have had various problems with it, and of
course there have been security problems with it.

Do you really want an init system that replaces a ton of functionality poorly
like dns, ntp, syslog, xinetd, etc etc etc?

~~~
happypants23
> _Do you really want an init system that replaces a ton of functionality
> poorly like dns, ntp, syslog, xinetd, etc etc etc?_

SystemD is not just an "init" system. You have to look at the larger vision
for it being a set of building blocks for Linux system and service management.

I think this is a very good thing. Its about time that Linux had a unified and
self-consistent approach to system management and configuration.

~~~
pmoriarty
_" SystemD is not just an "init" system. You have to look at the larger vision
for it being a set of building blocks for Linux system and service
management."_

This was not the way it was sold. It was originally sold as an init
replacement.

If at the start the systemd guys had come out and said, "We aim to replace
much of the core of Linux with some kind of Windows-Linux conglomoration:
replace not only init but dns, syslog, turn all system log files binary,
replace intetd, fuck with account naming, and oh, by the way, there'll be no
easy way for distros that go with systemd to give their users a choice to use
something else" then I'm not so sure how many people would have gotten
onboard.

Basically, systemd has been a retooling of the core Linux ecosystem by people
who think they know better. Well, this might be news for systemd fans, but
they don't necessarily know any better than the rest of us, and the rest of us
would like some choice in the matter. Instead we're force fed systemd and our
operating system are no longer Linux but some bastardization of Linux and
Windows.

 _" Nobody shoved anything down anyone's throat, least not the SystemD
developers."_

Tell that to the significan number of RedHat, Debian, and CentOS users who
strongly object to systemd yet are forced to use it because that's all their
distro supports.

 _" Anyway, in reality, software does not become mature and well-tested
without early exposure to real users and systems. Chicken-and-egg problem
there."_

You don't have to test it out on all RedHat users, all Debian users, all
CentOS, etc, etc. Getting a bunch of _volunteers_ and letting them use a
feature-complete version for five or six years before springing it on the rest
of us would have been a good start. Instead you get an ever-morphing and ever-
growing trojan horse.

~~~
dijit
You will likely have this argument again so I need to point out two things:

1) Debian does support sysvinit right now, although with some packages having
hard dependencies on systemd it might not be long before you hit a brick wall.

2) systemd (the init) was used by Fedora for a long time before getting into
RHEL/CentOS- which is of course before it got into Debian. So there were
volunteers.

------
wirrbel
Sure, it might be because System D integrates so tightly with the whole
ecosystem that it is hard to replace it.

Or: Those folks who write rants about system d are not very active in
contributing to a distribution that runs without it. With some discussions,
you have that feeling that there are "those who talk, and those who do".

~~~
slau
Or maybe we just gave up on Debian, and decided to spend our efforts somewhere
else?

I've been fairly vocal against systemd, and not because of Poettering. I'm no
fan of the man, and I strongly agree with a lot of the points he made in his
initial blog posts when introducing systemd. I think he's probably very
talented---I'm not in a position to judge that. I wouldn't be against systemd
if it were _just_ an init system. The problem is that it's not. That's the end
of my argument. If systemd were just an init system, I'd be very happy to have
it on my machines. Poettering even admitted this himself [1]:

> Well, systemd certainly covers more ground that it used to. It's not just an
> init system anymore, but the basic userspace building block to build an OS
> from, but we carefully make sure to keep most of the features optional. You
> can turn a lot off at compile time, and even more at runtime. Thus you can
> choose freely how much feature creeping you want.

In response, I've decided to not use systemd on my machines. I'm a patreon of
xtraeme [2], the creator of Void Linux [3] and xbps (the package manager used
in Void). Void Linux uses runit [4], which is exactly as simple and fast as
I'd like it to be [5]. Working in startups, I haven't had time to contribute
code to either xbps or Void, but at least I'm contributing in the way that I
can. I didn't contribute to Devuan (other than seeding a few hundred TB of
ISOs) for the same reason that I haven't contributed to Debian in a long time:
I don't use it any more.

[1]: [http://0pointer.de/blog/projects/the-biggest-
myths.html](http://0pointer.de/blog/projects/the-biggest-myths.html)

[2]: [https://www.patreon.com/xtraeme](https://www.patreon.com/xtraeme)

[3]: [https://www.voidlinux.eu/](https://www.voidlinux.eu/)

[4]: [http://smarden.org/runit/](http://smarden.org/runit/)

[5]:
[https://www.voidlinux.eu/usage/runit/](https://www.voidlinux.eu/usage/runit/)

~~~
jcrites
> I wouldn't be against systemd if it were just an init system. The problem is
> that it's not. That's the end of my argument.

All of systemd's functionality, as far as I'm concerned, is rationally
motivated by wanting to orchestrate and instrument system startup in a
sensible way. What functionality do you think systemd should not have that it
has today? What is a better way of achieving the result provided by that
functionality?

If your perspective is that you don't care about that functionality or the
related user experience, and so all of its complexity is dead weight to you,
OK. That makes sense. But as a software engineer and system administrator, I
find the features extremely useful, and I think many other people do too.
That's why systemd has been adopted by major distributions.

I wrote a comment earlier [1] about systemd that reviews Russ Allbery's
analysis of systemd [2], from when Debian was evaluating switching to systemd
and upstart. Russ's comment reviews a number of different systemd functions,
such as its integrated journal, and then explains the pros and cons of each.
Russ even said:

    
    
        Integrated daemon status.  This one caught me by surprise, since the
        systemd journal was functionality that I expected to dislike.  But I was
        surprised at how well-implemented it is, and systemctl status blew me
        away. [lots more details in linked comment]
    

[1]
[https://news.ycombinator.com/item?id=13387989](https://news.ycombinator.com/item?id=13387989)
[2] [https://lists.debian.org/debian-
ctte/2013/12/msg00234.html](https://lists.debian.org/debian-
ctte/2013/12/msg00234.html)

Sure, you could argue an init system "shouldn't" have an integrated journal
(for example), but then you'd lose the clear benefits that Russ outlines. You
wouldn't achieve the same benefits or user experience. Similarly, systemd's
configuration-driven approach to service definition makes it possible to
employ a wide variety of standard configuration options across all services.
From my previous comment:

> I can launch my service at the appropriate time during boot with
> configuration as simple as:
    
    
      [Unit]
      Description=Demo service
    
      [Service]
      Type=forking
      ExecStart=/usr/sbin/my-daemon
    

> Now let's say that I didn't author this daemon, but I'd like to run it with
> a private network, private temp folder, or a private /dev namespace. Or
> perhaps the daemon needs to run as root, but I want to drop all capabilities
> it doesn't need. It's as simple as adding these lines to the service's
> configuration:
    
    
      PrivateTmp=yes
      PrivateDevices=yes
      PrivateNetwork=yes
      CapabilityBoundingSet=CAP_NET_BIND_SERVICE
    

> The fact that systemd supports these configuration options means that
> there's a simple and standard way to employ them with any service. The
> service itself doesn't need to support them, and needn't complicate its own
> daemonization logic to do so correctly. Indeed, I don't need to trust the
> service to daemonize or drop capabilities, since I can tell the init system
> do that before launching the service.

> I can drop capabilities with CapabilityBoundingSet=, or limit resource usage
> with CPUSchedulingPriority=, IOSchedulingPriority=, etc. I could even tell
> systemd to open the listening socket for me so the service doesn't need
> CAP_NET_BIND_SERVICE! Moving these options into the init system makes a ton
> of sense, because it gives administrators the ability to employ these
> features from outside applications, not just by enabling them within
> applications that bother to explicitly support them via command line
> arguments. Systemd better encourages the principle of least privilege: if a
> system daemon does not need the ability to "ptrace" other processes, or bind
> to ports <1024, then as the administrator I can take those away with
> CapabilityBoundingSet= in the unit file. Chrooting the service is as easy as
> RootDirectory=. This is a huge step forward compared to the world where
> every service must be relied upon to expose these settings, and must be
> trusted to implement them correctly.

~~~
dozzie
> All of systemd's functionality, as far as I'm concerned, is rationally
> motivated by wanting to orchestrate and instrument system startup in a
> sensible way. What functionality do you think systemd should not have that
> it has today?

QR code generator? Message broker (IPC)? Underdeveloped, brittle, and
overcomplicated syslog replacement? Task scheduler?

~~~
jcrites
> Underdeveloped, brittle, and overcomplicated syslog replacement?

Are you referring to the systemd journal? I think you should lay out your
arguments in detail if you want to make claims like that, since there is a lot
of compelling evidence to the contrary. Russ Allbery's analysis of systemd [1]
lays out a convincing case that the journal is a highly developed, flexible,
and useful feature:

    
    
      * Integrated daemon status.  This one caught me by surprise, since the
      systemd journal was functionality that I expected to dislike.  But I was
      surprised at how well-implemented it is, and systemctl status blew me
      away.  I think any systems administrator who has tried to debug a
      running service will be immediately struck by the differences between
      upstart:
    
        lbcd start/running, process 32294
    
      and systemd:
    
        lbcd.service - responder for load balancing
         Loaded: loaded (/lib/systemd/system/lbcd.service; enabled)
         Active: active (running) since Sun 2013-12-29 13:01:24 PST; 1h 11min ago
           Docs: man:lbcd(8)
                 http://www.eyrie.org/~eagle/software/lbcd/
       Main PID: 25290 (lbcd)
         CGroup: name=systemd:/system/lbcd.service
                 └─25290 /usr/sbin/lbcd -f -l
    
      Dec 29 13:01:24 wanderer systemd[1]: Starting responder for load balancing...
      Dec 29 13:01:24 wanderer systemd[1]: Started responder for load balancing.
      Dec 29 13:01:24 wanderer lbcd[25290]: ready to accept requests
      Dec 29 13:01:43 wanderer lbcd[25290]: request from ::1 (version 3)
    
      Both are clearly superior to sysvinit, which bails on the problem
      entirely and forces reimplementation in every init script, but the
      systemd approach takes this to another level.  [...] the most useful
      addition in systemd is the log summary.  And that relies on the 
      journal, which is a fundamental  design decision of systemd.
    
      And yes, all of those log messages are also in the syslog files where
      one would expect to find them.  And systemd can also capture standard
      output and standard error from daemons and drop that in the journal and
      from there into syslog, which makes it much easier to uncover daemon
      startup problems that resulted in complaints to standard error instead
      of syslog.  This cannot even be easily replaced with something that
      might parse the syslog files, even given output forwarding to syslog
      (something upstart currently doesn't have), since the journal will
      continue to work properly even if all syslog messages are forwarded off
      the host, stored in some other format, or stored in some other file.
      systemd is agnostic to the underlying syslog implementation.
    

[1] [https://lists.debian.org/debian-
ctte/2013/12/msg00234.html](https://lists.debian.org/debian-
ctte/2013/12/msg00234.html)

The benefits that Russ describes above are meaningful, tangible improvements
to the user experience that helps system administrators like me. Far from
being brittle, the journal seems quite flexible (being "agnostic to the
underlying syslog implementation"); it seems highly-developed and well-
integrated into the systemd toolkit, and I see no evidence that it's over-
complicated.

> Message broker (IPC)?

Are you referring to dbus? Dbus is not part of systemd and predates systemd by
almost a decade! Dbus already had significant adoption when systemd took a
dependency on it. It's used by GNOME, KDE, and even the Upstart init system.
From Lennart Poettering's blog [2]:

    
    
      D-Bus has been a core component of Linux systems since more than 
      10 years. It is certainly the most widely established high-level
      local IPC system on Linux. Since systemd's inception it has been
       the IPC system it exposes its interfaces on. And even before
      systemd, it was the IPC system Upstart used to expose its
       interfaces. It is used by GNOME, by KDE and by a variety
      of system components.
    

[2] [http://0pointer.net/blog/the-new-sd-bus-api-of-
systemd.html](http://0pointer.net/blog/the-new-sd-bus-api-of-systemd.html)

~~~
dozzie
> [...] Russ Allbery's analysis of systemd [...] explains why it's such a
> compelling feature:

It's tightly integrated where it doesn't need to be (with zombie reaper), you
can't replace it (only slap something on top of it), and generally those few
last log lines that are displayed are useless anyway, as usually when
something breaks I still need to view full logs. Much more useful would be
debuggability (with syslog this part is easy), remote data collection (syslog
has it for decades already), and storage resilience (journald can corrupt log
files on system crash).

> Are you referring to dbus? Dbus development began in 2002 whereas systemd's
> first release was in 2010.

When what was started is pretty much irrelevant. I didn't say we don't need
D-Bus as a message broker (though I consider sub-par the way D-Bus works), I
said that we don't need it in neither zombie reaper nor daemon manager. It
worked well enough being a stand-alone service for a decade.

And now, you haven't addressed the glorified zombie reaper including a task
scheduler and QR code generator. Why do you think their integration is so
important that they needed to be included in systemd?

~~~
jcrites
Any discussion about this should be anchored with an answer to the questions I
asked earlier: What functionality do you think systemd should _not_ have that
it has today? What is a _better_ way of achieving the benefits provided by
that functionality?

Let's talk about logging. The systemd journal is designed to capture and log
the output of all programs. Some programs do not integrate with syslog: they
write to stdout/stderr instead. As a system administrator, I desire a way to
capture the logs of everything I run on a system, including programs that do
not integrate with syslog (e.g., ad hoc shell scripts and applications).

Who is in the best position to capture program stdout/stderr? The process
supervisor. Systemd does that with its journal facility, and other supervisors
work similarly. Personally, I value having a uniform method for capturing
program logs from all processes on a machine, and that's what systemd offers
me. Syslog by itself does not achieve this goal.

How do you recommend capturing and logging program output, if not by having
the supervisor do it? (And why is that approach preferable?)

The journal is an integrated part of the systemd toolkit, and that's a good
thing because it enables useful capabilities like capturing stdout and good
user experiences like `systemctl status`. But you don't have to interact with
it. If you prefer the syslog toolchain instead, then redirect the journal to
syslog via ForwardToSyslog=yes in journald.conf. The journal still adds value
with this approach because it captures logs that syslog can't (and without
kludges like `logrotate` and `logger`).

You can see more log lines with e.g. `systemctl status -n50`. If you want to
see everything, invoke `journalctl -u service-name`.

I view the systemd journal as a logging facade, a _complement_ to facilities
like syslog, not an _alternative_ to it. As Russ said in my quote above,
"systemd is agnostic to the underlying syslog implementation."

I'm not sure I grasp your point about dbus. I have not seen anyone make the
argument that dbus should be part of a zombie reaper or process supervisor.
However, IPC is a useful and sensible way for processes on the system to
communicate with their process supervisor, which is why both systemd and
upstart integrate with dbus. I've read that it is possible to use systemd
without dbus [1] (there are apparently Debian configurations that work this
way), though I haven't deeply looked into it.

If you want to launch supervised processes with clean environments,
namespaces, and cgroups, then it is necessary for a system process to be the
one that forks. Communicating that request between processes necessitates IPC.
As the most widely adopted IPC system for Linux, dbus was an appropriate
choice for systemd and upstart to integrate with. Again, going back to our
anchors: Is there another IPC framework you think systemd should have
integrated with instead? Or if not by using IPC, how do you think service
management commands (e.g., restart a service) should be communicated to the
system?

[1] E.g.
[https://github.com/mitchellh/vagrant/issues/7973](https://github.com/mitchellh/vagrant/issues/7973)
[https://github.com/aelsabbahy/goss/issues/101](https://github.com/aelsabbahy/goss/issues/101)

~~~
dozzie
> Any discussion about this should be anchored with an answer to the questions
> I asked earlier: What functionality do you think systemd should not have
> that it has today? What is a better way of achieving the benefits provided
> by that functionality?

I already replied to that.

> How do you recommend capturing and logging program output, if not by having
> the supervisor do it? (And why is that approach preferable?)

Supervisor is in appropriate place only for substituting STDOUT and STDERR
descriptors at the daemon's start, but once that is done, the descriptors are
easily transferrable to logging daemon. Obviously, the logging daemon should
understand that, so rsyslog would need a plugin to accept those descriptors,
or a generic intermediate daemon that reads all the output and sends it
further to syslog would be needed, but that's it. There's literally no benefit
in coupling log collector with zombie reaper as tightly as in systemd.

> The journal is an integrated part of the systemd toolkit, and that's a good
> thing because it enables useful capabilities like capturing stdout

...which is easily achievable without coupling...

> and good user experiences like `systemctl status`.

...which is useless anyway, as I said in my earlier post. Oh, but this "good
user experience" is indeed really excellent, because when I start a daemon and
it dies, `systemctl start' does not print the information that daemon failed
to start nor the daemon's initial logs. I get _nothing_ and I _don 't know_
that the daemon died. Indeed, great user experience. Or updating unit files,
again, great experience when the unit file acts in exactly the same way as
previously, because I forgot to reload systemd and it didn't inform me to do
so. Or systemd automatically chopping log output to my terminal. Or awful
command line syntax of journalctl. Top notch user experience indeed!

> But you don't have to interact with it. If you prefer the syslog toolchain
> instead, then redirect the journal to syslog via ForwardToSyslog=yes in
> journald.conf.

So this tight coupling is good, because it gives me nothing useful over not
coupled architecture and I can easily work around this coupling? This is what
you're trying to say?

> The journal still adds value with this approach because it captures logs
> that syslog can't (and without kludges like `logrotate` and `logger`).

First, syslog _can_ capture this output. It's not impossible and it's not even
difficult.

Second, these "kludges" that _worked well for decades_ are replaced with (a)
brittle disk format that breaks apart on system crash, (b) lack of control
over how long the logs are kept, and (c) lack of remote log collection.

> Is there another IPC framework you think systemd should have integrated with
> instead?

You're asking wrong question. It's not that systemd should not _integrate_
with D-Bus, it's that it should not _swallow_ it.

And now I'll ask _once again_ : what is the so important reason for systemd to
provide QR code generator and task scheduler?

------
korethr
This is why I've not been so enthusiastic about the systemd project. When I
first heard of systemd, it seemed like a good idea: A new clean-slate design
of an init system and service management that aimed to address shortcomings of
the present common init and service management. And had the systemd project
stayed just that, I'd have been fine with it, and may well have welcomed it
onto my systems with open arms.

But then it became necessary to subsume udev and syslog and cron and more
parts of the system, and each time, the justifications for this scope creep
have been unconvincing to me. It comes across as a bunch of added complexity
because it'd be nifty and because they can, without stopping to reflect on
whether trying to subsume the rest of the OS or forcing tight coupling and
integration is a good idea from a design or architecture perspective.

------
kazinator
Say I want to create an independent, 100% compatible implementation of systemd
which can just drop-in to any systemd installation, no matter how it is
configured and have it work?

Where can I download the precise set of requirements for this, so that I don't
have to peek at the systemd source code?

 _What_ is systemd?

~~~
ilovecookiestoo
Say I want to create an independent, 100% compatible implementation of Linux
which can just drop-in to any Linux installation, no matter how it is
configured and have it work?

Where can I download the precise set of requirements for this, so that I don't
have to peek at the Linux source code?

What is Linux?

Really, this doesn't make sense.

~~~
digi_owl
How can people drop a comparison between a kernel and a supposed init and not
stop and go "wait, what?!".

Having a would be second kernel metastasizing in userspace is no a good thing,
full stop.

------
zlynx
The linked page does not actually answer the question. Instead we get a long
bicycle analogy that does not really map to systemd in any way that I can
tell.

It would have been much more interesting to hear about what software had to be
rewritten or replaced to support desktop environments like Gnome.

------
bandrami
I mean, it is kind of strange that with one apt-get install, one apt-get
purge, one apt-get autoremove, and one pinfile in /etc/apt, you can replace
systemd in Debian. You'd think Devuan could just do that and redistribute or
something.

~~~
smhenderson
It's not systemd that's the problem, it's libsystemd0. Try removing it on
Stretch. It's doable [0] but it's a pain and makes future maintenance fragile
and error prone. And if I understand correctly once you do so you're not
running Gnome on that system anymore...

[0]
[http://lkcl.net/reports/removing_systemd_from_debian/](http://lkcl.net/reports/removing_systemd_from_debian/)

~~~
catdog
Which is a purely religious problem as libsystemd0 basically does nothing if
systemd is not running. Debian is a binary distribution, a lot of stuff is
linked against libs you probably never use, that's just how it is.

~~~
smhenderson
I didn't say it was being used but the Devuan project wants it out of their
distro and dependencies on it are much harder to resolve than dependencies on
systemd bins.

