
Structural and semantic deficiencies in the systemd architecture - vezzy-fnord
http://blog.darknedgy.net/technology/2015/10/11/0/
======
dang
Since it looks like this article is trying to treat a controversial topic
substantively, let's see if we can do the same. If you comment, please stick
to the substance and avoid dismissive comments and rudeness. We should all do
so anyway, but when the topic is polarized it can take more conscious effort.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

------
geofft
This is a very detailed article, and I appreciate it and will need to study it
carefully.

But I think that the major argument it is making is "systemd is a mediocre
service manager; similar tools existed before systemd, and were better." That
is a very different argument from "we would be better off with sysvinit,"
which seems to be the primary argument I see in practice: if you're arguing
things like "do one thing and do it well," "shell scripts are great and
debuggable," "init should reap children and nothing else," "there's too much
code in pid1," "I don't want to learn something new," etc., you're making that
form of argument.

If there is general consensus from studied opponents of systemd that something
systemd- _like_ is an obviously good idea, great. Let's move on, acknowledge
the network effects of systemd, and build something that can feasibly be the
shinier, newer, better, securer systemd. Which might even be based on an
existing system like s6. But then the path forward has to be either improving
systemd or replacing it, not continuing with sysvinit.

I am in practice a pro-systemd person, but that's only because my realistic
alternatives, for my use case, are sysvinit and Upstart. I have no idea how
I'd start to build a system that used something like s6. I would happily admit
there is extensive room for improvement in systemd, but we UNIX folks have
been working with worse-is-better and rough-consensus-and-running-code for
decades.

If the general consensus is that a systemd-style service manager is great, but
it should not be pid 1, that's an argument worth making, but I don't see much
focus in this article on that.

If the general consensus is that everything can be done with sysvinit, this
article doesn't make that case at all. (But I suspect that's not the general
consensus.)

~~~
lobster_johnson
Talking about "general concensus" is hard because there are a lot of different
viewpoints, and depends on how much of the debate one is exposed to.

I have seen people compare systemd unfavourably to daemontools, s6, runit,
Solaris' SMF, and Apple's launchd, to mention a few. I've never seen anyone so
reactionary that they suggest that SysV is in any way or shape the _better_
alternative. A lot of people are unhappy with SysV, but if they don't like
systemd, they would probably consider sticking with SysV to be the lesser of
two evils.

> Let's move on, acknowledge the network effects of systemd, and build
> something that can feasibly be the shinier, newer, better, securer systemd.

Unfortunately, I think that for those who don't like systemd, their criticisms
(its design is a monolithic, intrusive, all-consuming, all-or-nothing ball of
yarn contrary to Unix philosophy about small, composable programs cooperating
over standard protocols) cannot be solved by improving systemd itself, at
least not without a huge undertaking to redesign and break it apart, somehow.

~~~
rodgerd
> I've never seen anyone so reactionary that they suggest that SysV is in any
> way or shape the better alternative.

Unfortunately it's... a position that is not as rare as it ought to be. Some
of the frustration and heat undoubtedly comes from the fact that while
reasonable people can disagree about the best way to implement many of the
goals of systemd, it's hard to take anyone seriously when they propose that
SysV init is "more Unix" or technically better.

> ...cannot be solved by improving systemd itself

I suspect the most fundamental impediment is that upstream simply aren't
interested in radical changes to how it works - it would be like trying to
convince Linus that Linux needs to be a microkernel, or Theo that he needs to
start using Rust for OpenBSD.

Any replacement for systemd is probably going to have to do what systemd did
(and what daemontools and runit didn't achieve) and create a broad buy-in,
with the additional overhead of working with at least some of the systemd dbus
interfaces for compatibility.

~~~
skarnet
It's important to realize that "a replacement for systemd" is neither
achievable nor desirable.

systemd has grown fast and conquered large market shares because it was backed
up by a company, which put a lot of money and manpower into it - to write it,
to integrate it, to advertise it. The only thing that has not been correctly
funded in systemd is research and design.

systemd is only Open Source if you read the license; all its other aspects are
proprietary - it's software made by a company and aiming to capture a market.
It is impossible to compete with systemd on the same grounds, because no real
Open Source developers will have as many resources as Red Hat.

And even if it was possible, the result of such an attempt would simply be
another integrated behemoth, powered by money and marketing instead of good
technical decisions. (Or even worse, powered by ideology - can you imagine a
systemd-like controlled by the FSF?) In the end, the situation for the users
would be even worse than it is today. You don't fight Goliath with Goliath;
you don't fight Microsoft's hegemony by buying Apple products.

About interface compatibility: the author of the DnE article (vezzy-fnord) has
written uselessd, and finally abandoned the project because the systemd
interfaces are so tightly integrated with the systemd design in the first
place that it's impossible to be compatible without simply being a systemd
clone, which he did not want uselessd to be. No, interface compatibility is
not an option, because it would simply acknowledge the validity and
superiority of the systemd architecture, and nobody wants a copy of systemd.

I believe that the way to provide an alternative to systemd is to provide all
the functionality that the systemd users like, but in a technically better,
less integrated, more unixish way.

With s6, s6-linux-init and s6-rc, I now have respectively a process
supervision system, a simple init process and a service manager, which should
be sufficient for a majority of applications. The next important thing that
sysadmins like in systemd seems to be cgroup management, so I'd like to study
the thing soon and assess what needs to be done next; but for now, I believe
that the s6 family of programs is now viable as a serious alternative to
systemd, and I would love to give it a broader audience.

~~~
glogla
> systemd has grown fast and conquered large market shares because it was
> backed up by a company, which put a lot of money and manpower into it - to
> write it, to integrate it, to advertise it. The only thing that has not been
> correctly funded in systemd is research and design.

> systemd is only Open Source if you read the license; all its other aspects
> are proprietary - it's software made by a company and aiming to capture a
> market. It is impossible to compete with systemd on the same grounds,
> because no real Open Source developers will have as many resources as Red
> Hat.

This. The reason systemd is hated so much by "old unix guys" is because it
shows that Red Hat basically owns linux the way Microsoft owns Windows and
Apple owns OS X.

Many people protested against systemd but it was still pushed by force. It was
made specificaly incompatible to make experience on BSDs worse and it
sucessfuly removed Gnome from there.

So yeah. Some "haters" talk about unix this and complex that, but I believe
it's mostly about scummy company showing off it owns linux now.

~~~
derefr
I would say it's more like RedHat has actually stepped up and made a "Linux-
derived OS" the way that Android is a "Linux-derived OS" or OSX is a "BSD-
derived OS." They've thrown the "basic Unix principles"—the ones that make
Linux distributions effectively interchangable commodities, but with software
having to target a _very low_ lowest-common-denominator of functionality—to
the side, and instead adopted a complete "base system API", in the same sense
that CoreFoundation+Cocoa is for OSX, and Win32 is for Windows.

As with those other APIs, you can now "target systemd" in the same way you
could "target Win32" or "target Android"—and, because of this, we'll probably
see a virtualization library (like winelib is for Win32; like ARC is for
Android) spring up, giving other OSes the ability to provide the "systemd API"
without needing to run a Linux kernel.

From RedHat's perspective, they were just trying to copy OSX and Android:
making a "real OS" atop Unix-y foundations. The interesting thing is that a
large number of other providers in the Linux space has started to agree with
their design decisions—effectively unifying on the "systemd platform" rather
than on the "Posix platform" they were focusing on previously.

I think this has been coming for a long time; the large OS providers already
think of Desktop Environments as the GUI equivalent to a "platform" rather
than as application software for the OS; thus why Ubuntu, for example, has a
separate distribution for each GUI software suite.

In a discussion of systemd history held here previously, it was pointed out
that back in 2007-or-so, there was a large amount of interest expressed by
various distro makers in adopting OSX's launchd as the basis for a new
platform standard, if only it had been FOSS at the time. Early systemd was a
launchd clone; upstart also started as a launchd clone before mutating.

I have a feeling that although RedHat was the one to step up and _create_ a
platform, _all_ of the major distro providers were actually waiting to hop on
the first available FOSS "OS platform", no matter who it came from. If, for
example, Apple had created "Apple Linux": a version of Linux that runs the OSX
GUI and OSX apps—then I can bet you right now that every other distro would
have standardized on whatever "platform" gunk was in-between that GUI and the
Linux kernel; CoreFoundation et al would have become to Linux distros as
WebKit is to mobile web browsers.

As it stands, it's only an accident of history that this didn't happen with
the _Android_ platform. (If the Android Runtime ran native-speed binaries from
the start, and the GUI was developed five-or-so years later such that it was
developed from the start with Win8-like unified desktop/mobile support in mind
rather than getting type-cast as "only for mobile devices", I could bet that
Linux would "be" Android now.)

\---

With all that said, while all the _big, commercially-backed_ Linux providers
have been waiting on tenterhooks for a "platform" to latch onto, all the
little distros are looking to remain Unix-y, and that's a wonderful thing.

Think about it: when you've got the POSIX platform "for desktops and servers"
and the Android platform "for mobile", and nary the twain shall meet, it makes
little sense to care about cross-compatibility; why would you want to run
Android apps natively on your desktop? But if you add in a third platform
that's _also_ for desktop—the systemd platform—suddenly kernel developers and
distro makers and whatever you call the FreeDesktop people start thinking
about how to create a Linux-based OS that natively supports _multiple_
platforms, so that it can run code targeting systemd, _and_ code targeting
Plain-old-POSIX, without making a mess. And once you have that infrastructure
in place... why not let it run code targeting Android too? Why not first-class
Wine support? Etc.

One thing Windows has had forever (though lately in a degraded state) is
multi-runtime (or, given the above discourse, multi-"platform") support.
Windows can be Win32 to one program, OS/2 to another, and POSIX to a third.
OSX had the capability for multi-runtime support back when it was called
Rhapsody (the "Red Box, Blue Box, Yellow Box" architecture) and also co-
supported Cocoa and Carbon apps for about a decade. And these are just single
companies trying to meet small needs. Linux, the OS with a kernel that
supports every device a nerd ever cared to packet-probe, could easily become
"the multi-runtime OS", if we care enough to make it that way.

If we don't, though, at least systemd is decent. :P

~~~
vezzy-fnord
_In a discussion of systemd history held here previously, it was pointed out
that back in 2007-or-so, there was a large amount of interest expressed by
various distro makers in adopting OSX 's launchd as the basis for a new
platform standard, if only it had been FOSS at the time._

I don't think there ever was a _large_ amount of interest. In fact, the only
party that studied launchd at the time (circa 2006) was Canonical. It was
indeed licensing issues that stalled further research, but there was also
another contender at the time being considered called initng, which Canonical
ended up rejecting and went on to write Upstart instead, led by Scott James
Remnant. See SJR's proposal and introduction to Upstart. [1]

 _Early systemd was a launchd clone; upstart also started as a launchd clone
before mutating._

Upstart was never a launchd clone to the best of my knowledge. launchd was
likely a spark that influenced Canonical to take action, but the design is
pretty different.

Your musings about the "multi-runtime" convergence that systemd will allegedly
enable do not appear to pan out. systemd is nothing like the OS X Core
frameworks or like the Windows Runtime, it's much lower level than that. It's
more of a middleware than a runtime platform (think Hurd, not Core
Foundation).

There is also absolutely nothing implying that GNU/Linux will ever target
Android code. The divergences are plentiful, with my article about an Android
init porting attempt listing only a few. [2] Nor is there any expressed
interest from any Android vendor to have any serious GNU/Linux convergence to
begin with.

Not sure what "first-class Wine support" is supposed to mean. Wine is pretty
self-contained.

 _If we don 't, though, at least systemd is decent._

You're too big of a dreamer, I'm afraid.

[1]
[https://wiki.ubuntu.com/ReplacementInit](https://wiki.ubuntu.com/ReplacementInit)

[2]
[http://blog.darknedgy.net/technology/2015/08/05/0-androidini...](http://blog.darknedgy.net/technology/2015/08/05/0-androidinit/)

~~~
derefr
> systemd is nothing like the OS X Core frameworks or like the Windows
> Runtime, it's much lower level than that. It's more of a middleware than a
> runtime platform

You're not considering the full scope of the thing labelled "systemd." If you
only use parts of it, it's middleware, yes. If you "work with the grain" of
systemd, though, then you're packaging services in nspawn containers and so
forth, which _does_ constitute a separate "platform target", in the same way
that e.g. CoreOS is a (mostly-ignored attempt at a) "platform target."

Basically, I'm talking here about a Linux equivalent to Solaris's "Branded
Zones": within a container boundary, an app can be made by the system to think
it's running on "systemd Linux", or on "POSIX Linux", or on "Android Linux",
or on BSD or Win32 or Cocoa or whatever else. Runtime-virtualization is done
at the system container-management level (with more or less help from the
kernel), rather than expecting applications to "port" themselves by applying
their own proprietary virtualization wrappers ala Cider for OSX.

> Not sure what "first-class Wine support" is supposed to mean. Wine is pretty
> self-contained.

I mean, basically, support to the level of DOS applications run in Win32 VMM
containers, rather than to the level of X11 applications run on OSX:
management of Wine sandboxes as OS-level "runtime containers", such that you
could run and _maintain_ Wine apps alongside other apps, in production, using
the OS's maintenance tooling. A Linux "ReactOS Runtime" equivalent to
Windows's "POSIX Runtime."

> Nor is there any expressed interest from any Android vendor to have any
> serious GNU/Linux convergence to begin with.

I don't mean to suggest convergence. This is something very different: making
"running an Android binary" a primitive action of _the system_ (not the Linux
kernel), where "using an Android virtualization layer" is then an
_implementation detail_ of how the system accomplishes this. Think of Linux's
binfmt_misc and its ability to execute e.g. JVM code by booting a JVM, but
with an upcall to the init daemon to decide how to implement the policy of
running a particular binary format. If that init daemon is effectively a
container-manager that understands how to instantiate and manage different
"branded containers" for each runtime it supports, Linux multi-runtime support
just falls out.

~~~
vezzy-fnord
nspawn containers are yet not really used as a platform target, and currently
Docker is leading. Though nspawn has since acquired some notion of the Docker
image format, and even in light of the OCI standard, I do not foresee it
becoming a primary container solution in its present form. rkt being nspawn-
based may or may not take hold.

Indeed, systemd is as of yet not comprehensive enough to be a POSIX-parity
target. There is no "systemd Linux" as such, it's systemd/GNU/Linux or Linux
with the GNU and systemd suite. Android is a top-down integrated system on the
other hand and does not linearly track GNU/Linux.

As of yet, there is nothing like system call emulation or similar in nspawn to
have branded zones.

Nor does Red Hat's present actions imply something like this. The GNOME
project, affiliated with Red Hat, is working on a poor reinvention of Nix
called xdg-app to enable the "app frameworks and runtimes" design that Lennart
Poettering wrote about in "Revisiting How We Put Together Linux Systems," but
that too is firmly specific to GNU/Linux as the host.

Red Hat is also leading a container OS called Project Atomic, however nothing
like branded zones is seen there as a goal, either. Instead, they've made a
simple meta-framework for running various _Linux_ container images over
several orchestration frameworks and PaaS, called Nulecule. It's firmly a
layer over Docker and Kubernetes, however, so it is limited to that.

 _A Linux "ReactOS Runtime" equivalent to Windows's "POSIX Runtime."_

That would be quite a feat in of itself, systemd or branded zones aside.
ReactOS isn't Wine, but even with Wine it would be a sizable integration
effort.

 _If that init daemon is effectively a container-manager that understands how
to instantiate and manage different "branded containers" for each runtime it
supports, Linux multi-runtime support just falls out._

The init daemon is not a container manager in this case, but an object-
oriented resource management with a transactional job scheduler and some
limited execution environment modification that work with namespaces and
cgroups, but the container framework is outside. As it should be. An init
daemon as your container manager sounds dreadful and horrifying, though I hear
RancherOS boots from Docker as PID1...

But as there is no such init daemon yet nor anything like branded zones, I
still have to say you're a dreamer. This _might_ be a long-term strategy, but
with acts like Project Atomic it really doesn't look like it. I still think
it's more middleware than Core runtime. I pray it is...

~~~
derefr
Note that I wasn't suggesting that systemd is _itself_ an attempt at an "init
daemon with branded zones"—instead, it's a particular runtime that will
increasingly differentiate itself from POSIX Linux, and I believe that that
_friction_ will eventually cause developers to want to _create_ an init daemon
with branded zones to supercede systemd, where emulating systemd-as-it-stands
(for apps built to expect it) would be one of that multi-runtime system's
goals. With the POSIX people and the bigcorp Linux providers pulling in
opposite directions, a multi-runtime Linux would be the only Nash equilibrium.
(Though it'd need someone like FreeDesktop to suggest it and start a working
group for it, because neither "side" would care about it on their own.)

That may not be the way things play out, certainly. Things might just diverge
and stay that way, if nobody cares enough to change things. But there are
unknown unknowns that can give things a very hard shove in that direction. For
example: imagine that systemd decides to integrate deeply with GNOME, to the
point that you now have an integrated "systemd+glib+GNOME platform" with a
unified API, the way that OSX is a "launchd+CoreFoundation+Cocoa platform".
_That_ would be an extremely divergent target in the Linux ecosystem
introducing a lot of friction to everyone else's development efforts—similar
to the early stages of the Android project—and it would get a lot of people's
hackles up. But it's not out of the realm of possibility for RedHat and Debian
to agree on something like that.

~~~
vezzy-fnord
_instead, it 's a particular runtime that will increasingly differentiate
itself from POSIX Linux_

Ah, correct.

 _I believe that that friction will eventually cause developers to want to
create an init daemon with branded zones to supercede systemd, where emulating
systemd-as-it-stands (for apps built to expect it) would be one of that multi-
runtime system 's goals. With the POSIX people and the bigcorp Linux providers
pulling in opposite directions, a multi-runtime Linux would be the only Nash
equilibrium. (Though it'd need someone like FreeDesktop to suggest it and
start a working group for it, because neither "side" would care about it on
their own.)_

It's a hypothesis, then. Plausible, certainly. I assume you're presenting this
as a probable way out of the portability concerns from GNU/Linux becoming more
of a closed environment to the point of instigating a massive developer
schism? This still raises the question of why it would need to reside in the
init daemon, but I suppose crass architectural decisions are not a stranger to
Linux developers.

 _For example: imagine that systemd decides to integrate deeply with GNOME, to
the point that you now have an integrated "systemd+glib+GNOME platform" with a
unified API_

Such a decision might just backfire quite tremendously and prove to be
systemd's overturning, though it does sound like a similar integration is very
much in the spirit of the project. Then, yes, radical surgery would occur, but
it may not be quite as dramatic as branded zones and simply an abandonment of
systemd and related Freedesktop-ware.

 _RedHat and Debian to agree_

Debian aren't really that large of a player, more of a passive target
platform. Being a non-affiliated foundation, they're generally steered into
things by whoever is on the committee and seldom ever lead themselves. So, no
agreement strictly necessary.

------
adekok
My take-away is the following:

* systemd tries to solve a complicated problem

* systemd has a lot of complexity

* some of that complexity seems unnecessary

* parsing is hard (see DJB). Parsing (as systemd does) in PID 1 is bad. Complex systems in PID 1 is bad.

* systemd manages states and dependencies (this is good)

* some of the states are only internal, and can't be serialized to storage. The result is indeterminism and race conditions. (This is bad)

* as a result of the above point, systemd behavior is hard to debug, and hard to understand

* another result (not stated explicitly) is that systemd allows for (and apparently can't detect) ordering cycles and dependency cycles.

* The configuration files are simple for simple tasks (good), but not complex enough for complex tasks (bad)

... etc.

i.e. systemd is an engineering solution. It pretty much works, but it has
inherent design / implementation issues which cause real-world problems.

~~~
Jweb_Guru
This article pretty conclusively convinced me that systemd is actually good,
if these are the strongest arguments against it. Most of the dependency-
related stuff (process order related, cyclicstuff) seems eminently fixable. I
found his arguments about dependency management having to be linearizable
highly unconvincing.

~~~
sagichmal
> I found his arguments about dependency management having to be linearizable
> highly unconvincing.

What? Can you expand on this? To me it rang as a well-reasoned, and pretty
devastating, critique.

------
ncraun
Very interesting article, refreshing as well to see a criticism of systemd
based on technical details. Unlike most other criticisms of systemd, this
author does not resort to personal attacks and insults against Lennart
Poettering or fear-mongering appeals to tradition. I still think systemd is an
improvement over the previous sysvinit, but the author seems to have found
some real conceptual flaws with systemd that are somewhat unsettling.

I hope this article can start some actually useful conversations about init
systems that do not fall into the pitfalls of FUD and negativity that seem to
have surrounded this topic in the past.

~~~
striking
My explanation for the lack of technical (rather than political) articles
examining systemd is that it's very difficult to wrap your mind around because
it carries so many ideas. A quote from an article linked within the linked
article:

> All of the above definitions are technically correct. None of them are
> complete.

> It is thus that debates are doomed to go nowhere, as no one can agree on a
> single definition of systemd, and the developers have been largely unhelpful
> in that matter. Vagueness can be a strength, in that it allows for people to
> divide and conquer, but it is also unproductive and leads to unnecessary
> bloodshed.

Slowly, surely, people will begin to understand what systemd is. Only then
will people be able to figure out whether they want it or not.

------
ausjke
Read it through but hard to grasp all the details. One thing I'm sure is that
for non-systemd "legacy" init systems I know what I'm doing, for systemd I
most of the time don't know what's going on there, it's still like a blind art
after using it for a while, overly abstracted might be one reason for that.

~~~
kbenson
I think part of the reason for this may be that some types of systems lend
themselves to exploration and gradual learning through use, while others,
being very tightly coupled, generally require a larger upfront investment in
learning how they function as a whole, since any one subcomponent is only
marginally useful without the rest.

This is largely how I remembered studying for Microsoft certification back in
the day to be. There was only so much you could easily learn from playing
around with the system. At some point, well written technical documentation
was, if not required, _extremely helpful_ in making connections between how
systems worked together, or explaining exactly what a component was or was not
capable of. I don't think this is inherently worse or better, just _different_
, and with its own advantages and disadvantages. For example, people who went
through this training seemed to have a very high base-level of understanding
of how to configure these systems, what they could or could not do, and how to
troubleshoot them, but not necessarily how they _functioned_ in every case.
Conversely, the loosely coupled individual components of traditional UNIX
service handlers (sysvinit, inetd) promoted people learning how the actually
work, and what they do, but with scattered and not entirely comprehensive
documentation on how they worked. The average person responsible for using
these will have vastly different levels of skill and knowledge about them, but
with that comes how it actually functions, because that's needed to
troubleshoot problems, since it's so exposed.

The feeling I get mostly when reading about systemd is that it's very much in
the vein of MS products. Like I said, that isn't necessarily a bad thing, but
it is very different than a lot of UNIX admins are used to.

~~~
bgilroy26
>Conversely, the loosely coupled individual components of traditional UNIX
service handlers (sysvinit, inetd) promoted people learning how the actually
work, and what they do, but with scattered and not entirely comprehensive
documentation on how they worked.

Can you tease out this distinction a bit more? It reads like you are saying,
due to their nature, people learned how to use Unix service handlers, without
a good understanding of how they were designed. It could also mean that people
learned how Unix service handlers worked by using them, through trial and
error, because the documentation was scattered and not entirely comprehensive.

~~~
kbenson
The second interpretation. The documentation is not entirely comprehensive
because interface between components is not only a grey area of
accountability, but also continuously changing. When the system is composed of
multiple distinct sub-programs with their own development teams, update cycles
and planned feature sets, what is possible at any one point in time is largely
a matter of which versions of those sub-programs you are using, as well as
what the root program (e.g.sysvinit) allows in it's version. There is no one
set of features, nor one set of bugs, nor one set of directions. The best you
can get in this regard is whatever your distro put together for the
combination they chose, or alternatively to read the docs for every sub-
program and intuit what new capabilities may exist in how they interact.

I think the clearest example of this would be everything that makes up the
core Linux environment, compared OpenBSD. OpenBSD has amazing documentation,
and the systems are all designed to work with eachother because they were all
designed and developed as a working whole. Linux, on the other hand, often
exposes interfaces through the kernel, and some other set of developers (or
multiple set of developers) create systems to interface with it. As such, the
documentation can be haphazard or near non-existent. That's not to say the
Linux methodology doesn't have it's advantages. For example, I think there's a
better feedback loop in Linux in determining what is needed from the kernel,
and what can be done from it in userspace. It's a more market-like system, and
sees rapid advancement.

~~~
bgilroy26
Thank you!

------
nextos
The major problem right now if one decides to use a systemd alternative is
that most packages are shipping with systemd unit files, but will obviously
lack support for other init implementations.

I like dmd, implemented in Scheme, which is part of the GuixSD distribution.

~~~
baghira
1\. The solution to this is to write a translator from systemd units (those of
.service type) to the desired kind of unit. I haven't looked into dmd, but
this should be possible at least for the majority of units.

2\. This was going to be a problem in any case i.e. had most distros switched
to openRC one would still have been faced with the task of producing a
suitable collection of service configuration files for dmd and then adapting
them to dmd's specfic featureset.

~~~
mercurial
Yeah, well, if it's to go from unit files to Guile code, that's not what I'd
consider an improvement. Having configuration files for use cases where the
requirements are simple enough is nice.

Of course, when they're not, you end up with an abomination like MSBuild, but
that's not the case here.

~~~
baghira
True, but if that is the case you won't be running dmd in the first place. And
automated conversion/runtime translation will be a possibility even with
different service managers, e.g. systemXVI.

I'm not really convinced by the second, in that I believe that nothing in the
article proves that service management is something too complex to be done via
configuration files.

------
hueving
The article that lies herein, was, in a manner of speaking (but not literally
speaking - 'in a manner of speaking' is just a phrase), so overly verbose in
many places that it drowns the reader in verbal diarrhea as thick as pea soup.
That is to say, I, as a reader, felt that the author (of this article) could
have made these points in a more reachable, succinct fashion.

~~~
vezzy-fnord
I would, as a general principle, be cautious (in the sense of not being hasty)
to dismiss in a manner reminiscent of Parkinson's law of triviality (insofar
as the latter pertains to the bikeshed principle) the literary style of the
article in question, since the common interpretation of excessive verbosity is
flawed, in this case being seldom more than a preference with little in the
way of any concrete justification that would be of direct relevance to the
topic at hand, the topic being an architectural treatise. Any reservations
about succinctness, or a perceived lack thereof, must therefore be qualified
and left over for further analysis to draw proper and conceptually robust
remarks given the evidence presented.

(Yes, the writing style is somewhat obtuse. You'll have to excuse that. The
topic itself is obscure, so I suppose it fits.)

------
tadfisher
My response from another discussion on this article:

Much of these deficiencies boil down to the design criteria for systemd, which
is basically to avoid forcing the system administrator to perform dependency
graph resolution. This was a direct response to Upstart, which pushed this
work to the administrator (or distro packager) by design.

Thus the declarative syntax for unit files, which do not explicitly control
dependency resolution, parallelization, or startup order, which is simply not
necessary or cared about for 99% of use cases. The 1% case there is for
situations where you should not use a prebaked init system in the first place,
such as when you absolutely need deterministic boots, i.e. in an embedded or
realtime context.

> In systemd, however, the execution state is not based on explicit chain
> loading, but on serializing unit file options to a private ExecContext
> structure overlayed into Unit objects and set for service unit types.

This is not a problem with systemd's design. This is a problem with software
written without the assumption of reliable process supervision, which is a
reasonable assumption given the history of UNIX daemonization. systemd takes
the (apparently controversial) stance that processes should not manage their
own execution environment, or that of their dependencies. It provides numerous
prebuilt hacks to make things work for broken software, but nobody should
expect systemd to design for 100% of situations where processes do Weird
Shit™.

And yes, I'm including "delegated service restarters" under this umbrella. The
need for something like this absolutely smells of poor design. It also flies
in the face of the author's preference for deterministic execution behavior,
because by definition your process will execute differently depending on if it
was started or restarted.

The link to "systemd house of horrors" [1] only proves this point.

> A supervisor insisting on programmatic accommodations from service writers
> is not the most desirable state of affairs. A rarely discussed alternative
> to the two common approaches is, again as with startup speed optimization
> minus parallelism, checkpoint/restore: checkpoint a process image from a
> point where initialization is known to be complete and overlay it on
> startup, using a tool such as DMTCP or CRIU.

I don't know about anyone else, but this sounds like an absolute nightmare.
For one, it relies on the service exposing enough information to be
checkpointed at the right time; at which point, you might as well call
sd_notify or hook in a script to do it for you, so this is not a reduction in
requiring programmatic accommodations from service writers. Two, process-
snapshotting is not a panacea, and is a nightmare to debug when it breaks your
expectations; you'll need to test all sorts of process state, including open
pipes/fds/sockets, signals, system calls (!), shmem buffers, etc. At which
point it's best to just let the service notify you directly when the process
is done. Which is a facility that systemd provides.

> Of course we also have the issue of circular dependencies in the systemd
> architecture itself. We have the init, process manager, process supervisor,
> cgroup writer, local service tools, the Unit object structure (which might
> benefit from being made a protocol), timers, mounts, automounts and swaps
> all in the same module with ill-defined boundaries.

A worthy criticism. I would also like to see a bit more modularization from
systemd, but I can also see the point of systemd developers not expending
effort to do so. For one, it would drastically increase the number of contexts
for testing (i.e. can we verify the service manager works without the cgroups
module loaded, etc). I think systemd is at the point where it's rock-solid for
all of its use cases, and a pluggable/composable architecture would have
delayed this situation for not very much benefit.

In short, systemd imposes its opinions about process init and supervision.
Whether or not this is reasonable is up for debate. However, it is very useful
to have both a strong opinion and a solid implementation of this opinion.

IMO, the idealized world that systemd supports (i.e. non-forking services,
initialization notification, strict execution environment and dependency
declaration, centralized and opaque init/dependency-
resolution/parallelization) is a vast improvement over the status quo and
actually improves the lives of service writers (by removing an entire class of
responsibilities), so I think it's a reasonable opinion and thus I think
systemd has a reasonable design and architecture.

There are many known technical deficiencies with the implementation that are
mostly being worked on, but I'm not seeing anything damning of the whole idea.

[1]
[http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/sy...](http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/systemd-
house-of-horror/)

~~~
vezzy-fnord
It appears you received a response here:
[https://www.reddit.com/r/linux/comments/3od733/structural_an...](https://www.reddit.com/r/linux/comments/3od733/structural_and_semantic_deficiencies_in_the/cvwf201)

You do indeed misunderstand the point about execution composition. Chain
loading is indeed configuration-related, not service-related. It's a much more
flexible and visible method than internal structures. Delegated restarters
then can be deterministic as they reuse process management.

C/R solutions don't require special integration, that's the point. They work
on unmodified processes, unlike readiness notification. This is I think a
reasonable trade-off for the potential decrease in reliability, as the feature
comes with a double advantage of startup optimization and then readiness
notification is generally not _that_ critical, in light of regular Unix
process semantics. C/R is also constantly improving and reasonably stable for
base use cases these days.

~~~
skarnet
You seem to dislike the idea of asking for notification support in the daemons
themselves. Granted, it's impractical, but it's the only way to avoid race
conditions which are rare for most services, but _very real_ for a few
programs that may take ages to get ready. How would you address the problem of
slow starters _without support_ from the daemons, while keeping a "the
dependency graph is always respected" invariant?

~~~
vezzy-fnord
Right, if the problem is slow starters (say, RabbitMQ), then restoring from an
image where initialization has already been complete would resolve it. For
more complicated cases where there is a race with acquiring some resource,
then a readiness notification mechanism could be supported as a niche feature.
I'm still not convinced of it in the general case.

~~~
skarnet
But how would you go with restoring such an image? It won't be instant, there
will still be a window where the service manager is continuing its work but
the image hasn't been restored yet. One way or the other, you need a
synchronization mechanism, so how would you design such a mechanism without
support from the daemon?

You don't need a complicated case to have a race with acquiring some resource.
A server opening a socket is providing a resource; clients connecting to that
socket are using it. Clients should not be started before the server is
listening to the socket. How would you solve this case - are you advocating
socket activation by the service manager? ;)

------
chris_wot
Thank you for doing this - I'm going to read this with a lot of interest :-)

~~~
vezzy-fnord
Do return with commentary.

------
colin_mccabe
tl;dr: the author argues that:

* systemd is more similar to a traditional job scheduler than a traditional init system.

* systemd should not parse text files in pid1 because it might be a security risk.

* Because systemd handles order dependencies, it is sometimes susceptible to "ordering cycles" where a clear ordering cannot be established, "dependency loops" where jobs are continuously dropped and requeued, and race conditions if dependencies are accidentally underspecified.

* "Imbalance between promoting laziness or eagerness": launchd, a predecessor of systemd, operated purely "lazily," launching services only when other services needed them. systemd also supports launching services eagerly, which the author argues is more complex.

* Checkpointing a process image could have been used instead of systemd's readiness notification mechanism (?)

* Systemd does not have a plugin system

* Journald is criticized because it does too much

I would counter that:

* I agree that systemd is similar to a job scheduler, but I don't necessarily believe that that is a bad thing. Manual scheduling of jobs for startup and shutdown like we did in the rc.sh days is complex and error-prone.

* The text files systemd is parsing are owned by root, so even if the parser is exploitable, only root can take advantage? Seems like a weak criticism.

* I agree that dependency ordering makes things more complex. But upstart and launchd have the same issues. It seems to be a tradeoff worth making.

* I don't understand the argument against supporting both laziness and eagerness in dependencies. It seems like some things are naturally modelled eagerly, like needing to perform a bunch of somewhat unrelated actions to suspend the system. And some things are better handled lazily, like setting up a FUSE filesystem when a USB stick is inserted that needs it. Shouldn't we use the model that makes the most sense?

* I don't think checkpointing a process image can ever really replace having a notification system. Even if the checkpointing code could be make bulletproof somehow, some processes deal with state in the external world, or with hardware, that makes checkpointing infeasible.

* Regarding a plugin system: Systemd has the ability to run shell scripts, which can be useful in filling in gaps in functionality. I don't think a more complicated plugin system would be a good idea since it would add a lot of complexity (and potentially instability.)

* I didn't understand the criticism of journald, maybe someone can elaborate. The author presented some alternate approaches but I missed why these were better (other than the handwavey argument that journald was "a bottleneck")

I enjoyed reading this, and it's nice to see some more reasoned criticism of
systemd. I feel like the prose got a little bit purple at times. We had to
spend 10 paragraphs "descend[ing] into an inferno with the same dead horse
talking points", "culminating into some rather heterodox conclusions", and
"progressively introducing and elucidating on concepts, applying some a priori
reasoning at times, and backtracking to derive conclusions or reiterate on
prior stated knowledge" before we even saw a word of argument! This needed an
editor...

~~~
vezzy-fnord
Actually the job scheduler is one aspect, complimentary to the object-oriented
resource management which I made quite clear was the main point.

Regarding file ownership, not necessarily. If you run it as a session-based
init, they'd be owned by the user it serves as an agent to. (see dirs:
[http://www.freedesktop.org/software/systemd/man/systemd.unit...](http://www.freedesktop.org/software/systemd/man/systemd.unit.html))

Upstart and launchd do not have the same issues. Upstart doesn't have
dependencies, but named preconditions and postconditions. launchd has neither
that nor dependencies. They have many problems of their own, but different
ones.

My point regarding laziness v eagerness is the way systemd implements multi-
paradigm introduces undesirable externalities from both approaches, as I said:
"In systemd, one must endure both the non-determinism from aggressive
parallelism via dependency information, the object network needing the
transaction and job metaphors, all while having its lazy features be
thoroughly restricted to its internal Manager and not a flexible toolkit such
as UCSPI."

I'd assume the limited use of C/R to simply cut down initialization times _is_
feasible: [http://criu.org/Usage_scenarios#Slow-
boot_services_speed_up](http://criu.org/Usage_scenarios#Slow-
boot_services_speed_up)

Running shell scripts doesn't mean much when the underlying system has a large
surface and ill-defined module boundaries. A plugin system most certainly does
not have to be complicated: look at finit and initng for examples.

What about journald did you not understand? You lose control over the separate
stages of collection, rotation and processing. Particularly having custom
postprocessing is important for interoperability with foreign services that
might expect certain format constraints. Having the logs be per-service
instead of globally snarfed is also saner.

Yes, I did go overboard on the reservations. It's a flame war topic, so I felt
it necessary as a precautionary measure.

~~~
colin_mccabe
_Regarding file ownership, not necessarily. If you run it as a session-based
init, they 'd be owned by the user it serves as an agent to._

Interesting. I didn't realize systemd had a "user mode" where it doesn't run
as pid 1. Does systemd run as root (or root equivalent) in this mode? I wasn't
able to find this from a quick skim of the information available online.

 _Upstart and launchd do not have the same issues. Upstart doesn 't have
dependencies, but named preconditions and postconditions. launchd has neither
that nor dependencies. They have many problems of their own, but different
ones._

Sorry, but I have to disagree. The terminology may be different, but
ultimately upstart's "start on" directive establishes a dependency. Once you
have a dependency you can have circular dependencies. And once you rely on the
init systems to establish ordering rather than having everything just start up
in a deterministic order in simple single-threaded script, you can have race
conditions where things happen to work on one machine but not another. I'm not
as familiar with launchd; maybe its socket activation makes the circular
ordering harder to hit, but fundamentally there can always be races.

 _I 'd assume the limited use of C/R to simply cut down initialization times
is feasible: [http://criu.org/Usage_scenarios#Slow-
boot_services_speed_up](http://criu.org/Usage_scenarios#Slow-
boot_services_speed_up) _

Perhaps for some processes this is feasible, but nothing on that site
addresses my objections above: "some processes deal with state in the external
world, or with hardware, that makes checkpointing infeasible." You will always
need a notification mechanism even if you have checkpoint and restore. And why
have the complexity of checkpointing if you have a working notification
system?

 _Running shell scripts doesn 't mean much when the underlying system has a
large surface and ill-defined module boundaries. A plugin system most
certainly does not have to be complicated: look at finit and initng for
examples._

From a user's point of view, I do not want a plugin system. I want systemd to
work the same on SUSE vs. debian vs. Red Hat. It is usually better to put
things into the base system than to have them live in a half-supported state
outside the tree. That's why web browsers are killing plugins: in order to be
useful, they have to reach deep into the guts of the program, which is not a
place I want 3rd party code.

 _What about journald did you not understand? You lose control over the
separate stages of collection, rotation and processing. Particularly having
custom postprocessing is important for interoperability with foreign services
that might expect certain format constraints._

I don't understand why I would want separate daemons for collection and
rotation, at least for dmsg, kmsg, and syslog. logrotate always had a bunch of
race conditions; it was never a great design.

I agree that journald doesn't do custom processing. However, I'm a little
unsure whether I would really want it to, or whether I'd be better off running
my own syslogd implementation at that point. That is still an option, even
when you are using the rest of systemd.

 _Having the logs be per-service instead of globally snarfed is also saner._

You can always set up important services to log to a file rather than to
syslog.

My real criticism of systemd is that they made no effort at portability
outside of Linux (and I don't buy their arguments about why it's too hard.) It
would have been a lot less controversial if they had at least pointed out how
the BSDs could add support (even if they themselves didn't do the work).

~~~
vezzy-fnord
_Once you have a dependency you can have circular dependencies._

Then the question becomes whether an architecture obscures or exacerbates
them, and also how it deals with them. The systemd object model does not fare
well, as discussed.

(Also compare a system like s6-rc or serel where dependency graphs are
compiled from manifests and consistency checked before being applied. The
graph is persistent configuration and not an ephemeral artifact.)

 _And once you rely on the init systems to establish ordering rather than
having everything just start up in a deterministic order in simple single-
threaded script, you can have race conditions where things happen to work on
one machine but not another._

Systems having ordering dependencies does not inherently affect their
determinism, the startup discipline (sequential v. parallel) primarily does,
and then there are different ways to tackle parallelism. systemd's execution
model and transactional object system exacerbate non-determinism in ordering
results.

 _I don 't understand why I would want separate daemons for collection and
rotation, at least for dmsg, kmsg, and syslog. logrotate always had a bunch of
race conditions; it was never a great design._

They're not separate daemons.

 _My real criticism of systemd is that they made no effort at portability
outside of Linux (and I don 't buy their arguments about why it's too hard.)_

That's a _horrible_ criticism of systemd. Any cursory examination of systemd
will quickly reveal the task is next to insurmountable without doing what
amounts to a full reimplementation.

~~~
colin_mccabe
_Then the question becomes whether an architecture obscures or exacerbates
them, and also how it deals with them. The systemd object model does not fare
well, as discussed._

What specific aspects of the systemd dependency model do you feel exacerbate
race conditions and circular dependencies?

 _(Also compare a system like s6-rc or serel where dependency graphs are
compiled from manifests and consistency checked before being applied. The
graph is persistent configuration and not an ephemeral artifact.)_

I have seen people discussing systemd error messages about circular
dependencies online. It seems that systemd does perform some checking,
although I haven't found documentation on what checks it does (I probably
missed it). I'm not sure how the dependency graph could be persistent
configuration because hardware can be hotplugged these days. For example, are
you going to create nodes in the graph for all the hard drives that could
possibly exist over the lifetime of the machine?

 _Systems having ordering dependencies does not inherently affect their
determinism, the startup discipline (sequential v. parallel) primarily does,
and then there are different ways to tackle parallelism. systemd 's execution
model and transactional object system exacerbate non-determinism in ordering
results._

If the sequential vs. parallel startup discipline "primarily affects
determinism" won't all these systems have the same issues with nondeterminism?
I don't think anyone is seriously proposing starting up services on boot
sequentially in 2015. Or even when a USB stick is inserted. Systems have 12
cores now and people can be very sensitive to boot times.

 _[lack of portability is] a horrible criticism of systemd. Any cursory
examination of systemd will quickly reveal the task is next to insurmountable
without doing what amounts to a full reimplementation._

I understand that systemd relies heavily on cgroups and other Linux-specific
features for process isolation. And the BSDs are also unlikely to accept GPLed
code. However, I feel like they could have at least discussed some way of
reducing fragmentation here. As it is, I would be amazed if at least one or
two of the BSDs didn't implement a similar init system to systemd in the next
few years. And as it is, they probably won't share any code or concepts,
unfortunately.

~~~
vezzy-fnord
_What specific aspects of the systemd dependency model do you feel exacerbate
race conditions and circular dependencies?_

Well, the article discusses all of that...

(Speaking of which, JdeBP helpfully linked to Upstart's pre/postconditions not
being a proper dependency model, per the developers' testimony, and as I
asserted.)

 _I 'm not sure how the dependency graph could be persistent configuration
because hardware can be hotplugged these days._

The idea of device units is not well borne out, as I mentioned. You don't need
to create an object type out of a udev handle to order after devices or
trigger hotplug events (the device node manager itself does fine, of course in
the case of systemd, udev _is_ part of the project, so...)

 _If the sequential vs. parallel startup discipline "primarily affects
determinism" won't all these systems have the same issues with
nondeterminism?_

Not necessarily. It could either be more manageable if there wasn't so much
indirection, see above (I'm afraid you just didn't get the article, and your
original summary certainly implied as much). Additionally, one could forfeit
being _aggressively_ parallel for a more conservative approach like in OpenRC,
s6-rc or even Upstart.

* I don't think anyone is seriously proposing starting up services on boot sequentially in 2015. Or even when a USB stick is inserted. Systems have 12 cores now and people can be very sensitive to boot times.*

Parallelism is but a small part of the complicated topic of boot time
optimization. And yes, sequential strategies can still be really fast. Much of
the speed increase in modern systems is not from parallelism, but forfeiting
the overhead of the shell interpreter in favor of direct execution.
Checkpointing can do far greater wonders for your boot times than parallelism
can, anyhow.

 _I understand that systemd relies heavily on cgroups and other Linux-specific
features for process isolation._

That's only scratching the surface of Linux-specific features in systemd.

 _As it is, I would be amazed if at least one or two of the BSDs didn 't
implement a similar init system to systemd in the next few years. And as it
is, they probably won't share any code or concepts, unfortunately._

What will come of the BSDs is uncertain. launchd will be used in iXsystems
products like PC-BSD and FreeNAS, and there's not much "code sharing" you
could do there - it's already a complete product being ported.

~~~
colin_mccabe
_Parallelism is but a small part of the complicated topic of boot time
optimization. And yes, sequential strategies can still be really fast. Much of
the speed increase in modern systems is not from parallelism, but forfeiting
the overhead of the shell interpreter in favor of direct execution.
Checkpointing can do far greater wonders for your boot times than parallelism
can, anyhow._

No. I have worked on embedded Linux systems where we have carefully charted
boot time. We were able to get order-of-magnitude improvements by
parallelizing service startups during boot. Parallelism is far from "a small
part"\-- it is the only important part. The overhead of the shell is
negligible even on the slowest systems. Checkpointing is not an option for
most things (I have mentioned this several times) because most of the startup
processes initialize the hardware directly or indirectly, and hardware can't
be checkpointed.

 _The idea of device units is not well borne out, as I mentioned. You don 't
need to create an object type out of a udev handle to order after devices or
trigger hotplug events (the device node manager itself does fine, of course in
the case of systemd, udev is part of the project, so...)_

The important question is whether there is more or less complexity in systemd
than a system with a separate device node manager. I am not convinced that
systemd is more complex than the alternative. If the device node manager was
separate, you would have to debug the interaction between the two processes.
Having dealt with debugging d-bus interactions on Linux before, it doesn't
sound fun.

I'm not convinced that a static dependency graph is even feasible in a world
where users can reconfigure and install new services, hotplug hardware, and
join and leave wireless networks.

------
ised
"a controversial topic"

My observation: The "discussion" never seems to focus on systemd. Instead it
usually turns to comments on the former sysvinit system or other init systems.

That avoidance is perhaps something to ponder. And maybe it's why this author
felt the urge to put some focus on systemd itself.

Love the quote from djb. In sum, the best interface is no interface. Parsing
amounts to high margin for error and often an incredible nuisance.

In another thread today I wrote about the command line interface and
"expecting a reponse". Truthfully, djb's utilities that simply return an exit
value and no output are the best ones I have ever used.

(It seems djb himself is a systemd user. Not sure what if anything that
means.)

Imagine if the information dissiminated via www was as easy to "parse" as
text0. Writing a simple "web browser" might be easy enough that programmers
would not need to be paid to do it.

~~~
dang
> _The "discussion" never seems to focus on systemd_

All the more reason to try making this one better.

(We detached this comment from
[https://news.ycombinator.com/item?id=10370446](https://news.ycombinator.com/item?id=10370446)
and marked it off-topic.)

