
Broken by design: systemd - uggedal
http://ewontfix.com/14
======
bitwize
Nice article, but at this point it's game, set, match -- systemd has won. The
Linux community overwhelmingly supports it, and its advantages clearly
outweigh its perceived disadvantages.

"It's not Unix" is no longer a valid critique. The old Unix design philosophy
is dead. People want systems of components that function as an integrated
whole. They do NOT want pieces that are generally unaware of each other (this
is touted as the virtue of "loose coupling").

Yes, pid 1 does a lot. That's by necessity. It's not your grandpap's Unix
anymore. Do you want to control all the system services, including the "badly
behaved" ones? That functionality needs to be in pid 1. Do you want to bring
the system up safely and sanely, accounting for things like hotplugged disks
in /etc/fstab? That functionality needs to be in pid 1. Systemd is the correct
design for modern Linux. It also brings the advantage of better integration
and greater commonality across distros.

~~~
d0
daemontools doesn't run as pid 1 and achieves all of this.

[http://cr.yp.to/daemontools.html](http://cr.yp.to/daemontools.html)

~~~
FooBarWidget
It does not. Not even close.
[https://news.ycombinator.com/item?id=7210308](https://news.ycombinator.com/item?id=7210308)

~~~
Fasebook
That comment is insane. Why would log files need journaling? They should be
serialized.

How is adding a line to a run script "duplicating logic badly"?

Coupling is clearly bad, as evidenced by ... 30+ years of software design, but
is a necessary evil. I don't see any reason why it's necessary in such a low
level system component. "this is not rocket science" you don't impose
dependencies on something you want to be reliable.

Daemontools is clearly the most mature solution, from a perspective of
stability. It's probably why nobody wants to use it, there's nothing to play
with or exploit for profit because it already works.

~~~
bitwize
_That comment is insane. Why would log files need journaling? They should be
serialized._

systemd-journald provides tampering-attestation features that plain-text logs
do not, and cannot, provide. It's fundamentally more secure.

~~~
peterwwillis
Fallacy: having owned the user that writes logs, I cannot fuck with your logs
if journaled.

Reality: if I owned the user writing logs, I can do anything I want with the
logs, no matter how they're stored.

~~~
hdevalence
No, you cannot do anything you want with the logs.

Here is something you cannot do: you cannot alter the logs without anyone
noticing, since log entries have a rolling hash.

This prevents an attacker from being able to modify logs without detection,
and was implemented in systemd after the postmortem on the kernel.org breakin,
i.e., it was developed to counter an actual threat.

I would suggest you read up on what log attestation actually does.

~~~
peterwwillis
Logs can be modified in real time by being intercepted during writes to the
buffer before the new hash is applied. Because of this, only the logs _up to
the intrusion_ can be preserved, and even then if the old message/hash/key/etc
is still in memory, the old messages can be modified.

The only truly secure way to record messages up to the point of break-in is a
one-way remote log. Any new messages after break-in is subject to
falsification.

~~~
bct
> The only truly secure way to record messages up to the point of break-in is
> a one-way remote log.

And if you have this then you don't need a fancy cryptographic syslogd to
detect tampering.

~~~
peterwwillis
The fancy cryptologic syslogd is fallible as I mentioned previously, so why
depend on it? And really if you really want to cover your tracks you can just
cause random disk corruption over the whole disk (and just happen to damage
the journal in the process). Things will start failing rapidly, fsck will
later show itself fixing the disk corruption, and it will be assumed to be a
hardware error and the incident ignored.

Security half-measures do not mean you are secure. If it can be hacked, it
will be hacked, and then what was the point of sacrificing your whole system's
init system?

(Also you could just replace syslogd with a journaled syslogd, rather than
replacing your entire init system... but I guess that's off-topic?)

~~~
bct
I'm agreeing with you.

------
dschiptsov
It is broken by design because it contradicts with some general principles,
such as "single responsibility", "share nothing", which leads to "divide et
impera" and other "nice things".

Unix philosophy, according to which a system is a set of lose-coupled tools,
each of which, at least in theory, is supposed to do just one thing, but do it
well, and to follow certain conventions, such as using text streams as input
and output, which allows them to be hooked together into pipelines, is a
really great one. To sacrifice it just for very contradictional arguments of
people, who, perhaps, not fully understand it or its consequences for a
complex system's design, is, at least, shortsighted decision. (look what a
wonder is Plan9, at least from the system design perspective and "less is
more" principle).

As for so-called "problems", BSD people would tell you that most of these
"problems" are mere creations of distracted minds.)

Introducing various daemons and ctls merely adds unnecessary complexity
compared with the uniformity of "just shell scripts" and a very simple set of
conventions. At least _everything_ works fine with FreeBSD and its _huge_
ports collection.

~~~
Fasebook
What are these "general principles" of "unix philosophy"? Where can I read
about them?

edit: thanks, those both look like great books so far.

~~~
julie1
Everything is a file. Ted Nelson presentation on how it is quite broken by
design
[http://www.youtube.com/watch?v=Qfai5reVrck](http://www.youtube.com/watch?v=Qfai5reVrck)

~~~
Fasebook
Thanks

------
oofabz
> PID 1 brings down the whole system when it crashes. This matters because
> systemd is complex.

The Linux kernel is much more complex and also brings down the whole system
when it crashes.

> Attack Surface

Sysvinit runs shell scripts and has known race conditions. A system like that
is not secure by design. It is as secure as it is because exploits were fixed
one by one over the years.

> Reboot to Upgrade

You must also reboot to upgrade sysvinit.

~~~
dredmorbius
_It is as secure as it is because exploits were fixed one by one over the
years._

That's largely how things get secure. Though pre-emptive security through code
audits and library rewriting (the OpenBSD approach) also helps.

Init scripts have had those years. Systemd has not.

~~~
pipy
Systemd is controlled by Red Hat in a way in which critical system components
including kernel haven't been controlled before. Not by single corporate
entity.

That's what we know about this company from an old (2007) article:

> “When we rolled into Baghdad, we did it using open

> source,” General Justice continued. “It may come as a

> surprise to many of you, but the U.S. Army is “the” single

> largest install base for Red Hat Linux. I'm their largest customer.” [1]

It is better to go with a grass-roots solution, even the one technically
inferior, that isn't being influenced by one single vendor or government.

[1]
[http://archive09.linux.com/feed/61302](http://archive09.linux.com/feed/61302)

~~~
dredmorbius
I'd be more concerned about RH and its own incentives (look at the clusterfuck
that's GNOME) than the US Army specifically.

~~~
pipy
Sure. I have a gut feeling that GNOME reworking was done solely to make
trouble for Canonical.

The Interface Stability Promise [1] by systemd team is just a promise, nothing
more. I wonder if Red Hat will keep it if it decides that it no longer serves
their bottom line.

[1]
[http://www.freedesktop.org/wiki/Software/systemd/InterfaceSt...](http://www.freedesktop.org/wiki/Software/systemd/InterfaceStabilityPromise/)

------
chroma
My biggest concern with systemd has to do with its creator, Lennart
Poettering. He created PulseAudio and Avahi. Neither of those are particularly
reliable, stable pieces of software. Also, Lennart's opposition to making
systemd portable is worrying. The other unixes have it hard enough already.

I don't want to single him out and hate on him. Lennart's probably a nice guy,
and he's contributed a lot more to open source than I have. But having the
people responsible for PulseAudio involved in the next-generation init sounds
like a recipe for disaster.

~~~
Shish2k
> He created PulseAudio and Avahi. Neither of those are reliable, stable
> pieces of software

They weren't when they were first created, but since they became mainstream
and other people joined their teams, I've found them to be just as stable* and
far more featureful than the things they replaced

* not perfect, just not worse than what came before

~~~
makomk
They're still not. For instance, the latest PulseAudio release includes a
patch that contained incomplete changes that broke various resamplers but was
included anyway. It'll be reverted in the next major release which will
include other half-baked changes. (It turns out that wasn't even the cause of
the crashes I was experiencing; even before that patch it was broken in a way
that meant just having the official volume control application open caused
random crashes. I think anyway - unfortunately, all the variable names are
misleading and the comments non-existent, so it's impossible to be sure what
the code's meant to do.)

------
peterwwillis
I won't ever use a distribution that uses systemd as pid 1. There is only one
reason for this.

I want to be in control how my system works. To do that I need to be sure of
what it is doing and how. As the complexity of a system grows, the number of
things it does under the hood grows. At some point I can't keep track of
what's going on, and I no longer have confidence in my knowledge of the
system, without setting aside time for a code review.

With pid 1 being incredibly simple, and the executables it runs being
incredibly simple, I know what's going on, I know what the possible problems
are and I know how to fix them. The more complex it is, the more difficult it
is for me to know what's going on. That's all it comes down to for me.

As an aside: A lot of systemd is API-driven, meaning if you want a feature you
have to build it into the system as new code. This means you have to be a C
developer to leverage some new feature, instead of writing an interpreted
script and interacting with it via pipes/plugins/etc. If your current system
is dependent on some feature that doesn't yet exist in systemd, you now have
to build it in, instead of using what worked with sysvinit before.

Not only is systemd broken by design, it's still in its infancy, and does not
support the same features that have been built into other systems for years.
This additional load of development for many orgs serves no purpose, and is
essentially ignored by the leadership that decide to change init systems by
fiat.

~~~
hdevalence
> I want to be in control how my system works. To do that I need to be sure of
> what it is doing and how. As the complexity of a system grows, the number of
> things it does under the hood grows. At some point I can't keep track of
> what's going on, and I no longer have confidence in my knowledge of the
> system, without setting aside time for a code review. > With pid 1 being
> incredibly simple, and the executables it runs being incredibly simple, I
> know what's going on, I know what the possible problems are and I know how
> to fix them. The more complex it is, the more difficult it is for me to know
> what's going on. That's all it comes down to for me.

How is, say, sysv init better in this regard? Instead of having declarative
service files that say _what_ should be done, you have a collection of
disparate shell scripts, all with their own implementations of basically the
same things. The complexity isn't reduced, it's spread around (and IMO it's
more complicated).

If you want to have control over your system, wouldn't you want to use an init
system that can actually keep track of all of the child processes of a
service? Brittle hacks like pidfiles cannot ensure management over all child
processes of a service -- this is only possible with cgroups.

> As an aside: A lot of systemd is API-driven, meaning if you want a feature
> you have to build it into the system as new code. This means you have to be
> a C developer to leverage some new feature, instead of writing an
> interpreted script and interacting with it via pipes/plugins/etc. If your
> current system is dependent on some feature that doesn't yet exist in
> systemd, you now have to build it in, instead of using what worked with
> sysvinit before.

No, you just use D-Bus, from whatever your favorite language is, using that
language's D-Bus bindings. Systemd has a stable D-Bus API for this purpose
that's guaranteed to work in the future.

> Not only is systemd broken by design, it's still in its infancy, and does
> not support the same features that have been built into other systems for
> years.

Out of curiosity, what features are you referring to specifically?

> This additional load of development for many orgs serves no purpose, and is
> essentially ignored by the leadership that decide to change init systems by
> fiat.

This would be more convincing if it were just Lennart & co. making the
decisions, but in the absence of evidence to the contrary, I'm going to go out
on a limb and assume that the technical leadership of Fedora, RHEL, OpenSUSE,
Arch, and now Debian do not all make decisions by fiat for no reason.

~~~
peterwwillis
First, it's less complex because the technology involved (shell scripts) are
simpler to learn, design, implement, customize, debug, etc. Second, it's less
complex because it's based on known technology and there's no learning curve.
Third, it's less complex because the components it depends on are independent
and small/simple, versus monolithic and interdependent. Fourth,
troubleshooting sysvinit is incredibly simple, and you don't have to know
anything about the tools involved to debug them. Fifth, it's less complex
because there's less specialized operation to take account of.

Keeping track of child processes does not mean that the system is under
control, nor that I understand it better, which is necessary for me to have
control. And cgroups are not necessary at all to manage child processes. They
simply make it more compartmentalized, and help deal with zombie parents,
which also can be dealt with without cgroups.

So your language has to have D-Bus bindings, which hopefully are maintained
for your language. And you have to learn how to use those bindings. More
complex, more things going on under the hood, steeper learning curve.

The features I refer to are site-specific. I've worked for many companies that
heavily modify their services and system init, logging, etc to fit their
needs. I've worked on embedded applications, big server farms, and custom
tailored products. They all customize various parts of the operating system,
and have written custom features for them. Now it all has to be rewritten for
systemd.

I can't comment on the exact decision-making process behind adopting systemd,
but I can say almost all of what I have read about it is based on a mob
mentality that tries to impose its will on the organization rather than work
out compromises.

~~~
whatevsbro
_FLAWLESS VICTORY_

(Yeah yeah, Reddit, I know.)

------
VLM
The scariest part no one is talking about is the feature creep. All software
eventually dies; systemd will be no different. Some folks hope that happens
immediately, some hope we're "stuck" with it for a couple generations, maybe
30 years. Regardless of exactly when, its going to be replaced someday, and
massive feature creep seems intentionally designed to make the eventual
changeover as painful as possible.

Someday in the future, either 2016 or 2066 someone is going to want to
rearchitect the "thing that the kernel starts first" and is going to freak out
at the insane feature creep... so you're telling me that I not only have to
create something to start services which is no big deal, but also an entire
logging system, and a hotplug handler, and a cgroup server, and ... WTF does
this blasted thing have an MP3 player, a web server, and an IRC client in it
too?

Its an awful design but probably survivable, at least temporarily, and once
the flamewars die down a sensible replacement with a good design can be rolled
out to replace it. But mid-flamewar with all manner of business / marketing
money flying everywhere on all sides is not the ideal blue sky redevelopment
time. As much as it pains me to say it, rolling out an init that has almost
the worst possible inherent design possible does have some value to the
ecosystem as a counterexample, here's what not to do with 2.0. Obviously its
replacement will be better, I just hope it doesn't swing too far in the
opposite direction as a reaction. The point of this paragraph is the most
important thing to do at this time is to lay the seeds for the replacement,
while not over-reacting to the disaster of systemd.

So, the replacement for systemd, to be rolled out sometime between 2015 and
2065, should ... (insert feature list here).

~~~
marcosdumay
Great, now what will think the unluky sould that'll need to replace Linux?

The problem we are seeing is the framework vs. libraries dicotomy. Cut the
features in too many packages, and you won't be able to use them anymore.
Integrate them too much, and you won't be able to maintain them anymore.

Anyway, just like the kernel, some day we must be able to say "launching the
basic processes is a solved problem", and actualy solve it in a way that does
not need to be replaced. We are not there, but we are in a point where the
feature set is quite stable, so I guess we can improve usability a bit, at the
expense of maintanability.

------
chipsy
I get it now. The distro ecosystem has been moving towards the other half of
Sutherland's wheel of reincarnation - in this case, from modular systems
towards monolithic ones. First it was stuff in the desktop space - now it's
creeping into other areas.

You can state the particular point of inflection as a terrible thing, or a
good thing, but seen as a technology learning cycle, it becomes _necessary_
even if the implementation chosen is shit and you hate everything about it. At
the other end of it, it still gets broken up again into a different set of
modular components whose design wouldn't have been apparent at the beginning.

Since the initial discussion stage is over, I would say, if you really think
this is the wrong solution(and I don't consider myself expert enough on inits
to cast much judgment here), your work is best spent on the code environment
immediately surrounding systemd - not direct competitors, but stuff that it
has to deal with at some point. That will put pressure on the design to either
break entirely or evolve to support whatever you've come up with - and when
that happens you get to walk in with a patch to show and kick off another
firestorm.

------
nailer
The argument re: larger surface for updates is true. Systemd is smaller than
the things it replaces, and consistent in itself, but all the logic is in PID
1.

I fail to see a better alternative though.

Old, disparate init systems repeat logic for:

\- starting services (the same init script copied 90 times, often badly)

\- journalling (unmaintained syslogd facilities means apps all report
themselves as local0 and five different fixes for that)

\- a billion different 'sentinel / watch' type apps

\- various hacks on hacks for dependencies, parallel init, etc.

If you rip out all the redundancy and replace them with something unified,
that unified functionality can (and should) exist at low level.

Maybe the argument is that a better way of handling systemd updates is needed?

~~~
d0
These problems are all solved without destroying the Unix philosophy
completely:

starting services - daemontools.

journaling - daemontools

billion different sentinel/watch type apps - daemontools.

dependencies - just don't do it. This is the old age of power sequencers
again. It's coupling.

Daemontools is a set of tiny programs that talk to each other, are fully
privilege separated and guarantee reliability. Why they can't do this for
systemd, I don't know. If you remove the DJB path conventions from it (the
major objections), it's the right tool for the job.

~~~
nhaehnle
Dependencies between services exist, and for good reason (e.g. various
services depending on a database). How do you propose to deal with them?

~~~
d0
It's pretty basic software design. A few solutions:

1\. Web servers return error pages when databases are unavailable. They don't
go down.

2\. Postfix doesn't die if your virtual maps database server is not available.
It waits patiently.

This isn't rocket science.

~~~
nailer
I'd rather Postfix fail early, than wait until $TIMEOUT to discover that a
maps source isn't available.

~~~
mercurial
I think what parent is trying to say in so many words is "I have no use case
for tracking service dependencies, therefore it is pointless and should be cut
out". An unfortunately common attitude among technical people.

~~~
d0
No. I'm saying that relying on startup sequencing is a bad idea full stop.

~~~
the_mitsuhiko
That's why systemd does dependency tracking and socket activation instead of
sequencing.

~~~
d0
It needs to do neither of those as well.

~~~
mercurial
You're certainly entitled to your opinion about dependency tracking being an
unnecessary luxury, and I'm sure you will find a number of Linux distributions
and/or BSDs within your spartan requirements. Not to mention that you can at
least for the time being stay with sysvinit (though this won't solve your
issue on Debian, since it reads dependencies via a horrible hack in the
comments of the init scripts).

~~~
d0
I'm a FreeBSD and Windows person so yes you are right.

sysvinit dependency management is horrible as well so I cannot complain about
your point there.

------
julie1
The major problem is the requirements and the system dependencies it
introduces on a core component. [http://harmful.cat-v.org/software/dynamic-
linking/](http://harmful.cat-v.org/software/dynamic-linking/)

systemd introduces a major PITA for the OS packager by putting a lot of
dependency on a lot of parts with "moving components" making it both more
vulnerable as 1 and more likely to fail (it is not complex it is complicated).

Systemd also tends to become a system requirements for a lot of packages
making it unsuppressable.

It makes the problem of dependency solving logically totally messy without
justifying this cost by any real improvement.

It is introducing the DLL hell for the most crucial part of the system (PID
1).

------
brokenparser
Systemd makes my life difficult. Case in point: I want to stop cups, so I run
"sudo service cups stop". This appears to work, but then it prints out a
warning along the lines of:

    
    
      cups was stopped, but it can still be activated by:
      - cups.sprocket
      - cups.latch
    

What are these things and why are they there at all? How do I shut them off?
It doesn't tell, so stopping a service has become pointless. If I tell it to
stop, make it stop!

------
Aardwolf
> The desktop is quickly becoming irrelevant. The future platform is going to
> be mobile and is going to be dealing with the reality of running untrusted
> applications

This is off topic (tho from the article :p), but, is it really becoming
irrelevant? What will replace such things as: typing lots of text,
photoshopping/gimping, programming IDE, browsing with multiple browser windows
with dozens of tabs each, etc...?

Would Linux users abandon the desktop, if so, for what?

P.S. unless I can plugin my phone into something with a keyboard, mouse and
big monitors (or more productive equivalent), it is as fast as my current
desktop, and has desktop oriented software in that mode, I don't see myself
abandoning the desktop

~~~
tinco
Desktops are not really going away, they're just taking a roundabout. Instead
of desktops people are now buying laptops, and attaching external monitors,
keyboards, mice.

The next step could be people buying tablets, and attaching the external
devices for power use.

The difference is subtle, as it seems to be nothing is materially changing,
but the current tablet/mobile ecosystem is dominated by application container
operating systems like Android and iOS, and it seems unlikely that that is
going to change.

This is the movement that is obsoleting traditional Linux and Windows, at
least on consumer hardware.

So to answer your question: "typing lots of text, photoshopping/gimping,
programming IDE, browsing with multiple browser windows" will be replaced by
functionality in containerized applications or 'apps'. The apps are
bought/downloaded from app repositories that are loosely maintained by
commercial entities, and they could very well be untrustworthy.

As to your P.S.: yes, exactly, so you already know you are going to abandoning
the desktop.

~~~
pekk
because I really don't need actual Photoshop or development tools that have
been refined for many decades, I just need shitty little Android apps written
in the last few years.

No, that is wrong.

~~~
tinco
What makes you think that Photoshop or development tools can't be either apps
or web applications?

------
scrollaway
Discussion on reddit:
[https://pay.reddit.com/r/programming/comments/1xggg8/broken_...](https://pay.reddit.com/r/programming/comments/1xggg8/broken_by_design_systemd/)

On that note, nothing screams FUD like a screenshot of Windows behaviour on an
article like that. Is there a Linux-Windows equivalent to Godwin's law?

~~~
baq
the picture is very adequate when paired with this sentece:

> Unfortunately, by moving large amounts of functionality that's likely to
> need to be upgraded into PID 1, systemd makes it impossible to upgrade
> without rebooting. This leads to "Linux" becoming the laughing stock of
> Windows fans, as happened with Ubuntu a long time ago.

------
Tuna-Fish
This reeks of the same kind of argument that was used to push for
microkernels. Arbitrarily cutting up processes into smaller parts is not the
same as decoupled design.

~~~
d0
It's not arbitrary. There are logical security, reliability and convention
boundaries that systemd doesn't consider which are raised in this article.

Microkernels are a different subject altogether.

To be honest, the only service management solution I've seen that I actually
like for Unix platforms is BSD's simple rc and daemontools on top of it for
process management.

------
arca_vorago
Stuff like this is why I am increasingly getting back into BSD and recently
Minix3. I think this is just part of a larger issue with the *nix ecosystem,
where lots of philosophical and design assertions are proving to no longer
work, while some are falsely being said to no longer work but still do. With
the linux kernel now at over 10 million lines of code, the idea of peer review
of source code seems too big of a job for random unpayed volunteers. I think
we need to move back to simplicity and decentralization as much as possible,
because the problem with centralization is that it makes for a weak system.

------
ANTSANTS
This was posted earlier today, under the exact same title, but with a URL that
differed solely by a trailing backslash. It hit the front page, too! Way to
go, HN duplicate detector.

[https://news.ycombinator.com/item?id=7207655](https://news.ycombinator.com/item?id=7207655)

------
CopperWing
Lot's of FUD there on ewontfix.com.

A better source of information:

[https://wiki.debian.org/Debate/initsystem/systemd](https://wiki.debian.org/Debate/initsystem/systemd)

------
gegenschall
I guess this post has some relevance in this thread:
[http://0pointer.de/blog/projects/the-biggest-
myths.html](http://0pointer.de/blog/projects/the-biggest-myths.html)

~~~
pekk
Some of those jokes are very funny. I really liked the one about how systemd
is modular, just in the same way that the Linux kernel is modular, because you
can compile it with all different options.

~~~
gegenschall
I do not really understand what you're saying.

1) This is actually modular. I mean... those things are even called "modules".

2) Even if it wasn't modular, how would it matter? You're comparing to the
Linux kernel. If that one isn't modular then according to you, systemd also
isn't modular. Allright, so be it, did you have any problems with missing
modularity in the Linux kernel lately?

------
mwcampbell
The main systemd binary (version 208 for x86_64 as shipped by Arch Linux) is
1020 KB. I get that the systemd package is really several binaries, but still,
does the binary for PID 1 need to have _that_ much in it?

~~~
antientropic
That's about 1/8000th of the RAM in my laptop. Is 1020 KB really such a big
deal?

~~~
mwcampbell
The problem is that software complexity grows exponentially with code size.
Well, maybe not exponentially, but more than linearly. And the article
established that it's important to keep PID 1 simple.

------
njharman
systemd reminds me of 90's era Microsoft's "release a new service/api/whatever
that is almost the same, but different enough to create incompatibility and
lockin, include small addition or improvement to encourage adoption, make
everything require it to force adoption by dissenters."

I'm not joking.

------
Sammi
This feels like the Tanenbaum–Torvalds debate all over again.

------
olgeni
Why not copy SMF?

~~~
nailer
systemd shares much of its design with launchd and SMF, with the obvious
exception of using XML for service config. The same arguments against it would
apply.

