
Systemd and the crashing tweet - type0
https://tim.siosm.fr/blog/2016/10/02/tweet-systemd/
======
fjdkfjdkfjdk
There are remarkably _few_ people developing systemd (the whole project, not
just init) full-time.

No language is going to produce quality software in a project implementing too
much functionality with too few resources. The Go and Rust "safe systems
language" hype is strong right now so the language choice is being attacked.
The core cause is much simpler to explain.

Systemd (the project) is complex, broad, and deep in scope. It takes a lot of
time and effort to arrive at a quality implementation of something like that.

I fully expect the majority of the systemd code to be rewritten, even if the
language doesn't change. What we have today is a usable prototype.

When everyone is satisfied with the architecture, feature set, and
configuration schema of systemd, after the dust settles, this is all going to
be revisited, if there are any users left to warrant it.

There's much work to do, and I bet many of you reading this thread could
contribute to getting systemd to where it needs to be. If you use Linux, and
can code, clone the systemd repo and start helping!

~~~
mixedCase
If systemd devs cared about proper modularity it would be much easier to
collaborate.

I believe the first step towards a rewrite is just making an init
system+daemon supervisor that (at least partially) understands systemd unit
files or a similarly simple solution.

systemd's huge grapple on Linux systems was 95% caused by just by people not
wanting to maintain shell scripts (which is understandable).

~~~
thegeomaster
This. I see criticisms of systemd all around Hacker News, but I find it really
easy to create a unit file on a production system and just go systemctl start
<unit-file>. I sometimes even create shell scripts for some tasks and create
one-off unit files so I can run them easily as the right user, and have proper
logging for free.

I'm not a system administrator and I just appreciate that the user-system
interface is easy to work with, for the occasions when there's no dedicated
sysop in the team.

~~~
drinchev
I second this. As far back as I have been a Linux user ( back to Slackware 4 )
I was very pissed by the systemd ideas.

Nowadays I think it's priceless when I can just create a unit file for my
NodeJS and do journalctl -fu my_app.

I think the criticism goes to the leader of systemd, who has exceptional
arrogance ( as far as I have read ). Maybe if there is a competition between
systemd and you name it, the init system will feel more natural.

------
agwa
Although I don't agree with all of the analysis in this post, it's pretty
reasoned and balanced on the whole.

However, it's arguing a strawman that systemd should have been written in a
memory-safe language. I never argued this. I know that Rust did not exist and
Go barely existed when systemd started development. The reason I called out
the memory unsafety of systemd is to underscore how irresponsible it is for it
to be parsing messages from untrusted sources as root in PID 1. An alternative
to writing in a memory-safe language would have been to use privilege
separation, which was very much known in 2009.

> A possible path forward to solve this issue would be to start rewriting
> parts of systemd or libraries used by systemd in Rust. For example, take a
> look at the work in progress for syslog-ng (Syslog-ng and Rust).

Unfortunately, rewriting a component of systemd in Rust would require buy-in
from the systemd developers due to systemd's tight coupling. In a loosely-
coupled system, one can replace individual components with safer alternatives
as they become available. This fosters healthy competition and innovation. I'm
optimistic that we'll see some really awesome operating system components
written in Rust in the future. My fear is that systemd will slow or prevent
adoption of these safer replacements due to its tight coupling and the way
that it is spreading to more and more of the operating system.

~~~
srslack
>The reason I called out the memory unsafety of systemd is to underscore how
irresponsible it is for it to be parsing messages from untrusted sources as
root in PID 1.

systemd-notify is "parsing", in your scenario by passing the message in the
simple key-value format that systemd expects, which is the only "parsing" that
PID1 really does, along the notification socket. For everything else, there's
generators.

This wasn't a "parsing" bug in PID1, it was a logic bug.

And further, what would be a 'trusted source'? Something that would've done
the same thing, passed a data structure that PID1 has to receive and
understand?

Your ranting is nonsensical, considering that it's doing pretty much exactly
what you wanted.

~~~
papaf
_simple key-value format that systemd expects, which is the only "parsing"
that PID1 really does_

The important word in the grandparent comment is "untrusted". HTTP and SMTP
are simple key-value formats that most people would not consider parsing in a
root process in a language like C.

You make a good point about there being no safer alternatives though given the
architecture of systemd. What does that suggest about the architecture of
systemd?

~~~
srslack
>"untrusted"

So you're going to "trust" i.e. deputize some theoretical daemon that will
still have to pass the simple data structures to systemd, which won't actually
fix, or fix any future incarnations of, the bug that we're discussing.

>What does that suggest about the architecture of systemd?

It suggests that it is, at worst, the same as the status quo on Solaris and
macOS.

In reality, the community is filled with people waving their hands and
doomsaying against "complexity", ignoring the fact that their suggested
actions don't really solve anything.

They seem to be a cargo cult of modularization, if you catch my drift.

Along with all of the comments in the thread suggesting languages that can't
fork, and languages that can easily lose track of daemons even when the daemon
doesn't double fork, we can see that not every suggestion that sounds
reasonable is actually reasonable, or better.

Additionally, there's probably some lesson about `PR_SET_CHILD_SUBREAPER` and
what happens when the subreaper they so desperately want to exist crashes.

If systemd were running as PID2, and crashed, children would simply be
inherited by PID1, and now we need to communicate everything to the service
manager. You can't set subreaper again, after it crashed.

So really, what are we arguing about here?

------
i336_
It's like someone wanted to write the ultimate PHP OS-management framework.

PHP is installed virtually anywhere, unbelievably fast (really!), achieves
most tasks with a reasonable minimum of boilerplate (okay, okay, within
reason) while remaining verbose enough to be easy to learn, and makes it
possible for anyone to get started.

But the language _itself_ makes you want to... I don't know, exactly - scream,
or something? because of stuff like this gem:
[https://developers.slashdot.org/comments.pl?sid=204433&cid=1...](https://developers.slashdot.org/comments.pl?sid=204433&cid=16703529)
(highly recommended click)

I finally figured it out: this is exactly the same mindset behind systemd! D:

The OP article says that

> _systemd should be written in a memory safe language. The obvious picks are
> Rust or Go._

My immediate response was "no no no _no no NO_! You need to replace the
_developers_!"

I totally get the idea behind using a kitchen-sink-safe language for Important
Critical Stuff™, I really do. And I know that C is slowly going out of vogue
for system-level tools of the same order that systemd is in.

What I _don 't_ like is that the developers just blindly bumble along thinking
everything's fine, denying the importance of issues like these... and get all
confused when they get death threats
([https://plus.google.com/+LennartPoetteringTheOneAndOnly/post...](https://plus.google.com/+LennartPoetteringTheOneAndOnly/posts/J2TZrTvu7vd)).

It would be much nicer if the people with the power could just go "ooh, whoa,
thanks. Uhh, I don't feel like working on this, but PRs are welcome, and I'll
doublecheck if it's been fixed in two weeks." That acknowledgement is really
all people want...

~~~
digi_owl
> and get all confused when they get death threats

That once traced back to the source came from the Sailfish (aka the Jolla
smartphone OS derived from Nokia Maemo) IRC channel. And it was uttered as a
off color joke about hiring an assassin via Kickstarter after yet another late
night problem solving session because they were trying to update to Sailfish
to the latest Systemd version.

It didn't come from "haters" or "trolls", it came from within the systemd
community. From people actually trying to put systemd to productive use. But
it was used as an example of how horrid the Linux community was behaving
towards Poettering because they didn't like systemd.

Oh btw, Poettering gave a description of the big that all the recent hoopla is
about over at /r/linux. Except that he got it all wrong, and glossed
completely over that he was the person that did the change that caused the bug
in the first place. And what was that change? The removal of the very kind of
check that would stop the bug from happening in the first place.

In contrast we have a recent issue in the kernel, where Torvalds let a patch
in that made the kernel oops. The first thing we see is Torvalds blaming
himself for not rejecting the patch even though he had noticed tell tale signs
of potential problems.

Basically a repeating pattern with Poettering, and Sievers, his right hand man
in all this, is to deflect or try to blame others for their screwups.

~~~
gcr
No one deserves death threats.

Not systemd developers. Not you. Not me. No death threats.

~~~
hazeii
Back in the days when games were written by one person, it was not that
uncommon to get death threats or to have people say they were going to
firebomb your house. We just ignored them.

Which is not to say it's acceptable to make such threats; more a comment on
how times have changed.

------
stass
> Thus neither Go or Rust was an option at the time. Don’t tell me it should
> have been written in OCaml, Haskell, Java, ADA, whatever. All of those
> memory safe languages were either unfit for such a low level software or far
> too confidential to be a good pick.

Which of these are unfit or confidential? This is an extremely weak argument.
It's basically saying that none of the languages besides Rust and/or Go are
unfit for system-level software.

~~~
Avshalom
What the hell does confidential even mean in this context?

(Also it is of course written Ada)

~~~
fjdkfjdkfjdk
I suspect the author meant "controversial", since "confidential" makes zero
sense.

~~~
polotics
the author meant "not widely used" and assumed french~=english

------
ashwinaj
I find it hard to understand why people "expect" perfection from code written
by humans. Shit breaks, people make mistakes, deal with it. Is anyone in their
right minds to expect perfectly written software, while at the same time
adding new features without things breaking in some obscure (or obvious way)?

All this post-mortem of "I would have done it this way or that way or written
in this new super shiny language (which probably has bugs too)" is nothing but
a bunch of nonsense. Rant over!

~~~
agwa
> I find it hard to understand why people "expect" perfection from code
> written by humans. Shit breaks, people make mistakes, deal with it. Is
> anyone in their right minds to expect perfectly written software, while at
> the same time adding new features without things breaking in some obscure
> (or obvious way)?

Absolutely right, which is why software, especially critical components,
should be written in a fail-safe way that anticipates errors by programmers.
systemd is not written this way: just look at the unsafe-by-default umask and
the fact that a failure in message parsing can bring down all of systemd,
preventing a clean reboot.

------
ausjke
Systemd is probably the most controversial and intrusive project in linux
ecosystem. It may end up being good or awful. For me, I avoid it as much as I
could and hope to have nothing to do with it later.

~~~
petre
How exactly do you avoid it? It spread like the plague in almost every Linux
distribution. The only way to avoid it is by running BSD, Slackware or one of
those one man show Linux distributions.

I have issues with apache not being restarted by systemd for obscure DNS
resolving reasons.

~~~
dredmorbius
On apt-based systems, you can pin it oblivion. This will prevent systemd, or
any software which depends on it, from being installed.

There are some cases of non-systemd-specific packages which are affected,
though in my experience, anything brain-dead enough to rely exclusively on
systemd is best avoided (GNOME seems most affected).

There are several alternative init systems.

Debian and Ubuntu can be treated in this way, along with derivatives. Slack
and Gentoo as well as I recall. I've given consideration to BSDs, but much
prefer GNU userland (and yes, I know that's an option).

I'm hoping Red Hat collapses and/or Linux generally comes to its collective
senses.

~~~
atmosx
Gentoo lets you choose. The default is openRC though.

~~~
petre
Well, too bad openRC was not chosen when Debian picked an init system. I
really wished they would chose openRC instead, but no. It has an /etc/rc.conf
just like BSD's rc.subr. Devuan wasn't working the last time I tried to
install it, maybe that will change in a while. The prospect of a Debian fork
with openRC (even if it's optional) sounds pretty nice.

~~~
ausjke
alpine linux uses openrc too, which is the default for docker images.

------
phantom_oracle
I'm going to call it, on this day: 5/11/2016

 _Systemd is the new vim vs. emacs_

The road to the GNU/Linux unification, was forged by the great wizard of
Torvalds. Torvalds powerful use of the: fuck this code is ugly, fix your
fucking mistakes, don't break shit via the kernel ... kept the flame of the
kernel burning

Meanwhile, across the lake from Linux Use-Land, in the industrial city of
Enterpriseyness, a small section of Linux Use-Land citizens formed their
RedHat, to show the world how amazing life across the lake is.

They initially served Linux use-land as respected/renowned citizens, but
eventually the cities 'hungry' citizens, known as "shareholders" and
"government" consumed the Hat.

Beyond consumption, a brainwash-attack was launched by the dirty robber-baron
known as "MicroSoft", in which RedHat was tortured into the philosophy of
"embrace, extend, extinguish".

RedHat knew that Torvalds was growing weak, as his "Linux Foundation" depended
on Enterpriseyness for money. Together, a task-team from Enterpriseyness set
out to explore infiltration of Linux Use-Land. This task-team included: NSA,
CIA, GCHQ, MicroSoft, Google, Facebook, Cisco, et. al (from the leaked
documents of Enterpriseyness arch-enemy: WikiLeaks, a non-redacted version
also showed RedHat to be in this task-team).

Enterpriseyness started their exploration via the Gospel of Hacker News. This
Gospel was a powerful driver of the future, as reaching the "front page" of
said Gospel would ensure eternal bliss and success (and entry into tech-
heaven: Silicon Valley). A popular fable was the "Docker", in which this
whale-hunting organization showed that with enough good words from the Gospel,
hijacking PID1 was plausible.

As such, the CIA activated their sleeper-cell in the Linux Use-Land, known
only as "Poettering" in leaked-documents and the trojan-horse called "systemd"
was born. The systemd is now being built to consume Linux Use-Land and create
a new colony for Enterpriseyness.

PS. This is fiction + written for humour :)

~~~
i336_
[http://video.fosdem.org/2014/Janson/Sunday/NSA_operation_ORC...](http://video.fosdem.org/2014/Janson/Sunday/NSA_operation_ORCHESTRA_Annual_Status_Report.webm)
(46:05) (that link is a 302)

I was reminded of this video by Paol-Henning Kamp (2014) when I read this. It
sort of gets at the same set of ideas.

(Feel free to share similar videos, anybody...?)

------
digi_owl
His example about resolved is once more missing the point of the issue. Its
not about the protocols facing the rest of the world, its about the protocols
between the systemd "parts". Those are willfully unstable, and thus makes its
so you has to take systemd as a whole or not at all.

------
Grue3
The response from systemd guy was laughable. I can't believe this is 1) an
adult 2) developing a system-critical software 3) that is included in almost
every Linux distribution. Pathetic.

~~~
digi_owl
Say hello to the power of money and dependency chains.

------
kalefranz
Does anyone know of an alternate, semi-portable init system (I'm thinking
something like supervisord) writen in rust or go that could one day turn into
a systemd init alternative?

------
mmphosis
Maybe I am too much of a minimalist, but I think systemd is trying to solve a
dependency problem that should not be as bad as it is.

    
    
      # ps aux | wc | awk '{print $1}'
      200

~~~
valarauca1
The dependency issue is referring to start up process order.

One of the headaches of the old SysV init system was you had specify what
order daemons started and when they started in the boot process.

The login daemon started while the OS was still in single user mode. Then
you'd start the logger. Now you bring up multiple users, and the network
stack. Maybe NTP, DNS resolve,run as separately users? They definitely need
system log.

Systemd you just state what your daemon already needs running and it'll on
boot resolve the daemon start order.

If you've only ever lived in a package managed eco system and never rolled
several of your own daemons you likely never encountered this issue.

~~~
creeble
I'm a little lost on systemd myself. Seems like an attempt to solve the
problem of startup complexity by...adding complexity.

Maybe I'm just getting old and grumpy. But I have a Linux 2.0 kernel machine
that has been running a LAMP stack for over 10 years that boots in one tenth
the time my Ubu 16.04 machines - that run the same LAMP stack - do.

Ah for the old and simple days...

~~~
petre
> Maybe I'm just getting old and grumpy

Do try out FreeBSD then -- an OS for the old and grumpy. Systemd free. Uses an
init system written in sh(1) that just works.

------
arde
I think we systemd despisers are barking at the wrong tree. We already know
Poettering's social and technical abilities (or lack thereof). And we must
recognize his right to write whatever crap he wants. Focusing on systemd or on
Poettering is useless to us. The only people who should get our vitriol are
the ones that chose (and keep choosing) to use systemd in such main
distributions as Red Hat and Ubuntu despite its known defective design and
lack of safety measures. I want those people's names being known and
remembered by the community, and I want them to stand up to their harmful
decisions and I want them to change those decisions or step aside otherwise.
It is quite late for that but this can be changed and should be changed right
now, and such a change has already done once with Upstart (which was bad
enough to run away from too, BTW).

~~~
srslack
>or lack thereof

Coming from a group of people suggesting languages to write PID1 in that are
without the practical ability, and necessary in this context, to fork().

Or the suggestion to, at its core, make systemd a subreaper even though most
certainly don't know that they're suggesting it, or what it entails.

~~~
arde
That's a strawman, a tu-quoque and an ad hominem. Try arguments instead of
fallacies. If possible, arguments that don't ignore decades of knowledge
accumulated by software engineering, such as the need for low coupling and
high cohesion.

