
The bad side of systemd: two recent systemd failures - lreeves
http://utcc.utoronto.ca/~cks/space/blog/linux/SystemdCrashAndMore
======
ultramancool
I'm no fan of systemd, but do we really need to be nitpicking its every
failing? The alternatives are certainly not perfect either. If you don't like
it, don't use it. Many distros are still committed to supporting other init
systems along side systemd and many BSDs have a very good philosophy with
things like this and may be worth a look too.

Can we just agree to move on? I'm sick of the endless banter. If you like
systemd, use it, if you don't like it, don't. If you think it needs more time
to mature, stay away from it for a while and experiment occasionally. That's
my plan at least.

~~~
e40
_if you like systemd, use it, if you don 't like it, don't._

That is so naive. No one chooses to use systemd. People choose a distro. It's
being forced on __long time __users of distros without any of the users having
a say.

Yeah, I can switch from one distro to another. That's a choice I have. But
it's not so simple as you make it sound.

~~~
chimeracoder
> It's being forced on long time users of distros without any of the users
> having a say.

It's being adopted by distributions in accordance with the stated and implied
wishes of the developers, maintainers, and users of the distributions.

The reason that (e.g.) Arch dropped support for legacy init so quickly was
because they literally failed to find anyone who wanted to go through all the
work of maintaining the old init system. If none of the users want to do the
work, then nothing's being "forced" \- that's the way community-organized FOSS
projects work![0]

Same goes for Debian, who made the switch in accordiance with their normal
development process (including a general resolution in support of systemd!).
Debian has promised to maintain systemd-shim (supporting other init systems)
through at least Jessie, and possibly longer, depending on whether there is
enough demand for alternative init systems, and support from developers for
maintaining it in future releases.

Ironically, the biggest threat to systemd-shim in Jessie's successor are
people who claim they want to fork Debian - if all the people who hate systemd
leave Debian, then nobody who wants to actually do the work to maintain the
alternative init systems will remain with the Debian project[1].

[0] Even then, you _can_ still find an AUR package for it
([https://aur.archlinux.org/packages/initscripts-
fork/](https://aur.archlinux.org/packages/initscripts-fork/)), though as with
everything in the AUR, it's subject to much less testing. Note that the
package has only 27 votes - this means that only 27 people actually said that
they care about moving this package back into the main distro.

[1] This may be a bad or a good thing, depending on who you ask.

~~~
cthalupa
>The reason that (e.g.) Arch dropped support for legacy init so quickly was
because they literally failed to find anyone who wanted to go through all the
work of maintaining the old init system. If none of the users want to do the
work, then nothing's being "forced" \- that's the way community-organized FOSS
projects work![0]

Users != developers

I am not a developer. I'm an administrator. I know enough C to be able to poke
around source code and understand what is happening when I get an error, but
not enough of it or anything else to maintain an init system. A lot of admins
don't know this much. A lot of desktop users don't know this much.

I'm not disagreeing with your overall sentiment here - but it's not so simple
as "If the users of the product liked it, they'd maintain it" \- sometimes
they don't have the skills required to. Not everyone who drives is a mechanic,
etc.

~~~
jethro_tell
> Not everyone who drives is a mechanic, etc.

If you car requires more knowledge to maintain than you have, you can get a
different car, pay someone to maintain it or get a bike. Those are the only
options I see. I don't see how init is any different than a car. You have
those same options. I'm sure someone somewhere is paying RH or canonical or in
house devs/admins to maintain init scripts for their application stacks.

~~~
cthalupa
I hesitated to make the analogy because I expected a response like this,
stretching the analogy too far.

It isn't a matter of managing init scripts - that's easy. Continually updating
a distro to use your init system of choice is a wholly different matter. I can
change the oil in my car. I can't replace the transmission.

------
XorNot
I can't speak to the first problem, but the second is more of "this is strange
and I don't care to learn how it works".

journalctl --no-pager

Problem solved. I mean, you have the command, it's right there in the usage
text. Which, awesomely, does invoke a pager when you do journalctl --help so
if you're stuck on a 80x25 console (as you might be on a broken system) you
can easily actually read the help options.

~~~
mdpye
While I agree with your first paragraph,

> if you're stuck on a 80x25 console

What's wrong with `journalctl | less` in this case? The only command I use
regularly which invokes a pager _by default_ is `man`, and that seems like an
actual special case.

Edit: FTR, I agree with your assessment in the first paragraph, but not the
judgement I infer you passing. IMO, things should avoid being strange.

~~~
cesarb
> The only command I use regularly which invokes a pager by default is `man`

Git also invokes a pager by default. And it also sets $LESS when doing so. In
fact, it wouldn't surprise me if systemd's automatic use of a pager was based
on git; IIRC, it used to set $LESS to the exact same value git used (I believe
systemd has changed it since then).

~~~
camgunz
I super dislike this btw. If I want a pager I'll use a pager. I'm using Git, I
can handle `| less`.

------
lstyls
How much of the blame should systemd be taking for these bugs? Unless the same
segfault is happening on other distros, this seems like a problem in Fedora
rather than systemd.

I suppose this does give some substance to the systemd criticisms that it is
too system-critical for a process that is technically outside of the Linux
kernel.

~~~
kosinus
If you're saying, a bug should always be investigated downstream before
reporting it upstream, then I agree.

If you're saying, there are scenarios where segfaults caused by upstream code
are not upstream bugs, then I disagree.

------
darklajid
The first problem, PID 1 crashing, is certainly worth complaining about.

The rest seems mostly nitpicking. I wouldn't work myself up about the pager
issue as discussed elsewhere here. I do somewhat agree with a rather useless
default for the range of things to show (i.e. I'd prefer -b [1] to be implied
unless specified otherwise, for example).

1: -b means 'since the last boot' for people not using journalctl. The default
is instead to show the full journal from its beginning, as stated in the
article.

------
wtbob
Regarding the first problem, as someone else said, 'Remember how awesome
PulseAudio was? Imagine that for PID 1!'

~~~
cbd1984
PulseAudio gives me a better out-of-the-box experience than any other sound
system, bar none.

~~~
eots
Guess I'm the only one who had a cron job restarting the PulseAudio process
regularly.

~~~
bkeroack

      # keep systemd happy
      0 * * * * /bin/shutdown -r now

~~~
digi_owl
You know you are old when the first thing that comes to mind is a User
Friendly strip...

[http://ars.userfriendly.org/cartoons/?id=19990302](http://ars.userfriendly.org/cartoons/?id=19990302)

------
staticshock
Guys, the `less -S` problem is bunk, because you can easily change it at
runtime. From the OPTIONS section of `man less`:

 _Most options may be changed while less is running, via the "-" command._

So, just start `journalctl -l` as you would, then type in `-S` and scroll the
screen in any direction to force it to re-draw.

Discussing it any further is bikeshedding, and distracts from the much more
interesting segfault discussion.

~~~
deathcakes
To which I would say that you are completely missing the point of the
complaint levelled here - that you can work around a stupid design decision
does not negate the stupidity of that design decision. Personally I think the
default of show all logs ever is worse but both mean that what should be an
integral and useful tool is initially off putting at best and constantly
irritating at worst. I say this as an arch linux user for the past three
years.

------
Mister_Snuggles
It's interesting to see examples of where systemd falls down, though I'd be
interested to hear what the actual problem is. The Fedora bug report and
linked page basically say "systemd fell over during an upgrade", but is this
really a systemd problem or is it a Fedora problem?

~~~
siebenmann
Almost any time PID 1 segfaults, it's PID 1's fault. With PID 1 being systemd,
that makes it systemd's fault here. Based on gdb stack backtraces in the
Fedora bug report and looking at the systemd code, it seems like some sort of
memory corruption or overwrite, perhaps a use after free issue. The segfault
itself comes from dereferencing a clearly invalid pointer and said pointer was
obtained by dereferencing a structure field through another pointer, so you'd
get exactly this result if the structure was overwritten with other data at
some point.

(In my grumpy sysadmin view, it is PID 1's fault even if the distribution is
doing odd things around PID 1. Init processes need to be absolutely rock solid
and extremely defensively coded, precisely because the world basically dies if
they ever fall over.)

(I am the author of the original post.)

~~~
digi_owl
What i find interesting is that it is the traditional sysadmins that find
fault after fault with systemd.

On the other hand those that embrace it come from web/cloud development, with
an eye towards virtualization and containerization.

I think someone somewhere once compared it to pets vs cattle.

Traditional servers are the pets of sysadmins. Groomed and cared for to make
sure they never keel over unexpectedly.

To the cloud "admins" (or maybe i should call them devops?) servers are
cattle. they have X of them living in some cloud "farm" somewhere, and can
order any number of them to be slaughtered and replace at the drop of a pin.

------
raverbashing
" Oh, the full message is available, all right, but journalctl specifically
and deliberately invokes the pager in a mode where you have to scroll sideways
to see long lines. Under no circumstance is all of a long line visible on
screen at once so that you may, for example, copy it into a bug report."

Well, that's what you get by supporting a system built by obnoxious
incompetents that think they can replace the most important process in the
machine

"(Oh, and journalctl goes out of its way to set up this behavior. Not by
passing command line arguments to less, because that would be too obvious (you
might spot it in a ps listing, for example); instead it mangles $LESS to
effectively add the '-S' option, among other things.)"

Yes, of course.

------
angersock
Man, anybody else remember being able to use tail and grep to decode system
logs?

 _And not having the init process crash_?

Seriously, this is clownshoes. Their developers and evangelists should get hit
by a SIGBUS.

------
upofadown
I felt dirty, but this made me laugh out loud:

>While I'm here, let me mention that journalctl's default behavior of 'show
all messages since the beginning of time in forward chronological order' is
about the most useless default I can imagine.

~~~
cesarb
Yeah, that's an annoying default. At least it also uses a pager by default, so
it stops after the first screenful.

On the other hand, "journalctl -b" (show all messages since the machine has
been last powered on, in forward chronological order) is quite useful.

~~~
the_why_of_y
Yep... even better is that it takes an index, so you can "journalctl --list-
boots" to look up boot by timestamp and "journalctl -b -2" to look at the log
of the second-to-last boot to copy and report the kernel oops in there.

------
roller
With apologies for drifting off topic:

I love less's command line interface for some of this stuff.

The command line switch to chop long lines: -S The in-app command to toggle
chop lines: -S

The reverse is almost true for things like /search, F (follow) or G (go to
end). They can be used from the command line with +/search, +F or +G

------
userbinator
Looks like there is another bug open already with a similar backtrace, also
triggered by an upgrade:

[https://bugzilla.redhat.com/show_bug.cgi?id=1130633](https://bugzilla.redhat.com/show_bug.cgi?id=1130633)

------
jdlyga
All I've been seeing lately are posts very critical of systemd. I really don't
know much about it, and I would love to hear the other side of the story. Why
is it so popular lately?

