
First Impressions of Systemd, and They Are Not Good - alrs
http://changelog.complete.org/archives/9237-first-impressions-of-systemd
======
LeafStorm
It seems to me that the author's system was in an incredibly unstable state to
begin with. He's running a Debian prerelease with an unsupported filesystem,
that had gone without updates for months at a time. I have a hard time seeing
how changing the init system would _not_ cause problems in a setup like that.

~~~
javert
I don't understand why people would choose a non rolling-release distro (e.g.
Debian) over a rolling release distro (e.g. Arch). Because the former come
with the implicit guarantee that every few months, you have to update
EVERYTHING all at once. Seems like a needless risk and pain.

Genuinely curious here. There must be an upside to this model, can someone
tell me what it is?

~~~
thaumaturgy
I switched from a rolling release distribution (Chakra) to Debian because I
got tired of the rolling release system constantly breaking things. If you
fall too far behind on updates in Chakra, the maintainers basically tell you
you're screwed. I finally complained about it in their forums and was politely
told to go away.

My computer is just a tool. I just want it to work so I can get things done.
Debian has been good about that in the past and enjoys the support of a very
large, dedicated, active developer community.

Maybe Arch is better than Chakra at managing their updates (although Chakra
was basically a modified Arch distribution), but my experience with Chakra
left a really bad taste in my mouth for the rolling release approach.

FWIW, Debian's "update EVERYTHING" every few months never broke things as
badly for me as Chakra's frequent smaller updates did.

~~~
keithpeter
_" My computer is just a tool. I just want it to work so I can get things
done."_

Then Debian _stable_ or even CentOS (10 years support) is definitely what you
want, you are almost the definition of an Enterprise Linux user/adopter.

How in your opinion could we signpost that idea to people who are new to
Linux?

~~~
thaumaturgy
Yeah, Debian stable is basically what I'm on now. (Sort of. It's complicated.)

If I can be unabashedly honest for a moment, I think the problem with Linux
advocacy isn't the software any more, it's the advocates. The advocacy seems
to be coming from a lot of people that are always after the latest, newest,
greatest thing, people who are actually uncomfortable when features aren't
changing all the time. They tend to be very vocal about recommending anything
that comes with frequent updates, whether it's browsers or Linux
distributions.

But that's not what the larger untapped home market wants. They just want
things that work, and they want things not to change on them all the time.
They only have enough space in their daily lives to learn new features every
once in a while; any more than that, and they get frustrated. I hear this from
our customers all the time, and as I get busier, I'm really starting to
understand their point of view better. (As an example, we've recently been
seeing more of our customers switching _back_ to Internet Explorer on Windows
8 systems.)

So as far as signposting goes, I think boring, stable, slow-moving
distributions like Debian stable should be the default recommendation, and
then Arch or similar recommended as the really cool cutting edge thing for
hobbyists and early adopters. That is, Linux by default should be marketed as
stable and safe and not changing all the time, with the option of making it a
bit more fun.

We _almost_ opted to recommend Linux to a bunch of our customers who were
moving off of Windows XP this Summer, but the feedback wasn't great on a
couple of experiments and finally we decided we were too small to really
afford the support costs. It was a big missed opportunity though and I'm going
to take another look at doing it next year.

~~~
keithpeter
Very interesting.

So perhaps some demos using an Ubuntu LTS usb stick (xubuntu or stock Unity)
might be best. 5 years, pretty stable.

I find Stella Linux 6 (CentOS remix with multimedia) has had some acceptance
with a friend, but the NuX repositories are 'one man and a git account'.

------
nisa
There is a new blogpost from the author that explains the problems:
[http://changelog.complete.org/archives/9241-update-on-the-
sy...](http://changelog.complete.org/archives/9241-update-on-the-systemd-
issue)

TL;DR: systemd breaks badly (at the moment) when having to deal with non-
standard setups (ZFS, cryptsetup)...

As for the eternal systemd debate... if someone as knowledgeable as the author
(his publications:
[http://www.complete.org/Publications](http://www.complete.org/Publications))
can't debug it... we'll have fun times ahead.

~~~
pilif
_> TL;DR: systemd breaks badly (at the moment) when having to deal with non-
standard setups (ZFS, cryptsetup)..._

So you're saying that upgrading a custom tweaked configuration grown over the
years to be just perfect for that user is going to break when converted to
non-customized stock systemd packages?

The configuration in question has grown over the months and likely used
multiple iterations to get right. You can't expect a stock systemd package to
be a perfect drop in replacement.

What we really need in order to form an opinion of how badly or how well
systemd is going to work, we need real-world examples of systems being
upgraded into which only minimal tweaking of the init system has gone.

Like all the Arch and Fedora users who are not really complaining about
systemd making their systems unstable or causing issues with day to day usage.

~~~
nisa
No. I think the linked article is pretty clear where the problems are. It's
basically assumptions made by systemd that are not always true and from the
looks of it a possible bug regarding the ordering of service files and
reaction to events. Besides that broken or unusable console output without
plymouth (I can't really tell from the article if systemd is responsible for
that).

That's all fine and possible expected.. my only concern is just that this
appears to be very difficult to debug. The author is as knowledgeable as you
can hope for in a Linux user and is unable to get meaningful debug output.

All of these things are IMHO legitimate concerns that should be addressed by
the systemd community or the Debian systemd packagers.

~~~
digi_owl
>It's basically assumptions made by systemd that are not always true

Surprise surprise. Assume nothing, test everything is a old mantra in various
engineering circles iirc.

------
angersock
_" Have we made a vast mistake that can’t be undone? (If things like even _
brightness keys * now require systemd…)"*

Truly, _this_ will be the year of Linux on the desktop--because it probably
won't be on the server much longer, right?

------
nemo
"Linux can turn any song into death metal because all you'll be hearing is
your own screaming." \- Infosec Taylor Swift

my other favorite of her's:

"Mommy, what's Linux?" "That's what systemd used to be called, Becky."

------
hapless
Every time I touch systemd, I find another bug. I never wanted to break out
GDB to manage my freaking init.d scripts.

 __* Error in `systemctl ': double free or corruption (fasttop):
0x00007f26ef8089c0 ___

~~~
nisa
You are not alone.. after playing around for some hours with CentOS7/RHEL7:
[https://i.imgur.com/sCiqS2O.png](https://i.imgur.com/sCiqS2O.png) \- The bug
is fixed now (nmtui: changing the hostname crashed all of NetworkManager...)
now I'm always a little scared when using a freedesktop.org project ;)

~~~
anonbanker
Look for the names Lennart Poettering, Kay Seivers, and Ulrich Drepper amongst
the fd.o project's developers. If you see them, avoid the project. As 90
percent of freedesktop.org projects do not include these developers, you're
probably safe, but always good to be sure. The majority of developers on fd.o
will happily accept and action bug reports, so don't let these bad apples
spoil you on the rest of the bunch.

~~~
nisa
I'm tired of that name calling. These are just bugs. All these projects fix
bugs and honestly I'm personally glad for the work they've done. I don't like
their attitude or some design choices but I also lack knowledge. I wouldn't be
able to write these software and my complain is mainly a lack of thorough
testing. Using C for all these things makes debugging not exactly easier.

~~~
anonbanker
Just curious, but what names did I call them? Is using their names not
allowed? I recognize your nick from systemd's forums and IRC channel. Nice to
see you here defending your favorite project.

------
VLM
"But I can’t stay at some early jessie checkpoint forever."

We're all moving to freebsd.

~~~
angersock
What would make you pick FreeBSD over, say, Illumos?

~~~
VLM
More eyes and hands means better faster security patches and similar issues.
That's why I don't use "three guys and a github" specialized linux distros. If
you follow a similar outlook (and what works for me might not work for you)
then a crude google search result indicates the community for freebsd is about
30 times more talkative (30 times bigger?) and some more wikipedia searches
reveal freebsd is five times the age WRT stability.

I have never heard of Illumos which also implies I've never heard anything bad
about it. Although I do personally know people who swear by freebsd, so ...

I would probably use a *bsd to avoid all possibility of systemd issues, and

[http://en.wikipedia.org/wiki/Comparison_of_BSD_operating_sys...](http://en.wikipedia.org/wiki/Comparison_of_BSD_operating_systems)

implies the big "thing" for freebsd is "usable for any purpose" which is the
old Debian "universal OS" concept I like so much.

I would strongly consider openbsd for something bare out on the internet not
behind a stateful firewall and my limited experience with netbsd is its pretty
awesome but being able to boot on a PDP11 necessitates some excessive old
code. I'm astounded by retroBSD on PIC32 arch.

~~~
bingaling
> I have never heard of Illumos which also implies I've never heard anything
> bad about it.

It's the open fork of OpenSolaris after the Oracle takeover of Sun. Many ex-
Sun engineers work at companies that use/contribute to Illumos.

------
guylhem
Long story short - you did it the wrong way.

systemd on debian tries to reuse far too many /etc/init.d scripts, whose
quality and performance I find questionable.

Follow the guide on
[http://www.holgerschurig.de/linux/systemd.html](http://www.holgerschurig.de/linux/systemd.html)
to get a good experience. You can also use my scripts (cf
[http://libreboot.org/docs/future/#fastboot](http://libreboot.org/docs/future/#fastboot))
to compile your systemd

For the specific zfs problem, I can't say for sure (I use xfs), but a simple
script you wrote yourself is usually the best. I also had fstab related errors
for swap (I use luks) until I disabled the automatic stuff and wrote a
replacement script.

systemd is well done and such fixes don't take long, once you understand the
logic of it. (usually a symlink to /dev/null to disable the script and a few
lines of ascii text for the replacement)

My results: on a thinkpad x60 (2006 era laptop), after coreboot, I boot in
about 3 seconds, cf
[http://www.coreboot.org/pipermail/coreboot/2014-July/078215....](http://www.coreboot.org/pipermail/coreboot/2014-July/078215.html)
(link to a youtube video of another person doing the same thing). Suspend
works very well out of the box, and even better (ex: renewing dhcp leases)
after adding a few scripts.

systemd makes many things possible and easy, but with debian at least, the
out-of-the-box experience has to be improved. Nothing impossible if you have
say a day to spare to learn how systemd works and write your fixes.

~~~
nknighthb
Time and again proponents of systemd belittled those of us with compatibility
concerns and assured us of its compatibility[0] with existing scripts and
setups. Now that it's been adopted, using that supposed compatibility is
"doing it wrong". Imagine my surprise.

[0]
[http://www.freedesktop.org/wiki/Software/systemd/](http://www.freedesktop.org/wiki/Software/systemd/)
("compatible with SysV and LSB init scripts", "It can work as a drop-in
replacement for sysvinit.")

~~~
CoffeeDregs

        Now that it's been adopted, using that supposed
        compatibility is "doing it wrong".
    

It has only been adopted in non-Stable distributions of Debian... I haven't
seen anyone belittle anyone over "compatibility concerns"; I have seen folks
belittled (especially in this thread) when they complain about _imperfect
stability_ in distributions which are _not Stable_.

At the risk of appealing to authority, I'd be interest to see posts from
Debian/Ubuntu maintainers in which they express frustration at the various
problems systemd is causing them. I haven't seen any and would appreciate
hearing from them or getting some links to their posts.

EDIT: clarified area of "belittlement"...

~~~
nknighthb
You said elsewhere:

> _I run Debian Testing and was able to upgrade ~5 machines without a hitch._

So which is it? Is systemd in Debian hitch-free or not? Or do you only care
about your own systems and no one else's?

> _I have seen folks belittled_

Meaning you've already lost the argument.

> _I 'd be interest to see posts from Debian/Ubuntu maintainers_

Yeah. Screw users, right?

systemd, land of ever-shifting rhetoric. You guys should get out of the
software development business and run for Congress.

~~~
bkor
It seems you're rather upset with people who like systemd. That's fine.
However, you're putting a lot of words into my mouth. I like systemd.

According to you because I like systemd: \- my opinion is "screw users" \-
belittled you with compatibility concerns \- arguments are nothing more than
rhetoric.

I find such discussion style rather poor. Instead of diving people into
pro/con or black/white (with us or against us), try expressing things a bit
more granular. Further, it seems you're out to win arguments by selectively
quoting and changing the meaning of what you've quoted.

If there are bugs in systemd, they have to be fixed. I'm one of the biggest
systemd advocates. This combined proves you are incorrect on what people
liking systemd say or suggest.

~~~
Karunamon
_If there are bugs in systemd, they have to be fixed._

Philosophically, yes. Realistically? The behavior of some of the people on the
systemd bug tracker[1] is the single most convincing argument against adopting
it.

It might break things? Sure, it's a pretty massive change.

People are going to have to re-learn how to do a lot? Again, it's a pretty
fundamental shift in a lot of ways. This is expected.

The developers will fucking backtalk you[2] and make political issues out of
clearcut things? Uh.. no. That's forkworthy behavior. That's dysfunctional
enterprise behavior, which is something I'd imagine the average hacker hates
instinctually.

And sure, in the end, those bugs were addressed, but it took an outburst from
Linus and widespread, public coverage (and I'm sure no small amount of offline
discussion among the systemd folks) for it to happen.

Poettering is on record as wondering why systemd attracts so much hate. He
only need look in the mirror.

[1]:
[https://bugs.freedesktop.org/show_bug.cgi?id=76935](https://bugs.freedesktop.org/show_bug.cgi?id=76935)

[2]: [http://www.muktware.com/2014/04/linus-torvalds-happy-
systemd...](http://www.muktware.com/2014/04/linus-torvalds-happy-systemd-
author-kay-sievers/25151)

~~~
bkor
That bugreport is often misunderstood. You only gave one example, and that
example is flawed. The bugreport was to change systemd behaviour to avoid a
kernel bug. The bug should be fixed at kernel, that is why the notabug was
there. Further, it is ack'ed that this should've been handled much better.

I have seen loads and loads of examples where "no" is said in some email
conversation based on technical arguments.

> And sure, in the end, those bugs were addressed,

The bug was fixed in systemd git before Linus got involved.

> Poettering is on record as wondering why systemd attracts so much hate. He
> only need look in the mirror.

Here you're being an ass. This bugreport was not handled by Lennart. Lennart
is also not talking about disliking systemd, he's talking about people taking
things way too far towards him. He talked about people raising money to hire
an assassin.

~~~
Karunamon
The bugreport was because systemd misused the kernel command line. If your
code causes something else to break, _it is your code 's fault!_

~~~
not_with_retard
It was a bug in the kernel, as Torvalds has already admitted.

"Something else" shouldn't break in the first place.

It's obvious you're not a programmer.

~~~
nknighthb
> _It was a bug in the kernel, as Torvalds has already admitted._

Where?

> _It 's obvious you're not a programmer._

I certainly am, and he's right. Stop trying to deflect blame. The kernel isn't
responsible for preventing init from doing stupid things, init shouldn't do
stupid things.

And most importantly, when your code does stupid things that breaks users, it
is your responsibility to fix it, _no matter what you think anybody else 's
code should be doing_. If you can't accept that, you're a menace, and need to
stop writing code.

~~~
not_with_retard
> Where?

From G+

"I guess this does mean that we have to apply my patch to rate-limit messages
into the kernel."

......

"+Greg Kroah-Hartman: I really hoped that you could push the trivial and
obvious one-liner through. What’s going on here?"

From LKML

"Steven, Borislav, one thing that strikes me might be a good idea is to limit
the amount of non-kernel noise in dmesg. We already have the concept of rate-
limiting various spammy internal kernel messages for when device drivers
misbehave etc. Maybe we can just add rate-limiting to the interfaces that add
messages to the kernel buffers, and work around this problem that way instead
while waiting for Gregs fix to percolate?"

He even goes so far as to say:

"I don't think we should try to protect against willful bad behavior unless
that is shown to be necessary. Yeah, if it turns out that systemd really does
that just to mess with us, we'd need to extend it, but in the absence of proof
to the contrary, maybe this simple attached patch works?"

Showing that even if systemd tried to get around the rate-limiting just to
piss off the kernel developers, that it's a bug they would fix (which is the
wrong stance for whatever it's worth; you should be using defensive
programming, not waiting for someone to abuse it first, but at the very least
shows that it's a kernel issue).

> I certainly am

Certainly not a very good one.

When a library has an error, is it your fault for triggering that problem, or
do you log a bug with the library?

This isn't rocket science.

> And most importantly, when your code does stupid things that breaks users,
> it is your responsibility to fix it, no matter what you think anybody else's
> code should be doing. If you can't accept that, you're a menace, and need to
> stop writing code.

Greg Kroah-Hartman himself (on G+) said "the original bug that was causing the
syslog spew to stop the box has been fixed in systemd.".

Please don't contribute to a conversation if you know absolutely fuck-all
about it. Spreading misinformation is one of the world's worst problems, and
you're simply contributing to it.

~~~
nknighthb
No mention of a bug in the kernel anywhere. I see systemd proponents just
can't stop lying. Please don't contribute to a conversation when you can't be
at all truthful.

------
CoffeeDregs
His follow-up is at: [http://changelog.complete.org/archives/9241-update-on-
the-sy...](http://changelog.complete.org/archives/9241-update-on-the-systemd-
issue)

------
vezzy-fnord
I'm not sure to what extent this is systemd, the author's setup, or the
integration work on part of Debian.

To be honest, I wouldn't be surprised if it's the latter. It turns out that a
lot of Linux distros really don't have very adequate systemd bootstraps at
this point, which you'll find the systemd developers admitting (including, and
in my personal experience, especially for Fedora):
[http://freedesktop.org/wiki/Software/systemd/Optimizations/](http://freedesktop.org/wiki/Software/systemd/Optimizations/)

When you factor in the fact that Debian and other distributions have some
absolutely strange packaging policies concerning when and to where to include
systemd as a dependency, coupled with the fact that most distros still don't
have good mechanisms for separating hard and soft dependencies, and you get
some pretty confused default configurations.

I once tried compiling and booting from systemd on a Lubuntu box, out of my
own curiosity. It worked, and there were already some predefined unit files
from systemd-shim and various system packages. Performance was good. I tried
upgrading my system... the process is absolutely broken, due to
preinst/postinst scripts assuming Upstart. Sometimes a package will get
successfully unpacked, sometimes it won't. Error messages galore.

Considering systemd's extensively rapid evolution, I don't see integration
work in most distros stabilizing or being optimized very soon.

~~~
hamburglar
I have been half-following this whole saga and as far as I can tell, it's a
shit-show on a scale that runs the risk of turning entire distro major
releases into complete write-offs. I just hope they only go one release cycle
before deciding to abandon it. I'll get on the last pre-systemd LTS and hold
my breath.

They don't see the benefit of keeping critical system components simple, they
don't listen to criticism, they make promises they can't keep, and they've
bitten off way more than they're capable of chewing. The only part I really
can't figure out is why major distros have actually decided to make the switch
way before any of the tech is ready.

~~~
makomk
One of the things they're planning next is to replace the current kernel VT
implementation with a userland terminal emulator that depends on systemd, so
you won't even be able to boot with init=/bin/bash in a pinch because without
systemd there won't be any terminal to run it on anymore. No idea how people
will be expected to diagnose and fix issues that break boot at that point.

~~~
mjevans
Pick a 'usb stick' distro of choice (my choice is usually sysrescuecd) and
boot that image off of media or virtual media if you have one of those
remotely managed systems. Though it sure would be nice if said systems could
pull the image off of a mirror local to the DC.

Alternatively, build a similar rescue option in to your boot menu; if that's
broken you've got no choice anyway.

~~~
pdkl95
USB rescue distros are nice, but that assumes you have a USB stick to use and
already had the rescue distro installed. That doesn't help when you are trying
to fix some failed update late at night with the computer in question being
the only hardware available. Contrived example? Yes, but I've been in more or
less that situation a couple times in the past and was _very_ thankful that
init=/bin/bash worked as a fallback.

// yes, most of the time I just put fallback/rescue options in the boot menu

------
perlpimp
systemd intention is great but aftermath is that of talented albeit
inexperieced developer writing it. Fact that i have misconfigured one of the
services, should not lock me out of machine to the point of needed to boot
from usb flash stick. I am not as advanced but that was what I got when put
coreOS with systemd on one of our servers - thankfully not production.

~~~
XorNot
Which service was it?

~~~
perlpimp
I think it was hello world service but after tooling around starting and
killing it , rebooted my machine and it machine froze solidly on reboot said
something like unable to find mountable systems. but the systems were there
with flash drive everything was intact. I've just picked up coreos and I think
stuff like while(1); echo "hello world"; sleep 1; maybe it was not that or
something else but setup was stock stable 431.

------
jauer
I don't use Linux on the desktop so my experience with systemd is strictly
server-side but it just feels like too much magic. I'm used to (and like)
Solaris/OI's SMF so I'm certantly not in the "init scripts for life" camp.

------
systemdowntime
Just saw the all-hands call on the IRC channels and systemd fanbois groups for
an epic show of whiteknighting required yet again on HN where rational folks
are trying to discuss the actual and real downsides of systemd - quick, post
the rebuttals, choose from:

1\. You're doing it wrong 2\. Of course, if you use <insert something here>,
then it won't work! 3\. If you spend a month, it is simple! What, are you
lazy? 4\. If you hack these bits, disable these bits, run some pointless
commands here, then presto it works!

~~~
vacri
Making a throwaway account to post a snide comment doesn't do anything to
raise the quality of conversation.

