
Systemd is not portable and what this means for our ports - tshepang
http://people.debian.org/~stapelberg//2013/07/13/systemd-not-portable.html
======
peterwwillis
If there ever was a "red herring" in a discussion of Linux distributions, it's
the init system.

The distro maintainers package the software for their system. It's therefore
also their responsibility to make sure a service can be started/stopped and
have its status checked.

It _does not matter_ what kind of init system a distro has. A single Linux
distribution is an Operating System independent of all others. There is no one
true portable Linux way, much less for non-Linux-based Operating Systems. You
have to treat them all as separate because to do otherwise would not only add
complexity to a monolithic "portability" system, it would eventually get so
complex you'd have to split it up anyway.

Portable software makes 3rd-party software developers' lives easier. But the
init system is not managed by 3rd-party developers. It is managed by the OS
maintainers. So it does not matter what method they use.

Furthermore, it's fucking Debian, the second-oldest distribution still in
release. The oldest (Slackware) holds onto the old ways for as long as
technically possible, and they're still going strong. So why the hell they
care about being "modern" and "remaining relevant" is anyone's guess.

(disclaimer: I wrote my own init system for my own Linux distribution, and
have modified most others)

~~~
makomk
All that's true of init systems that follow the traditional Unix approach,
where each part of the system has one task to do and does it well. The point
of systemd is to break from that approach. So systemd isn't just an init
system, it also creates device nodes, handles system-wide logging, manages
both desktop and console login sessions, replaces inetd (but with support for
Unix sockets and DBus), provides resource management functionality for
virtualised containers, and a bunch of other stuff I've forgotten. Making full
use of these features requires changes to a large number of other
applications.

~~~
peterwwillis
Holy shit, that sounds horrible. As an application developer who already knows
how to make programs work for Linux, and a distribution maintainer who can
take care of all that crap without a new framework, and a systems
administrator who can edit it all on the fly using just shell scripting, I
struggle to understand why this system is necessary at all, or why anyone
would push for its adoption.

Oh, wait. I found it. From
[http://0pointer.de/blog/projects/why.html](http://0pointer.de/blog/projects/why.html)
:

 _" Specialized professional consulting and engineering services available:
yes"_

~~~
zanny
You don't actually need to use all the different parts of systemd, though.
They are a suite of different binaries. You could _just_ use the /sbin/init
replacement at /usr/bin/systemd, under consideration it _will_ require a
rewrite of your services, etc. But logind and journald are optional
dependencies, you could redirect journald output to a syslog.

The purpose of a lot of systemd was to say "there are these old utilities
based on shell script hacks that would be better served in a Linux-specific
framework" like power management, device plugging handling, etc. So they put
them under one project. Same way under the Linux kernel is hundreds of megs of
device drivers, and a lot of features end up in kernel space (rendering
interfaces, networking stacks, filesystems, virtualization) yet nobody calls
the kernel anti-unix (some might call it bloated, though).

~~~
peterwwillis
Not to get too side-tracked, but Unix-like kernels aren't very Unix-y. They're
big monolithic beasts that are difficult to configure and impossible to modify
on-the-fly, save for the few modules that can be unloaded, recompiled, re-
inserted and configured. In general it was thought of as bad design, and still
is by many "experts".

I'm not sure why our system management tools/frameworks have historically been
interpreted scripts. It could be due to the low barrier to entry. Or the
dynamic way it can be modified - even on embedded systems - without any tools
other than a text editor. Perhaps relying on the infinite possible
combinations of simple tools just gave us more power than we could ever get
from trying to make libraries for every possible need. Or maybe it was just
way simpler to manage.

I'm not anti-systemd, mainly because I haven't used it. But any time someone
says "let's replace the old system wholesale" without really REALLY good
reasons, I suspect it's premature. It seems really useful for tightly
integrated embedded systems, but hell for general admin use.

~~~
zanny
I used Debian for a year before moving to Ubuntu and eventually Arch. I've
tried all 3 flavors of init, and I vastly prefer systemd :

1\. It is noticeably faster. 2\. It is braindead easy and intuitive to write
service files, and you can write service files or a surrogate for almost any
system function from hot swap behavior to load balancing. 3\. The entire
systemd execution model is fs based, and you have a large library of services
to run at your discretion. 4\. powerctl is supremely more robust in its
usability than pm-utils and shell scripts around suspend / shutdown states.

Only thing that really bugs me is the binary log. I want to pipe it and
manipulate it and read it outside a terminal in say, kate, but I have to
install syslog on top to do that.

~~~
peterwwillis
Okay, well to play devil's advocate:

1\. Legacy init systems were never tuned for performance. Just look at 'grep
sleep -r /etc/init* /etc/rc*' as an example.

2\. Easier is nice, but you could write scripts for all those system things
using hotplug, udev, and a variety of other things (remember devfs?) I never
bothered to learn because they just worked.

3\. I'm not sure what you mean here; what execution model is not fs-based in
some way? If you mean scripts would execute other scripts based on a
filesystem hirearchy, yes, all the old systems did that too.

4\. That's nice, but it's kind of irrelevant to how your system
initialization/service management works. Different purposes and all.

And yeah, binary logs are dumb without a tool to manage them, but relying on
syslog is smart. It's an industry standard. It's the only industry standard
for collecting and managing logs from every kind of device, other than maybe
snmp (yuck).

------
jmillikin

      > This blog post is the third of a series of posts
      > dealing with the results of the Debian systemd survey
    

Previous posts in the series

[http://people.debian.org/~stapelberg//2013/06/09/systemd-
blo...](http://people.debian.org/~stapelberg//2013/06/09/systemd-bloat.html)

[http://people.debian.org/~stapelberg//2013/07/01/systemd-
tra...](http://people.debian.org/~stapelberg//2013/07/01/systemd-
transition.html)

------
sherr
I sympathise with the fact that Debian considers itself the "universal
operating system" but there are limits and the Debian developers have to
consider the impact of their support decisions on their major consumer: the
Linux OS. The Linux OS is the first and primary OS they should support. And as
Michael says, to stay relevant, Debian needs to meet the needs of a modern and
dynamic OS. That's why I want to see systemd supported and considered first-
class in Debian, my OS of choice.

~~~
ealexhudson
Exactly the same here. I like the idea that startup semantics will be much
more unified across a range of distributions and that more sysadmin knowledge
will be common - particularly in terms of writing good 'init scripts' for
custom services.

------
_delirium
Especially considering that Debian/kFreeBSD and Debian/HURD are closer to
interesting experiments than systems used for real work, holding back the init
system solely for the non-Linux Debian variants doesn't seem justified. So
this roadmap seems pretty sane.

~~~
asveikau
But what exactly is "holding back"?

I feel like Linux distros today come with too many crappy, overthought, over-
engineered daemons that don't really offer much over the previously available
interfaces. I guess parallelizing init scripts or making dependencies more
explicit is a good thing in some sense, but I have to say, it's a breath of
fresh air to see something like a *BSD system, which has an /etc/rc shell
script that is easy to follow. Unlike Linux which frequently seeks out complex
solutions from simple problem spaces.

~~~
Jasper_
Process and service management is not a simple problem space, especially with
POSIX semantics.

Try doing something like "stop apache". It turns out that trying to simply
kill the Apache processes isn't going to work right, as Apache forks out lots
of worker processes that won't be guaranteed to be killed when the Apache
process stops. And now since the parent has gone away, they're reparented to
PID1, so there's all these worker processes littering your system.

systemd uses cgroups to tag every single forked out process with a service
file, so that all of the processes associated with a service can be stopped at
once.

~~~
regularfry
Right, but is systemd necessary for that, or can you use a cgroup from within
a conventional sysvinit script?

~~~
bkor
systemd also logs stderr + stdout, so if some daemon fails to start up, you
can easily see it in 'service $SERVICE status'.

A lot of things could theoretically be done by putting that all in a sysvinit
script. That means duplicating that in every sysvinit script. Bugs galore :P

~~~
bashinator
Oooh, if I'm understanding right, if a daemon fails to start correctly for
whatever reason, that status request will give me the stdout/stderr of the
failure? That's a genuine improvement if systemd handles that consistently for
all services.

~~~
Jasper_
That's part of what the systemd journal provides, yes.

------
mcclung
It's getting harder and harder to find a distro where systemd isn't a hard
requirement. I am hoping Debian will keep the choice of sysvinit available.

~~~
crististm
All of a sudden everybody and his dog requires systemd or upstart or whatever
flavor of rebranded and upgraded and extended and whatnot initd of the day.

I failed to see a reason behind this movement except of the "you're an old
fart and this is how we're doing it today" explanation. And when BSD comes up
and say this is not portable, everybody say we're on Linux and we don't care
and you should jump the shark as we do. What?!

~~~
sanderjd
I think the newer alternatives are more standard, easier to understand, easier
to write for my own services, and less error prone than sysvinit scripts. You
may disagree with all of those points, but it really isn't a "you're an old
fart and this is how we're doing it today" explanation. By all means, argue
against the specific reasoning, but claiming there is none and it is just
progress for its own sake is spurious.

~~~
crististm
> argue against the specific reasoning

What's with the /usr that should be mounted at start? What was wrong with /?
And why should initd depend on the kernel?

~~~
VLM
I may be misreading you, but I think you're trying to ask why the survey was
making technical decision justifications based on purely subjective opinions
rather than a survey based on objective facts, balancing the cost to humanity
of the few remaining /usr splitters having to reconfigure, vs the slightly
increased effort of the very small number of people who write initscripts
having to continue to use the old system plus or minus the increased effort by
the existing stock of programmers having to learn systemd which the other
distros having switched has been forced on everyone anyway. I think both sides
of the balance have pretty small numbers.

A la the famous Landley rant from about 4 years ago, I had a NFS mounted
/usr... back in 97 as an experiment which I rapidly terminated. Shared and RO
/usr is an interesting hack, but probably not worth holding init development
back, especially since so many other boot time things demand /usr anyway now
(like the pulseaudio, like the networkmanager thing)

If you were aiming more at why systemd needs different stuff, you can google
for Landley's email around 2010 on the topic and on the other side google for
read only /usr and NFS /usr. Also google for cgroups, especially systemd and
cgroups.

~~~
crististm
I'm saying that just because a bunch of people say they have a new shine
thing, I should not be _forced_ to use it. They are free to do whatever they
want as long as they don't force me on their bandwagon.

BSD should not have to implement Linux semantics just to start the userland.

~~~
Shish2k
You will never be forced to use systemd. You may have to stick with running
debian 7 forever and backport your own bugfixes though :P

Unless you're suggesting that developers forcing users to upgrade is bad;
users forcing developers to continue providing support for deprecated software
forever is good?

> BSD should not have to implement Linux semantics just to start the userland.

If neither BSD nor Linux supported files, and then Linux added support for
files, would you suggest that we refrain from making file-based software, and
continue giving all apps raw disk access for sake of compatibility?

Using cgroups for service management is that kind of fundamental good idea -
the BSDs don't have to copy the linux API, but they should really provide
something similar

------
drill_sarge
I could not care less about systemd because sysvinit is working fine here. I
hate that you are forced in most distros to choose systemd without any
alternative. I want my own initscripts, if a distro doesn't allow me that I
won't use that distro.

~~~
cnvogel
Using systemd still allows you to have custom scripts run on boot up. Given a
shell script you want to have running on boot you'd have linked that script to
/etc/rc3.d on sysvinit. With systemd the proper way to activate it is a
service file in /etc/systemd/… or /lib/systemd/… .

It's different but I wouldn't call it more or less complicated than sysvinit.
There are a lot of example/skeleton files floating around, especially
archlinux's collection is useful.

Of course, systemd allows for much more configuration than simple running of
scripts on boot, but you can fine tune later.

------
bbanyc
Previously the non-portability of upstart was seen as a reason for Debian not
to switch to upstart.

Now the non-portability of systemd is seen as a reason for Debian to drop the
ports.

~~~
rquirk
Debian isn't going to drop the ports! The blog post seems a sane roadmap for
eventually switching to systemd on Linux and maintain a more traditional init
process on the BSD ports. The fact that scripts and systemd can work in tandem
on Linux makes this possible. Seems pretty reasonable to me.

The only downer is that now there'd be 2 init setups/configs to maintain. But
that would be the case even if systemd were not the default, just available as
a choice.

------
synchronise
I wonder why Debian didn't go with OpenRC
([https://en.wikipedia.org/wiki/OpenRC](https://en.wikipedia.org/wiki/OpenRC))
as it is compatible with both Linux and *BSD, which also offers 'Process
segregation through cgroups'.

~~~
bkor
Change happens by people doing work. There are at least 3 people actively
working on systemd support. Meaning: they help out other packagers, file bugs,
look at bugreports, coordinate with upstream, etc.

For OpenRC, there is just one person advocating it. There is a summer of code
thing, but seems not much work is done. Furthermore, OpenRC seems nice, but
parallel starting of services is actually experimental/buggy (just ask one of
the OpenRC maintainers!). So although you can have a similar boottime, one
init system (systemd) will be running in a supported configuration, the other
(OpenRC) in something that is known to have issues.

That all said, I don't think Debian really makes decisions. What is being
proposed is maintaining two.

Aside from above, there is also the practical bit in that systemd provides
something to replace the unmaintained and deprecated ConsoleKit with. This bit
can be used on another init system (Canonical is using or will use it on
Upstart), but at one point seems easier to rely on the existing systemd
supporters in Debian to get things working correctly.

~~~
synchronise
Surely the Debian Foundation can see that an init project that can support
more than one kernel/OS, pretty much as Debian does with Hurd and kFreeBSD,
would be a better platform to support through the long term.

------
Qantourisc
Wasn't looking forward to this "upgrade" ... The fact that it's from the same
dude as PulseAudio is enough to make me anxious, the pulseaudio stuff didn't
go smooth. (I'm still of ALSA, yes also is a mess to.)

So I hope I won't be forced, to a more crappy system !

The fact that it will be handling a lot of stuff more, sounds like a major
pita ...

~~~
pyre

      | the pulseaudio stuff didn't go smooth
    

That wasn't the fault of the developer so much as it was the fault of the
distributions for jumping on the bandwagon (making it the default audio
system) too early.

~~~
nieve
Poettering oversold its state of stability and then seems to have largely lost
interest during the most critical period of adoption. The distributions are
responsible for believing the hype and shipping too early, but that doesn't
change the fact that the developers were hyping it well beyond what was
justified. PulseAudio has a lot of nice features, but it's still less stable
and has more bizarre failure modes for me _9 years_ after its initial release
than plain old ALSA. I literally know no other linux users who have ever had
anything good to say about PulseAudio. Entirely anecdotal, but when it comes
to trusting developers to do a good job in the first place, much less help
with a major transition I'm not inclined to discount that.

~~~
pyre
After the initial transition period, I have not had any PulseAudio issues
other than on maybe 2 occasions where the pulseaudio daemon ran amok,
consuming 99% CPU for some reason, until it was killed. I'll admit that I'm
not running any exotic configurations though, but built-in audio works just
fine, and plugging my USB camera/microphone[1] has worked without issue for a
long time too.

At one point I was also able to stream audio from my HTPC to my laptop, so
that I could use my headphones when everyone else was sleeping. Though I'll
admit configuration was a pain in the ass, I don't recall being able to do
that with ALSA (though I recall someone creating a shell script a while back
that read from /dev/dsp and piped to a socket).

Prior to PulseAudio, sound on Linux was always an uphill battle for me (don't
get me started on regressions in the PowerBook audio driver that no one cared
about since "it works on the newer PowerMacs"). It was a crap-shoot which
audio system an application would use. OSS? Alsa? eSound? KDE's sound daemon?
Do they all use the same mixer app? What happens when I write to /dev/dsp?
Does all other audio block? Which applications support Jackd? etc...

[1] Referring to the mic in the pulseaudio context.

EDIT:

    
    
      > I literally know no other linux users who
      > have ever had anything good to say about
      > PulseAudio.
    

I'll mention that the same could be said of OSS or ALSA too. They may have
worked better for you, but I doubt that prior to PulseAudio, you were singing
their praises from the rooftops.

~~~
zanny
It's the difference between a sound server and having kernel audio device
APIs. Pulseaudio is an actual daemon running that is sinking audio and
managing it in realtime - ALSA (sans dmix, and piling other alsa plugins on
top of that) was just an API into the kernel audio interface.

Sound servers are a modern, I'd argue, necessary convenience for the average
joe. API wise, pulse is much nicer to use than alsa
(pcm_sink_thing_getbuffer_readhardware_processpcm() x50. I've debugged the sdl
implementations of alsa and pulse, and pulseaudio literally takes a tenth the
code to use).

~~~
nieve
Thanks, that's a definite win for pulseaudio and I hadn't really considered
the developer side of it at all (I'm primarily backend/network/db). I
certainly don't think pulseaudio is an inherently bad architecture, but I do
feel it's matured too slowly in terms of stability. On the flip side as you
it's more user-friendly than Network Audio Server (which has fewer features)
and and as you point out supports more of the "I've got an hdmi output playing
music through my tv, but a bluetooth headset for voip" use cases that a
decent-sized class of users run into.

It also occurs to me for the first time that the rise of pulseaudio was not
that many years after laptops largely switched to integrated codec chips
instead of discrete sound and I had all sorts of problems with those above and
beyond the "Damn it, give me hardware volume controls" frustrations. It could
have inherited some of the problems around quirky/flaky hardware that I
remember ALSA facing in that period. At this point I just want something that
can reliably play music or take audio input and for whatever reason pulseaudio
isn't it for me. Hopefully that'll change before most distros switch to
whatever the new hotness in sound setups is...

------
simonz05
I recently switched from Arch Linux to Debian mainly due to the hard systemd
requirement. I hope the Debian transition will be better than what Arch Linux
did.

~~~
mateuszf
Haven't heard about lot of users complaining about the way Arch developers did
it. In fact I think it works flawlessly.

~~~
alecthomas
It hosed my machine. The problem was I hadn't upgraded in a while, so when I
did upgrade some of the hacks they had in place to upgrade seamlessly no
longer worked due to library incompatibilities. Even the Arch IRC channel
couldn't help :(

~~~
mateuszf
I understand. Well, at least they did it early. It shouldn't happen once
again.

------
dschiptsov
systemd is, in some sense, a very modern-day product - a mess of different
"alien" concepts and functionality stuck together into one project with an
intention to show other the only true way of doing things. It like some
(freedesktop guys, perhaps) will come to Plan9 and would say that all they
need is gconf* -like registry to have a uniform configuration, or something of
the same level of absurdity.

The movement out of shell-based initscripts for non-specialized (non-
mobile/embedded) distributions is a cancer.) Remaining BSD guys are much more
sane and "conservative" in this respect.

There is absolutely no necessity to break what is good-enough and _well-
balanced_ \- having a standard shell scripts to do the job they were designed
to. FreeBSD's rc.d system is a very good example.

~~~
Jasper_
systemd has one purpose: to be about system and service management. It will
replace everything that deals with services: starting them (whether based on
traditional init, scheduling, socket activation, or from a desktop user
session), logging the stdout/stderr in a consistent and well-defined location,
and about giving you flexible tools to manage, debug, and inspect system
services.

Besides that, yes, it does have a few things (hostnamed, timedated), which are
mostly about some POSIX infrastructure that wasn't there already: notification
when the system hostname or timedate changes (yes, it happens quite often on
desktop systems, due to NTP, user setting, and VPN). They're just utilities
they needed along the way, and systemd can be built without them.

If the system or service fails to start, instead of hunting around in
/var/log/, syslog, dmesg, etc. for anything that looks relevant, you simply
use the journal which aggregates most everything. It's already a very good
breakthrough.

I've debugged countless bad Debian service scripts (which usually comes down
to bash's terrible whitespace and quote management), and inserted echo
statements into my Arch Linux to debug why it wouldn't boot when I upgraded,
hacking out clear statements here and there designed to make the boot pretty.
Shell scripts aren't a good solution for robust system and service management.

~~~
dschiptsov
We have no services in UNIX-like systems, we have daemons and it is perfectly
OK to use a templated shell script to start/stop restart it, modified
according to the nature of a daemon, as needed.

That other operating system from which the concept of a "service" came from
(along with lots of other crap) have different philosophy and founded on
different design decisions, so blindly trying to bring stuff from it here is
not always reasonable.

And, of course, shell scripts are OK when well-written - there are other UNIX-
like systems beside Linux.)

~~~
Jasper_
I don't know why you think "service" is a word that systemd got from Windows.
The name "service" actually came from the usage of the "service" script to run
initscripts, like "sudo service httpd start".

A systemd "service" is just a declarative description of a daemon that allows
systemd to start and stop it.

------
VLM
Is a cgroups emulator in the future for kFreeBSD and the HURD?

I'm curious if faking cgroups (or heck, porting it) would be an easy solution.

Not talking about a fake cgroups providing all the real world advantages of
systemd; merely a a fake not being too much worse than existing sysvinit.

Maybe rephrased actually porting THE cgroups to HURD might be impossible due
to design issues or license issues. OK well how about writing a stub cgroups
using the API that always gives a pleasant and completely meaningless response
to all calls. What, if anything, would be the overall system wide net negative
effect of systemd on HURD with a fake cgroups? I have not had much success
googling for that.

This would not fix the "my /usr is not on my / partition" problem which is a
serious concern to a extremely small number of people.

~~~
Jasper_
I don't know why you think cgroups has anything to do with the /usr split.

systemd uses cgroups to tag processes with metadata about services, and to
provide resource limits. If you added a stub cgroups, then resource limits
wouldn't apply, and you might run into issues with services not fully shutting
down as you do right now with the POSIX service management.

It's entirely possible, but you wouldn't get any of the upsides of systemd.

~~~
VLM
I don't have that belief. My interpretation of the opposition against systemd
revolves primarily around:

1) /usr split

2) Can't run on kfreebsd and hurd because they don't speak cgroups so either
"we" can't run systemd or need to drop ports or need to support two init
forever.

You can eliminate #1 with some sysadmin work. A stub cgroups compatible API
emulator type thing for kfreebsd and hurd would eliminate complaint #2 if the
overall result were not much worse than existing sysvinit.

One interesting solution is there is no particular reason to flick the power
switch on kfreebsd over and over as quickly as possible. So one incredibly
non-optimized way to "fake" cgroups process monitoring on kfreebsd would be to
replace actual cgroup process monitoring with something ridiculous like "sleep
30". In 30 seconds, anything will have started or stopped, correct?

Obviously a dev roadmap would be to replace/emulate/port real cgroup-like
support, once people get tired of all cgroups api calls being replaced with
"sleep 30".

I can't be the first one to come up with this "great" idea.

~~~
Jasper_
First of all, systemd does not depend on anything relating to how /usr is
mounted. Other components don't work properly with it; it simply detects the
case and sends a warning. systemd itself will continue to work fine, but
components that systemd launches will not. It's just a messenger.

I also want to note that cgroups is not the only Linux-specific interface
that's necessary for systemd. There's lots of them, and if you stub all of
them out, then you've achieved portability with none of the benefits, and have
potentially introduced a lot of bugs. You simply make systemd runnable, but
unusable.

I think you have the wrong impression of cgroups entirely, though. It does not
do "process monitoring". Think of it more like a hierarchical processes tree.
With a traditional hierarchical filesystem, to recursively delete everything
in a folder, you use a combination of readdir and unlink. In cgroups, it's
similar: you put processes in a "cgroup", and to kill everything inside a
"cgroup", you iterate over everything and call kill.

And much like in some filesystems you can say "make sure this subtree doesn't
go above 1GB of storage", you can apply similar resource limits to cgroups:
"make sure this subtree doesn't go over 2GB of memory and 25% CPU time", etc.

A stub interface that doesn't iterate over any of the processes won't really
do you much, and thus when you try to stop a service, it won't get shut down.

------
bch
"Maintaining portable code increases complexity

Since systemd is written in C, the canonical way to write portable code is by
using conditional compilation, for example with ifdef statements. That makes
the code harder to understand and reason about, but more importantly it blows
up the test matrix."

Take a page from NetBSDs effort...
[http://www.netbsd.org/about/portability.html](http://www.netbsd.org/about/portability.html)

------
16s
When higher-level software begins requiring systemd (Gnome, KDE, etc.), the
BSDs and other non-Linux, but Unix-like operating systems will no longer be
able to use that software.

------
cmccabe
This is great. It will give Debian access to features like a faster startup, a
sane way of restarting daemons that die, better power management and process
management.

~~~
sciurus
What is great? Debian hasn't decided to switch to systemd.

~~~
shmerl
But systemd already works in Debian. Though it's optional:
[http://wiki.debian.org/systemd](http://wiki.debian.org/systemd)

~~~
asb
But the latest systemd even in sid is version 44, which is _ancient_.

