
The Debian init system general resolution returns - mariuz
http://lwn.net/Articles/616571/
======
vezzy-fnord
From what I can tell, all this means is that Debian packagers will have to
drop hard dependencies on components of systemd in many cases. I see nothing
objectionable with this, as there have been some absolutely inane things going
on at the packaging sphere. For example, treating the fact that upstream
provides a systemd unit file as a dependency on systemd. To some extent, this
can also be blamed on inherent weakness in most contemporary package managers
that lack the ability to easily express multiple classes of dependencies, and
treat everything as a hard one.

That said, we still keep saying the same shoddy sentiments from systemd
proponents. "Can't we just move on?" "systemd hasn't had any problems for me.
Don't like it? Leave it!", "This must be Canonical.", so on and so forth.

The /r/linux thread is a depressing clusterfuck of uninformed proponents, as
always.

~~~
_delirium
One thing that is not making the discussion clearer is that many people are
failing to distinguish between two different kinds of dependencies. Depending
on a systemd _library_ versus depending on the systemd daemons _running_ are
quite different scenarios. It would be ideal for the latter to be minimized,
but the former is not a problem at all.

To take an analogy, plenty of Debian packages depend on libselinux1, because
they feature optional selinux functionality, and not depending on libselinux1
would result in broken symbol tables. But most of these packages work
perfectly fine without SELinux configured (as you might guess, since very few
people actually have SELinux configured). Having libselinux1 on your system
does not require you to use SELinux, and imo it would be unreasonable to
complain that your system is now "infected" by SELinux because you can't
uninstall libselinux1. It's just a library, like many others, and having it on
your system doesn't impose policy requirements on you as administrator or
user. Yet people are making exactly that argument with systemd, complaining
that some systemd library or another is on their system and thereby
"infecting" it. Rather than focusing on the more legitimate issue of which
packages actually require systemd daemons to be configured/running.

~~~
digi_owl
Do systemd even have such a "inert" lib?

Best i can tell everything happens via dbus (to be replaced with kdbus), and
that again requires that systemd is running as pid 1 (as it will refuse to run
as anything else, unless it is inside a namespace governed by another systemd
as pid 1).

~~~
_delirium
Sure, that's what libsystemd0 is. It's just a shared library, like
libselinux1, that lets applications be compiled with systemd support. It
doesn't depend on systemd itself, or start up any daemons. It's quite possible
to have a package that's compiled with support for two different logging
systems, for example. In that case it would depend on the libraries for both
of them, but would only actually use the one that is configured. On the other
hand, it's possible an application actually does require systemd running to
function, in which case it would depend on 'systemd' as well, not only
'libsystemd0'.

------
Shish2k
So the impression that I'm getting is that some people just want Jessie to be
released, even if some packages only run correctly under systemd; others wish
for debian to enforce "every package must run correctly under every init
system" (excluding things like init-system management tools, which are clearly
tied to their system).

I wonder, how much difference does it make either way? What packages which
only support systemd are we actually talking about, and how much work would it
be to untie them?

~~~
pantalaimon
> What packages which only support systemd are we actually talking about

Gnome, Wayland, KDE

> how much work would it be to untie them?

a lot

~~~
KayEss
That sounds like a pain for my desktop, but implies there's still hope to use
a simpler init on my servers so long as the coupling doesn't increase.

~~~
lucian1900
But servers is precisely where systemd is particularly useful. Having tools to
interact with the init system and proper logging are invaluable.

~~~
edwintorok
Some may disagree that binary logs or putting coredumps into your journal are
the "proper" way to handle logging. Sure there is some additional information
that systemd stores in logs that syslog wouldn't, but implementing that
could've been done while keeping the main log text-only (eg: have an id for
each logline in your human-readable _text_ logs, and have the binary/index etc
separate that refer to that id).

~~~
lucian1900
Structured data is very useful. Just because many unix folks have managed to
extract some meaning out of random text for so long doesn't mean it's a good
idea.

This is the same thing I see about HTTP/2\. It's _good_ to finally remove a
huge number of parsing errors and inefficiencies!

~~~
edwintorok
I'd be less opposed to journald if it used SQLite instead of its own format,
but I still think that logs don't belong in a binary file by default:
[https://news.ycombinator.com/item?id=7210570](https://news.ycombinator.com/item?id=7210570)

~~~
KayEss
The problem with sqlite for this is that as you write more and rows into a
table then the writes get progressively slower -- it simply isn't able to do
O(1) writes over a long period of time so you'd have to rotate the database
files themselves.

I've been solving this problem by writing a non-pretty printed JSON blob to
one line in a log file. Structured, but still text and very easy to stream. I
build a separate binary index that gives more efficient time based lookups,
but it isn't needed to understand and process the log file.

~~~
edwintorok
Did you release your code that does that?

~~~
KayEss
No. It's in a monitoring system we've been working on, but the principle is
pretty simple. We may well end up open sourcing quite the rest of it sometime
in the future -- a lot of the code already is.

------
ausjke
To make Debian init-neutral is a great idea in my opinion, just like Linux
distro we have many flavors, there is no real reason for mandating systemd to
be the sole init system I think. New changes are good or bad, let's give
people more choices, the better one will eventually win, or they just co-
exist, that's how the nature works, systemd is no different.

~~~
acdha
How much are you willing to pay for those choices? Each init system doubles
your testing load, which is already a concern, but more importantly it also
means that there are features which are either need to be dropped or which
will require time developing abstraction layers for multiple init systems.

That might still be worth doing but this is a lot more than saying “Oh, if you
don't like vim don't install it”. All of that will require time, effort and be
a source of bugs / UI clunkiness which no other operating system has to deal
with – I'd much prefer that people pick 1 system and spend time getting closer
to the “it just works” level which OS X and Windows have been at for years
rather than maintaining a stable of patches and extra code.

~~~
felixgallo
Linux has been at the "it just works" level for decades, where it matters --
on servers and appliances. It's "Linux on the desktop" where apparently boot
times are a big deal requiring screwing up the entire operating system for.

~~~
acdha
The first few trivial counter-examples which come to mind:

* someone adds a storage device. Now your server can't boot because what used to be /dev/sda1 is now /dev/sdc1. Yes, UUIDs partially address that but unless you were defining "decades" as "within the last year or so" you can't assume that they were used.

* init script starts something, which crashes / is killed by OOM / etc. and you'd like it to be restarted. Note that this is _still_ unsolved with upstart if the process crashes for more than your respawn limit, as is common with e.g. networking-related startup failures but, hey, it's the same braintrust which required multiple major releases to accept that sysadmins need logging and daemons to be started as users other than root or that "hang forever" is not a valid error handling strategy.

~~~
felixgallo
Nobody adds a storage device and ends up changing the boot drive on a server.
I've never even heard of that in three decades of changing drives on servers.
It never even happened with SCSI.

And in the rare case where automatic restart is a good idea, I wrap that
script in a supervisor, like monit, as god intended.

If a process is crashing for more than the supervisor's respawn limit, then
you have a problem that computers can't solve. Not sure how you thought that
could be resolved automatically.

~~~
acdha
> Nobody adds a storage device and ends up changing the boot drive on a
> server. I've never even heard of that in three decades of changing drives on
> servers.

It's not especially common, particularly if you don't work with large servers
or tons of local storage. Here are three variations on this theme which I had
to deal with:

* The Linux kernel changed the order of device enumeration in a point release (early 2.6, if memory serves). This caused some of our servers to start enumeration the on-board IDE CD-ROM ahead of the SCSI drives and on the next reboot the ones which had been setup before Debian started using UUIDs in /etc/fstab failed because /dev/sda was now the CD drive.

* A very similar problem happened involving USB mass storage, which was a problem because our remote administration tools could forward a local drive or ISO image to the remote server as a USB device. This was particularly annoying on one early KVM system which only forwarded the device when you were connected so the logical device ordering changed if you rebooted it while connected to the console.

* If you have multiple controllers in a system, the drives are enumerated in order per-controller. If the bays were not fully populated when the system was first setup, a new drive added to controller 1 will be enumerated ahead of the drives from controller 2 and all of the subsequent device names will be shifted by one.

All of these are why it's now routine to use UUIDs to identify drives. It's a
solvable problem but hasn't been so for even a single decade.

> And in the rare case where automatic restart is a good idea, I wrap that
> script in a supervisor, like monit, as god intended.

Uptime and resilience are baseline requirements for what I work on, not "rare
cases". I've used external supervisors but that's introducing a third-party
tool with its own bugs and means that you can't assume that another sysadmin
is familiar with it. If you're the monit person in a supervisord shop (or vice
versa) you'll probably have something take longer to debug or break at some
point because of a subtlety in the differences between the two.

This is such a common case that it makes sense to solve it once at the system
level, particularly since that allows me to e.g. not have to code an extra bit
in my daemon that knows how to ask arbitrary third-party supervisors about a
different process.

> If a process is crashing for more than the supervisor's respawn limit, then
> you have a problem that computers can't solve. Not sure how you thought that
> could be resolved automatically.

That's a fine approach if the failures are caused by something like a broken
config file but it doesn't handle environmental failures gracefully. For
example, consider a daemon which fails if it can't resolve a hostname or
connect to a server (e.g. a database) or which exits if it can't write to
disk. In either case, you're going to have downtime if the server / data-
center drops offline or a partition fills up.

The question is whether the service comes back quickly after the underlying
problem is fixed or requires someone to localize it and restart the process.
If you have one server and you happen to be on-duty, this isn't a big deal but
for anyone with many servers running different apps it can be a headache after
a network partition or power failure (oops, web servers all came online well
before the database server finished checking its RAID array).

In the case of upstart, the default respawn limit is only 10 times within 5
seconds after which it will stop the job until you manually restart it. Since
they didn't add throttling or delays, there's no way to deal with this within
upstart without either increasing downtime for every error (e.g. adding "||
sleep 10" to every script) or wasting more CPU restarting jobs as quickly as
possible until you hit a larger limit.

Again, not an unsolvable problem but had they not ignored a huge amount of
prior art and perhaps even consulted with actual sysadmins there would be
things like a respawn delay and backoff so you could say things like "restart
this no more than n times per minute". Instead, most places either convert
everything to use a different supervisor or support an external monitor which
requests restarts for services periodically, both of which add maintenance
overhead and risk for what is a standard feature on other systems.

------
felixgallo
The fundamental problem here is that desktop Linux users, who have absolutely
had a bad experience over the decades, have gotten so tired of incompetently
executed half-assed solutions that they now grab at any other solution with
the fervor of a drowning victim.

Why have they gotten to this point? Because the people working on desktop have
generally been the D-team. The A-team is working on kernels and
infrastructure, the B-team turns out packaging, and the D-team has been coming
up with hair-raising stuff like gnome, 'desktop bus', pulseaudio,
enlightenment.

So this weird little monoculture has emerged in which no other concern matters
except that they get rescued from drowning -- that their boot times (!?!?) go
to 5 seconds from 10, that hot-plugging USB drives (!?!?!?!) be wrapped in
with the init system if it helps at all, that DHCP (!!??!?!?!?!!) and core
files (!!!!!!!!) and all logging (!!!!?!?!?!?!?!?!) be taken over in the name
of helping even 1% on the core experience.

But at the same time, two basic axioms of reality still bear:

1\. Linux on the desktop is irrelevant and will never win, both in terms of
the Linux story and in absolute terms; and every scrap of effort that goes
into it has been objectively wasted for two decades and will continue to be
wasted.

2\. Linux everywhere else has been working fine for longer than most of the
desktop people have been alive.

following from (2), the people who run servers recognize that although systemd
has some good ideas, it forcibly bundles together a grip of super bad ones
too, and is clearly implemented by people who don't understand systems outside
of the desktop.

The attempt to turn Debian back into a distribution for everyone, rather than
a small collection of desktop people, is completely valid and sensible. That
said, it's totally clear why the desktop people want it this way, and why they
have such a misguided emotional investment in their terrible new windows
emulator.

~~~
pyre
> pulseaudio

To be fair, some of the ideas behind pulseaudio are good:

\- Unify all of the different audio systems that existed: OSS, ALSA, esound
daemon, etc...

\- Network accessible sound. Years ago I was able to setup pulseaudio to
stream audio from my HTPC to my laptop where I had headphones plugged in. The
process to do this wasn't entirely user-friendly, but prior to pulseaudio,
people would setup _yet another audio subsytem_ that every app would then need
to support on a case-by-case basis.

------
buster
Not again... I think this was already decided.. Can we just move on? I've
switched to systemd on my laptop months ago and had not a single problem. Boot
times are noticably faster.

Does Linux have to stay at software from the sixties forever? Can't we just
move on to systemd and wayland and whatever fits better into the 21st
century?!

I really feel this is all too political and FUD after months and months of
senseless complaining about systemd and particularly far too personal
regarding Lennart Poettering. If people like SysV init so much there will
always be alternatives but this shouldn't hinder the majority of users to move
forward with systems that just work with systemd (e.g. Debian+Linux). Nobody
really uses Debian/kFreeBSD anyway, for example. Yet, it's a common statement
that is brought up.

~~~
ultramancool
Please do not dismiss legitimate issues as personal attacks. Just because you
did not hit issues, doesn't mean others didn't. My experience was quite
negative:

\- Binary log files, which are also prone to corruption and crashes

\- Unnecessarily complicated and unfamiliar init script structure. I feel like
I went from one that was arguably too verbose to one which is too vague.

\- The need for unnecessary things like dbus to be run at startup

\- Complication of some simple configuration tasks (in my experience on Arch
at least, network configuration was a bitch)

\- udev has apparently been hijacked and may cause issues for those who don't
want systemd

All in all, it feels like change for the sake of change to me, I see no
advantages which I desire, and several minor to moderate disadvantages, but
learning and using it is pushed on me for no reason. Personally, I've moved to
running FreeBSD, it's simple and stable, though I've been considering
Debian/kFreeBSD, I'd be sad to see it go. But it's the farthest of the issues
with systemd IMO.

As far as your boot time improvement goes, that's great, but modernly, my
systems are either always online or suspended to RAM or disk, rarely rebooted.

~~~
nodata
> \- Unnecessarily complicated and unfamiliar init script structure.

Unfamiliar, yes. Complicated? You have to be kidding me. systemd "scripts"
remove the crud and hacks that the old init system built up over years.

~~~
andmarios
Run a cron job on sysvinit; add one line to /etc/crontab.

Run a cron job on systemd; create two files: one timer unit (about 5-7 lines)
and one service unit (about 5-7 lines) that you have to name identically
(minus the extension) with the timer unit.

~~~
JeremyNT
sysv init can launch a cron daemon which itself can run cron jobs.

systemd can launch a cron daemon which itself can run cron jobs.

 _in addition_ , systemd can also execute timer units periodically. This is
functionality that sysv init lacks.

If you select the same cron daemon, the configuration of cron jobs may be
performed in an identical manner, regardless of which init system has launched
that daemon.

Are you complaining that the optional systemd timer functionality is difficult
to use and that you would prefer to run cron jobs via a dedicated cron daemon?
That may be a valid criticism, but I am a systemd user and I have never tried
this functionality, so I cannot comment on it. I personally continue to use a
cron daemon since this is what I am familiar with.

Do you have a philosophical objection to the init system providing such
capabilities? Maybe there is a valid reason to have such an objection, but I
do not hold this perspective personally.

Regardless, your comparison here is not valid, since you are comparing the
process of configuring optional functionality in systemd (which has no
equivalent with sysv) to the process of configuring a cron daemon (which you
could just as easily be using with either init system).

~~~
thwarted
_in addition, systemd can also execute timer units periodically. This is
functionality that sysv init lacks._

You present this as if this makes sysvinit somehow lesser because of it lacks
the ability to execute timer units. Sysvinit doesn't have that because that
capability is completely unrelated to what sysvinit does. Although one could
argue that the system as a whole provides that functionality through cron
anyway, so there'd be no need to have sysvinit do it.

------
sandGorgon
Systemd has been trying to drag desktop linux into a world that is as seamless
as OSX and it is so sad that most people blast it as "not the linux way". And
even sadder that people type this on Macbooks.

The truth is that the West has moved on from Linux - Apple has won. Linux is
most relevant in Asia, where we want to run it on cheap, locally assembled
desktops. SystemD is bringing so much to improve the state of desktop linux -
it is enabling fundamental security like XWindows-without-root[1], service
dependency management ,etc.

there is a lot of fud around binary logs (which incidentally are append only
and have a grep like interface built around it) and hard dependencies on
systemd for any cgroup access. The second is actually kernel mandated - all
cgroups have to have a single arbitrator for external applications. Direct
access to cgroups will be dropped in the future. AFAIK, as on today systemd is
the ONLY arbitrator that complies with the kernel-dev requirements.

I'm keen to see where desktop Linux can go to - I believe that it is going to
be transformative for India. I strongly believe desktop Linux initiatives like
SystemD and Mir/Wayland is going to take us there

That said, if you have worked with Docker - I find myself wishing very
frequently for SystemD as pid 1. Right now I use supervisord as PID 1 (for a
multiprocess container) and it is definitely more difficult to use than
SystemD. Plus the ease of imposing resource limits (using systemd.resource-
control) is simply unparalleled [1]
[http://fedoraproject.org/wiki/Changes/XorgWithoutRootRights](http://fedoraproject.org/wiki/Changes/XorgWithoutRootRights)

~~~
Zardoz84
Have you any information about that with systemd, updates of some software
that run as service on the machine, would need now to reset the machine ??

For my, it is the only big issue that I see for my personal usage.

~~~
sandGorgon
Actually, that is another systemd FUD. SystemD enables stateless systems - now
that I think of it, very similar to Docker - by clearing out the /var and /etc
directories. Which means that when you reboot the system, it acts as a system
reset. This is actually, pretty hard to actually implement and systemd is
trying really hard to do this.

Imagine being able to build your own private DigitalOcean and able to have
pristine machines for a new fleet deployment (possibly even avoiding the stale
confidential data issues plaguing DigitalOcean).

[http://0pointer.net/blog/projects/stateless.html](http://0pointer.net/blog/projects/stateless.html)

~~~
thwarted
_[stateless] is actually, pretty hard to actually implement and systemd is
trying really hard to do this._

Support for stateless has existed in RedHat based distributions since at least
2003-2004. I, in fact, used it to run a ~50 seat callcenter using read-only
NFS root systems, with a dynamically populated /etc, a decade ago.

~~~
sandGorgon
And that pretty much encapsulates, what systemd is attempting vs what people
think it is attempting.

Systemd is not innovating on ideas, but on implementation. How do you make
things like service dependencies, stateless systems, etc consistently
possible.

It is bringing something which was possible only for a person of your skill
level, down to mine.

I would say it is the same difference between Ubuntu and Arch - idiotproofing
the Linux ecosystem.

