
Problems with Systemd and Why I Like BSD Init - mariuz
http://bsdmag.org/randy_w_3/
======
jmspring
I've been FreeBSD wherever I can (mostly personal machines), but did work on a
high availability VPN based off FreeBSD that a major Telco bought years ago.

There is another post mentioning the desktop/web thinking and the
server/sysadmin thinking.

My thinking - the server/sysadmins are right, but the majority are people in
the desktop/web crowd.

I actually expect things to get worse once you rope in the container
(docker/deis/kubernetes/mesos/etc) crowd.

The general attitude seems to be "this doesn't work for me, let me invent
something new".

If you've spent anytime trying to develop on top of containers and have a deep
systems background, you are resisting a hell we had in the late 90s and early
2000s...because "new shiny thing".

SystemD is endemic of the new generation trying to solve problems on top of
linux. Rather than understand and address the problem, invent something
monolithic that isn't better than what it replaced.

Oh then force it on distros and not be open to criticism.

I really hate this debate and probably should keep quiet.

~~~
jamespo
I'm an admin and really like a lot of the features SystemD brings, my main
concern is that it seems to be overreaching.

~~~
creshal
I'm mainly sysadmin, but I also do some occasional embedded development, and
use systemd-enabled Arch on the desktop.

Systemd solves _so many pain points_ in _all_ areas that I don't care about
ideological debates.

And most of the touted "advantages" of sysvinit are, in practice, a nightmare.
"people should just know what they're doing"? Well, too bad, nobody does!
Distribution maintainers don't know how to write robust init scripts, software
vendors don't know how to write robust init scripts, _nobody does!_

Taking away the "freedom" to introduce the halting problem to service
definitions is, in practice, _godsent_.

"With BSD init, I can just edit the /etc/rc script and replace it with
arbitrary shell code."

And now try to debug it one year later when you don't even remember that you
did any changes to it. The shit I've seen on Linux systems' /etc/rc.local
would give Lovecraft nightmares.

And the ability to sprinkle echo statements all over your init system does not
make a system "easy to debug", unless you're a PHP "devop" who thinks that OOP
is the source of all evil in the world.

"The way systemd does parallel startup bothers me. If service A depends on
service B, they were traditionally started in sequence; A only starts after B
is done starting."

Unless it doesn't, because init scripts _cannot_ reliably determine whether a
daemon has started up. Grab, say, debian's sysvinit sctipts and count the
amount of sleep statements sprinkled all over them. That's not sequential
startup, that's a race condition waiting to happen!

"Why should systemd implement NTP when ntpd already exists?"

Because ntpd is a steaming, insecure pile of crap. And no, openntpd is not a
solution, it's just a nicer problem. Why do I need an ntp _server_ on a pure
client machine? Just because ntpdate is an even bigger pile of crap?

"Being unable to shutdown the system."

As if that never happened with sysvinit. Badly behaving init scripts can cause
a lot of havoc, at least systemd has a configurable hard cut-off.

"Debian should not have adopted it for at least a few more years; they’re
supposed to be the slow, steady, and stable distro. "

Debian profits the most from ripping out the unusable joke that was their own
"parallelized startup enabled" sysvinit. I've seen more Debian stable+sysvinit
systems unable to boot up than I've seen Arch+systemd systems.

~~~
_yy
Full ACK. It makes so many things so much easier and, notably, much more
resilient. Ever had to debug a broken init script?

The ability to "simply edit init scripts so that it works" often results in
nasty, broken hacks ending up in production.

~~~
digi_owl
And once more the whole debate goes of into file format lala land.

------
digi_owl
This article seem to reinforce that there is a split in thinking going on
within the Linux community.

On the one hand you have the traditional server/sysadmin thinking about uptime
and stability.

On the other hand you have the desktop/web thinking of rebooting often (or
shutting down when not used, starting up before using) and being on the
bleeding edge.

Another thing is that his mention about standard ways of doing something. I
think that was why Freedesktop was set up in the first place.

But they seem to have gone massively off the rails almost from the outset, or
it was simply too dominated by Gnome/Red Hat people (it was after all founded
by once such person, Havoc Pennington) to really get things properly generic
and modularized.

Asi write this i get to thinking about politics and thinktanks, and how they
can make political decisions seem like they are scientifically based, when
they really are rubberstamping preconceived political notions.

Similarly it may well be that Freedesktop exist to rubberstamp Red Hat/Gnome
notions about Linux, by virtue of appearing to be a independent entity
(similarly Gnome has its own foundation, but a large segment of its core
developers are under Red Hat employ).

BTW, i seem to recall that while the BSD people had implemented most of the
systemd APIs for various settings changes, they had run into a problem with
logind.

Probably because logind depends heavily on PAM to get its job done, and PAM
seems to fly in the face of basic unix-isms.

~~~
zzalpha
_On the other hand you have the desktop /web thinking of rebooting often (or
shutting down when not used, starting up before using) and being on the
bleeding edge._

I actually find this line of thinking baffling... I don't remember the last
time I actively chose to shutdown/reboot voluntarily. Spending time optimizing
the boot sequence strikes me as a total waste of effort.

Modern computers support sleep and hibernate... why would anyone choose to
reboot except to apply OS updates on certain operating systems that insist on
a reboot?

I would _much_ rather focus on uptime stability and flawless sleep/hibernate
so we can encourage people to never reboot unless they're forced to.

That'd better serve phones, tablets, and other devices that share this
infrastructure and need this capability, and it would cater to a public that
is getting used to always-on, always-available technology.

~~~
digi_owl
Best i can tell, it comes from the -sec community. This because a encrypted
HDD in a laptop is at its safest when the laptop is turned fully off. At that
point you have no keys in RAM or anything like that. Thus when you do not use
your computer you turn it off, you do not sleep, you do not hibernate, you
turn it off.

The systemd devs have a thing about listening to the -sec community. You can
see this in how while the Linux default is to treat namespace mounts at
private, systemd mount them as shared unless told otherwise. This because the
-sec people recommend it that way (allows a sysadmin to override mounts from
outside the namespace, to avoid dos-ing apparently).

Then again they also have a bit of theater in them. For example they accepted
a patch to lock the computer if a HID devices was plugged in. But while the
original patch did it on any device, they modified it so that it only happens
on the device ids of known mouse wigglers used in police computer seizure
kits.

~~~
JoshTriplett
> Then again they also have a bit of theater in them. For example they
> accepted a patch to lock the computer if a HID devices was plugged in. But
> while the original patch did it on any device, they modified it so that it
> only happens on the device ids of known mouse wigglers used in police
> computer seizure kits.

There was a series of two patches: one to logind to add a mechanism to tell
all active user sessions to lock, and one to the udev rules provided with
systemd to add a policy that plugging in a known mouse wiggler causes all
active user sessions to lock.

(It wouldn't make any sense to lock the screen when inserting any HID device.)

Separately from that, logind also implements other policies for when to lock
the screen, such as when closing a laptop lid without an external monitor
connected.

~~~
digi_owl
And how bloody hard it is to make the wiggler change the ID to look like any
old mouse? Or one gets kitbashed out of a existing mouse? Or a new one comes
to market that is not part of the IDs systemd have on file?

------
krylon
Over time, I have come to think that systemd probably should have been two
projects. One to replace init, watch services as they run, restart them if
needed; if you get dependencies out of that, being able to start services in
parallel sort of falls out of that for free, but I feel that few people really
need that (OTOH, if it comes for free, it doesn't hurt, either).

The other project could include all the rest, basically. Again, having a
unified API for applications to e.g. get notified when the network connection
changes, time changes or a USB flash drive is connected, or a new display, or
whatever - that makes sense; having it effectively mixed in with the init
system does not neccessarily. As a bonus, if it was an independent project, it
might have been portable across different Unixoid systems.

For reasons totally unrelated, though, I installed FreeBSD on the new home
server I got recently, and it was a really pleasant experience. Plus I
installed OpenBSD on an old laptop I saved from becoming landfill, which was
fun, too. There are good reasons to like BSD even if one gets along with
systemd just fine.

------
anonbanker
This man makes valid points about the state of GNU and BSD. I've been tempted
to leave linux completely as of late, but the truth of the matter is that I
love GNU too much to go.

Yes, RMS is pretty pegged on the autistic scale, but he's not wrong, and
Snowden proved that GNU as an ideology is pretty much our only hope against
nation states intending to deprive us of our freedoms.

GNU is the closest thing to a soul the software industry could have, demanding
nothing less than freedom for the users of software and writers of software.
Their (not) unix is a beautiful thing to look at, and companies like
Novell/SuSE and Red Hat built their empires on its foundation.

Now, Red Hat is taking an active role in rewriting everything but the kernel,
in order to push-out and/or marginalize the GNU crowd. If they were honest,
they would have Poettering come out and state publicly that they are creating
a replacement Operating System for the linux kernel. but they're not. They're
wrapping themselves in the GPL (and the goodwill that comes with it) while
trying to destroy Stallman's creation.

Red Hat has quietly declared war against GNU, and those that support it,
embracing and promoting the watered-down (and far more profitable) "open
source" concept as they go.

In that war, systemd is merely another salvo that (I think) began either with
Havoc Pennington writing dbus off the back of Nat Friedman of Ximian's hard
work and research, or their complete takeover of freedesktop.org.
NetworkManager, Avahi, and PulseAudio being pushed into all the distributions,
always long before they were stable (and all written by Poettering), proved
with a paper trail that they had worked the kinks out of their embrace-and-
extend plan long before systemd was ever mentioned on a debian mailing list
for adoption.

I wish the author well, but some of us are gonna stick around to try and keep
GNU strong against this new threat. We've lost Debian, Arch, and Ubuntu, but
this war is far from over.

~~~
pdkl95
Well said. It's too bad some people seem to believe down-voting the truth will
make their cognitive dissonance go away.

> They're wrapping themselves in the GPL

...while encouraging a development style that isolates processes so they use
IPC when they should be simply linking to a library and making a function
call. Systemd depends heavily on dbus, because it ties developers into this
model.

This enables a rather convenient way for freeloading propriety software to
_use_ GPL code, without having to be subject to the _requirements_ of GPL
code. If you get some package to provide their services over dbus, you only
have to "use" the software instead of linking against a library.

Even ignoring systemd's bad design that tightly couples together services that
worked fine as separate specs for decades, and even ignoring the _terrible_
"we don't give a crap if we break your environment for no reason" attitude
that is common among systemd's authors, this end-run around the GPL is a very
good reason to avoid systemd.

Now, I'm sure some people in this thread think I'm making crap up, being
paranoid, or similar. To those people I say: when are you going to start
paying attention? This is _obvious_ when you look at the Linux community over
the last 20 years. Stop being distracted by _technical_ features instead of
investing in your future. Are you going to be happy when all distros are _de
facto_ variants of Red Hat? Some of you might be, but if you're going to
ignore the vast majority of Linux and Unix, could you at least _fork the
project_ instead of dragging the rest of the community down with you?\

~~~
the_why_of_y
So let me get this straight - you are arguing _simultaneously_ that systemd's
DBus APIs should be library calls, and that systemd should be less tightly
coupled?

Please tell me this is a parody?

~~~
e12e
You're aware there are a number of different libc-implementations one can
choose from, right?

~~~
anonbanker
in case he/she isn't, I'll just mention musl and uClibc.

------
_ZeD_
Well, if people would like to try a linux-kernel OS without systemd - or in
general if you like to try a _good_ OS - I will always recommend Slackware.
It... just run. It's no-frill, no-political choices always prefers semplicity.
On a desktop machine a slackware-current it's a joy.

~~~
progman
Agree. I just set up a Slackware system for testing purposes, and I am amazed
how easy it was. Slackware is one of the oldest Linux distros ever, and it was
actually my first distro. After a long journey to many other distros Slackware
brought back the good feeling of having control over my machine. There are
still a lot of customizations to be done (enable TRIM for SSD etc.) but
nevertheless I like it because I know that my machine does exactly what I want
it to do.

KISS is a magic principle that just works. The Unix/Linux world should never
let it go.

------
dbalan
>> Many of the problems they described with init systems are already solved by
both OpenRC and BSD init

which problems is he talking about?

~~~
tremon
Service babysitting.

This includes for example environment control (env, ulimit, cgroups, chroot),
auto-restarts, changing uids.

------
_Codemonkeyism
As I migrate more and more (production) services to Docker, I have a Linux
split into two distinct usages.

A minimal system as a base image for Docker, and then install by apt or curl
(would never, never have used curl for installing software 5 years ago) into a
service image. And a second Linux as a base system for running Docker (and
nothing else). Start/stop is managed by Docker ( /Nomad/Swarm/...). Next step
might be moving the base image away from Linux.

------
INTPenis
I guess he makes a good point because he's saying that the standard APIs are a
good thing and BSD should also focus on that to ease development for large
projects like GNOME and KDE.

I can completely understand why systemd need their own timers as part of those
APIs, I however cannot see the point of having their own NTP service. Maybe
the old one was broken with systemd?

But that's just one valid point, not being able to see the necessity of
timesyncd does not invalidate the underlying goal of systemd, nor does it
warrant moving to another distro.

Because bottom line here, bsdmag is a BSD oriented zine so its focus is to
promote BSD distros. I used to use 100% BSD 8 years ago but my migration to
Linux is now complete.

The BSD license in my opinion hurts open source more than GPL. There are three
major examples that I can think of where big corporations have used BSD code
to avoid re-inventing the wheel and piggybacking on open source. Microsoft
when they used some BSD userland tools, Apple when they built OS X and Juniper
when they made JunOS.

I used to interpret these decisions as flattering, but they are in fact simply
taking the code they need, with the license that allows them to do it. They
would never take GPL code as freely because they would risk being locked in by
its requirements.

~~~
hhw
>I used to interpret these decisions as flattering, but they are in fact
simply taking the code they need, with the license that allows them to do it.
They would never take GPL code as freely because they would risk being locked
in by its requirements.

1) Have you not considered that it was the intention of the developers of that
code that it could be used in this way, without giving back? 2) Have you
considered that the rest of the world benefits from the use of BSD code in
Apple and Juniper products, even if they don't give any code back? Anyone who
uses these products receives gets to use better code than what may have been
written from scratch. 3) Apple and Juniper both give code back to FreeBSD 4)
Even if they didn't, it does not hurt FreeBSD any to have companies use their
code and not give anything back. 5) We're agreed that Microsoft, Apple, or
Juniper would never have used GPL code in their kernels. So how exactly has
the open source world lost out due to the existence of the BSD license?

~~~
progman
> So how exactly has the open source world lost out due to the existence of
> the BSD license?

iOS (based on BSD Unix) is proprietary. Android (based on GPL Linux) is open.
iOS came first and seemed to be the huge winner. Then Google took GNU/Linux
(GPL), built Android around it, and outpaced iOS by far.

Now the BSD developers have to pay (!) for their own (!) developments when
they buy idevices. They have no option to modify anything. They are just
consumers.

The Linux developers can take the code, tweak their devices (Cyanogenmod) and
do a lot of other things with it. There is Android for x86 for instance.

BSD is good for hardware producers. GPL is good for hardware consumers.

------
CrLf
My main criticism of systemd is that the project doesn't seem to have any
bounds. They are actively trying to re-implement everything instead of having
a clear definition of what problems they're trying to fix. That tends to end
badly, usually in a unportable web of dependencies and components being shoved
in the wrong places (D-BUS inside the kernel just doesn't seem right to me).

Having said that...

It does have some nice things. It's very easy to create robust service startup
definitions (units) without resorting to brittle shell scripts checking PID
files and whatnot. Also, services start in their own resource groups for
better resource allocation between them.

These alone have changed my opinion about systemd. These solve real sysadmin
issues.

------
the_why_of_y
Meh. In the literary genre of systemd criticism there are better examples,
mostly from people fond of the microkernel design aesthetic.

"Systemd will start them both at the same time, but buffer any messages A
tries to send to B until B is ready."

It's not systemd which is buffering messages, it's the kernel's IP or UNIX
domain sockets. (Also, this means that daemons don't need to run setuid root
just to bind a low socket.)

"With systemd, the order is not deterministic, so you don’t know what’s going
to happen next time you boot."

All you need to do is define the dependency graph properly, then any possible
ordering will be fine. You might as well argue that "make -jN" is somehow
harmful.

"A server spends several minutes in the BIOS during POST anyway, before the
bootloader is even run; making the OS boot faster doesn’t change very much"

Never used virtualization or containers?

"That doesn’t feel modular to me." "Cleaning up or re-implementing ntpd is a
good thing, but OpenBSD didn’t feel the need to make it part of init."

The systemd project is not (any more) an init system, it's a collection of
tools to create a Linux-based OS from.
[http://www.openntpd.org/](http://www.openntpd.org/) says that OpenBSD
includes OpenNTPD as part of its base system.

"OpenNTPD was primarily developed by Henning Brauer as part of the OpenBSD
Project and gets released as a base component of OpenBSD every six months."

Clearly this is the same as what systemd does? Do we conclude from this that
OpenBSD feels "not modular", since it has an NTP implementation in the same
repository as its init, or is it only bad if systemd does it?

"Regular journal corruption that Lennart says is not a problem. It seems
obvious that a non-transactional binary log is a terrible idea."

As opposed to plain text files, which magically survive having random blocks
overwritten by hypothetical filesystem bugs or hardware issues without losing
a single line of log content?

"Numerous reports of systemd hanging on boot because of something in fstab
that violated POLA."

Does this refer to systemd implementing the "nofail" keyword exactly as it was
documented? The author will be very surprised when he discovers that FreeBSD
does _exactly_ the same thing, except that the keyword is called "failok".

"To my knowledge, no one has ever lost data as a result of a bug in ZFS."

Does the author have access to Sun/Oracle's internal bug tracker where such
customer bugs would be reported, in particular during the early years of ZFS's
existence?

~~~
erpellan
I feel like you're being deliberately disingenuous here. Most of your
rebuttals are straw-man arguments or they take a very narrow interpretation of
the original statement.

~~~
jamespo
The virtualization / containers criticism is completely valid

~~~
GalacticDomin8r
Um, not really. On BSD, you're using jails and startup is virtually
instantaneous unless you want to define start orders and staggering.

------
riffraff
isn't FreeBSD considering switching to nosh[0]? I seem to remember reading
about it but cannot find any reference now.

[0]
[http://homepage.ntlworld.com./jonathan.deboynepollard/Softwa...](http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html)

------
rfraile
Seeing the way that Linux have and comparing it with FreeBSD, I think that
Linux is the best OS for developers and FreeBSD for sysadmins.

