
Systemd needs official documentation on best practices - zdw
https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdNeedsBestPractices
======
wibble10
I tend to approach such tasks by just copying and editing a unit that shipped
with the os (e.g sshd) — the example the post gives as being problematic is
using some timer feature instead of cron and I suppose my question is really:
who is doing that anyway?

I know you can rig up systemd to act like inetd or cron, but perhaps you would
be better with an actual service (how hard is quartz?) than baking your jobs
into systemd units..

That said, I only use Linux in prod for enterprise stuff that depends on it,
everything else is on bsd so you can probably guess my opinion on systemd ;)

~~~
Skunkleton
I'm one of the (rare?) people who actually likes systemd. On Arch Linux, it is
uniformly applied to the whole system and is actually a nice experience IMO.
This contrasts with other distros that use a hodgepodge of sysv-init, upstart,
and others. Even still, I think that systemd has discoverability and verbosity
issues that should be addressed.

~~~
jandrese
I spent the time to learn it and yeah, it's pretty nice when it's working.

The problem is of course when you need to do something that isn't supported by
SystemD, in which case you're in for a world of hurt as the system constantly
fights you.

For example, I had a requirement to make sure some machines randomized their
MAC addresses when connecting to untrusted WiFi in Ubuntu 14. Turns out this
functionality was broken in NetworkManager and trying to go around its back
was a huge huge headache because the system was relentless in checking the MAC
of the interface and setting it back to the physical one.

There are other times where it bites you in the ass too, like writing a raw
image to a SD card. In the old days you would just dd the image onto the card,
but you can't do that anymore because shortly after you write the filesystem
header to the card SystemD will notice the new filesystem attempt to mount it,
killing off your dd process.

I always try to work with SystemD now because it's so much harder to fight it.

~~~
Twirrim
Ubuntu 14 doesn't use systemd. It was the last LTS prior to the systemd
switch.

~~~
jandrese
Ack, I misspoke. Ubuntu 16 was the one with the busted MAC randomization. It
worked in 14, but was broken in 16 due to the SystemD deployment.

[https://bugs.launchpad.net/ubuntu/+source/network-
manager/+b...](https://bugs.launchpad.net/ubuntu/+source/network-
manager/+bug/1573856)

------
tharne
Here are some best practices:

\- Small is beautiful.

\- Make each program do one thing well.

\- Build a prototype as soon as possible.

\- Choose portability over efficiency.

\- Store data in flat text files.

\- Use software leverage to your advantage.

\- Use shell scripts to increase leverage and portability.

\- Avoid captive user interfaces.

\- Make every program a filter.

~~~
beefhash
I know this is only tangentially related, but when you're bringing up UNIX
philosophy...

> \- Store data in flat text files.

Storing encryption keys or the like in flat text files is surprisingly hard,
at least in C. I don't think it's even possible to use sscanf(3) without
unexpected explosions, and otherwise you get to hand-roll your hex parsing
code yourself (Base 64? Get yourself a library).

~~~
avar
Then don't do it the hard way and store something like a password in some
general text file that needs to be parsed, just have the password in its own
file. This is also good practice when that's otherwise a sane idea because you
can apply separate permissions to that password file.

Working with that sort of file in C is trivial, you either open() and read()
from that file, or mmap() it.

------
pmoriarty
The lack of clear, thorough, reliable documentation is one of my biggest
gripes with systemd.

~~~
beefhash
The man pages that ship with systemd seem very thorough to me. The problem is
rather that they're very thorough in the first place. They're a very hard,
lengthy read and you kind of require to understand all of it before things
begin to make sense.

~~~
bashinator
That's exactly the point the article is making. In addition to the
comprehensive documentation, systemd users need best practices for a good
range of common-to-uncommon use cases.

------
arminiusreturns
I agree, and I am one of the people who has some major gripes with systemd
through experience. I admit it's a reality and people should learn it, but the
problem is not many people have, and sometimes it really makes things that
used to be much easier not necessarily more difficult (though sometimes), but
rather more complicated (there is a difference). For example, I always found
writing a sysvinit service pretty straight forward, but systemd units can get
unweildy especially when doing anything complex, though I will admit some of
the features are nice if you can figure them out, such as when in the init
proc to start, vs having to do that manually in sysvinit scripts.

It also often fails to admit when it is failing, for example, nobody I know
uses journald as the main or only logging system, and are still piping to
local (r)syslog(ng) and then to elk/splunk, etc, because journald is simply
not up to par.

In general, and I'm just being honest here, besides these mostly work-
aroundable issues, it is the seeming disingenuousness of those who
automatically rail against anybody who criticises systemd that makes me not
like it. Call me a contrarian, but I hate being told I must like X and if I
don't I'm dumb and behind the times, and my nature is to push back against
that.

In that vein, I support the efforts of Devuan and similar projects if only to
support a counter-balance.

~~~
habarnam
> [..] elk/splunk, etc, because journald is simply not up to par.

Could you expand on what's not up to par in journald vs. rsyslog? Watching
textfiles for changes vs. receiving data on a non blocking socket is
definitely _not_ more efficient (nor "better" in my opinion).

~~~
arminiusreturns
Mostly related to the tooling around those things, especially when dealing
with groups of machines sending logs to a central system. It's worth
remembering you are still using journald when using one of the syslogs, it's
just forwarding all the journal messages to the syslog socket.

As the tooling catches up this gap will close. For example the json output of
journald can be really useful, that said, most of the tools still require some
change of the output from the binary format which they can't read to a text
format whether that be json or syslog... and the binary format in general
being a general pain point if (read: when) things break (such as corruption of
the whole journal past corruption point vs one section of text in syslog).

It's worth remembering that journald was designed for local system use. It's
just one more of the reasons I had some hesitation with systemd but got
shouted down by everyone on the systemd bandwagon.

------
etaioinshrdlu
What's the general consensus on the relevance of systemd when you've migrated
to a fully containerized architecture? I know that I run almost everything in
docker nowadays and barely touch systemd. I use fargate on AWS too and there
is no systemd to touch. I use a simple python based init process inside most
of my containers.

~~~
markstos
If you are not using systemd, it is not relevant.

~~~
etaioinshrdlu
To be more specific, is it worth learning nowadays?

~~~
markstos
For DevOps, yes. Systemd can be a simpler solution than say Docker for some
applications.

------
catalogia
But if they codify best practices, then they won't be able to invent "best
practices" on the spot to cover their asses whenever somebody discovers some
flaw in their software.

Prime example: the "best practice" of not letting usernames start with numbers
is invented on the spot by a SystemD dev to cover his ass:
[https://github.com/systemd/systemd/issues/6237#issuecomment-...](https://github.com/systemd/systemd/issues/6237#issuecomment-311900864)

~~~
Xylakant
Isn’t it shadow-utils that came up with that rule? And rhel/centos patching
that out?
[https://gist.github.com/bloerwald/a482791395114fa82636e2ab20...](https://gist.github.com/bloerwald/a482791395114fa82636e2ab207cdb11)

~~~
catalogia
It was never a rule, only a quirk a few other tools had. The kernel has always
permitted these usernames; it is plainly systemd's responsibility to not shit
the bed when it encounters such a username.

------
1996
I love systemd. I use it extensively in production. The constraints handling
(ex: service depends on database which depends on network) is great. You can
have a systemd timer restart another timer that was suspended when the system
load was too high, etc.

Systemd is the best way I know to handle programs that can fail randomly, with
complex dependencies. Just tweak the restart limits on the service. You can
manage the whole set of dependencies. It can even be instructed to reboot the
server.

However, documentation could be better. For example, I have not been able to
use yet the new feature allowing services to join existing namespaces not
created by systemd (so not JoinsNamespaceOf=, but NetworkNamespacePath= ; cf
otherwise [https://cloudnull.io/2019/04/running-services-in-network-
nam...](https://cloudnull.io/2019/04/running-services-in-network-name-spaces-
with-systemd/))

But systemd is still a bit new. Documentation and best practices will improve.

EDIT: actual technical content get downvoted as usual, while opinion pieces
about systemd being the antechrist fly.

------
api
The best practice would be to replace it with something simpler and more
elegant.

------
markstos
The article complains about not getting email when a timer-activated service
fails.

Like all software, if you didn't test it to work, why should you expect it to
work? Cron can fail silently as well if don't set MAILTO. Not all servers
start out with working outbound mail, which is something you have to setup for
both Cron and Systemd as well.

I agree the OnFailure= recipe could be documented better, although I found an
example easily enough when I looked up how to have systemd timers send email
on failure. I use a script which mails the output of `systemctl status` on the
service that fails, so I get logs of what happens.

I've a number of questions about systemd on StackOverflow sites and there are
definitely some FAQs about best practices related to getting started with
setting up services.

~~~
upofadown
>Cron can fail silently as well if don't set MAILTO.

From the cron man page:

> Any output produced by a command is sent to the user specified in the MAILTO
> environment variable as set in the crontab(5) file or, if no MAILTO variable
> is set (or if this is an at(1) or batch(1) job), to the job's owner.

~~~
1996
With a non working mail configuration, it goes nowhere.

I think the author meant that

~~~
upofadown
If they did mean that then they were deliberately missing the point of the
original complaint.That isn't the sort of thing we are allowed to assume
here...

~~~
1996
So you prefer not assuming good fait? Ok.

Without mail, journalctl and even better, systemctl list-timers will tell you
about failed crons.

Can you do something as good with at and cron?

