
The importance of Devuan - telmich
https://blog.ungleich.ch/en-us/cms/blog/2017/12/10/the-importance-of-devuan/
======
badrabbit
So, the whole thing with systemd is not that "it's bad" but that "I don't like
it and I am stuck with it". Moreover, it leaves those of us who don't like it
disenfranchised by distros that insist on using _only_ systemd

Opensource is about alternatives. Forks are a good thing,I wish devuan great
success so people like myself won't have to whine about systemd all the time.
If you like systemd and it's philosophy,I wish you the best of luck with it.
Just keep in mind that others don't have to like it and different people or
organizations have different needs.

~~~
buu700
My thoughts exactly. I personally have nothing against systemd and haven't
been directly affected since I rarely need to touch init scripts, but I still
appreciate Devuan and loosely follow its progress for this reason. (Kind of
like ReactOS — 99% I'll never want or need it, but the fact that it exists is
still really cool.)

~~~
cjsuk
I was a systemd proponent until I had a hosed box. At that point I learned
that alternatives are important. It made my life very difficult due to how
opaque it is.

Whike problems are surmountable, rehearsing for failure is advised.

------
dvfjsdhgfv
I spprt Devuan because I support choice. And I hope one day - be it 5 or 10
years - we arrive to the point where Devuan is no longer needed - you will be
able to replace one init system with another in Debian. Currently it's not
possible and the system I've been using since 1998 became something else that
I can't control in a way that was possible until now.

When I ask my fellow colleagues (mostly sysadmins), some of them like Systemd,
some don't, but we all agree that in this particular case the freedom of
choice is much needed.

~~~
icebraining
Why can't you run Debian with another init system? Some things probably won't
work, but wasn't that always the case?

Sincerely asking, I never tried running a non-default init system.

~~~
rini17
Debian decided to not support using any other init system. If it was just
"same breakage as always", devuan would not be created.

~~~
bigon
That's actually not true at all...

    
    
      * https://packages.qa.debian.org/s/systemd.html
      * https://packages.qa.debian.org/u/upstart.html (it's gone from stretch that's true but still supported in jessie)
      * https://packages.qa.debian.org/o/openrc.html
      * https://packages.qa.debian.org/s/sysvinit.html

------
wicket
> But first, let me put some warning out here: Dear Devuan friends, while I
> honor your work, I also have to be very honest with you: in theory, you
> should not have done this. Looking at creating Devuan, which means splitting
> of Debian, economically, you caused approximately infinite cost.

I thought the same at first but after trying to maintain a Debian installation
without systemd, I was convinced otherwise. The "infinite cost" was not caused
by Devuan developers but by Debian developers.

I felt the Debian transition to systemd was handled very badly and was rushed
in order to get Jessie out of the door. Debian still provides the sysvinit-
core package as if it were a supported init system however if you use it
you'll find the operating system has many quirks and many things are broken.
Before Jessie was released, I commented on one bug report related to a hard
dependency on systemd for NetworkManager. Instead of fixing the regression,
the package maintainer gave the excuse that they don't have the resources and
would need more maintainers. I sympathise with the package maintainer but this
is the sort of thing that led to this mess. I remember there being a lot of
hostility in that period from Debian developers towards those who did not wish
to use systemd.

Based on my experience, I think the right decision was made to "fork" Debian
to create Devuan. It's a shame that they had to do this. I still hold hope
that maybe one day all of the work put into Devuan can be reintegrated into
Debian and it can return to be the "Universal Operating System" that they
still claim to be.

~~~
icebraining
I'm pretty agnostic towards systemd, but I don't understand this complaint
that Debian now doesn't support other init systems. Did they support other
systems besides sysv before?

~~~
wicket
They also supported upstart but that was dropped in Stretch. We are left with
systemd and System V init to provide /sbin/init. Debian still provides System
V init which itself works fine but other packages, many of which previously
worked, have come to depend systemd only. This type of regression would not be
acceptable in other projects but Debian has done nothing about it. I think
they should stop pretending that Debian is still usable System V init and just
drop the package altogether.

~~~
icebraining
Debian officially supported Upstart until Stretch? That's news to me. Do you
have a link?

~~~
wicket
[https://tracker.debian.org/pkg/upstart](https://tracker.debian.org/pkg/upstart)

~~~
icebraining
The package existed, but as you pointed out regarding SysV, that doesn't mean
it's actually _supported_. Was it?

~~~
wicket
That really depends on your definition of "supported". It does not form part
of the base distribution, nor was there ever any paid support for it.

A Debian developer built a deb package for it and added it to the "main"
repository. Everything under "main" is normally considered to be part of the
distribution. As you can see from the link I provided, the package was
maintained for several years and received several updates until it was dropped
for Stretch. That is exactly what I would expect from a supported package in
Debian.

------
moreentropy
I think most systemd criticism misses the point and revolves too much around
usage semantics.

Systemd allows actual resource management on a level that simply didn't exist
with traditional init [1]. Yes we can argue about implementation, but the
necessity of better resource management in more dense and shared environments
can't just be ignored, especially when building a shared VM hosting platform
like the author is talking about.

[1]
[http://0pointer.de/blog/projects/resources.html](http://0pointer.de/blog/projects/resources.html)

~~~
pmontra
The conclusion of the post is:

> the reason to use Devuan is hard calculated costs. We are a small team at
> ungleich and we simply don't have the time to fix problems caused by systemd
> on a daily basis.

They lament

> servers that don't boot, that don't reboot or systemd-resolved that
> constantly interferes with our core network configuration

Using systemd was costing them too much. Moving back to the previous init
system looks like a rational choice for them.

My experience is different but I only manage a handful of virtual servers and
not on a daily basis, plus my laptop. Systemd configuration files are not
difficult to write and they restart the daemons if they crash. For complex
stuff I make systemd run a bash script that eventually executes a daemon, kind
of cheating. My laptop still runs well. Booting time is definitely not an
issue, it went from fast once per month or so (kernel upgrades), to fast still
once per month so. I didn't notice any difference after the change of the init
system. A good thing but maybe it means that the return on investment was
dubious for this use case.

Binary log files are objectively worse than text ones. cat, less and tail were
good enough and shorter to type than journalctl. Ok, I could alias it to a
four characters word but it was still a lot of work that could have been
invested on some other goal. Instead I've got servers with possibly compact
binary logs made of very few lines and large text logs from web applications.

I'm also puzzled by the philosophy of bundling more things together and
tighten dependencies. It's somewhat disconcerting and I'd like a system where
we could swap components out more freely, but this is an opinion and not
facts.

~~~
gerbilly
> Systemd configuration files are not difficult to write and they restart the
> daemons if they crash.

This I have a problem with, if it crashes, I'd prefer it stay crashed, instead
of crashing over and over.

At least when it crashes, a human can come and see why it's crashing, rather
than doing the caveman thing and just restarting it over and over.

~~~
rkangel
The restart 'strategy' is a one line configuration change in the unit file.
"Restart=always" vs "Restart=no". See:

[https://www.freedesktop.org/software/systemd/man/systemd.ser...](https://www.freedesktop.org/software/systemd/man/systemd.service.html#Restart=)

------
jchw
I don't like systemd, but it works pretty well nowadays in my experience and
we use it for servers at work.

Truth be told, I'm fine with monoliths. They just have to be good. The Linux
kernel is a monolith.

The real problem to me is how systemd makes the rest of the system more
complex and integrated to systemd. Dbus is also not a great IPC layer, and yet
it's the current state of the art.

We need better APIs and foundations. Systemd doesn't provide those; it just
lets everyone depend on systemd. Good for Redhat, bad for innovation.

------
turbinerneiter
Soooooo, I'm part of a CubeSat team and we use systemd. Extensively. We us
their dbus library for our daemons to communicate, some of the ops is
basically done in service files.

I have to get our systems people to write a paper on it. My experiences with
systemd are pretty much perpendicular to what some people claim.

~~~
telmich
It's not about claiming here, it's about experiences. I guess we should open
up a youtube channel for filming our most catastrophic experiences with
systemd, because otherwise somebody will claim again that its brokeness is
just a claim ...

~~~
lima
Let me tell you about my most catastrophic experiences with sysv... I've had
quite a few.

Anecdotal evidence is useless.

What you're suggesting is basically spreading FUD. Your article didn't point
out a single _actual_ issue with systemd.

I run large scale VM infrastructure on a systemd distro (RHEL, in particular)
and I've yet to encounter a single issue that was caused by systemd or, for
that matter, any issue with systemd that wasn't easily resolved.

~~~
fao_
> What you're suggesting is basically spreading FUD. Your article didn't point
> out a single actual issue with systemd.

That doesn't mean that there _aren 't_ any. Also, FUD (as I understand it) is
people spreading unfounded rumours, not people documenting their own
experiences and problems with a piece of software. The intent is incomparable.

If you are actually interested in tangible structural flaws in systemd, look
no further:

[https://suckless.org/sucks/systemd](https://suckless.org/sucks/systemd)

[https://www.theregister.co.uk/2017/07/28/black_hat_pwnie_awa...](https://www.theregister.co.uk/2017/07/28/black_hat_pwnie_awards/)

Also, see how Poettering handled this zero day (Which grants root to any user
with a numeral):
[https://github.com/systemd/systemd/issues/6237](https://github.com/systemd/systemd/issues/6237)
HINT: He doesn't think it's a bug...

The fact that this even fucking exists in the first damn place:
[https://latesthackingnews.com/2017/06/29/a-systemd-
vulnerabi...](https://latesthackingnews.com/2017/06/29/a-systemd-
vulnerability-allows-attackers-hack-linux-machines-via-malicious-dns-
response/)

See also: [http://without-
systemd.org/wiki/index.php/Arguments_against_...](http://without-
systemd.org/wiki/index.php/Arguments_against_systemd)

~~~
bennofs
> Which grants root to any user with a numeral

This formulation may be misunderstood. The bug did not grant users root
rights, it ignored the user= setting on a service if the service was intended
to run as a user stating with a numeral. So you already needed to be root to
create a service. The bug allows a service to run with more permissions than
intended though.

------
gargravarr
I thoroughly support Devuan. I switched my home servers to it around the
systemd introduction and kept their complete stability, and now I've been able
to influence my company to run it instead of pure Debian. You have to make a
few allowances for packages that assume they're running on systemd Debian, but
overall the system works exactly as intended. And as commented in the article,
it's stable and easy to understand.

At the risk of igniting something, the only serious recommendation I hear
about systemd is improved boot times. My computers are rebooted only for
kernel updates - _this is the entire point of running Linux_. I want uptime. I
can stand a boot time of a minute or so if it's once a month or less. If I
have to reboot regularly enough that boot times become a serious advantage,
someone has done something very wrong with Linux.

In my opinion, the Debian developers were wrong to fully embrace systemd; it
goes quite against the Debian principle of being able to adapt your machine to
do anything. Yes, Red Hat held a lot of sway in convincing the community to
adopt it, but Debian is a big distro as well - just look at the install base
of Ubuntu. They could have held out and pushed back, but instead caved and now
we're in the systemd mess. It intrudes so massively into userspace that it's
impossible to get away from, and we wind up with the Windows model as this
article points out - a massively complex spaghetti model of processes, all
interlinked, all hinging on everything playing nicely. Just like Windows, it's
a neat house of cards that could topple if any one of those crashed. This
isn't Linux.

I'm not defending sysvinit, there's no denying it's showing its age and harks
back to a much simpler time, but systemd is not the answer.

~~~
lima
> At the risk of igniting something, the only serious recommendation I hear
> about systemd is improved boot times. My computers are rebooted only for
> kernel updates - this is the entire point of running Linux. I want uptime. I
> can stand a boot time of a minute or so if it's once a month or less. If I
> have to reboot regularly enough that boot times become a serious advantage,
> someone has done something very wrong with Linux.

That's far from the only advantage systemd has.

Even then, boot times are extremely important for modern VM and container
infrastructures.

------
majewsky
> We tried to build Data Center Light on Debian and Ubuntu, but servers that
> don't boot, that don't reboot or systemd-resolved that constantly interferes
> with our core network configuration made it too expensive to run Debian or
> Ubuntu.

Can you expand on that? I work at a cloud provider, and I never heard of such
issues before. Our default images are Ubuntu 16.04, RHEL 7 and SLES 12, so
systemd all around.

~~~
telmich
Regarding not rebooting: sometimes systemd hangs and does not reboot in our
setups, but we were never able to sufficiently debug this, as all terminals
are already gone and it seems that systemds is waiting in a loop for
dependencies that don't solve.

~~~
moreentropy
Have you tried passing 'debug' instead of 'quiet splash' to the kernel cmdline
and doing halt instead of poweroff/reboot? That way you should see what's
going on during shutdown.

~~~
telmich
In terms of booting: we are aware that systemd just kills fsck after a
timeout. And that you can adjust the timeout. It is just something additional
that we need to take care of, when running a system with systemd that is not
the expected behaviour of a Linux system (like: why would you ever want to
abort fsck?)

~~~
justinclift
If you have a large (multi TB) filesystem, sometimes those can take a long
time to fsck.

And if the fsck is just a scheduled one (after x reboots) instead of being
triggered due to filesystem weirdness, then getting the server back up and
serving can be the higher priority.

That's just an example because you asked though. Wouldn't want that same fsck
to be skipped if it was actually started due to filesystem weirdness being
detected. :)

------
zbentley
Every time I read something complaining about systemd from a
centralization/reducing-choices POV, I mentally replace the word "systemd"
with "glibc" and check to see if the argument still holds water.

Mind you, there are plenty of other complaints (stability, back-compatibility,
etc.) that are useful/valid regarding this software, but the above litmus
serves to eliminate rather a lot of FUD and hand-wringing in my experience.

~~~
greenhouse_gas
>Every time I read something complaining about systemd from a
centralization/reducing-choices POV, I mentally replace the word "systemd"
with "glibc" and check to see if the argument still holds water.

Actually, I replace systemd with *BSD. You know, the OS which has blessed core
packages which are not replaceable?

~~~
petre
Actually the BSDs with their non-replaceable core packages are a sane
alternative to Linux w/ systemd.

~~~
dijit
The difference is that in the BSDs (especially OpenBSD) a lot of care is taken
to be configurable. Consistent. Clear and transparent. OpenBSD has the most
extensive suite of tools built specifically for systems administrators that
exists in a base system.

SystemD has opaque behaviour and binary interfaces. It's the antithesis of the
BSD model. Despite BSDs being quite coupled groupings of software.

Anecdotally we're attempting to move to BSDs instead of upgrading to RHEL7 due
to SystemD.

~~~
justinclift
One gotcha (for me) with _BSD is that I 'm developing on Golang. It doesn't
really debug well on anything other than Linux... so had to move to that for
development.

... and for deployment, the Golang debugger doesn't even run on _BSD at this
point. So, going to have to setup new Linux (likely CentOS 6 - no systemd - or
7) boxes soon now that we're getting closer to first initial real world
deployment for the project I'm on.

------
cosarara97
You can run a traditional init in debian. Systemd is just the default.

~~~
telmich
That __might __be true, but you cannot get rid of systemd in Debian

~~~
duckerude
You can just uninstall it. It looks like apt remove systemd would work on my
system, at the cost of GNOME and NetworkManager. APT automatically fills the
gap with sysvinit and consolekit. OpenRC is also available. Debian even has
ports to kFreeBSD and HURD that don't support systemd at all. There were some
rough edges last time I ran Debian without systemd, and it might have gotten
worse since then, but it surprises me how Debian ended up at the center of the
debate when it doesn't seem that bad as systemd-defaulting distros go.

~~~
greenhouse_gas
>kFreeBSD

It seems dead.

------
telmich
Btw, I wanted to thank everybody for upvoting this post!

For me it is not only a Sunday morning essay, but also important that people
understand, why the Devuan movement is so important to all of us.

------
bitwize
Oh, suck it up.

From a business perspective, the most important thing -- indeed, the only
thing that truly matters -- isn't the software itself. It's support. When it
comes to open source, you either use a supported configuration or you assume
ALL the responsibility for supporting your systems yourself. Who would you
rather handle the support on the off chance your machine goes tits up? You? Or
Red Hat?

No wait, scratch that. Who would YOUR BOSS rather handle the support?

The supported init system for most flavors of Linux (that aren't in LTS) is
systemd. Systemd makes the most sense from a business perspective, and "but
muh Unix philosophy" doesn't hold when there's money on the line. May as well
get used to it.

~~~
gerbilly
>From a business perspective

That's just it, some of us aren't interested in the 'business perspective' and
have other uses for our systems.

------
crististm
What makes the door responsible for the failures of engine ignition?

This is a good analogy to remind Devuan why they rejected systemd in the first
place.

------
lima
What irks me more about this blog post than the systemd part \- which has been
discussed in-depth countless times in the past, with valid arguments for and
against - is the decision making and risk assessment process.

I've run large-scale infrastructure on both Debian and RHEL, with and without
systemd. I've been involved in many distro decisions in the past. The init
system was never a major consideration, and I've always been able to work
around whatever issues I've had with both sysv and systemd.

Here's some considerations which were important for my team the last time we
had to make a choice:

 _\- Vendor stability and long term support_

If you're running a business, you'll want long term stability guarantees -
both from a technical, and a business point of view. Running a small community
distro with few - if any - commercial users is a huge liability. Yes, you
always have the option to fork it or take it over, but unless you're in the
business of building a Linux distro, it's almost certain to be a bad business
decision since your competitors are spending their time on their products
instead.

Anyone who doesn't appreciate long term vendor stability hasn't yet been
burned by the lack of it. Your Gentoo wizard left the company? Too bad, good
luck maintaining your infra now (I've actually seen that exact scenario play
out not once, but twice!). A few core maintainers left for another project and
you're left building your own packages? D'oh.

Having a mature and well-funded organization or a large community of
commercial users (in the case of Debian) supporting your distro is extremely
valuable, even if you aren't the ones paying for it.

 _\- Ecosystem_

The ecosystem is also really important. It's often the main motivation for
using a particular distro. Development tools, third party package repos, 3rd
party enterprise software, troubleshooting resources, documentation and much
more depend on a healthy ecosystem.

With Devuan, this isn't as much of an issue, but it's still sufficiently
different from Debian in sometimes subtle ways that it will nullify some of
the advantages of being in a common ecosystem.

This also includes hiring - you'll have a harder time staffing your company if
you're using exotic stuff and you'll unnecessarily spend your time training
them. Ever wondered why companies like Google publish so many papers, talk
about their infrastructure and even publish books about it[1]?

It's because it means they can - by advancing the state of the industry - hire
people who are already familiar with their architecture and concepts. This is
a real problem for them - Google, for instance, is in some cases so far ahead
of others that they have to spend a significant amount of resources to bring
their new hires up to speed. I don't know about Amazon and Facebook, but I'm
sure they have similar problems.

[1]:
[https://landing.google.com/sre/book.html](https://landing.google.com/sre/book.html)

 _\- Security_

Security follows the supply chain. Your security is only as strong as your
distro's - no matter how good your security controls and processes are, if
your distribution vendor is compromised, you won't stand a chance unless you
have a world-class security team.

Likewise, your customers fully rely on you - if you're compromised, they will
be, too.

Security is much more than signing your packages and publishing security
advisories. In fact, those are merely the results of a properly implemented
security management framework - risk assessment and mitigation needs to be an
integral part of your vendor's (and your) processes. Where do they keep their
PGP keys? Who has access to it? Is is stored on a HSM or just sitting on a
server somewhere? Is there a process for revoking it? Is there a change review
process? Does the build system verify source code integrity? Is there two
factor authentication for production? Do the team members keep their SSH keys
on a smart card? This list could go on for miles.

 _\- Quality Assurance_

Even with the main distros, QA quality varies wildly. In my personal
experience, Red Hat has by far the best QA, followed by Debian and then (with
some distance) Ubuntu. I've had particularly bad experiences with Ubuntu, a
sentiment shared by many operations people I've talked to (the worst one being
the grub timeout bug that made me spend a few days pressing the return key a
lot - basically, they removed the default timeout in a minor update and all my
machines sat there waiting for someone to manually continue the boot process).

Attention to details, even if it's minor and seemingly insignificant, really
helps productivity if you sum it up at the end of the day - each bug your
vendor catches and each quirk they document is time you won't have to spend
with debugging.

With smaller distros, this is even worse - in some cases, they don't even
_have_ proper QA.

Now, let's assume that the issues they had with systemd were as significant as
they say they are (I'm doubting it, but unfortunately, they didn't talk about
any particular ones), it still might very well have been a better business
decision to stick with Debian and spend their time fixing the issues they had
instead of migrating to Devuan, which is a HUGE liability compared to plain
Debian.

~~~
jancsika
In your "security" section you completely skipped over the distro's process
for deciding when to patch and when to push/outsource to upstream. Especially
given that you're discussing Debian that's a glaring omission.

~~~
lima
True, I forgot to mention it.

Debian does a particularly good job with security patching (better than Red
Hat and Canonical, IMO).

~~~
jancsika
More to the point-- what's Debian policy on applying local patches in their
packages?

For example-- if you do it today on a core app/lib without consulting upstream
is that enough to get your credentials revoked?

I ask because Debian famously patched a library to quiet a valgrind memory
error which resulted in their userbase generating predictable key material for
a few years[1].

I trust them not to do that again (at least with libs that are obviously
security related). But is there explicit policy in this regard?

[1]
[https://www.schneier.com/blog/archives/2008/05/random_number...](https://www.schneier.com/blog/archives/2008/05/random_number_b.html)

------
raziel2p
The fact that they weren't able to set up computers to boot using systemd is
enough to warrant that I'd never want to buy or rent servers from this
company, sorry to say.

~~~
telmich
Question: did you every try to start a system with > 60TB storage attached
with systemd?

~~~
hpcjoe
I have. At my previous company, I built single system storage units for our
large scale distributed storage systems that had between 100-500TB of physical
storage in a single box. Linked together via infiniband, 40/100 GbE, and
others. RAID LUNs via both hardware and/or software (tuned MDRAID, or ZFS).

Mounting/unmounting wasn't the biggest problem with systemd. Actually, once we
figured out the right ordering of units, it went pretty well.

The greatest pain was the networking.

Systemd is a highly opinionated system. And it is opinionated in ways it
really shouldn't be. Happily some aspects are configurable, and in much of our
system setup, I reconfigured some of the more egregious settings. Some aspects
were simply painful, such as networking.

Years ago, I dealt with other highly opinionated, and similarly broken
systems. These systems insisted on doing things in their order to bring up
their services, even if I didn't need them, because they maximized dependence
radii. Which, for the life of me, I did not understand for my use case. But I
could see it for other use cases.

My approach to dealing with this was to provide the absolute minimal basis for
that system to operate, and then exit its configuration as rapidly as
possible, given our experience with its bugs. That is, have it execute for the
bare minimum possible time, before we transfer control to something we've
developed, that actually works.

I adapted that to systemd. While I had to put up with all sorts of timeouts
for systemd services that I could not adequately control, and could not remove
due to these insane dependence radii, I could tune those timeouts way down.
Which enable me to escape the systemd startup within reasonable timeframes.
And then allow my code to take over.

Customers didn't notice unless they looked at bootlogs. They simply saw a
reliable service. Which was made reliable after working around systemd's
myriad of shortcomings. I could not get systemd to do what I needed, my
opinions were different than its, and its control plane couldn't fathom what I
needed to do. So the idea was simply push it out of the way as rapidly as
possible.

~~~
lima
> The greatest pain was the networking.

Last time I checked, systemd - the init system - wasn't involved with
networking at all.

There's systemd-networkd, but that's an optional component and no major distro
is actually using it as a default. RHEL 7 - which runs systemd - supports both
NetworkManager and their legacy networking scripts.

You said there were "myriad of shortcomings" and "egregious settings", some
examples would help.

~~~
hpcjoe
Specifically in Debian 8 and Debian 9, systemd-networkd is the default used to
bring up interfaces. Controlling it is very hard for any less "traditional"
use cases.

On egregious settings, google is your friend. However, here are just a tiny
smattering of what I had to deal with. These were dealt with over a few years
of delivering and supporting systems that had to work reliably and
predictably, in a supportable manner. Each line has often a significant amount
of debugging time invested behind it before we came up with the line you see.
$TARGET is the target install directory for the image.

This is just a small sampling of the open source build system, that I grabbed
from some of the configs we used.

Yes, systemd is broken, but not irretrievably. It is fixable.

# fix some systemd timeout brokenness

    
    
            sed -i 's|^#DefaultTimeoutStartSec=.*|DefaultTimeoutStartSec=15|g' ${TARGET}/etc/systemd/system.conf
    
    
            sed -i 's|^#DefaultTimeoutStopSec=.*|DefaultTimeoutStopSec=15|g' ${TARGET}/etc/systemd/system.conf
    
    
            sed -i 's|^#ShutdownWatchdogSec=.*|ShutdownWatchdogSec=2min|g' ${TARGET}/etc/systemd/system.conf
    
    

# fix systemd journaling. Yeah, really

    
    
            sed -i 's|^#Storage=.*|Storage=persistent|g' ${TARGET}/etc/systemd/journald.conf
    
            sed -i 's|^#SystemMaxUse=.*|SystemMaxUse=250M|g' ${TARGET}/etc/systemd/journald.conf
    
            sed -i 's|^#RuntimeMaxUse=.*|RuntimeMaxUse=250M|g' ${TARGET}/etc/systemd/journald.conf
    
            sed -i 's|^#ForwardToSyslog=.*|ForwardToSyslog=yes|g' ${TARGET}/etc/systemd/journald.conf
    

# fix the INSANE logind.conf per user directory size ... hard code it to 256M

    
    
            sed -i 's|^#RuntimeDirectorySize=.*|RuntimeDirectorySize=256M|g' ${TARGET}/etc/systemd/logind.conf
    

# # fix the INSANE logind.conf KillUserProcesses problem, which nukes
nohup/tmux/screen ...

    
    
            sed -i 's|^#KillUserProcesses=.*|KillUserProcesses=no|g' ${TARGET}/etc/systemd/logind.conf
    

# mask off systemd-udev-settle ... yes it is broken

    
    
            chroot ${TARGET} systemctl mask systemd-udev-settle
    

[edited for formatting]

~~~
lima
> _Specifically in Debian 8 and Debian 9, systemd-networkd is the default used
> to bring up interfaces. Controlling it is very hard for any less
> "traditional" use cases._

Are you sure about that? To my knowledge, Debian 9 is still using the good old
network scripts. I'm 100% about Debian 8 since that's why I use in production,
with systemd-networkd nowhere to be seen.

I agree about it being less flexible. Either way, it's a separate daemon not
tied to systemd.

> _fix some systemd timeout brokenness_

Why are you decreasing the timeouts? I actually increased them in my case to
give services more time to exit (upstream default is 90s).

ShutdownWatchdogSec is rightfully set to 0 by default - having a hardware
watchdog reboot your system may not always be a good idea.

> _fix systemd journaling_

Debian runs a non-persistent runtime journal by default and leaves the long-
term storage responsibility with the syslog daemon. What's wrong with this? If
anything, they're being too conservative.

> _RuntimeDirectorySize_

What's wrong with the 10% of physical RAM default? Hard coding this to a small
value is potentially messing with things like flatpak's portals which store
large-ish data there.

> _KillUserProcesses_

Off by default in Debian. I specifically turn it on, since it kills any rogue
user processes when they log out and makes sure nobody leaves anything running
in a screen instead of doing it properly (I've even seen a crashed vim
instance max out CPU on a server)

> _mask off systemd-udev-settle ... yes it is broken_

Masking this introduces race conditions into your boot process. The systemd-
udev-settle unit calls "udevadm settle", which waits for all pending udev
actions to complete before any services are started which may depend on it. If
anything is broken, it's the particular subsystem which takes too long to
initialize/does not support async events.

Important examples are LVM and iSCSI. If you mask the settle unit, systemd
won't wait for all devices to be ready. This may or may not work and the safe
default is to wait.

~~~
hpcjoe
> Are you sure about that? To my knowledge, Debian 9 is still using the good
> old network scripts. I'm 100% about Debian 8 since that's why I use in
> production, with systemd-networkd nowhere to be seen.

The last job developed/supported Debian based high performance storage systems
in production, starting with Debian 7, then Debian 8, and Debian 9.

root@localdns:~# cat /etc/debian_version

8.10

root@localdns:~# locate systemd-networkd

/lib/systemd/systemd-networkd

/lib/systemd/systemd-networkd-wait-online

/lib/systemd/system/systemd-networkd-wait-online.service

/lib/systemd/system/systemd-networkd.service

/usr/share/man/man8/systemd-networkd-wait-online.8.gz

/usr/share/man/man8/systemd-networkd-wait-online.service.8.gz

/usr/share/man/man8/systemd-networkd.8.gz

/usr/share/man/man8/systemd-networkd.service.8.gz

Its there, but I disabled it in this case.

> Why are you decreasing the timeouts? I actually increased them in my case to
> give services more time to exit (upstream default is 90s).

This comes from long debugging/tracing sessions where we were trying to
understand why we wound up with effectively repeating timeouts.

[...]

On systemd-udev-settle, allowing it to run was actually the cause of a race
condition on start. The only way to fix this that I found was to disable this.
Once we did this, no more race condition. The system booted normally.

It sounds like your experience is significantly different than mine on this.
We built very large high performance computing and storage systems for many
users, where the things that people used simply had to work, without surprise.
Every line I showed came from a set of long and often painful debugging
sessions where we had to figure out what new change had broken a key bit of
software.

The RuntimeDirectorySize issue actually tickled another bug, and caused some
incredible paging problems when used with a specific software package. Enough
end users on a system (we had many) could launch an effective DoS against the
system by using these tmps.

Generally, my observation, and I've seen others make a similar set elsewhere,
is that the default and general distribution settings for systemd parameters
appear to be generally set for desktop users (single or very few users per
machine). Not for large shared resources.

~~~
someone12345
systemd-networkd is not enabled by default (even when its binaries might be
installed).

------
vgy7ujm
There is also Slackware!

~~~
Wildgoose
And Gentoo.

------
arca_vorago
I still think systemd was a NSA/Redhat backdoor in the midnight for Linux.
I've forced myself to deal with it because its everywhere, but my Spidey-
senses tingle everytime I find some new way its trying to takeover command of
a core function.

