
Systemd v230 kills background processes after user logs out, breaks screen, tmux - polemic
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394
======
kazinator
Several years ago I was developing a "robust serial console over USB" feature
for Linux. THis feature allows you to have a console on, say /dev/ttyUSB0. The
device doesn't have to be plugged in. When you plug in the USB-Serial dongle,
you have a console. You can unplug your serial dongle right in the middle of a
session, plug in one that uses a _different chip_ , and your session is
intact! Just "Ctrl-L" to refresh your Vim window where you were editing and
there you are. The /dev/ttyUSB0 did actually disappear when unplugged and did
re-appear on the re-insertion, but the TTY session was isolated from that,
held in a suspended state.

Chopping apart the USB-Serial and TTY code in the kernel was relatively a
piece of cake; but systemd threw up a wall, ironically! I got good help from
the mailing list:

[https://lists.freedesktop.org/archives/systemd-
devel/2013-Ma...](https://lists.freedesktop.org/archives/systemd-
devel/2013-May/011011.html)

At first I was stumped: everything should work, but what the heck is killing
my login sessions when I unplug the device? I went over the code to make
absolutely sure nothing would be getting a hangup signal from the TTY or the
like. Through kernel printk messages, I traced it to systemd, slapping my
forehead. Basically, even though the tty session was being kept intact, it was
the fact that the USB device went away that systemd killed the session.

Once I solved the systemd issues, everything worked great.

systemd: _it just likes to kill things_ , at least in its default
configuration.

~~~
0x0
I can't believe the attitude you received in some of those replies :-/

~~~
JdeBP
I suspect that you're all talking about the reply from Christian Rebischke,
who is indeed now talking about a "hate-report" and "a reason to cry".

* [https://twitter.com/Sh1bumi/status/735973258793779200](https://twitter.com/Sh1bumi/status/735973258793779200)

In fairness, please note that Christian Rebischke _is not_ one of the Debian
system maintainers, who wrote some of the other replies and whom you will find
listed on this Debian WWW page:

* [https://packages.qa.debian.org/s/systemd.html](https://packages.qa.debian.org/s/systemd.html)

Nor is xe a systemd developer, apparently.

* [https://github.com/systemd/systemd/commits/master?author=shi...](https://github.com/systemd/systemd/commits/master?author=shibumi)

Unless filing bug reports against systemd makes xem part of "the systemd
crew", which, considering the headline of this page, would be a huge double
standard.

* [https://github.com/systemd/systemd/issues?q=shibumi](https://github.com/systemd/systemd/issues?q=shibumi)

~~~
dkuntz2
I don't think that's related at all. From what I can tell, Christian isn't
involved in the GP's thread about systemd and usb serial communications.

~~~
JdeBP
Gah! You're right. I thought that we were still talking about the headline on
this page.

------
madmax96
This violates the rule of least surprise. Honestly, it looks like the Systemd
maintainers operated without regards to the community at all. There's probably
going to be some commenter who says "it only is surprising to __you __" , but
as evident by the email, it's surprising to a lot of users.

Does anyone know if there's any precedent for a UNIX init system to behave
like this? In my knowledge there isn't.

~~~
wallacoloo
> This violates the rule of least surprise.

Maybe I'm completely misunderstanding things here, but when I log out, I
_expect_ all of my processes to be killed. When I'm logged off, and if another
user logs on, it would be kind of weird if the system was still playing my
music, etc, wouldn't it? If your processes don't die when you log off, then
what exactly does it _mean_ to "log off"?

Or maybe the issue is that it's the init system doing this, rather than some
other component?

~~~
Nursie
No, if I background and nohup something, it ought to stay there until or
unless I kill it or it exits.

What it means to log off is to end the session you're in. A desktop session
would end all the windows and systray-type stuff you're running, a terminal
session would end the shell. but any processes you have explicitly daemonised
or nohup'd should carry on.

Maybe you have a headless linux box that you want to set some command-line
music player playing on, why should you need to keep the terminal open to keep
the music playing?

~~~
wallacoloo
> Maybe you have a headless linux box that you want to set some command-line
> music player playing on, why should you need to keep the terminal open to
> keep the music playing?

If that behavior is desired, then it would make more sense to have the music
player be a _system_ service rather than a user service, in my opinion.

As a non-user of nohup, it seems like a bizarre way to achieve the desired
permanence. I guess maybe it was originally conceived as a way to run a long
command from a terminal, but then effectively 'log out' of the terminal to
prevent anyone else with physical access from running a command as you? Under
graphical environments, this is solved via "locking" the computer. Under non-
graphical environments, I would personally find something functionally
equivalent to `my_command && exit` (but in a way that doesn't allow other
people to ctrl+c and then steal your session) more intuitive - i.e. run the
command _and then log off_ , rather than "log off, but keep this command
running".

Or maybe I'm misunderstanding why things like `nohup` exist? In any case, I
can agree that making significant breaking changes in this way is pretty
irresponsible. But I'm still not sure it violates the rule of least surprise.

~~~
ams6110
nohup dates way back, most unix computers were not owned by a single person
but were used by dozens if not hundreds of users. People had needs to run jobs
(even compiles could take hours) that they would not want terminated if their
dial-up connection dropped (common occurrence). Among other reasons.

~~~
kevin_nisbet
Agreed, at work we use nohup / screen for many important tasks. These tasks
are items like doing an upgrade, or migration, or a long running script that
you want to protect from a dropped ssh connection. Networks are unreliable,
your local system may be unreliable, and the last thing we want to happen is
for a script or program to die while making changes to the system that leave
it in an inconsistent state.

In my limited understanding, nohup was a very simple mechanism to achieve
this. So to me, when you logoff or drop a connection, the system will send a
HUP to all the processes in the tree belonging to that session. If the process
doesn't accept the HUP, well then it doesn't want to die right now, which is
what nohup allowed you to do by intercepting / wrapping the HUP. I assume
screen worked on a similar principle and just ignores the HUP since it knows
you may want to re-attach later.

Now it sounds like they are working on or have another way to achieve the same
result in SystemD, I'm just very curious what the thoughts or reasoning is
that this should be a thing.

Anyways, all I can say is that to me, changing this behaviour if I didn't see
the release notes or this bug, would cause some very unexpected behaviour for
me interacting with a SystemD system, where some of the basic mechanisms I use
and encourage for change safety would no longer work. I suspect many others
also use screen/nohup, and will be in for a big surprise.

------
dijit
This came up about a week ago, I questioned why it was a useful feature and
people were quick to bark "security, if you don't like it you should be
writing service/unit files for your programs."

To me, I can't think of a single other operating system that works this way,
even Windows. But I'm sure the pro-systemd supporters will "correct" my
thinking.

Additional: "You can just disable it, don't worry about the defaults" will be
another argument that comes up, seems to be a nice way of stopping
discussion.. it's the same thing people espoused when binary
logging/journalctl was enforced.

~~~
0x0
Can you write service/unit files for irc clients like irssi, or one-shot file
transfers like rsync or wget?

This is crazytown.

Edit: I guess I should take the massive amount of downvotes to mean that it is
in fact not crazytown to write service and unit files for one-off downloads or
backgrounded TTY clients, and anyone asking for more information on this does
not deserve a reply. Ok.

~~~
eeperson
You wouldn't have to write service/unit files for one off commands. The
release notes mention that you can use 'systemd-run' to start these[1].

[1] -
[https://github.com/systemd/systemd/blob/master/NEWS#L29](https://github.com/systemd/systemd/blob/master/NEWS#L29)

~~~
0x0
Looks like that's not enough, you also need to set some sort of "lingering"
option or the last logout will kill everything despite having used systemd-
run?

Sounds like a lot of pitfalls with these new defaults. Wonder how many
background tasks will be ruined before somebody patches screen to
automatically issue all these new systemd commands.

~~~
eeperson
I don't think the 'lingering' option is necessary. That controls the default
behavior for processes. Systemd-run basically makes a one off user level
service.

~~~
eeperson
Looks like I was wrong. You do have to enable lingering[1].

[1] -
[https://wiki.archlinux.org/index.php/Systemd/User#Automatic_...](https://wiki.archlinux.org/index.php/Systemd/User#Automatic_start-
up_of_systemd_user_instances)

------
TheGuyWhoCodes
WOW. That will break/surprise a lot of people unless the distros disable it at
compile time. I use tmux regularly to start long running jobs.

On the upside you can use loginctl with lingering to keep the processes after
the user logs out.

~~~
codehusker
This seems to be a common story. Systemd breaks one thing, but it's okay, they
have their own built-in thing.

~~~
toyg
Or rather, systemd breaks _an entire class of things_ , but it's okay, they
have _one and only one_ built-in thing.

~~~
misnome
It seems like the attitude "If you don't closely follow systemd development
then you are an idiot and deserve what you get" is the most common. You
shouldn't expect thousands and thousands of users to have to reconfigure to
maintain existing, standard behaviour.

------
AnonymousPlanet
This is what you get if you have people in charge of developing an init system
who never did any professional admin work and who apparently have a disdain
for everything Unix stands for.

~~~
IshKebab
Maybe they want to improve Unix? This is undeniably the _correct_ behaviour.
It's only bad because everyone has gotten used to how things worked in the
70s.

~~~
AnonymousPlanet
I believe we have to discern between Unix as a concrete implementation (what
is a process, where do certain files go, etc.) and the design philosophy. My
remark focused on the latter. I would very much like to see modernization in
Unix as a technology. At the same time, I believe that Linux owes its success
to the adherence to a number of Unix design philosophies. This has kept Linux
flexible across a wide spectrum of use cases and prevented it from running
into a dead end, becoming stale.

Abandoning these foundations in order to optimize towards a very structured,
hierarchical engineering ideal, will make Linux go more in the direction of
BeOS or Windows. And I'm afraid that we don't have the resources that
Microsoft uses to keep Windows competitive and backwards compatible.

------
ChuckMcM
This feels like another land grab for the user experience by systemd. They
have successfully screwed up a lot of sysadmin's habits and now they are going
after the everyday users. Sigh.

~~~
digi_owl
Pretty much. More and more it feels like a bunch of disgruntled ex kernel devs
have found a way to circumvent Torvalds, basically using Linux as a source of
drivers and base plumbing.

I'm just waiting for them to reimplement coreutils and sh as systemd binaries
(they did have plans for a user space tty implementation, iirc).

~~~
ChuckMcM
To be honest I had not considered that angle. If you were going to write the
equivalent of "Windows 3.1" for Linux how would you start?

For those to young to remember (or care :-) Windows 3.1 was the commercially
successful version of Windows where all of the "user experience" had been
abstracted into the Windows libraries, leaving the MS-DOS kernel underneath
for legacy APIs and driver support. The DOS kernel, once the "center of
attention" became irrelevant.

If the systemd folks can get everyone to write applications to the systemd
APIs, then the Linux kernel people are just plumbers who you can yell at if
they don't support the latest networking chip or USB gizmo, but aren't really
contributing things to the system that users see.

Its brilliant in a way, and the more I think about it the more plausible it
seems.

~~~
digi_owl
Well it is in large part what Android is doing already. Most apps there live
inside the Android JVM after all.

------
jeanpralo
I still can't find a good reasons why you should use systemd, this things
grows out of control and is way out of the initial scope.

Upstart was a reasonable replacement for system V I reckon, ...

~~~
chatmasta
Socket activation is pretty cool.

[http://0pointer.de/blog/projects/socket-activated-
containers...](http://0pointer.de/blog/projects/socket-activated-
containers.html)

If you're hosting a bunch of apps for clients, but not all of them are being
used all the time, then you can fit more onto one server by only activating
them when they're used.

I know Pantheon does this, for example. [https://pantheon.io/open-
source](https://pantheon.io/open-source)

There has also been some discussion of integrating this functionality into
kubernetes:
[https://github.com/kubernetes/kubernetes/issues/484](https://github.com/kubernetes/kubernetes/issues/484)

~~~
jeanpralo
Thats is indeed pretty cool, the whole question is more under the way
decisions are beeing taken regardless of what the community says.

But I don't really see what this sort of feature has to do in an init process.
I really dislike the idea of having one project handling everything on a box
.... The day you have a security issue in systemd we are all fu __*.

But yeah there are some real good things in systemd, maybe they should just be
taken out of it and spin up in a new project ;)

~~~
SwellJoe
_" I really dislike the idea of having one project handling everything on a
box .... The day you have a security issue in systemd we are all fu...."_

I've heard this argument a bunch, but looking at the actual systemd approach
doesn't really convince me that it's the case. There are dozens of binaries,
dozens of maintainers, hundreds of contributors. It's comparable to a project
with a bunch of parts (think Gnome or KDE only at the systems-level), rather
than one big black box that does everything.

If there were a single "systemd" program that did all the hundreds of things
that the systemd system does, it would be alarming and impossible to maintain.
But, it's not. They may be more tightly coupled than some like (I quite like
systemd and still find it a little uncomfortably coupled in many places), but
that's partially a symptom of doing a bunch of things in new ways. No one
built the interfaces needed to do what systemd does in the past, so building
systemd took reinventing the universe in some regards. That's actually some of
what you're getting when you get "systemd"; things that aren't part of the
init system, but existed in some form before and are now being replaced by
systemd-provided services.

 _" But yeah there are some real good things in systemd, maybe they should
just be taken out of it and spin up in a new project ;)"_

That becomes more feasible over time. Right now, there's a lot of moving
parts. Another reason for the tight coupling is that the way things work
together is still evolving as people learn the shortcomings of this new
architecture.

systemd is not perfect, but it _is_ a bunch of little pieces working together.
A security issue in one of those pieces doesn't necessarily mean we're all
fucked any more than a security issue in one part of any of the old systems
necessarily meant we were all fucked.

~~~
digi_owl
Then why is it then being pressed into service across distros?! Why not let it
churn on in its own world somewhere out of the way until it settles down and
the interfaces are more defined?

~~~
SwellJoe
Because it would never fit a real world use case, if isn't designed in and for
the real world. Software built in a secret lab and trotted out to people after
the fact is not the way Open Source works. We build the plane as it flies.
It's always been that way in Linux.

I'm surprised there's still so much rage about systemd. It's not a bad
solution to the problem. I mean, hell, just the move from procedural to
declarative for service configuration is _huge_. The size of that improvement
really cannot be overstated.

~~~
wtallis
> "I'm surprised there's still so much rage about systemd. It's not a bad
> solution to the problem."

You can only be surprised if you've got blinders on. Systemd has some real
technical superiority but also some questionable or worse technical decisions.
Beyond that, it has a truly horrible community around it that has an
incredible track record for being bad at justifying shaking things up and bad
at easing migration pains and bad at responding to valid criticisms or
accommodating use cases different from their own. A project can't trigger
flamewars this reliably and be innocent in the matter. You don't see the same
kind of social and organizational conflict surrounding the major Linux display
stack overhauls over the past decade.

~~~
SwellJoe
GPL vs BSD has triggered flamewars reliably and for decades. Which one is
guilty?

------
otterley
I think this is actually a useful change in default behavior for server
environments, which comprise the vast majority of Linux installations.

Being able to assure by default that all of a user's spawned processes are
terminated when the user logs out is a much more secure default behavior than
the opposite. This means if I disable a user in LDAP and force the user to log
out, I don't have to worry about lingering processes that might be malicious
or pose some sort of threat to my server or data.

Keep in mind that there is a switch in logind to enable lingering that you can
enable if you don't like it.

Besides, the release notes provided the workaround in line, so that interested
users wouldn't have to hunt for a solution.

~~~
wtallis
> Besides, the release notes provided the workaround in line, so that
> interested users wouldn't have to hunt for a solution.

Only if the interested user is astute enough to put systemd at the top of the
suspect list for why screen and tmux stopped working.

~~~
otterley
Individual users aren't the target audience, though. System integrators are.
Your distribution maintainer is responsible for reading the change log and
making the necessary decisions with respect to default configuration,
notifying users, etc.

------
dkarapetyan
Time to switch to one of the BSDs. I hear they're a better unix than unix and
these systemd clowns are out of control.

~~~
anonbanker
try Calculate Linux first before going full BSD. you'll thank me.

~~~
dkarapetyan
Will do.

------
mcguire
The response to the bug report:

Hello, You should quote the full changelog and not just the part that is 'bad'
in your mind.

>systemd-logind will now by default terminate user processes that are part of
>the user session scope unit (session-XX.scope) when the user logs out. This
>behavior is controlled by the KillUserProcesses= setting in logind.conf, and
>the previous default of "no" is now changed to "yes".

For debian it would be enough to set this to "no" again with \--without-kill-
user-processes option to "configure"

>This means that user sessions will be properly cleaned up after, but
>additional steps are necessary to allow intentionally long-running processes
>to survive logout.

Here comes the important part. Seems like the systemd-devs are working on a
way to allow intentionally long-running processes in a specific user scope.

And here is another way for allowing these long-running processes:

>While the user is logged in at least once, user@.service is running, and any
>service that should survive the end of any individual login session can be
>started at a user service or scope using systemd-run. systemd-run(1) man
>page has been extended with an example which shows how to run screen in a
>scope unit underneath user@.service. The same command works for tmux.

And another way for allowing long-running processes.

>After the user logs out of all sessions, user@.service will be terminated
>too, by default, unless the user has "lingering" enabled. To effectively
>allow users to run long-term tasks even if they are logged out, lingering
>must be enabled for them. See loginctl(1) for details. The default polkit
>policy was modified to allow users to set lingering for themselves without
>authentication. > >Previous defaults can be restored at compile time by the
>\--without-kill-user-processes option to "configure"

You see? No reason to complain about.

Best regards

Christian Rebischke.

 _tl;dr_ : You can configure it not to, or you can accept that Linux is about
as much a "Unix" as AIX, Irix, or HP-UX and join them in the wonderful land of
the future, which suspiciously resembles the time before all the non-standard
vendors died.

I'll stop now, before I make Linus on a backwards compatibility rant look like
a choir boy in church.

~~~
TwoNineFive
Wow. Christian Rebischke's response to this reasonable, mild-mannered, bug
report, is unbelievably hostile.

Now he's out on Twitter describing the bug as a "hate-report".

With any hope, a future employer will find this, put it in front of him, and
ask if that's the kind of behavior that should be expected of him. Before
turning him around and out the door.

------
tbyehl
So if I set /etc/systemd/logind.conf KillUserProcesses=no the current behavior
will be preserved whenever this change finds its way downstream?

------
kazinator
Even "nohup" programs?

~~~
mcguire
Apparently, according to the third message in the report.

~~~
kazinator
That's treading on hallowed grounds man. How can you teach newbies to use
nohup now?

"Oh, first you have to talk to the machine's administrator to reconfigure
systemd, if you're on Linux. Or just use FreeBSD, MacOS, Solaris, or ...
Cygwin".

~~~
justinsaccount
> or ... Cygwin

Or not, because windows will kill all processes that are part of the users
session when they log out.

~~~
wtbob
> Or not, because windows will kill all processes that are part of the users
> session when they log out.

Ah-HA! Now we know what inspired this change.

Is Poettering financed by Microsoft, or is he just a fan of their ease-of-use,
high security and all-round wonderfulness?

------
cmurf
As far as I've experienced on OS X, any running process does not survive
logout. Granted it's a single session, and that session is also GUI.

For a GUI login, I'd expect systemd to do exactly this. Right now, wayward
processes ignore or just don't grok whatever "quit" notification the DE sends,
and this will cause delayed restart/shutdown for 1m30s at which point systemd
kills the user session. 1m30s is too long.

Whereas on GUIless server, where I use tmux, the idea I'd login, run tmux and
create a session or two, detach, logout, and by logging out all my tmux
sessions go poof is not expected. But even in the cited thread there's a
mechanism for such processes to get a separate user session.

~~~
wtbob
> For a GUI login, I'd expect systemd to do exactly this.

Processes have always done this when they receive a hangup (HUP) signal
(unless they do something else, of course, but that's up to them).

> But even in the cited thread there's a mechanism for such processes to get a
> separate user session.

That mechanism has existed for decades: it's called nohup.

------
shirro
I like systemd. It solves a lot of problems and does some cool things. But it
always worries me when they go around breaking stuff. Too often the response
is something like tmux has been doing it wrong - use the new systemd-tmux
service instead.

------
digi_owl
And now tmux is asked to accommodate the change.

[https://github.com/tmux/tmux/issues/428](https://github.com/tmux/tmux/issues/428)

------
d3ckard
Well, I guess that is a useful feature for init system.

~~~
otterley
It's not a feature of the init system. It's a feature of the login daemon that
ships with it.

~~~
digi_owl
Potato, potato...

------
Scarbutt
I wonder if the debian folks feel regrets about settling with systemd.

~~~
iso-8859-1
Debian will disable this if they want.

~~~
morganvachon
But they shouldn't have to. Changing a default, known, long used feature of
UNIX-like OSes without any sort of announcement or debate is dangerous and
irresponsible. This kind of thing is why I feel the encroachment of _systemd_
into most Linux distros is a bad thing.

It's a great system in theory and I am excited about its potential, but it has
some extremely egotistical and irresponsible developers who don't give a damn
about what they break, as long as they can keep pushing code out without
community testing or review to slow them down. Debian was flat out wrong to go
with _systemd_ until it reaches a certain level of maturity and stability, but
there's no going back now.

~~~
justinsaccount
> without community testing or review

This is a bug report in debian unstable against a bleeding edge version of
systemd released 5 days ago.

~~~
morganvachon
They pushed this "feature" without any advance notice to developers of
userland apps like screen and tmux, leaving them to pick up the pieces and
figure out what happened. That is irresponsible and reckless. Common sense
dictates that when you're going to introduce severe breakage, you give advance
warning so it can be properly tested and dealt with.

And it's having a negative effect on their own pet distro as well, see this
thread:

[https://lists.fedoraproject.org/archives/list/devel@lists.fe...](https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/ZNQW72UP36UAFMX53HPFFQTWTQDZVJ3M/)

~~~
justinsaccount
Oh, and here is your "advanced notice"

[http://tmux-users.narkive.com/LXp72CHV/pam-support-in-tmux](http://tmux-
users.narkive.com/LXp72CHV/pam-support-in-tmux)

~~~
morganvachon
That was notice that it would be a compile time option, not set to the
unexpected and harmful setting as default.

------
gargravarr
I am certainly not surprised by this change. I'm just glad I switched to
Devuan a few months ago. I don't read every changelog, but I would have been
furious if I'd discovered my screen session gone. Having this enabled by
default is moronic and shows real short-sightedness on the part of the Debian
packagers.

------
mrmondo
To be fair, Debian Jessie is not exactly the most stunning of Debian releases
over the past decade either. Between the missing and ancient packages we ended
up having to move to CentOS 7 w/ epel+elrepo.

~~~
rincebrain
Which packages were you finding "missing" or "ancient?"

~~~
mrmondo
Corosync, Pacemaker, OCF, they're off the top of my head but we found a LOT
when looking at upgrading from Wheezy.

------
vxxzy
What about user-specific CRON jobs? Does this prevent them from running?

------
digi_owl
I noticed the entry for this in their release note, and my first thought was
that this would catch many off guard...

------
gdamjan1
This is a good feature, and a good default. Processes in the session-xx.scope
should be stopped when the session ends.

there's an easy sollution for processes that don't need to be stopped too.

~~~
caf
No, there was _already_ a mechanism in place to end processes when you log out
- when the controlling tty disappears, every process in the session is sent a
SIGHUP and SIGCONT signal.

The processes started under screen(1) are deliberately created with a new pty
as controlling tty and in a new session, because _they are intended to survive
the ending of the original session where screen was started_.

~~~
gdamjan1
not for processes that don't have a controlling tty.

processes deliberatelly started with screen can be started with systemd-run
screen.

