Chopping apart the USB-Serial and TTY code in the kernel was relatively a piece of cake; but systemd threw up a wall, ironically! I got good help from the mailing list:
At first I was stumped: everything should work, but what the heck is killing my login sessions when I unplug the device? I went over the code to make absolutely sure nothing would be getting a hangup signal from the TTY or the like. Through kernel printk messages, I traced it to systemd, slapping my forehead. Basically, even though the tty session was being kept intact, it was the fact that the USB device went away that systemd killed the session.
Once I solved the systemd issues, everything worked great.
systemd: it just likes to kill things, at least in its default configuration.
That's why the systemd crew generates such great hatred.
In fairness, please note that Christian Rebischke is not one of the Debian system maintainers, who wrote some of the other replies and whom you will find listed on this Debian WWW page:
Nor is xe a systemd developer, apparently.
Unless filing bug reports against systemd makes xem part of "the systemd crew", which, considering the headline of this page, would be a huge double standard.
The one that the linked to email is a reply to.
The replies from other people (A. Borzenkov) seemed OK.
But then he seems to straddle the kernel/userspace line more than most other kernel devs i know of.
Does anyone know if there's any precedent for a UNIX init system to behave like this? In my knowledge there isn't.
Don't act surprised. Those asshats stomped over a whole lot of shit and everyone acts like they're the savior of the init (and a dozen other) systems.
I feel like every time I read about something with systemd someone is acting like there is a gun to their head to use it. If your distro of choice _chose_ to use systemd then your beef is with them forcing it on you...not systemd devs. They made an init system - your distro maintainers are the ones who decided it was the superior choice.
Third-party software, particularly from commercial vendors, typically uses Red Hat as the base system requirement.
So you are correct, no-one holds a gun to distro maintainers' heads and forces systemd. They just find that there's no path forward without either  committing resources to keep abreast of the changes that systemd is creating both directly and indirectly or  just adopting systemd.
Red Hat has won at this point through sheer persistence and clout, so most distros are going with option .
Personally I don't see what everyone is so upset about here. As pointed out by message 15, they've simply changed the default behavior and provided a long-opt setting to restore the old behavior if required; the hundreds or thousands of people who don't closely follow systemd will just be inconvenienced temporarily until they figure out wtf is going on, then they are back in business. And, maybe next time they'll pay closer attention.
Anyway I'm sure there will be a systemd-mux soon enough where you can declaratively define your long-running post-logout processes and register them via muxctl or some other existing sd-bus client.
I think most of us agree to this, and I haven't said anything we all don't already know. We just all disagree where that complexity tipping point is, and that's why some of us hate systemd and some of us don't see the problem.
(I'm not saying tmux is bad, I use it all the time. I just don't use it as a daemon manager -- If I understand correctly, they do provide a way to fix screen/tmux.)
Systemd generally tries to standardize how to declare common things about a process:
- should it be automatically restarted if it does? at what interval?
- where are its logs?
- how to tell if the process is alive or not?
- can we isolate the process? (tmp files)
- and now: should this process continue to run even if the user has logged out?
And yep, I'm an old schooler who just got tired of dealing with duct tape solutions.
The way we do development where I work is by having two people ssh into a server, and open a joint tmux session. These sessions generally last a week or more, and you generally log out of ssh daily.
Or even if you're not pairing, just being disconnected over lunch is a long enough time for SSH to disconnect. Finding ways to overcome that (besides changing the systemd config) are even more duck tape fixes.
Automatically terminating processes when a user logs out is a bad default. Sure, leave it as an option because there might be some people that would find it useful, but changing the default without a good reason is bad policy.
I've seen people run "nohup [program]" and logout, not even run it in a tmux. If systemd killed their program, I would be quite happy about it.
The problem with your sarcasm is that it actually matches current reality. This is how various programs are busily being rearchitected to operate, or indeed already have been rearchitected to operate.
Take gnome-terminal, for example. Nowadays, a gnome-terminal-server service runs under the per-user systemd service manager, and requests to open more terminals generated by the gnome-terminal program arrive at it over the per-user Desktop Bus. gnome-terminal is a Desktop Bus client.
The gnome-terminal-server service used to be Desktop Bus activated, rather than simply run as a per-user systemd service. Ironically, given the headline on this page, this too hit a problem with a systemd release turning things on a while back. systemd version 228, in fact. All of the terminal emulator processes and the shells and login sessions that they spawned ended up sharing a single 512 maximum threads limit.
And yes, gnome-terminal-server too suffered from the kills-screen-and-tmux problem:
Because that brings to mind a recent project i learned about that plans to stream video via dbus...
This actually describes a lot of my problems with Docker as well: I __hate__ that I have to use `docker ps`, etc. This is due to a weakness in traditional UNIX architectures. I ran HURD in a virtual machine a while back and it has the solution to this problem: translators.
I can't wait until they hit a 1.0 release of HURD.
My guess is that previous init systems didn't do this because it wasn't really possible before cgroups. I'm certain that this behavior would be desired on multi-user systems to ensure that logged out users don't leave old processes lying around.
What I find strange about systemd tying processes to an active session is that was something that was only needed in multiuser situations. The systemd advocates (and others) keep telling me that Linux isn't really multiuser anymore as justification for various incompatible changes. In the case of single user desktop (including netbook, tablet, etc) systems there's no need to kill off processes.
 if you count idling in Pine as "active"
It may have started as Poetterings way to handle some transient daemons on his laptop, but has since grown to cover containerized servers (hello CoreOS) and multiseat (effectively turning a single X86 computer into a desktop terminal server).
I sometimes wonder if the systemd people encourage the confusion about what systemd really is about, so that they can shortcircuit any debate by claiming their opposition is misunderstanding and therefore not fit to comment.
Maybe I'm completely misunderstanding things here, but when I log out, I expect all of my processes to be killed. When I'm logged off, and if another user logs on, it would be kind of weird if the system was still playing my music, etc, wouldn't it? If your processes don't die when you log off, then what exactly does it mean to "log off"?
Or maybe the issue is that it's the init system doing this, rather than some other component?
What it means to log off is to end the session you're in. A desktop session would end all the windows and systray-type stuff you're running, a terminal session would end the shell. but any processes you have explicitly daemonised or nohup'd should carry on.
Maybe you have a headless linux box that you want to set some command-line music player playing on, why should you need to keep the terminal open to keep the music playing?
If that behavior is desired, then it would make more sense to have the music player be a system service rather than a user service, in my opinion.
As a non-user of nohup, it seems like a bizarre way to achieve the desired permanence. I guess maybe it was originally conceived as a way to run a long command from a terminal, but then effectively 'log out' of the terminal to prevent anyone else with physical access from running a command as you? Under graphical environments, this is solved via "locking" the computer. Under non-graphical environments, I would personally find something functionally equivalent to `my_command && exit` (but in a way that doesn't allow other people to ctrl+c and then steal your session) more intuitive - i.e. run the command and then log off, rather than "log off, but keep this command running".
Or maybe I'm misunderstanding why things like `nohup` exist? In any case, I can agree that making significant breaking changes in this way is pretty irresponsible. But I'm still not sure it violates the rule of least surprise.
In my limited understanding, nohup was a very simple mechanism to achieve this. So to me, when you logoff or drop a connection, the system will send a HUP to all the processes in the tree belonging to that session. If the process doesn't accept the HUP, well then it doesn't want to die right now, which is what nohup allowed you to do by intercepting / wrapping the HUP. I assume screen worked on a similar principle and just ignores the HUP since it knows you may want to re-attach later.
Now it sounds like they are working on or have another way to achieve the same result in SystemD, I'm just very curious what the thoughts or reasoning is that this should be a thing.
Anyways, all I can say is that to me, changing this behaviour if I didn't see the release notes or this bug, would cause some very unexpected behaviour for me interacting with a SystemD system, where some of the basic mechanisms I use and encourage for change safety would no longer work. I suspect many others also use screen/nohup, and will be in for a big surprise.
I actually have some services (like Transmission) in /etc/systemd/system that are run on boot as arbitrary users, and they're completely detached from any user sessions (i.e. logging off will never kill them). It seems comparable to logging in, running `nohup transmission` and logging out, but without the weird "he logged out - how are his processes still running?" confusion.
Something like a daemon is basically meant to run in the background and never be directly interacted with. It's a different thing to tell a process "hold on; I'll be back later".
As an example: I've got some SBCs that are on all the time. It's useful to be able to pop on, pick up where I left off from, do some experiments, then log out, without having to set my environment back up every time I do it. There isn't really a better way to achieve the same goal.
I just find the whole session thinking an ill fit for unix, outside of a shorthand for a login process and the shell child process it has spawned.
Anything beyond that is just another process.
There was a case for the init system doing this when unix was young. But changing it now breaks convention, and compat with other distros, without good cause.
Let me explain where I come from. At times I write distributed systems that expect to be based in a non-specific unix.
Whenever I mess around in an area such as process control, I carefully read the relevant section of Stevens and my distro's own docs (usually linux and until now I have tried to avoid caring about choice in init system). And - in these phases - I write in C because I don't trust that runtimes like python won't have wacky defaults that make the system hard to reason about.
This issue is a groaner, because it suggests that mainstream linux is becoming harder to reason about during development and debugging. Instead of being able to think clearly and tell the computer what to do, you will be constantly looking over your shoulder for unnecessary vagueries of your local systemd configuration and whatever quirks have been introduced by the neckbeard who owns commit rights to its default behaviours.
This bug and the abrasive german's response in the ticket suggest to me that there is a point in the near future where people who care about developing against the unix api will Bozo-bit systemd-affected distros altogether. Which would be a shame for all the good citizens who have steadily done low ego work towards those distros.
> This reminds me of a character of windows-based development that I find exhausting: opaque layers of unnecessary complexity.
This is pretty much exactly how I feel about this whole thing. While designing NT, MS made a number of assumptions and decisions that were sensible from an engineering point of view back then. Especially if you're very much into OO (wich was the new rage back then). But from an administrator or system programmer point of view, these byzantine structures get more in the way than they help. On top of that, it makes integrating new ideas harder and you end up with layers and layers of API sediment. The people behind systemd are making very similar decisions and assumptions, deviating from the very principles of Unix that prevent this short sightedness.
> I log into multiple remote systems every day and connect to my ongoing tmux sessions in other countries.
In this case, it sounds to me that you aren't logging out. It sounds like you're effectively just suspending the connection to your session and then coming back to it later. i.e. you didn't SSH in and then run 'exit' from within your sesson, right? Or is that indeed the way it's done?
I'm not sure if you'd call it an init system (but I guess any collection of shell scripts is a traditional UNIX init system), but MIT Athena has worked this way for many years: when you log out of a graphical session, there's an attempt to kill all your processes, because the machine should be fully for the use of whoever next logs in. Otherwise people would do long-running computing on machines in the computer labs ("clusters") and slow down interactive use for the next person -- and that's the non-malicious case.
It used to be possible to get around this until some point in the late 2000s, when as part of unrelated rewriting, the logout script was set to just reboot the machine if it was unsure if it managed to clean everything up.
The systemd people knew that this impacted the likes of screen and tmux, and there are discussions of this going back to 2010.
Here's Jóhann B. Guðmundsson, erstwhile owner of the System 5 rc to systemd conversion project in Fedora, writing in July 2010, for example:
To me, I can't think of a single other operating system that works this way, even Windows. But I'm sure the pro-systemd supporters will "correct" my thinking.
Additional: "You can just disable it, don't worry about the defaults" will be another argument that comes up, seems to be a nice way of stopping discussion.. it's the same thing people espoused when binary logging/journalctl was enforced.
This is crazytown.
Edit: I guess I should take the massive amount of downvotes to mean that it is in fact not crazytown to write service and unit files for one-off downloads or backgrounded TTY clients, and anyone asking for more information on this does not deserve a reply. Ok.
 - https://github.com/systemd/systemd/blob/master/NEWS#L29
Sounds like a lot of pitfalls with these new defaults. Wonder how many background tasks will be ruined before somebody patches screen to automatically issue all these new systemd commands.
 - https://wiki.archlinux.org/index.php/Systemd/User#Automatic_...
Windows has stopped all the programs I had ran when I log out for as long as I remember.
But then again, the closest equivalent I can think of to an SSH session is Enter-PSSession, which does close all processes started under it when closing the session with Exit-PSSession.
For an ssh session on Linux, the traditional equivalent would be to run tmux/screen in the background, but systemd will happily kill them.
Speaking of which, if you run an Xvnc server in the background out of your shell and log out, this systemd problem will now kill it.
So, like this one: https://bugzilla.redhat.com/show_bug.cgi?id=753882#c29
On the upside you can use loginctl with lingering to keep the processes after the user logs out.
Abandoning these foundations in order to optimize towards a very structured, hierarchical engineering ideal, will make Linux go more in the direction of BeOS or Windows. And I'm afraid that we don't have the resources that Microsoft uses to keep Windows competitive and backwards compatible.
I'm just waiting for them to reimplement coreutils and sh as systemd binaries (they did have plans for a user space tty implementation, iirc).
For those to young to remember (or care :-) Windows 3.1 was the commercially successful version of Windows where all of the "user experience" had been abstracted into the Windows libraries, leaving the MS-DOS kernel underneath for legacy APIs and driver support. The DOS kernel, once the "center of attention" became irrelevant.
If the systemd folks can get everyone to write applications to the systemd APIs, then the Linux kernel people are just plumbers who you can yell at if they don't support the latest networking chip or USB gizmo, but aren't really contributing things to the system that users see.
Its brilliant in a way, and the more I think about it the more plausible it seems.
Upstart was a reasonable replacement for system V I reckon, ...
I don't like its effective ownership by redhat and its monolithic nature however this can be solved in the long run by forks and standardisation.
I keep hearing this refrain, but the reality of the situation right now is that Systemd (and the other processes under the same development umbrella) are moving too quickly and are too complex for any fork to ever have a realistic chance.
Does there exist a group of people outside of RedHat who understand the internals of the whole thing well enough to even chance a fork?
Example: I love service files.
If they would just make everything modular/optional, I wouldnt hate it nearly as much
Do you currently have a need for a piece of systemd that would work better as a standalone component?
If you're hosting a bunch of apps for clients, but not all of them are being used all the time, then you can fit more onto one server by only activating them when they're used.
I know Pantheon does this, for example. https://pantheon.io/open-source
There has also been some discussion of integrating this functionality into kubernetes: https://github.com/kubernetes/kubernetes/issues/484
Used in the "olden" days to allow a single server to serve a bunch of services, not all of which are being used all of the time, to a bunch of clients, by only activating them when they are used.
This can work for simple services like sshd which fork for each connection anyway, but would never work for something like nginx or redis.
I haven't seen the code that does this, but my hope is that it is perfect in every way - if not then its an open door from the internet into my server.
But I don't really see what this sort of feature has to do in an init process. I really dislike the idea of having one project handling everything on a box .... The day you have a security issue in systemd we are all fu*.
But yeah there are some real good things in systemd, maybe they should just be taken out of it and spin up in a new project ;)
I've heard this argument a bunch, but looking at the actual systemd approach doesn't really convince me that it's the case. There are dozens of binaries, dozens of maintainers, hundreds of contributors. It's comparable to a project with a bunch of parts (think Gnome or KDE only at the systems-level), rather than one big black box that does everything.
If there were a single "systemd" program that did all the hundreds of things that the systemd system does, it would be alarming and impossible to maintain. But, it's not. They may be more tightly coupled than some like (I quite like systemd and still find it a little uncomfortably coupled in many places), but that's partially a symptom of doing a bunch of things in new ways. No one built the interfaces needed to do what systemd does in the past, so building systemd took reinventing the universe in some regards. That's actually some of what you're getting when you get "systemd"; things that aren't part of the init system, but existed in some form before and are now being replaced by systemd-provided services.
"But yeah there are some real good things in systemd, maybe they should just be taken out of it and spin up in a new project ;)"
That becomes more feasible over time. Right now, there's a lot of moving parts. Another reason for the tight coupling is that the way things work together is still evolving as people learn the shortcomings of this new architecture.
systemd is not perfect, but it is a bunch of little pieces working together. A security issue in one of those pieces doesn't necessarily mean we're all fucked any more than a security issue in one part of any of the old systems necessarily meant we were all fucked.
I'm surprised there's still so much rage about systemd. It's not a bad solution to the problem. I mean, hell, just the move from procedural to declarative for service configuration is huge. The size of that improvement really cannot be overstated.
You can only be surprised if you've got blinders on. Systemd has some real technical superiority but also some questionable or worse technical decisions. Beyond that, it has a truly horrible community around it that has an incredible track record for being bad at justifying shaking things up and bad at easing migration pains and bad at responding to valid criticisms or accommodating use cases different from their own. A project can't trigger flamewars this reliably and be innocent in the matter. You don't see the same kind of social and organizational conflict surrounding the major Linux display stack overhauls over the past decade.
Look at the kernel, where larger charges are developed and tested outside the main tree, and then applies for inclusion.
Never mind Torvalds' mantra about not breaking user space. Sadly user space seems highly adept at breaking itself repeatedly...
Didn't you hear? It hangs up processes which have been run with nohup!
For more details about the advantages of this can be found here and here.
 - http://0pointer.de/blog/projects/socket-activation.html
 - http://0pointer.de/blog/projects/inetd.html
Whatever you do, don't switch to a BSD.
A pro for systemd doing it is maybe consistent nomenclature for service startup, but a con is that it couples in yet another use case to systemd which may lead to a higher risk of security, maintainability problems down the line.
There have been init bugs in the past. There have been xinetd bugs in the past (and inetd was occasionally notorious with regard to security, though it was usually the services it provided access to rather than inetd itself). There will be bugs in systemd. But, how does consolidating service startup into one project increase the risk?
Being able to assure by default that all of a user's spawned processes are terminated when the user logs out is a much more secure default behavior than the opposite. This means if I disable a user in LDAP and force the user to log out, I don't have to worry about lingering processes that might be malicious or pose some sort of threat to my server or data.
Keep in mind that there is a switch in logind to enable lingering that you can enable if you don't like it.
Besides, the release notes provided the workaround in line, so that interested users wouldn't have to hunt for a solution.
Only if the interested user is astute enough to put systemd at the top of the suspect list for why screen and tmux stopped working.
I'm only half joking.
When the user has explicitly requested that processes not terminate when he hangs up by using the nohup command, that is precisely what not to do. This is fundamentally broken behaviour.
> Besides, the release notes provided the workaround in line, so that interested users wouldn't have to hunt for a solution.
In honour of Towel Day:
Mr Prosser: But, Mr Dent, the plans have been available
in the local planning office for the last
Arthur: Oh yes, well as soon as I heard I went
straight round to see them, yesterday
afternoon. You hadn’t exactly gone out of
your way to call attention to them had you?
I mean like actually telling anybody or
Mr Prosser: But the plans were on display…
Arthur: On display? I eventually had to go down to
the cellar to find them.
Mr Prosser: That’s the display department.
Arthur: With a torch.
Mr Prosser: The lights had probably gone out.
Arthur: So had the stairs.
Mr Prosser: But look, you found the notice, didn’t you?
Arthur: Yes yes I did. It was on display at the
bottom of a locked filing cabinet stuck in
a disused lavatory with a sign on the door
saying beware of the leopard.
http://unix.stackexchange.com/a/136552/34801 it's 2 lines.
It's also fine when systemd allows it as an easy knob and cgroups might catch something that pam doesn't (but you could also use cgroups in the pam script and cleanup/atd/cron or systemd timers).
But doing it by default behavior is kind of surprising.
The Hitchhiker's Guide to the Galaxy approach to announcements ;)
But it was announced on the CHANGES file on that github repository works really well for breaking POSIX.
Mind you, I have also started investigating the BSD family seriously (rather than just to have a play with), just in case.
You should quote the full changelog and not just
the part that is 'bad' in your mind.
>systemd-logind will now by default terminate user processes that are part of
>the user session scope unit (session-XX.scope) when the user logs out. This
>behavior is controlled by the KillUserProcesses= setting in logind.conf, and
>the previous default of "no" is now changed to "yes".
For debian it would be enough to set this to "no" again with
--without-kill-user-processes option to "configure"
>This means that user sessions will be properly cleaned up after, but
>additional steps are necessary to allow intentionally long-running processes
>to survive logout.
Here comes the important part. Seems like the systemd-devs are working on a
way to allow intentionally long-running processes in a specific user scope.
And here is another way for allowing these long-running processes:
>While the user is logged in at least once, user@.service is running, and any
>service that should survive the end of any individual login session can be
>started at a user service or scope using systemd-run. systemd-run(1) man
>page has been extended with an example which shows how to run screen in a
>scope unit underneath user@.service. The same command works for tmux.
And another way for allowing long-running processes.
>After the user logs out of all sessions, user@.service will be terminated
>too, by default, unless the user has "lingering" enabled. To effectively
>allow users to run long-term tasks even if they are logged out, lingering
>must be enabled for them. See loginctl(1) for details. The default polkit
>policy was modified to allow users to set lingering for themselves without
>Previous defaults can be restored at compile time by the
>--without-kill-user-processes option to "configure"
You see? No reason to complain about.
tl;dr: You can configure it not to, or you can accept that Linux is about as much a "Unix" as AIX, Irix, or HP-UX and join them in the wonderful land of the future, which suspiciously resembles the time before all the non-standard vendors died.
I'll stop now, before I make Linus on a backwards compatibility rant look like a choir boy in church.
Now he's out on Twitter describing the bug as a "hate-report".
With any hope, a future employer will find this, put it in front of him, and ask if that's the kind of behavior that should be expected of him. Before turning him around and out the door.
"Oh, first you have to talk to the machine's administrator to reconfigure systemd, if you're on Linux. Or just use FreeBSD, MacOS, Solaris, or ... Cygwin".
Or not, because windows will kill all processes that are part of the users session when they log out.
Ah-HA! Now we know what inspired this change.
Is Poettering financed by Microsoft, or is he just a fan of their ease-of-use, high security and all-round wonderfulness?
The nohup utility is standardized by POSIX: Look here:
For a GUI login, I'd expect systemd to do exactly this. Right now, wayward processes ignore or just don't grok whatever "quit" notification the DE sends, and this will cause delayed restart/shutdown for 1m30s at which point systemd kills the user session. 1m30s is too long.
Whereas on GUIless server, where I use tmux, the idea I'd login, run tmux and create a session or two, detach, logout, and by logging out all my tmux sessions go poof is not expected. But even in the cited thread there's a mechanism for such processes to get a separate user session.
Processes have always done this when they receive a hangup (HUP) signal (unless they do something else, of course, but that's up to them).
> But even in the cited thread there's a mechanism for such processes to get a separate user session.
That mechanism has existed for decades: it's called nohup.
It's a great system in theory and I am excited about its potential, but it has some extremely egotistical and irresponsible developers who don't give a damn about what they break, as long as they can keep pushing code out without community testing or review to slow them down. Debian was flat out wrong to go with systemd until it reaches a certain level of maturity and stability, but there's no going back now.
This is a bug report in debian unstable against a bleeding edge version of systemd released 5 days ago.
And it's having a negative effect on their own pet distro as well, see this thread:
You act like every single systemd user on the planet upgraded to the bleeding edge version of systemd and had this configuration option changed on them overnight.
FWIW "HandleLidSwitch=ignore" is another back-to-normal that will probably save you some walking.
there's an easy sollution for processes that don't need to be stopped too.
The processes started under screen(1) are deliberately created with a new pty as controlling tty and in a new session, because they are intended to survive the ending of the original session where screen was started.
processes deliberatelly started with screen can be started with systemd-run screen.