Hacker News new | past | comments | ask | show | jobs | submit login
Systemd v230 kills background processes after user logs out, breaks screen, tmux (debian.org)
152 points by polemic on May 26, 2016 | hide | past | web | favorite | 182 comments

Several years ago I was developing a "robust serial console over USB" feature for Linux. THis feature allows you to have a console on, say /dev/ttyUSB0. The device doesn't have to be plugged in. When you plug in the USB-Serial dongle, you have a console. You can unplug your serial dongle right in the middle of a session, plug in one that uses a different chip, and your session is intact! Just "Ctrl-L" to refresh your Vim window where you were editing and there you are. The /dev/ttyUSB0 did actually disappear when unplugged and did re-appear on the re-insertion, but the TTY session was isolated from that, held in a suspended state.

Chopping apart the USB-Serial and TTY code in the kernel was relatively a piece of cake; but systemd threw up a wall, ironically! I got good help from the mailing list:


At first I was stumped: everything should work, but what the heck is killing my login sessions when I unplug the device? I went over the code to make absolutely sure nothing would be getting a hangup signal from the TTY or the like. Through kernel printk messages, I traced it to systemd, slapping my forehead. Basically, even though the tty session was being kept intact, it was the fact that the USB device went away that systemd killed the session.

Once I solved the systemd issues, everything worked great.

systemd: it just likes to kill things, at least in its default configuration.

I can't believe the attitude you received in some of those replies :-/

Actually, that attitude is exactly what one receives from the systemd crew. They, and only they, know what you want, when you want it, and they always know better than you do.

That's why the systemd crew generates such great hatred.

I love that the long, flaming reply so obviously didn't read the entire report, went on to lecture how to change the behaviour, and completely ignored the questions of expectation.

I suspect that you're all talking about the reply from Christian Rebischke, who is indeed now talking about a "hate-report" and "a reason to cry".

* https://twitter.com/Sh1bumi/status/735973258793779200

In fairness, please note that Christian Rebischke is not one of the Debian system maintainers, who wrote some of the other replies and whom you will find listed on this Debian WWW page:

* https://packages.qa.debian.org/s/systemd.html

Nor is xe a systemd developer, apparently.

* https://github.com/systemd/systemd/commits/master?author=shi...

Unless filing bug reports against systemd makes xem part of "the systemd crew", which, considering the headline of this page, would be a huge double standard.

* https://github.com/systemd/systemd/issues?q=shibumi

I don't think that's related at all. From what I can tell, Christian isn't involved in the GP's thread about systemd and usb serial communications.

Gah! You're right. I thought that we were still talking about the headline on this page.

I read the thread, and it seemed rather cordial. Which message in particular?

The one from GregKH, natch.

The one that the linked to email is a reply to.

Ah oops indeed. Skimmed too quickly :)

The replies from other people (A. Borzenkov) seemed OK.

Yeah i am sitting here confused about why Greg got involved at all, especially in such a rough manner.

But then he seems to straddle the kernel/userspace line more than most other kernel devs i know of.

This violates the rule of least surprise. Honestly, it looks like the Systemd maintainers operated without regards to the community at all. There's probably going to be some commenter who says "it only is surprising to you", but as evident by the email, it's surprising to a lot of users.

Does anyone know if there's any precedent for a UNIX init system to behave like this? In my knowledge there isn't.

> the Systemd maintainers operated without regards to the community at all

Don't act surprised. Those asshats stomped over a whole lot of shit and everyone acts like they're the savior of the init (and a dozen other) systems.

Don't you feel like that's a little melodramatic? I mean they created open source software and forced exactly zero distro's or people to use it.

I feel like every time I read about something with systemd someone is acting like there is a gun to their head to use it. If your distro of choice _chose_ to use systemd then your beef is with them forcing it on you...not systemd devs. They made an init system - your distro maintainers are the ones who decided it was the superior choice.

systemd development is led by Red Hat, the $2 billion per annum gorilla in the corner.

Third-party software, particularly from commercial vendors, typically uses Red Hat as the base system requirement.

So you are correct, no-one holds a gun to distro maintainers' heads and forces systemd. They just find that there's no path forward without either [1] committing resources to keep abreast of the changes that systemd is creating both directly and indirectly or [2] just adopting systemd.

Red Hat has won at this point through sheer persistence and clout, so most distros are going with option [2].

Certainly not everyone. There's a whole host of people who are angry about systemd the project primarily because of the (maybe misunderstood) behavior and communication from systemd the development team.

Personally I don't see what everyone is so upset about here. As pointed out by message 15, they've simply changed the default behavior and provided a long-opt setting to restore the old behavior if required; the hundreds or thousands of people who don't closely follow systemd will just be inconvenienced temporarily until they figure out wtf is going on, then they are back in business. And, maybe next time they'll pay closer attention.

Anyway I'm sure there will be a systemd-mux soon enough where you can declaratively define your long-running post-logout processes and register them via muxctl or some other existing sd-bus client.

I think people are upset because we're creating a lot of technical debt here. Every new "feature" we add creates a larger learning curve, makes a system that much more complex, and adds more to our maintenance costs. A simple and modular system that allows the user to combine easy-to-understand-parts in a succinct fashion to achieve functionality that __meets their requirements__ usually is superior to monolithic one-size-fits-all solutions. In short, composition generally is preferable to monoliths (after a certain point in complexity.)

I think most of us agree to this, and I haven't said anything we all don't already know. We just all disagree where that complexity tipping point is, and that's why some of us hate systemd and some of us don't see the problem.

Speaking of technical debt, I'm not sure if "nohup as a process management scheme" is a good one. Similar to "screen/tmux as a daemon manager".

(I'm not saying tmux is bad, I use it all the time. I just don't use it as a daemon manager -- If I understand correctly, they do provide a way to fix screen/tmux.)

Systemd generally tries to standardize how to declare common things about a process:

- should it be automatically restarted if it does? at what interval? - where are its logs? - how to tell if the process is alive or not? - can we isolate the process? (tmp files) - and now: should this process continue to run even if the user has logged out?

And yep, I'm an old schooler who just got tired of dealing with duct tape solutions.

You don't have to use tmux as a daemon manager to find this change annoying.

The way we do development where I work is by having two people ssh into a server, and open a joint tmux session. These sessions generally last a week or more, and you generally log out of ssh daily.

Or even if you're not pairing, just being disconnected over lunch is a long enough time for SSH to disconnect. Finding ways to overcome that (besides changing the systemd config) are even more duck tape fixes.

Automatically terminating processes when a user logs out is a bad default. Sure, leave it as an option because there might be some people that would find it useful, but changing the default without a good reason is bad policy.

As I said, I'm a huge fan of tmux. I would be concerned if they broke tmux, but it does not seem to be the case.

I've seen people run "nohup [program]" and logout, not even run it in a tmux. If systemd killed their program, I would be quite happy about it.

Requiring use of systemd-specific configurations and tools is absurd. People who want to start a long-running process and check on it shouldn't even need to know the computer is running systemd. Straightforward usage of nohup, tmux, and other equally standard and/or portable tools should be the most that a system administrator who wants to kill unwanted background processes requires of end users.

Well I thought it was obvious, but my reply was intended to be sarcastic. For it to be taken at face value by anyone kind of highlights the problem with the systemd approach, IMO.

> Anyway I'm sure there will be a systemd-mux soon enough where you can declaratively define your long-running post-logout processes and register them via muxctl or some other existing sd-bus client.

The problem with your sarcasm is that it actually matches current reality. This is how various programs are busily being rearchitected to operate, or indeed already have been rearchitected to operate.

Take gnome-terminal, for example. Nowadays, a gnome-terminal-server service runs under the per-user systemd service manager, and requests to open more terminals generated by the gnome-terminal program arrive at it over the per-user Desktop Bus. gnome-terminal is a Desktop Bus client.

The gnome-terminal-server service used to be Desktop Bus activated, rather than simply run as a per-user systemd service. Ironically, given the headline on this page, this too hit a problem with a systemd release turning things on a while back. systemd version 228, in fact. All of the terminal emulator processes and the shells and login sessions that they spawned ended up sharing a single 512 maximum threads limit.

* https://news.ycombinator.com/item?id=11675129

And yes, gnome-terminal-server too suffered from the kills-screen-and-tmux problem:

* https://lists.freedesktop.org/archives/dbus/2015-January/016...

Am i right in thinking that the client end talks to the server via dbus only?

Because that brings to mind a recent project i learned about that plans to stream video via dbus...

I noticed the sarcasm in the bit about the hundreds of thousands of "inconvenienced" users, but the forecast about systemd-specific tools appears bitter and realistic, perfectly aligned in its defeatism with my modest hearsay knowledge of systemd from outraged news reports.

I agree with this.

This actually describes a lot of my problems with Docker as well: I __hate__ that I have to use `docker ps`, etc. This is due to a weakness in traditional UNIX architectures. I ran HURD in a virtual machine a while back and it has the solution to this problem: translators.

I can't wait until they hit a 1.0 release of HURD.

> Does anyone know if there's any precedent for a UNIX init system to behave like this?

My guess is that previous init systems didn't do this because it wasn't really possible before cgroups. I'm certain that this behavior would be desired on multi-user systems to ensure that logged out users don't leave old processes lying around.

For that, couldn't any old shell script run from cron or whatever and kill any process tree not attached to a tty or X console?

I've seen that a few times over a decade ago on a couple multiuser boxen at a university. As you suggest, someone simply wrote a custom script that killed off jobs for a few reasons. It cause problems, of course, especially if you didn't now about the server's limitations. However, in that specific situation - A shared server with hundreds of active[1] sessions - it was justified.

What I find strange about systemd tying processes to an active session is that was something that was only needed in multiuser situations. The systemd advocates (and others) keep telling me that Linux isn't really multiuser anymore as justification for various incompatible changes. In the case of single user desktop (including netbook, tablet, etc) systems there's no need to kill off processes.

[1] if you count idling in Pine as "active"

Systemd is undergoing continual scope creep.

It may have started as Poetterings way to handle some transient daemons on his laptop, but has since grown to cover containerized servers (hello CoreOS) and multiseat (effectively turning a single X86 computer into a desktop terminal server).

I sometimes wonder if the systemd people encourage the confusion about what systemd really is about, so that they can shortcircuit any debate by claiming their opposition is misunderstanding and therefore not fit to comment.

Wouldn't that kill off anything started by init as well?

> This violates the rule of least surprise.

Maybe I'm completely misunderstanding things here, but when I log out, I expect all of my processes to be killed. When I'm logged off, and if another user logs on, it would be kind of weird if the system was still playing my music, etc, wouldn't it? If your processes don't die when you log off, then what exactly does it mean to "log off"?

Or maybe the issue is that it's the init system doing this, rather than some other component?

No, if I background and nohup something, it ought to stay there until or unless I kill it or it exits.

What it means to log off is to end the session you're in. A desktop session would end all the windows and systray-type stuff you're running, a terminal session would end the shell. but any processes you have explicitly daemonised or nohup'd should carry on.

Maybe you have a headless linux box that you want to set some command-line music player playing on, why should you need to keep the terminal open to keep the music playing?

> Maybe you have a headless linux box that you want to set some command-line music player playing on, why should you need to keep the terminal open to keep the music playing?

If that behavior is desired, then it would make more sense to have the music player be a system service rather than a user service, in my opinion.

As a non-user of nohup, it seems like a bizarre way to achieve the desired permanence. I guess maybe it was originally conceived as a way to run a long command from a terminal, but then effectively 'log out' of the terminal to prevent anyone else with physical access from running a command as you? Under graphical environments, this is solved via "locking" the computer. Under non-graphical environments, I would personally find something functionally equivalent to `my_command && exit` (but in a way that doesn't allow other people to ctrl+c and then steal your session) more intuitive - i.e. run the command and then log off, rather than "log off, but keep this command running".

Or maybe I'm misunderstanding why things like `nohup` exist? In any case, I can agree that making significant breaking changes in this way is pretty irresponsible. But I'm still not sure it violates the rule of least surprise.

nohup dates way back, most unix computers were not owned by a single person but were used by dozens if not hundreds of users. People had needs to run jobs (even compiles could take hours) that they would not want terminated if their dial-up connection dropped (common occurrence). Among other reasons.

Agreed, at work we use nohup / screen for many important tasks. These tasks are items like doing an upgrade, or migration, or a long running script that you want to protect from a dropped ssh connection. Networks are unreliable, your local system may be unreliable, and the last thing we want to happen is for a script or program to die while making changes to the system that leave it in an inconsistent state.

In my limited understanding, nohup was a very simple mechanism to achieve this. So to me, when you logoff or drop a connection, the system will send a HUP to all the processes in the tree belonging to that session. If the process doesn't accept the HUP, well then it doesn't want to die right now, which is what nohup allowed you to do by intercepting / wrapping the HUP. I assume screen worked on a similar principle and just ignores the HUP since it knows you may want to re-attach later.

Now it sounds like they are working on or have another way to achieve the same result in SystemD, I'm just very curious what the thoughts or reasoning is that this should be a thing.

Anyways, all I can say is that to me, changing this behaviour if I didn't see the release notes or this bug, would cause some very unexpected behaviour for me interacting with a SystemD system, where some of the basic mechanisms I use and encourage for change safety would no longer work. I suspect many others also use screen/nohup, and will be in for a big surprise.

Err unless i am totally off target, unix do not have anything like "system". The closest you will get is something running as the root user. And doing everything as root is bad juju, as we can see from the DOS legacy that plagues Windows to this day.

By "system service", I meant analogous to everything under /etc/systemd/system - it should be operated the same way things like networking and printers, etc are handled.

I actually have some services (like Transmission) in /etc/systemd/system that are run on boot as arbitrary users, and they're completely detached from any user sessions (i.e. logging off will never kill them). It seems comparable to logging in, running `nohup transmission` and logging out, but without the weird "he logged out - how are his processes still running?" confusion.

What, exactly, is confusing about it? It's just how it works, and there are good reasons for it. Screen allows you to do some cool things like keep your session running when you log off, so that you can log back in later and pick up where you left off.

Something like a daemon is basically meant to run in the background and never be directly interacted with. It's a different thing to tell a process "hold on; I'll be back later".

As an example: I've got some SBCs that are on all the time. It's useful to be able to pop on, pick up where I left off from, do some experiments, then log out, without having to set my environment back up every time I do it. There isn't really a better way to achieve the same goal.

There is no confusion unless one is clueness about unix.

I didn't come here to be insulted.

Ok, sorry.

I just find the whole session thinking an ill fit for unix, outside of a shorthand for a login process and the shell child process it has spawned.

Anything beyond that is just another process.

One of the primary use cases of tmux/screen is to act as a base for Process Groups so that this does not happen.

There was a case for the init system doing this when unix was young. But changing it now breaks convention, and compat with other distros, without good cause.

Let me explain where I come from. At times I write distributed systems that expect to be based in a non-specific unix. Whenever I mess around in an area such as process control, I carefully read the relevant section of Stevens and my distro's own docs (usually linux and until now I have tried to avoid caring about choice in init system). And - in these phases - I write in C because I don't trust that runtimes like python won't have wacky defaults that make the system hard to reason about.

This issue is a groaner, because it suggests that mainstream linux is becoming harder to reason about during development and debugging. Instead of being able to think clearly and tell the computer what to do, you will be constantly looking over your shoulder for unnecessary vagueries of your local systemd configuration and whatever quirks have been introduced by the neckbeard who owns commit rights to its default behaviours.

This bug and the abrasive german's response in the ticket suggest to me that there is a point in the near future where people who care about developing against the unix api will Bozo-bit systemd-affected distros altogether. Which would be a shame for all the good citizens who have steadily done low ego work towards those distros.

In an earlier version of your post you said

> This reminds me of a character of windows-based development that I find exhausting: opaque layers of unnecessary complexity.

This is pretty much exactly how I feel about this whole thing. While designing NT, MS made a number of assumptions and decisions that were sensible from an engineering point of view back then. Especially if you're very much into OO (wich was the new rage back then). But from an administrator or system programmer point of view, these byzantine structures get more in the way than they help. On top of that, it makes integrating new ideas harder and you end up with layers and layers of API sediment. The people behind systemd are making very similar decisions and assumptions, deviating from the very principles of Unix that prevent this short sightedness.

Thanks. I removed this because I felt it was unfair on Windows. Windows has strange things in it, but then Raymond Chen will write a post clearly explaining it, and they are strong on backwards compatibility. Everything you say is true. But you can read docs and write in C and work to a conclusion. Systemd is shaping up to be a layer of complexity that constantly shifts underneath you. It is going to be a nightmare for people who build on unix. The unix api and conventions are our bedrock. This is not sustainable.

Systemd might become stable at some point, but I'm more and more getting the idea that yanking Linux away from that Unix bedrock is the very purpose of it.

It may well be. Not sure if i can produce a link off hand. But Poettering seems to be on record about having no interest in unix.

UNIX was designed to run background processes and do offline computing. I log into multiple remote systems every day and connect to my ongoing tmux sessions in other countries. This isn't exactly an unusual use case. Banning this is a big downgrade in functionality.

Yeah, I get this. I don't disagree that it shouldn't be possible to run persistent sessions. I think it's just an issue of me finding the terminology counterintuitive.

> I log into multiple remote systems every day and connect to my ongoing tmux sessions in other countries.

In this case, it sounds to me that you aren't logging out. It sounds like you're effectively just suspending the connection to your session and then coming back to it later. i.e. you didn't SSH in and then run 'exit' from within your sesson, right? Or is that indeed the way it's done?

I'm not a tmux user, but what you describe is pretty much how it works for screen. You use a keyboard shortcut to detach from your screen session, then `exit` to terminate the login shell that ssh ran and close the ssh connection. Later, you can log back in through ssh or locally and run `screen -r` to resume the detached session.

Depends what you mean by 'session'. Unix has long had a concept of sessions, where the login shell will send SIGHUP to everything in the session when it exits; daemons like tmux (the server side) indicate their intent to start a separate session using setsid(). Apparently systemd has its own concept of session which is the same idea but completely separate, and requires some systemd specific magic to split off. Yay!

In fairness, X desktops have their own concepts of sessions that are not congruent with kernel sessions, similarly.

* http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/wi...

* https://www.x.org/releases/X11R7.6/doc/man/man1/xdm.1.xhtml#...

> Does anyone know if there's any precedent for a UNIX init system to behave like this?

I'm not sure if you'd call it an init system (but I guess any collection of shell scripts is a traditional UNIX init system), but MIT Athena has worked this way for many years: when you log out of a graphical session, there's an attempt to kill all your processes, because the machine should be fully for the use of whoever next logs in. Otherwise people would do long-running computing on machines in the computer labs ("clusters") and slow down interactive use for the next person -- and that's the non-malicious case.

It used to be possible to get around this until some point in the late 2000s, when as part of unrelated rewriting, the logout script was set to just reboot the machine if it was unsure if it managed to clean everything up.

There is, if six years of systemd having this counts as precedent. (-:

The systemd people knew that this impacted the likes of screen and tmux, and there are discussions of this going back to 2010.

Here's Jóhann B. Guðmundsson, erstwhile owner of the System 5 rc to systemd conversion project in Fedora, writing in July 2010, for example:

* https://fedoraproject.org/w/index.php?title=User:Johannbg/QA...

This came up about a week ago, I questioned why it was a useful feature and people were quick to bark "security, if you don't like it you should be writing service/unit files for your programs."

To me, I can't think of a single other operating system that works this way, even Windows. But I'm sure the pro-systemd supporters will "correct" my thinking.

Additional: "You can just disable it, don't worry about the defaults" will be another argument that comes up, seems to be a nice way of stopping discussion.. it's the same thing people espoused when binary logging/journalctl was enforced.

Can you write service/unit files for irc clients like irssi, or one-shot file transfers like rsync or wget?

This is crazytown.

Edit: I guess I should take the massive amount of downvotes to mean that it is in fact not crazytown to write service and unit files for one-off downloads or backgrounded TTY clients, and anyone asking for more information on this does not deserve a reply. Ok.

You wouldn't have to write service/unit files for one off commands. The release notes mention that you can use 'systemd-run' to start these[1].

[1] - https://github.com/systemd/systemd/blob/master/NEWS#L29

Looks like that's not enough, you also need to set some sort of "lingering" option or the last logout will kill everything despite having used systemd-run?

Sounds like a lot of pitfalls with these new defaults. Wonder how many background tasks will be ruined before somebody patches screen to automatically issue all these new systemd commands.

I don't think the 'lingering' option is necessary. That controls the default behavior for processes. Systemd-run basically makes a one off user level service.

Looks like I was wrong. You do have to enable lingering[1].

[1] - https://wiki.archlinux.org/index.php/Systemd/User#Automatic_...

Wow, talk about insane...

I actually do wish I could write a unit for things like irssi... if it dies, I want systemd to restart it, and if my entire machine reboots, I want systemd to restart it on boot as my UID.

You can do this with user level systemd instances: https://wiki.archlinux.org/index.php/Systemd/User

To me, I can't think of a single other operating system that works this way, even Windows. But I'm sure the pro-systemd supporters will "correct" my thinking.

Windows has stopped all the programs I had ran when I log out for as long as I remember.

For RDP - Windows doesn't spawn a new session every time you remote into a machine if there's already an existing one, and closing the RDP window without logging out leaves the session running.

But then again, the closest equivalent I can think of to an SSH session is Enter-PSSession, which does close all processes started under it when closing the session with Exit-PSSession.

Closing the RDP session is equivalent to locking your session (lock screen) and not equivalent to log out. However, in Linux exiting an SSH is like a log out, is it not?

Yes, that is what I meant.

But Windows also allows a session to disconnect without logging out. (This is the default when a remote session disconnects from the client side or due to network errors, and is prominently displayed in the UI otherwise.)

For an ssh session on Linux, the traditional equivalent would be to run tmux/screen in the background, but systemd will happily kill them.

You can similarly disconnect a VNC to a GNU/Linux desktop without logging it out.

Speaking of which, if you run an Xvnc server in the background out of your shell and log out, this systemd problem will now kill it.

Yes, you can disable it if you have root on that system where you're trying to use tmux or whatever.

But what if I don't have root? Like on production let's say.

Then start your process with `systemd-run`. nohup is so three decades ago.. :(

This is more a case of "nohup was broken anyway, so we are well within our rights to do whatever we want. We'll write our own new API for it in a version or two."

So where's the explanation of how nohup is broken enough to be worth breaking the rest of the way and replacing with a more complex non-portable alternative?

I was speaking in general, you probably won't be getting that explanation until enough people post a bug about it on Red Hat's bugzilla. Then we'll get the usual dance of systemd maintainers saying it's working as intended, users saying it used to work differently, systemd maintainers saying that the old behaviour was wrong and users were stupid to rely on it, etc.

So, like this one: https://bugzilla.redhat.com/show_bug.cgi?id=753882#c29

WOW. That will break/surprise a lot of people unless the distros disable it at compile time. I use tmux regularly to start long running jobs.

On the upside you can use loginctl with lingering to keep the processes after the user logs out.

This seems to be a common story. Systemd breaks one thing, but it's okay, they have their own built-in thing.

Or rather, systemd breaks an entire class of things, but it's okay, they have one and only one built-in thing.

It seems like the attitude "If you don't closely follow systemd development then you are an idiot and deserve what you get" is the most common. You shouldn't expect thousands and thousands of users to have to reconfigure to maintain existing, standard behaviour.

This is what you get if you have people in charge of developing an init system who never did any professional admin work and who apparently have a disdain for everything Unix stands for.

Maybe they want to improve Unix? This is undeniably the correct behaviour. It's only bad because everyone has gotten used to how things worked in the 70s.

I believe we have to discern between Unix as a concrete implementation (what is a process, where do certain files go, etc.) and the design philosophy. My remark focused on the latter. I would very much like to see modernization in Unix as a technology. At the same time, I believe that Linux owes its success to the adherence to a number of Unix design philosophies. This has kept Linux flexible across a wide spectrum of use cases and prevented it from running into a dead end, becoming stale.

Abandoning these foundations in order to optimize towards a very structured, hierarchical engineering ideal, will make Linux go more in the direction of BeOS or Windows. And I'm afraid that we don't have the resources that Microsoft uses to keep Windows competitive and backwards compatible.

This feels like another land grab for the user experience by systemd. They have successfully screwed up a lot of sysadmin's habits and now they are going after the everyday users. Sigh.

Gotta love systemd, they are always solving problems that you don't have, then yelling at you for not having them.

Pretty much. More and more it feels like a bunch of disgruntled ex kernel devs have found a way to circumvent Torvalds, basically using Linux as a source of drivers and base plumbing.

I'm just waiting for them to reimplement coreutils and sh as systemd binaries (they did have plans for a user space tty implementation, iirc).

To be honest I had not considered that angle. If you were going to write the equivalent of "Windows 3.1" for Linux how would you start?

For those to young to remember (or care :-) Windows 3.1 was the commercially successful version of Windows where all of the "user experience" had been abstracted into the Windows libraries, leaving the MS-DOS kernel underneath for legacy APIs and driver support. The DOS kernel, once the "center of attention" became irrelevant.

If the systemd folks can get everyone to write applications to the systemd APIs, then the Linux kernel people are just plumbers who you can yell at if they don't support the latest networking chip or USB gizmo, but aren't really contributing things to the system that users see.

Its brilliant in a way, and the more I think about it the more plausible it seems.

Well it is in large part what Android is doing already. Most apps there live inside the Android JVM after all.

I still can't find a good reasons why you should use systemd, this things grows out of control and is way out of the initial scope.

Upstart was a reasonable replacement for system V I reckon, ...

After using systemd for a few years on arch linux I no longer despise it. It does have some good features like unifying system jobs and cron jobs, faster boot, journal forward secure sealing (to detect log tampering).

I don't like its effective ownership by redhat and its monolithic nature however this can be solved in the long run by forks and standardisation.

> this can be solved in the long run by forks and standardisation.

I keep hearing this refrain, but the reality of the situation right now is that Systemd (and the other processes under the same development umbrella) are moving too quickly and are too complex for any fork to ever have a realistic chance.

Does there exist a group of people outside of RedHat who understand the internals of the whole thing well enough to even chance a fork?

I do have to admit to being impressed by the boot speed of Arch under Systemd.

The history of uselessd (q.v.) indicates that it cannot be solved by forks.

I tried really hard to learn systemd, but I am in the process now of moving to a systemd free distro. It's such a mess.

Gentoo uses OpenRC. It works awesome. I like not being surprised and I like avoiding conversation around systemd cause I don't see it (till I shell into a Debian)

I hate hate HATE systemd. That doeant mean it doesnt have its positive points, however.

Example: I love service files.

If they would just make everything modular/optional, I wouldnt hate it nearly as much

> If they would just make everything modular/optional, I wouldnt hate it nearly as much

Do you currently have a need for a piece of systemd that would work better as a standalone component?

Not so much standalone, but being able to substitute other services. Logging and service-management are two issues off the top of my head. I'd love to be able to disable those, and manage them with another service/package.

Optional, like the KillUserProcesses option?

Yes, I'm glad they made it optional. I don't like it, but being able to easily turn it off makes it a non-issue to me.

Socket activation is pretty cool.


If you're hosting a bunch of apps for clients, but not all of them are being used all the time, then you can fit more onto one server by only activating them when they're used.

I know Pantheon does this, for example. https://pantheon.io/open-source

There has also been some discussion of integrating this functionality into kubernetes: https://github.com/kubernetes/kubernetes/issues/484

And yet, socket activation is nothing new:

inetd: https://en.wikipedia.org/wiki/Inetd

xinetd: https://en.wikipedia.org/wiki/Xinetd

Used in the "olden" days to allow a single server to serve a bunch of services, not all of which are being used all of the time, to a bunch of clients, by only activating them when they are used.

Does systemd socket activation offer any advantages over xinetd?

Afaik something like xinetd forks and execs a copy of the daemon for each connection and communicates over stdin/stdout.

This can work for simple services like sshd which fork for each connection anyway, but would never work for something like nginx or redis.

That is untrue. Read about "wait"/"nowait" (in the xinetd and inetd manuals, for starters).

Ah, I must have been thinking of something like


Then you'll enjoy reading about the nosh toolset and s6-networking, which both separate the listen-accept model into separate programs and have the ability to invoke "wait" servers with the listening socket.

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwa... * http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/UC... * http://skarnet.org/software/s6-networking/

Avoiding extra processes is good, but it's not the end of the world to exec netcat.

Yes, it can also support local sockets as well as network sockets.

It has one terrifying disadvantage. The socket is opened and appears to be owned by pid 1.

I haven't seen the code that does this, but my hope is that it is perfect in every way - if not then its an open door from the internet into my server.

Thats is indeed pretty cool, the whole question is more under the way decisions are beeing taken regardless of what the community says.

But I don't really see what this sort of feature has to do in an init process. I really dislike the idea of having one project handling everything on a box .... The day you have a security issue in systemd we are all fu*.

But yeah there are some real good things in systemd, maybe they should just be taken out of it and spin up in a new project ;)

"I really dislike the idea of having one project handling everything on a box .... The day you have a security issue in systemd we are all fu...."

I've heard this argument a bunch, but looking at the actual systemd approach doesn't really convince me that it's the case. There are dozens of binaries, dozens of maintainers, hundreds of contributors. It's comparable to a project with a bunch of parts (think Gnome or KDE only at the systems-level), rather than one big black box that does everything.

If there were a single "systemd" program that did all the hundreds of things that the systemd system does, it would be alarming and impossible to maintain. But, it's not. They may be more tightly coupled than some like (I quite like systemd and still find it a little uncomfortably coupled in many places), but that's partially a symptom of doing a bunch of things in new ways. No one built the interfaces needed to do what systemd does in the past, so building systemd took reinventing the universe in some regards. That's actually some of what you're getting when you get "systemd"; things that aren't part of the init system, but existed in some form before and are now being replaced by systemd-provided services.

"But yeah there are some real good things in systemd, maybe they should just be taken out of it and spin up in a new project ;)"

That becomes more feasible over time. Right now, there's a lot of moving parts. Another reason for the tight coupling is that the way things work together is still evolving as people learn the shortcomings of this new architecture.

systemd is not perfect, but it is a bunch of little pieces working together. A security issue in one of those pieces doesn't necessarily mean we're all fucked any more than a security issue in one part of any of the old systems necessarily meant we were all fucked.

Then why is it then being pressed into service across distros?! Why not let it churn on in its own world somewhere out of the way until it settles down and the interfaces are more defined?

Because it would never fit a real world use case, if isn't designed in and for the real world. Software built in a secret lab and trotted out to people after the fact is not the way Open Source works. We build the plane as it flies. It's always been that way in Linux.

I'm surprised there's still so much rage about systemd. It's not a bad solution to the problem. I mean, hell, just the move from procedural to declarative for service configuration is huge. The size of that improvement really cannot be overstated.

> "I'm surprised there's still so much rage about systemd. It's not a bad solution to the problem."

You can only be surprised if you've got blinders on. Systemd has some real technical superiority but also some questionable or worse technical decisions. Beyond that, it has a truly horrible community around it that has an incredible track record for being bad at justifying shaking things up and bad at easing migration pains and bad at responding to valid criticisms or accommodating use cases different from their own. A project can't trigger flamewars this reliably and be innocent in the matter. You don't see the same kind of social and organizational conflict surrounding the major Linux display stack overhauls over the past decade.

GPL vs BSD has triggered flamewars reliably and for decades. Which one is guilty?

Not so much a secret lab, as a development fork that is maintained in parallel to shake down and stabilize before being put into the wider use.

Look at the kernel, where larger charges are developed and tested outside the main tree, and then applies for inclusion.

Never mind Torvalds' mantra about not breaking user space. Sadly user space seems highly adept at breaking itself repeatedly...

> I'm surprised there's still so much rage about systemd.

Didn't you hear? It hangs up processes which have been run with nohup!

I think socket activation makes sense as part of the int process because of the 'activation' part. That's basically what you rely on init systems to do.

For more details about the advantages of this can be found here[1] and here[2].

[1] - http://0pointer.de/blog/projects/socket-activation.html [2] - http://0pointer.de/blog/projects/inetd.html

> I really dislike the idea of having one project handling everything on a box

Whatever you do, don't switch to a BSD.

It's an interesting feature but inetd (or xinetd) did it for a long time decoupled from the boot process.

A pro for systemd doing it is maybe consistent nomenclature for service startup, but a con is that it couples in yet another use case to systemd which may lead to a higher risk of security, maintainability problems down the line.

I'm not sure why starting services two different ways would lead to fewer security problems. Seems like a classic example of reducing surface area for attack. One way to start services means one pile of code to audit, one configuration syntax to understand (that it's new is a negative, but that it is declarative rather than procedural in the case of initscripts is a huge win; xinetd is also declarative, so systemd is converging on the known better way to configure starting services), one place to look when figuring out what is running and what is supposed to be running on your system.

There have been init bugs in the past. There have been xinetd bugs in the past (and inetd was occasionally notorious with regard to security, though it was usually the services it provided access to rather than inetd itself). There will be bugs in systemd. But, how does consolidating service startup into one project increase the risk?

Well for one, systemd hooks to a large number of functions/subsystems that used to be more isolated. Logging, udev, dbus, churn, networking functions, etc. So it's a bit of a judgement call to say it has a larger or smaller attack surface than separating the functions out. I don't know how to really quantify if in the end it ends up more risky one way or another.

I find it frustrating that systemd's socket activation API is via library calls and not a protocol. This makes it necessary for separate support in programs to support systemd rather than having a generic mechanism that all init systems might use.

`libsystemd` just provides an implementation of the socket-activation protocol. It is documented and can be implemented easily without libsystemd: after all it's in the end just reading which file descriptors to use as listening sockets from an environment variable.

Anyone taking bests on said frustration being intentional?

Systemd is useful; assuming your goal is to turn Linux into Windows. Personally, I would rather use a well crafted Unix system.

I think this is actually a useful change in default behavior for server environments, which comprise the vast majority of Linux installations.

Being able to assure by default that all of a user's spawned processes are terminated when the user logs out is a much more secure default behavior than the opposite. This means if I disable a user in LDAP and force the user to log out, I don't have to worry about lingering processes that might be malicious or pose some sort of threat to my server or data.

Keep in mind that there is a switch in logind to enable lingering that you can enable if you don't like it.

Besides, the release notes provided the workaround in line, so that interested users wouldn't have to hunt for a solution.

For most development server use-cases, it's pretty terrible though. Screen is how a lot of people work.

Yep -- I use tmux myself, but I'll have tmux sessions running for months on my work machine, so that I can ssh in and be immediately productive from anywhere.

It's a POLA violation. Documenting this gross violation of expectations is not a sufficient apology.

(I assume you meant POLS, not POLA?)

"POLA" would be for "astonishment", instead of "surprise". Based on the Wikipedia citations, POLA looks like the older form of the phrase.

Ah, thanks. I parsed that as "authority", which is a different thing altogether, but yes, I've heard "astonishment".

> Besides, the release notes provided the workaround in line, so that interested users wouldn't have to hunt for a solution.

Only if the interested user is astute enough to put systemd at the top of the suspect list for why screen and tmux stopped working.

Individual users aren't the target audience, though. System integrators are. Your distribution maintainer is responsible for reading the change log and making the necessary decisions with respect to default configuration, notifying users, etc.

Assume anything wacky is a result of systemd. Problem solved.

I'm only half joking.

> Being able to assure by default that all of a user's spawned processes are terminated when the user logs out is a much more secure default behavior than the opposite.

When the user has explicitly requested that processes not terminate when he hangs up by using the nohup command, that is precisely what not to do. This is fundamentally broken behaviour.

> Besides, the release notes provided the workaround in line, so that interested users wouldn't have to hunt for a solution.

In honour of Towel Day:

    Mr Prosser:	But, Mr Dent, the plans have been available
                in the local planning office for the last
                 nine months.
    Arthur:	Oh yes, well as soon as I heard I went
                straight round to see them, yesterday
                afternoon.  You hadn’t exactly gone out of
                your way to call attention to them had you?
                I mean like actually telling anybody or
    Mr Prosser:	But the plans were on display…
    Arthur:	On display? I eventually had to go down to
                the cellar to find them.
    Mr Prosser:	That’s the display department.
    Arthur:	With a torch.
    Mr Prosser:	The lights had probably gone out.
    Arthur:	So had the stairs.
    Mr Prosser:	But look, you found the notice, didn’t you?
    Arthur:	Yes yes I did.  It was on display at the
                bottom of a locked filing cabinet stuck in
                a disused lavatory with a sign on the door
                saying beware of the leopard.
End users cannot be expected to read the release notes for each release of something their distribution forces on them.

As I said elsewhere, the distribution maintainers are the target audience, not the individual users. This isn't yet a problem for the vast majority of users yet because the consequences have not made it to them. It's reasonable to expect appropriate arrangements will be made before then.

Is this really an unsolved problem? If you need that functionality can't you do it with ps or pkill -U and an alias? This approach also allows you to send SIGTERM first.

http://unix.stackexchange.com/a/136552/34801 it's 2 lines.

It's also fine when systemd allows it as an easy knob and cgroups might catch something that pam doesn't (but you could also use cgroups in the pam script and cleanup/atd/cron or systemd timers).

But doing it by default behavior is kind of surprising.

> Besides, the release notes provided the workaround in line, so that interested users wouldn't have to hunt for a solution.

The Hitchhiker's Guide to the Galaxy approach to announcements ;)

But it was announced on the CHANGES file on that github repository works really well for breaking POSIX.

Time to switch to one of the BSDs. I hear they're a better unix than unix and these systemd clowns are out of control.

Gentoo still lets you avoid systemd - I've switched to Gentoo and have been pleasantly surprised at how nicely everything runs.

Mind you, I have also started investigating the BSD family seriously (rather than just to have a play with), just in case.

try Calculate Linux first before going full BSD. you'll thank me.

Will do.

The response to the bug report:

Hello, You should quote the full changelog and not just the part that is 'bad' in your mind.

>systemd-logind will now by default terminate user processes that are part of >the user session scope unit (session-XX.scope) when the user logs out. This >behavior is controlled by the KillUserProcesses= setting in logind.conf, and >the previous default of "no" is now changed to "yes".

For debian it would be enough to set this to "no" again with --without-kill-user-processes option to "configure"

>This means that user sessions will be properly cleaned up after, but >additional steps are necessary to allow intentionally long-running processes >to survive logout.

Here comes the important part. Seems like the systemd-devs are working on a way to allow intentionally long-running processes in a specific user scope.

And here is another way for allowing these long-running processes:

>While the user is logged in at least once, user@.service is running, and any >service that should survive the end of any individual login session can be >started at a user service or scope using systemd-run. systemd-run(1) man >page has been extended with an example which shows how to run screen in a >scope unit underneath user@.service. The same command works for tmux.

And another way for allowing long-running processes.

>After the user logs out of all sessions, user@.service will be terminated >too, by default, unless the user has "lingering" enabled. To effectively >allow users to run long-term tasks even if they are logged out, lingering >must be enabled for them. See loginctl(1) for details. The default polkit >policy was modified to allow users to set lingering for themselves without >authentication. > >Previous defaults can be restored at compile time by the >--without-kill-user-processes option to "configure"

You see? No reason to complain about.

Best regards

Christian Rebischke.

tl;dr: You can configure it not to, or you can accept that Linux is about as much a "Unix" as AIX, Irix, or HP-UX and join them in the wonderful land of the future, which suspiciously resembles the time before all the non-standard vendors died.

I'll stop now, before I make Linus on a backwards compatibility rant look like a choir boy in church.

Wow. Christian Rebischke's response to this reasonable, mild-mannered, bug report, is unbelievably hostile.

Now he's out on Twitter describing the bug as a "hate-report".

With any hope, a future employer will find this, put it in front of him, and ask if that's the kind of behavior that should be expected of him. Before turning him around and out the door.

So if I set /etc/systemd/logind.conf KillUserProcesses=no the current behavior will be preserved whenever this change finds its way downstream?

Even "nohup" programs?

Apparently, according to the third message in the report.

That's treading on hallowed grounds man. How can you teach newbies to use nohup now?

"Oh, first you have to talk to the machine's administrator to reconfigure systemd, if you're on Linux. Or just use FreeBSD, MacOS, Solaris, or ... Cygwin".

> or ... Cygwin

Or not, because windows will kill all processes that are part of the users session when they log out.

> Or not, because windows will kill all processes that are part of the users session when they log out.

Ah-HA! Now we know what inspired this change.

Is Poettering financed by Microsoft, or is he just a fan of their ease-of-use, high security and all-round wonderfulness?

Yes, but you can use 'systemd-run' to do something similar[1]


As a regular user, I don't want to know anything about systemd. I want the system to behave like Unix.

The nohup utility is standardized by POSIX: Look here:


As far as I've experienced on OS X, any running process does not survive logout. Granted it's a single session, and that session is also GUI.

For a GUI login, I'd expect systemd to do exactly this. Right now, wayward processes ignore or just don't grok whatever "quit" notification the DE sends, and this will cause delayed restart/shutdown for 1m30s at which point systemd kills the user session. 1m30s is too long.

Whereas on GUIless server, where I use tmux, the idea I'd login, run tmux and create a session or two, detach, logout, and by logging out all my tmux sessions go poof is not expected. But even in the cited thread there's a mechanism for such processes to get a separate user session.

> For a GUI login, I'd expect systemd to do exactly this.

Processes have always done this when they receive a hangup (HUP) signal (unless they do something else, of course, but that's up to them).

> But even in the cited thread there's a mechanism for such processes to get a separate user session.

That mechanism has existed for decades: it's called nohup.

And now tmux is asked to accommodate the change.


I like systemd. It solves a lot of problems and does some cool things. But it always worries me when they go around breaking stuff. Too often the response is something like tmux has been doing it wrong - use the new systemd-tmux service instead.

Well, I guess that is a useful feature for init system.

It's not a feature of the init system. It's a feature of the login daemon that ships with it.

Potato, potato...

I am certainly not surprised by this change. I'm just glad I switched to Devuan a few months ago. I don't read every changelog, but I would have been furious if I'd discovered my screen session gone. Having this enabled by default is moronic and shows real short-sightedness on the part of the Debian packagers.

I wonder if the debian folks feel regrets about settling with systemd.

You mean the folks that are left? I doubt it. The vote within Debian to go to systemd was very close. It would not be close today, because a lot of those that voted against it left.

Debian will disable this if they want.

But they shouldn't have to. Changing a default, known, long used feature of UNIX-like OSes without any sort of announcement or debate is dangerous and irresponsible. This kind of thing is why I feel the encroachment of systemd into most Linux distros is a bad thing.

It's a great system in theory and I am excited about its potential, but it has some extremely egotistical and irresponsible developers who don't give a damn about what they break, as long as they can keep pushing code out without community testing or review to slow them down. Debian was flat out wrong to go with systemd until it reaches a certain level of maturity and stability, but there's no going back now.

> without community testing or review

This is a bug report in debian unstable against a bleeding edge version of systemd released 5 days ago.

They pushed this "feature" without any advance notice to developers of userland apps like screen and tmux, leaving them to pick up the pieces and figure out what happened. That is irresponsible and reckless. Common sense dictates that when you're going to introduce severe breakage, you give advance warning so it can be properly tested and dealt with.

And it's having a negative effect on their own pet distro as well, see this thread:


No, it's having a negative effect on rawhide which is "the current development version of Fedora".

You act like every single systemd user on the planet upgraded to the bleeding edge version of systemd and had this configuration option changed on them overnight.

Oh, and here is your "advanced notice"


That was notice that it would be a compile time option, not set to the unexpected and harmful setting as default.

We used to make jokes about Emacs being bloated and how it would eventually be its own operating system, etc. systemd has gotten much closer to that point in a much shorter time.

I would be happy to replace systemd on my system with sys-emacs-d.el

I think this is what happens when the "web 2.0" cancer leaks from the casino :/

FWIW "HandleLidSwitch=ignore" is another back-to-normal that will probably save you some walking.

To be fair, Debian Jessie is not exactly the most stunning of Debian releases over the past decade either. Between the missing and ancient packages we ended up having to move to CentOS 7 w/ epel+elrepo.

Which packages were you finding "missing" or "ancient?"

Corosync, Pacemaker, OCF, they're off the top of my head but we found a LOT when looking at upgrading from Wheezy.

What about user-specific CRON jobs? Does this prevent them from running?

I noticed the entry for this in their release note, and my first thought was that this would catch many off guard...

This is a good feature, and a good default. Processes in the session-xx.scope should be stopped when the session ends.

there's an easy sollution for processes that don't need to be stopped too.

No, there was already a mechanism in place to end processes when you log out - when the controlling tty disappears, every process in the session is sent a SIGHUP and SIGCONT signal.

The processes started under screen(1) are deliberately created with a new pty as controlling tty and in a new session, because they are intended to survive the ending of the original session where screen was started.

not for processes that don't have a controlling tty.

processes deliberatelly started with screen can be started with systemd-run screen.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact