Firstly, it's not true: most systemd services don't link to libsystemd. It's useful for supporting the sd_notify feature, but plenty of things support it without using libsystemd.
Rich attack surface: there was like... four dependencies that aren't glibc, and irrespective of this backdoor attempt they were almost all already being made dlopen dependencies, which would've prevented this backdoor from working since it would've never been triggered for sshd. (Note, also, that linking libsystemd into sshd was a downstream patch.)
The truth is that libsystemd and systemd in general don't have ridiculous dependency trees or attack surfaces. Most likely the big reason why this backdoor was being pushed heavily to try to make it into Debian and Fedora releases was because the clock was ticking on the very small in they had found in the first place.
There are a lot of criticisms of systemd that I think are reasonable, but this really isn't one.
> (Note, also, that linking libsystemd into sshd was a downstream patch.)
I feel like a lot of people are really glossing over this point. What's happened here is that Red Hat and Debian have made a choice to patch OpenSSH in a way that opened it up to this attack.
It's a little ironic that e.g. Arch, which actually shipped the malicious code to end users since it publishes changes so much faster, as shipped never would have executed the payload (because they didn't patch OpenSSH).
>It's a little ironic that e.g. Arch, which actually shipped the malicious code to end users since it publishes changes so much faster, as shipped never would have executed the payload (because they didn't patch OpenSSH).
The backdoor was designed to only be injected during the building of an RPM or Debian package. Arch never would have been impacted no matter what choices they made. They were trying to hit production systems while minimizing their potential exposure to other less important types of users beforehand.
Honestly, I'm a bit curious why sshd startup would ever be significant. Not sure.
The most intensive operation I can think of that might happen at sshd startup would be host key generation, but at least in NixOS it looks like this is actually handled in an ExecStartPre= script, which I believe means it will happen before After= units execute.
Of course I sincerely doubt that distributions decided to add sd_notify support just for the hell of it, so I am sure there is a reason, it's just not overtly obvious, especially considering plenty of distros don't do this and sshd (and dependent units presumably) certainly seems to work absolutely fine.
So on the one hand libsystemd provides what both you and the manual page describe as "useful" functionality for implementing this feature, but on the other you're implying that Debian shouldn't have used it to do what it's designed for?
Maybe libsystemd shouldn't provide sd_notify then, if you're not supposed to use it?
No, I never said you should or shouldn't use sd_notify. I said that Debian and Fedora added it downstream, that it was not a decision of OpenSSH.
Generally, I think using libsystemd is probably a good idea, if you are a C program that wishes to have support for sd_notify. Even better, even before they moved things to dlopen, the dependency chain was very clean; nobody would've batted an eye at the idea that something, even something as critical as the SSH daemon, would have a very popular and well-regarded compression library which is already trusted in many package managers loaded into the process space. There is absolutely nothing unreasonable about considering that to be trustworthy. Your sshd probably has zlib linked in and it is fine.
Seriously. We're not talking about the difference between zero dependency and one dependency, but rather something more like 34 dependencies vs 38 dependencies. Here's my sshd, noting that libtree excludes some glibc stuff by default:
But there is one thing that is questionable, and that's whether or not downstreams like Debian and Fedora should really be making patches like this that add new dependencies to security-critical programs like OpenSSH. It's one thing if OpenSSH takes on this dependency itself, but downstreams adding it is scarier specifically because it's so unexpected. If you didn't know the way OpenSSH was packaged specifically in Debian and Fedora, you would have absolutely no way of figuring out that liblzma5 could be in its process space and thus it wouldn't be part of the threat model. Certainly the upstream developers would have zero chance of ever noticing this.
Of course, I'm sure Debian and Fedora have their reasons for the specific patch that adds sd_notify support, it probably does improve something somewhere, but this incident absolutely showcases how the consequences of innocuous-looking patches like that can domino into something absolutely devastating in a non-trivial manner. I strongly suspect whatever it improves on would not have been worth it if the backdoor had succeeded in proliferating. Nothing is a panacea, but you definitely gain some advantage by sticking to configurations supported by the upstream where possible.
That said: Debian and Fedora already (largely) dodged this bullet, libsystemd already (even before this news) plugged the hole that made it possible in the first place, and everything is fine for now. Debian and Fedora thus have no real reason to remove this patch other than if their sensibilities regarding it have changed, because the risk that it posed is basically gone now. Now for this specific patch to pose a threat, you'd have to compromise either libsystemd or libcap2. On any distro that uses systemd to begin with, compromising systemd already gets you root, and libcap2 IIRC is maintained by the Linux kernel folks, so of course if anyone compromises that, it's way, way game over already.
Is this some grand lesson about systemd bloat potentially enabling a horrible sshd backdoor? You can read it that way, but it's a pretty silly take, considering all of the factors. But if it makes people who get emotionally attached to hating init systems happy, more power to them I suppose. As for normal risk analysis, there is no particular factor here that completely dominates for what really enabled this. It was a combination of decisions that, on their own, are completely justifiable and reasonable, but when combined, led to near disaster. Not a new story and not one that's going to stop occurring either...
Firstly, it's not true: most systemd services don't link to libsystemd. It's useful for supporting the sd_notify feature, but plenty of things support it without using libsystemd.
Rich attack surface: there was like... four dependencies that aren't glibc, and irrespective of this backdoor attempt they were almost all already being made dlopen dependencies, which would've prevented this backdoor from working since it would've never been triggered for sshd. (Note, also, that linking libsystemd into sshd was a downstream patch.)
Seriously:
That's what you get today in NixOS unstable.The truth is that libsystemd and systemd in general don't have ridiculous dependency trees or attack surfaces. Most likely the big reason why this backdoor was being pushed heavily to try to make it into Debian and Fedora releases was because the clock was ticking on the very small in they had found in the first place.
There are a lot of criticisms of systemd that I think are reasonable, but this really isn't one.