It's a not so well guarded secret that the claims by many linux distributions that they support old systems with security fixes for many years are not really true. The number of security flaws constantly found and fixed is just too large to make this even remotely feasibile. This is just another instance of this.
Can confirm this is a very real problem with a long history. Quoting from a sort of Arch-specific security hardening page[1]:
> A common misconception about the Linux kernel is that it's secure, or that users can go a long time without worrying about security updates. Neither of these are even remotely true. New versions of Linux are released almost every week, often containing important security fixes among the other changes. These releases typically don't make explicit mention of which commits have security implications. As a result, many "stable" or "LTS" distributions don't know which commits should be backported to their old kernels, or even that something needs backporting at all. If the problem has a CVE assigned to it, maybe your distribution will pick it up. Maybe not. Even if a CVE exists, at least in the case of Ubuntu and Debian especially, users are often left with kernels full of known holes for months at a time. Red Hat and similar "enterprise" distros have the same problem and have been called out publicly about it on more than one occasion. Moreover, the Linux kernel security team doesn't request CVEs for any vulnerabilities, partly because there are just too many to track. Downstream's culture of trying to cherrypick security fixes in the name of stability does not work. Arch doesn't play the backporting game, instead opting to provide the newest stable kernels shortly after their upstream release.
Nothing prevents you from running the latest kernel. The kernel build scripts can even generate .deb and .rpm directly so it coexists nicely with the package manager, though even that isn't strictly necessary.
Ubuntu LTS has a package called linux-hwe that backports the latest Ubuntu kernel to the last LTS release. I find it's a good compromise between maintaining a mainline kernel and using an ancient one. And since the hwe kernel is provided by Ubuntu and used in its release, it's been tested by thousands of users and updating it is a breeze.
My only complaint is that I wished it went back 2 LTS releases. For example, the current project I'm working on settled on 20.04 and there are no more new hwe kernels being released for it despite the rest of the userland receiving updates.
Nothing prevents you from running the latest kernel, but the project is introducing new bugs with security implications all the time even as it is eradicating old bugs, which is why one guide recommends an LTS kernel as the better of two bad options (the other option's being a stable kernel):
Who cares about a few reboots? You can reboot a workstation overnight and if you have a single point of failure in your production system the reboot ain't the problem.
That depends entirely on how well your state serializes to disk and restores. If nothing else, I occasionally make the mistake of getting a setup going that would just be a pain to recreate in terms of getting everything open again (too many windows open in a specific arrangement).
A few years ago we couldn't upgrade the kernel because XFS had a regression. Our choices were: revert the XFS changes in the newer kernel; backport security fixes to the version we were already using.
We went with the latter because our minimal attack surface made it unlikely that we'd need to backport many fixes, but it was always looming.
This is one reason why I don't mainline a Linux desktop anymore. It's just too much sysadmin work to stay on top of it, in addition to all of the desktop issues.
When even the creator of the Linux kernel has a very lax approach to security, and so the culture of Linux is the same way - I consider it an issue.
Experience has taught me not to be the first to beta-test a new package update, and don't blindly update. I used to run Arch and have run into scenarios multiple times where I've been unable to boot into a desktop environment because of an update. They were recoverable issues, but still cost me time and motivation for that day.
That, and updates are just a constant stream - there is always a new package to update every single day.
And as a separate issue, but the quality of software has taken a sharp decline lately as we all start switching to flatpaks and snaps (a correlation, not causative). More and more software is Electron-based and the quality of the non-Electron based software has more or less tanked.
Not many hobby programmers are making desktop software anymore.
You could try using Nix OS stable with "boot.kernelPackages = pkgs.linuxPackages_latest;". That way, you will get the latest kernel version and if there is an issue with you being unable to boot into a desktop environment, you can very easily roll back the update by rebooting and selecting an older generation in the boot manager's menu.
I second that. On few occasions kernel update in Fedora on my Dell XPS from 2016 cannot boot. The last time it was an upstream regression that was not fixed for months. In the mean time a bad WiFi bug was discovered. I could in theory compiled my own kernel with a fix but without the boot regression, but I rather decided to use Debian. At least they do backport fixes
Edit: I have been corrected on the support time frame. Also had the bad luck of checking the CentOS wikipedia page to validate my memories - one of the Tables makes it look as if both CentOS stream 8 and 9 are already out of support.
As far as I can tell IBM officially nuked CentOS long term support in the same step it killed Red Hat compatibility with 7 being the last supported version.
The replacement for CentOS seems to be Rockit Linux, which continues with tracking Red Hat packages like CentOS did in the past.
CentOS Stream is just as "LTS" as Debian, Ubuntu, or OpenSUSE - supported for 5 years. This is admittedly not as generous as the 10 years CentOS provided previously (and what Alma / Rocky linux have promised to provide), but it's definitely still "long term" by the same definition everyone else uses.
And while CentOS Stream {8,9} does trend slightly ahead of RHEL {8,9} respectively in terms of bugfixes and features, it still maintains equivalent ABI with RHEL {8,9} and is compatible, though not precisely "bug-for-bug" compatible, which even CentOS / Alma / Rocky have never quite accomplished.
That is probably what I get for checking my assumptions against wikipedia. One of the tables on the CentOS page makes it looks as if only CentOS 7 is still supported. On a closer check the table may just be incomplete. https://en.wikipedia.org/wiki/CentOS
Scientific Linux was killed in 2019 although security releases for SL 6 and 7 will probably continue as long as Centos 6 and 7 get them. I'd go with AlmaLinux or Rocky Linux if you want to something that's a free copy of RHEL 8 or 9
RockyLinux is by the guy who did CentOS, and seeks to follow it's principles.
Scientific Linux is no longer a thing, essentially; from wikipedia:
> In April 2019, it was announced that feature development for Scientific Linux would be discontinued, but that maintenance will continue to be provided for the 6.x and 7.x releases through the end of their life cycles. Fermilab and CERN will utilize CentOS Stream[4] and AlmaLinux[5] for their deployment of 8.x release instead.
> RockyLinux is by the guy who did CentOS, and seeks to follow it's principles.
Not quite. Back in 2003, when folks were first coming up with the idea of creating a distro that matched RHEL as closely as possible, Greg Kurtzer (the Rocky founder) explicitly stated that he would be interested in helping host it, but that he was "totally not interested" in leading it.
Oracle Linux 7 is guaranteed binary compatible with RHEL/CentOS 7, although there is only (a little over) a year of support left on the whole platform.
Scientific Linux was also on v7, but I thought there was some kind of ongoing apoptosis with this repackaged distro.
Neither Alma nor Rocky implemented v7. I thought that CentOS left that version untouched.
I wonder if the Oracle UEK has addressed these problems. If so, it would be a reason to run Oracle's kernel on CentOS, supported or not.
For those poeple who can tolerate neither these bugs nor Oracle's kernel, another option is ElRepo's "Kernel of Last Resort."
"We stress that we consider such kernels as a last resort..."
UEK can run into the same issue. It is just a picked upstream kernel with a bunch of stuff that Oracle cares about dropped on top of it, and managed by a much smaller team.
It wouldn't matter if they didn't. Research and disclosure of vulnerabilities is an absolute good; they don't need to earn the right to publish bugs (it's closer to the other way around).
This mentality is becoming passé. And frankly this article/post is a good example of the reason. It was long thought that you could name & shame people into good security, but it turns out they just purchase cyber insurance and retreat deeper into the shadows, which increases exposure to the end user.
But the new reality is that malicious uptake on publicized POCs is so great, and the negligent refusal to patch is so great that a widespread culture of reporting has actually increased harm to the public. That is the reality, and until there are laws that hold companies liable for the software they choose it will remain the case.
I came up in an era of CERT's anti-disclosure regime, with no auto-update anywhere, and there is simply no comparison. No, to all of what you said; I don't think I could disagree with anything more strongly.
From the project zero posting [1] it would seem like most of these would be relatively low scoring CVEs and hence maybe less of a reason to prioritize a fix.
The CVEs are for CentOS Stream 9, not the RHEL derived CentOS 7 or Rocky/Alma 8/9. If the RHEL derived distributions were patched but not Stream it would seem that IBM-RedHat's promises about CentOS Stream being just as good are now provably false.
I'm not sure if such a promise ever existed. After all, before CentOS Stream, all security fixes went out with RHEL first and were imported into CentOS (non-Stream) only afterwards. And under the Stream development process, the fixes have to be merged into Stream eventually for the next minor RHEL release, otherwise they would be missing from that, too.
So the Neowin headline is a bit misleading because Red Hat disclosed these issues publicly (presumably after consultation with the reporters) even before the 90-day timer expired in the Google bug tracker.
(Disclaimer: I work for Red Hat, but are not involved in security anymore.)
Additionally these are moderate CVEs and Red Hat, per their lifecycle page, patches them at their will.
During the Full Support Phase, Red Hat defined Critical and Important Security errata advisories (RHSAs) and Urgent and Selected (at Red Hat discretion) High Priority Bug Fix errata advisories (RHBAs) may be released as they become available. Other errata advisories may be delivered as appropriate.
Rocky and Alma are vulnerable to these CVEs too, as is RHEL. As a matter of fact, CentOS Stream 9 already has the fix to one of the CVEs. The rebuilds have to wait for it to show up in RHEL before they can rebuild it.
RHEL doesn't have these patches either. Because they're local exploits and not all that important in the grand scheme - they're everywhere and you have to already be pwned for these to benefit an attacker.
Typically fixes like these would just be bundled together and wait for a regular release rather than be pushed out immediately.
What I don't see mentioned is that IBM would not have to back port anything from the 5.15 tree if they just used an LTS kernel in the first place. But instead they like to differentiate(?) their product by using non-LTS kernels and handling all the back porting themselves, rather than collaborating with distros like Debian and Ubuntu which use the LTS kernels.
Reasoning I've heard for this is since they backport complete subsystems and features from mainline kernel over the years, it turns into a frankenstein and it wouldn't resemble the origin version. So it doesn't matter if they'd used LTS version or a non-LTS one.
I've wondered about this too. Like, what's the advantage of it?
It looks like they've gone back and forth on this. RHEL 7 and 6 used LTS kernels. Before that, RHEL 5 did its own thing instead of using the LTS. Of course, RHEL also supports these kernels for much longer than the original maintainers. RHEL 6 (released in 2010) still has another year of Extended Life-cycle Support into 2024, even though end-of-life for the 2.6.32 LTS kernel was in 2016.
Looking at everything now, I see that Ubuntu 18.04 and 14.04 also chose non-LTS kernels. Interesting.
Red Hat doesn’t fix moderate CVEs immediately. Per there RHEL Lifecycle web page:
“During the Full Support Phase, Red Hat defined Critical and Important Security errata advisories (RHSAs) and Urgent and Selected (at Red Hat discretion) High Priority Bug Fix errata advisories (RHBAs) may be released as they become available. Other errata advisories may be delivered as appropriate.”
Seeing how all of these CVEs are considered Moderate they fall under the “delivered as appropriate”.
As usual people over react and try to make companies look like they don’t care about security, back ports, etc when all Red Hat is doing is following its own policy.
Absolutely agree, but one small point of clarification. One of the referenced CVEs (CVE-2023-1249) is rated low. The other two (CVE-2023-0590 and CVE-2023-1252) are rated moderate.
As of today Google still has not fixed the vulnerability on Pixel 6 a device that Google promised it would provide security fixes for 5 years disclosed by the Project Zero which is a Google suborg.
It was patched before the public disclosure, go read the security bulletin [1]. CVE-2023-24033, CVE-2023-26496, CVE-2023-26497, and CVE-2023-26498 are the big modem vulnerabilities.
There is an enormous difference between a vulnerability in CentOS Stream, and RHEL. The article title is about CentOS, but the body says RHEL is affected too.
Exactly, I followed the CVE link through, and all affected RHEL8/9.
Soo…… Red Hat isn’t patching their live, currently supported distributions? That seems bigger news than their approach to CentOS, which they are clearly trying to kill.
CVE-2023-0590 is already patched in C9, meaning the fix will likely will be shipped in RHEL 9.2. The other CVEs will probably get fixed as well (in CentOS first!), but when the RHEL maintainers are ready to, not based on some arbitrary deadline from a third party. Lower severity CVE fixes are routinely delayed until future minor versions, so this is nothing new, just an example of a security researcher not understanding how RHEL works.
You're obviously a Red Hat employee. It might be worth disclosing it.
I don't quite agree though. Sure, RH can decide when to patch, but a researcher isn't wrong just because they say "Hey, RHEL isn't patched".
Should RHEL be patching sooner? Maybe. Though I get patches can have unintended consequences. However, I like the idea of a third part scrutinizing this stuff. Otherwise companies will do the wrong thing and claim their security posture is perfect.
LTS is a dangerous concept, really, especially in the way Debian or Red Hat do it, that is, by freezing versions at a completely random point and then attempt to backport fixes.
This approach is outright dangerous, because in my experience a distro maintainer is often not as accustomed with the codebase as an upstream maintainer. I understand the need to keep versions stable, but I would rather use a real long term support version from an upstream maintainer than random patches from Red Hat.
That's why in my experience ironically rolling releases tend to be less finicky to use than long term support distros.
Personally at my work I've been trying to navigate migrating our CentOS 7 production servers to EL8 (Looking at Rocky Linux) and it's been a bloody nightmare thanks to RedHat prematurely killing CentOS8. It seems like there isn't any kind of tools available on the internet to migrate off CentOS to another RHEL clone, so we may end up having to rebuild of our affected production servers before doomsday in 2024. Rocky has a bash script, but it only supports systems running EL8 or higher.
Yup. I was working on a migration to CentOS ~8.2 before we got rugpulled. Took the opportunity to move to Ubuntu/Debian. If we have to do all this work, might as well change distro's.
With RHEL, it has always been recommended to migrate your entire system to a new install of EL. They are rather notorious, historically anyways, for not providing their customers with an easy upgrade path.
I basically learned to decouple the services from the OS it's running on (back before containers and all were a thing) so I focused on making it easy to spin up a new machine and restore a service. Over time, this made OS upgrades immaterial and made it easy to spin up an additional instance.
This led to containerization and virtual machines being a good fit for my habits.
Converted my homelab media server from CentOS 8 to Stream 8 early last year, haven't noticed an appreciable difference in stability or use-case. Stream 8 is EOL next year though, so I'm considering options of what to move it to.
At work an intentional decision was made to avoid migrating any prod/dev systems (previously on CentOS 7/8) to CentOS Stream.
