I think there was a very large knee-jerk reaction to the CentOS announcement, myself included. I immediately began looking into alternatives, completely discounting Stream as a viable production OS, a perception shared by many commentators around the web.
But then I actually tried out Stream, and thought deeper about it. I don’t really need to exactly match RHEL’s point releases. I don’t need 100% binary compatibility with RHEL. My installs don’t break between point releases or errata updates, why would a rolling release change that? I just need a rock-solid OS to run my applications on, and Stream seems to fill that need. Just ‘cause the update model was changed doesn’t mean the OS is unstable.
I agree with the authors claim that 95% of CentOS users will be perfectly happy with Stream. Of course, there is that remaining 5% gap, where I understand the frustration of those in that situation.
I already roll several other Red Hat upstream projects in production, including AWX (Ansible Tower) and oVirt (RHV). I’m sure RH would love for me to pay for support, but I don’t really need it. The upstreams serve me well enough.
The one thing I can’t get over though, is the decision to roll out this change in the middle of CentOS 8’s lifecycle, trimming out what, 8 years or so of support. That seriously stung. This decision should have been made a year or two ago, before CentOS 8 was released.
Based on my experience working for a hardware vendor and now in another role where I work with clients of all sizes in the enterprise, I think saying this works for 95% of users is an intentionally gross mischaracterization of the user base.
Literally every enterprise user I work with that has CentOS uses it specifically because it directly tracks RHEL. They don't want to pay for RHEL licenses in test/dev but need a binary compatible version to ensure they can get enterprise support in production should they hit an issue. Having different drivers/kernels/libraries is a non-starter.
There's absolutely no way that Redhat is so blind to their user base that they don't know this is happening. This is a pure money grab, if they can get even 5% of the users doing what I've seen paying for licenses, it's a net-win for them.
I'm 100% onboard with Redhat doing things to make lives difficult for companies like Oracle that are trying to make money off their hard work. I have no time for user-hostile moves like this, every justification they've given is hollow IMO which is why they're still trying to do damage control and persuade the public that it's not what it looks like.
RHEL is a subscription mess when working across its zoo ("Is our dev mirror the Azure's managed RHEL repository with mismatching kernel headers? Or pass through to RHEL's with epol disabled? Or RHEL 8 with docker repos disconnected so they can push their podman alternative?"). Losing centos hurts. As soon as you get into basics like drivers or build tools, RHEL quickly goes off the rails, and centos gave a semi-predictable escape hatch.
GPU land is directly on this path, so I feel awful for enterprise IT admins we work with who just want to do a good job for their client hospitals, schools, businesses, etc. We help shield them most of the way by reducing the install to just a gpu driver + docker, but those deps still need basic pinned version ranges. RHEL makes getting those hard, and worse, as life happens, make it hard to get the right diagnostic tool versions (gcc 7, ...). It's a bad look for a 2020 infra vendor to charge an enterprise for reliable and predictable rollout sw/support when RHEL makes it a marathon to get the right major release of base layers like docker/gcc/drivers. Contrast this to our cloud users, who one-click launch, or our self-managed cloud/ubuntu/etc. ones, who can generally get the right drivers + docker stack up in 10-15 provided commands.
Watching the chaos spread into centos makes me even more nervous about critical software relying on RHEL, and we already had to have The Talk with enterprises wanting HPC/ML on RHEL before this..
You're saying it's a "gross mischaracterization." Does that mean you have more/better data than Red Hat/CentOS about CentOS usage?
> Literally every enterprise user I work with that has CentOS uses it specifically because it directly tracks RHEL. They don't want to pay for RHEL licenses in test/dev but need a binary compatible version to ensure they can get enterprise support in production should they hit an issue. Having different drivers/kernels/libraries is a non-starter.
Unless you are doing kernelly things then the small difference in kernel version will not be that big a deal. Also for most applications, dev/test is better on Stream since it represent RHEL next. If the next version of RHEL will break your app, you find out in dev/test instead of in production like you used to. I think it's worth considering if this isn't better for your use case than previously.
It's also not very hard to compile your own kernel if you really need an exact version. I do that sometimes on Fedora when I'm testing a driver I maintain. You can also just keep the RPMs from either previous releases as well (that's not a bad idea so even if you're building your own kernel you can extract the config file from it).
The entire value proposition of CentOS/RHEL was you don't have to do things like compile your own kernel. You have a relentlessly tested operating system with the certainty of knowing it is compatible with anything you could run on it. You don't want to manage multiple versions in the first place.
> Unless you are doing kernelly things then the small difference in kernel version will not be that big a deal.
> It's also not very hard to compile your own kernel if you really need an exact version.
It isn't about having a specific version because the RHEL kernel backports features anyway. If you are compiling the RHEL kernel you might as well use the actual distribution that has been tested with it. What would be the point otherwise?
Yes, I think that use case falls in the 5% gap that had the rug pulled under them. But I really do believe 95% of CentOS users don’t really need that, hosting stuff like HTTP servers or containers. Stuff that really doesn’t need to match RHEL.
Binary compatibility is a myth: if you actually need 100% binary compatibility, then you need the actual binary from Red Hat because neither Red Hat nor any other commercial distro offers reproducible builds.
CentOS had to reverse engineer code drops from Red Hat just like Oracle or Amazon. Oracle will sell you a support contract claiming "binary compatibility" even when they are using a newer kernel than what's in RHEL.
They are trying to make lives easier to collaborate with their frienemies who all maintain a RHEL clone. That means calling CentOS what it actually is: upstream of RHEL.
> I don’t really need to exactly match RHEL’s point releases. I don’t need 100% binary compatibility with RHEL.
That's good, because regular old CentOS is not bit-for-bit identical either. CentOS might strip out all the RHEL trademarks, but they had to reverse engineer RHEL code drops just like Oracle, Amazon, Google, Facebook, SuSE, etc.
> The upstreams serve me well enough.
What you don't appreciate is that the distinction between upstream/downstream is a broken metaphor across small time scales. Bug fixes and backports will show up in any number of RHEL clones from Oracle|Amazon|Facebook|Google|Baidu|Samsung before making their way into Fedora|CentOS and finally into RHEL. In the future, RHEL will release their own bugfixes to subscription customers before those fixes find their way into CentOS Stream.
That's the meat grinder of a Linux distribution: uptime "guarantees" are just refunds you get to cash in when a large devops team can't fix a problem fast enough.
> The one thing I can’t get over though, is the decision to roll out this change in the middle of CentOS 8’s lifecycle, trimming out what, 8 years or so of support. That seriously stung. This decision should have been made a year or two ago, before CentOS 8 was released.
If you have a kernel that hasn't been updated in over three years and you need very low downtime, you either have a dedicated operations crew or you don't let it touch the internet.
But if you are so cheap that you don't want to pay for bandwidth/build servers yet aren't willing to violate the "developer" license agreement ... then use a RHEL clone from Amazon|Oracle. They are already undercutting Red Hat support packages, why not make them foot the bandwidth bill too?
Everyone else should be celebrating: Red Hat is basically turning CentOS Stream into a spot where they can collaborate with their frenemies and ship more software faster.
Qubes developers for example, have been spread very thin by Fedora's six month release cycle. Switching away from an RPM distro would be too difficult but CentOS lags too far behind mainline (5 years!) to be useful. If Red Hat can step up their game and push out a major release very 3 years and get the "pipeline" between Fedora and CentOS working ... maybe they can win some market share back from Debian and Ubuntu.
> That's good, because regular old CentOS is not bit-for-bit identical either. CentOS might strip out all the RHEL trademarks, but they had to reverse engineer RHEL code drops just like Oracle, Amazon, Google, Facebook, SuSE, etc.
Do you have a source for SUSE reverse engineering RHEL code drops? Last I checked they were unrelated.
"[A]fter approaching and discussing [their problems] with the CentOS Project core team, Red Hat and the CentOS Project agreed to 'join forces.' We said joining forces because there was no company to acquire, so we hired members of the core team and began expanding CentOS beyond being just a rebuild project."
At the time, did Karsten et al. disclose this to the CentOS community? This language suggests that Red Hat had long-term plans for CentOS and its core team all along, but to my knowledge, the community was not made aware of these plans so that they could take defensive action if needed.
The intention behind this was obviously a business decision[^1]. You don't need to read an article with quotes from redhat executives to understand the reasoning. Putting out articles like this to obfuscate the reasoning just creates distrust. Obviously Redhat wants to move as many people as possible over to subscriptions in light of cloud offerings hurting them with their core enterprise market.
I think one should give the new Centos Stream a chance and try it out before judging it. What is CentOS stream will later become Red Hat enterprise Linux, so Red Hat and the Centos community is interested in Centos being good and stable.
I completely agree. CentOS Stream will only receive updates that have passed RHEL QA and CI testing. These are patches that would normally go straight to RHEL in the next point release, so these are stable.
Even Fedora is pretty damn stable. Stream will be far, far more so. I'm planning to run Stream for all but one of my use cases (my home router, which is a SPOF for my internet connection. Not sure what I'll run there yet).
I suspect that RedHat is actually planning on making a no-cost version of RHEL available without support. I think they are still in the process of deciding exactly how to pitch it without cannibalizing existing sales.
They are using this fallout as an opportunity to gauge the market. Risky to be sure, but they didn't do it without considering the risks.
From the article:
> I am confident that CentOS Stream can cover 95% (or so) of current user workloads stuck on the various sides of the availability gap. I believe that Red Hat will make solutions available as well that can cover other sides of the gap without too much user heartburn in the end
They have already said they will be expanding the license for the developer edition, but there’s no way they will be allowing it on production systems.
And they should have had this all worked out before making the announcement. As it is, they are just promising a bunch of things that don’t yet exist, after they literally just destroyed all trust in them.
I work for Red Hat but have zero inside info on this. This is just my opinion.
I personally think they will be allowing some production use of free RHEL (assuming we get free RHEL. I expect we will but don't want to count chickens before they hatch). Enterprise production I don't know, but personal production or prod for non profits and/or open source projects, I could totally see that being an option. I'm personally excited for the announcement. I hope they get it out soon.
If that's true, this would go down as one of the most monumentally stupid executive decisions in Open Source history. It would make them look really, really stupid, like they had no clue why their users used their products.
Nobody wants to use CentOS. We have to use it because it's the cost-effective alternative to RHEL. And nobody wants to use RHEL either, but we have to, because vendors won't support anything else.
Part of this is our (the users'/businesses') own fault. We allowed our businesses to get away with depending on an OS with no support until we needed it, which is not really sustainable.
What we maybe should have been doing is creating vendor-neutral open standards for the platform, like a standard kernel (vanilla, anyone?) and a standard userspace distribution (vanilla, anyone??). That way any Linux distro could "meet the standard" ("Install package standard-distro-20200816 to use packages and settings aligned with the standard", and then every vendor could support "the standard". Vendors wouldn't need to lock themselves into one distro, and we wouldn't either. Basically a POSIX for Linux. But there's no money to be had in that.
(It is in fact possible to run a Linux distro without patches. It's buggy as all hell, but it can be done.)
Yep Red Hat is in the process of figuring out some no-cost options (and lower cost options as well). I work for RH but am outside of this so have no internal insight.
> They are using this fallout as an opportunity to gauge the market. Risky to be sure, but they didn't do it without considering the risks.
Yes you're right, but it wasn't done this way intentionally. I agree they should have coincided these announcement, but it didn't happen that way. They're doing their best to work with what they have, but they certainly didn't intend to stoke the outrage in hopes to study it :-D
Lots of people use either Debian testing or significant amounts of backports on servers. I'm down with the idea that a non-enterprise distro with a contribution pathway is useful, and enterprise stability is mostly unnecessary.
But functionally, I don't think this helps resolve the issue. The issue is the business model, and RH will never make people happy while it attempts to charge for the OS. The basic thing people are after is RHEL itself, at no cost. That feels unreconcilable.
But then I actually tried out Stream, and thought deeper about it. I don’t really need to exactly match RHEL’s point releases. I don’t need 100% binary compatibility with RHEL. My installs don’t break between point releases or errata updates, why would a rolling release change that? I just need a rock-solid OS to run my applications on, and Stream seems to fill that need. Just ‘cause the update model was changed doesn’t mean the OS is unstable.
I agree with the authors claim that 95% of CentOS users will be perfectly happy with Stream. Of course, there is that remaining 5% gap, where I understand the frustration of those in that situation.
I already roll several other Red Hat upstream projects in production, including AWX (Ansible Tower) and oVirt (RHV). I’m sure RH would love for me to pay for support, but I don’t really need it. The upstreams serve me well enough.
The one thing I can’t get over though, is the decision to roll out this change in the middle of CentOS 8’s lifecycle, trimming out what, 8 years or so of support. That seriously stung. This decision should have been made a year or two ago, before CentOS 8 was released.