Hacker News new | past | comments | ask | show | jobs | submit login

>> Imagine if you were running a business, and deployed CentOS 8 based on the 10 year lifespan promise. You're totally screwed now, and Red Hat knows it.

The hypothetical you posed is the actual situation, I am now learning, I have apparently forced on my team. We've ramped up labor 3x revenue preparing product launch in 90 - 180 days. We created an image containing centos 8 , Java , postgres and tomcat a year ago and that what is deployed to beta clients and what we've been testing.

What's ironic is that I sort of went out on a limb with my team by forcing us to go with Linux over Windows and the way I allayed concerns was to ask them to just "wait and see" in hopes that the performance differential would make it a moot point.

edit: after a little thought it seems that moving to RHEL might cost us the least amount of money and downtime. Still sucks and not what we need to be working on right now.

What I don't get is this:

You build software, you ship it on an OS so it works, cool, this makes sense (I assume you need hardware and VM support, not just VM).

Why would you accept additional risk on the OS if you can easily reduce the risk, and ultimate cost, by going with an OS that has vendor support written into the actual contract? RHEL is 11-13 years total, Windows is... I'm guessing my grandkids will be using some form of Windows 10. CentOS is, and always was a community "best effort", with some serious delays occasionally (not often, but it happened).

A RHEL server license starts at $349, I have to assume that's at least an order of magnitude (or two or three) less than the cost of your software based on the technologies involved (sounds enterprise-solutiony). In other words a rounding error overall.

As a sysadmin in academia, this is not so straightforward. Since number of servers/VMs are in ballpark of over one hundred, RedHat license costs will be over 30,000$/year. This is not insignificant amount of money and not easy to get the money suddenly.

$30k is a significant amount, especially in academic environment, I don't deny it. But it's not much for > 100 servers. In my previous job, I had licenses which were about 15k€ for one server (thanks Oracle...) and the Windows Server licenses were also pretty expensive. $300-500 per server/year is not much more than a Photoshop license for one designer. And if we talk big business (1000's of servers and even more), you will get very special pricing anyway.

Not just price, but ability to function without constant access to the satellite portal. It's darn near impossible to upgrade a RHEL box from a work at home VPN environment.

> edit: after a little thought it seems that moving to RHEL might cost us the least amount of money and downtime. Still sucks and not what we need to be working on right now.

And that is exactly what IBM is counting on. Vendor lock in.

We are switching to Debian-based distros because, frankly, we don't trust IBM not to knife us in the back even on RHEL for more money. Of course we have the advantage of being able to take the time to convert.

This is not about IBM. It may not be a pleasant change for everyone, but it's a change that has strictly technical motives.

CentOS was acqui-hired because Red Hat's upstream for layered products (at the time mostly RDO/OpenStack and oVirt/RHEV) could not use Fedora because it was too far from RHEL a year of two after RHEL was released, could not use RHEL because upstream contributors would have to pay, and could not use CentOS because its releases had too large delays. The solution was to make CentOS releases happen timely by paying people to make them.

These days a RHEL downstream is not enough for the layered products. Some of them require the kind of bleeding edge feature that is backported every six months to the RHEL kernel, and corresponding userspace changes (BPF, virtualization, etc.) and cannot afford waiting for the CentOS release because development must be done in parallel with RHEL. So the solution was to move CentOS from happening after RHEL to before RHEL which is what CentOS Stream is.

I can confidently say that the reasons are technical because other CentOS downstream have the same needs (e.g. Facebook's) and they also want to send patches to CentOS for bugfixes or features themselves, instead of waiting for Red Hat to find out about that bug, or decide they need the same feature. Plus there's no reason for rebuilds to disappear. The SRPMs will still be released by Red Hat.

That in no way explains why they don't continue to have both. They indeed will have both for another year. There was no requirement the Stream product even use the CentOS name.

CentOS was a community project whose leadership and control was taken over (acqui-hired as you say) by Red Hat and then it's core use case for the majority of people actually using it was discontinued. That is a statement of facts that happened as I understand them, not some spin on my part.

If Red Hat had not stepped in, perhaps some of CentOS problems (trouble getting releases out on time) would have been worse, or perhaps some other companies would have stepped in. We don't know, but we do know that CentOS has not been changed to be something different than it was before. It used to be a free re-spin of RHEL. Going forward it's something entirely different.

Red Hat always had the option to stop funding/providing resources to CentOS and name their new thing something else, but they didn't, and now they've effectively co-opted CentOS to be something different than it was originally intended to be.

> That in no way explains why they don't continue to have both

Because they don't need it anymore. CentOS Linux or other rebuilds can still exist (just not using the name; I disagree with that but I can understand Red Hat doesn't want its name attached to something that might have large delays in security fixes in the future) if somebody funds it or volunteers to do it, just like CentOS still supports Xen but RHEL does not.

Also for what is worth there have been lots of engineering changes to RHEL in the past couple of years that make nightlies (and CentOS Stream) much more stable than they used to be, especially with respect to regressions. Running CentOS Stream is not going to be like Fedora Rawhide or Debian sid.

>>> it's a change that has strictly technical motives.

I understand the business reasons for doing so. I don't agree with anyone branding this as done for purely technical reasons. Having CentOS Stream may be needed for technical reasons. Stopping CentOS 8 is in no way a technical decision. They are unrelated in any technical sense.

If Red Hat just doesn't want to put resources towards CentOS as it traditionally existed anymore, that's their option, but they deserve any flak they get for taking over an open source project just to extinguish it, since CentOS is in no way really needs to be linked to their Stream product. They could just as easily called it RHEL Stream and said it's free, and it would be a less confusing and more direct funnel of people that want RHEL stability into RHEL subscriptions. Using the CentOS name is just a mind-share grab and screwing over an open source community. They control it so can do it, but that doesn't mean I'm not going to call them out for doing so.

The thing is, Red Hat never considered the distro more than a side effect of providing a base for developing "things" that will run on RHEL. It's even written on the centos.org home page, the distro is not why CentOS existed in 2020. So the fact that users (including myself!!) enjoyed a free distro as a result was not a part of Red Hat's RHEL strategy in any way.

That only makes sense if Red Hat started CentOS. They didn't. The fact that they took control of it then changed it, even the web page, and then are not effectively killed the reason people are using it, is the thing I'm upset about.

If I effectively took control of the EFF and then a couple years later changed the website copy to say that the EFF is a vehicle for litigating cases that kbenson thinks are important, and then actually changed its actions to do so, would you argue the same points? How is this any different? Something that was a net good for many people has been taken over and eventually killed. I think we're all worse off for that.

> I think we're all worse off for that.

I don't disagree.

IBM is all about expensive support contracts. We cannot afford RHEL. So we went with CentOS 7 and recently CentOS 8. I migrated some machines from 7 to 8. Turns out I did that for one year. My boss wasted precious money there, and it isn't getting us closer to RHEL. My take is its getting us closer to Debian Stable instead. Which is RedHat's loss because a migration to RHEL is then more out of the picture. They know this. They thought of the above. And they are fine with it. That is not a technical decision, as you said. It is a business decision.

> Because they don't need it anymore.

Isn’t that the crux of the problem? CentOS used to be about “us” (the users), not “them.”

Fire up the wayback machine and find when the centos.org home page changed the mission statement. At this point it started to be about a different "us" than you think, an "us" that doesn't include you and me and presumably doesn't mind CentOS Stream at all.

In particular I started doing that and I got to the Sep 2019-Dec 2019 range, around the time CentOS Stream was launched. At that time this:

> For users, we offer a consistent manageable platform that suits a wide variety of deployments. For open source communities, we offer a solid, predictable base to build upon

was changed to this:

> CentOS Linux is a consistent, manageable platform that suits a wide variety of deployments. For some open source communities, it is a solid, predictable base to build upon.

> That in no way explains why they don't continue to have both.

They're a business ... are you honestly and in good-faith actually confused about this outcome?

Oh come now, this is Red Hat, a company based on Open Source software. We're not wrong for wanting to hold it to a higher standard. They have benefited from the OSS community just as they have contributed to it.

Red Hat didn't make CentOS they acquired it. This is very similar to a large corporation acquiring a smaller competitor, promising to continue supporting it, creating a new, different product using the brand, then dropping the original product.

Centos was always a free alternative to Redhat. I and many other people used it because it was a way to get the benefits of Redhat without paying for it.

This is a really smart more on their part to get people to stop doing precisely this, and getting them to pay money for RH. Pepople who are not willing to pay (me included) are now rightly annoyed, but we were never their customers in the first place, so we don't really matter.

I’m in the same situation, and I recently had started to plan on upgrading a small HPC cluster from CentOS 7. Before today, I had planned on CentOS 8.

Now? “¯\_(ツ)_/¯“ Probably Debian.

I’ve always used CentOS for clusters, but part of the reason for that is that there are some research packages that support RPM installation, but not deb. At least this gas historically been the case.

If a large amount (maybe even a majority) of users have to switch away from CentOS and RPM packaging, I think we’ll see an acceleration away from RPM as a default option.

So, in that way, I think we do matter, but just not on the balance sheet.

I disagree. Having a pool of sysadmins / developers who know how to manage CentOS makes RHEL a more appealing alternative than Debian for many companies. The three companies I've worked at have primary used CentOS for development boxes. Shifting to an alternative will change who is familiar with CentOS / RHEL significantly for the worse.

We do matter. We are part of the OSS community that RedHat and IBM benefit from.

I'm not confused at all about that. I was responding to the point that said "it's a change that has strictly technical motives." It's a business decision that's based on profit and positioning and name mind-share, so let's not hide that.

Yes i am, redhat had a really good ride being the trustfully and good guy.

I'm wondering, what ties you so strongly to a single OS? Nay, to a single Linux distribution?

I have a hard time imagining, and the only scenarios that come to mind are those where things have gone awfully haywire.

I'm sure I'm missing something. Enlighten me?

Software doesn’t run on its own. They rely on bunch of other software that is typically provided by distributions. So unless there’s some magic piece of technology that can take CentOS RPMs and make it work flawlessly on any Linux distro, the entire software industry is suddenly going to have to spend significant amounts of time on repackaging work.

There is alien[1], but I'm not sure if it is flawless, because of slightly different ways of doing things across distros like /etc/default vs /etc/sysconfig for example.

Depending on how good is the source, its usefulness and dependencies, packaging it to Debian is pretty straight forward. The dh_* helpers[2] does the job automatically most of the time. There is also tools for helping with specific languages, like dh-make-golang, dh-python, etc...

[1] https://packages.debian.org/buster/alien

[2] https://manpages.debian.org/testing/debhelper/debhelper.7.en...

From the looks of it, alien is just a tool to convert between different package formats and is not a replacement for proper packaging work. Very much like how you can’t take a deb package from different releases of the same distro and expect things to work properly, you can’t just take RPMs from CentOS to a whole different distro and expect things to work.

It’s not only the differences in the packaging format that you have to take care of. There’s also version differences, path differences, dependency handling, and many other stuff to take care of. These are the kind of tasks which can’t be automated away and require non-trivial amount of work.

For organizations that maintain tens, hundreds, or thousands of CentOS packages that spans multiple teams, moving to other distros would be time-consuming and costly. It would certainly pay off if it was driven by technical reasons, but for organizations that are forced to switch by this announcement, this is just pure overhead.

>Java , postgres and tomcat

That's his stack...

>So unless there’s some magic piece of technology that can take CentOS RPMs and make it work flawlessly on any Linux distro,

It's not magic, just alien.

> >Java , postgres and tomcat

> That's his stack...

What’s your point and when did I ever mention Java?

> It's not magic, just alien.

Dumping the contents of an RPM archive on random distros is never going to work except for the most simple of cases.

>Dumping the contents of an RPM archive on random distros is never going to work except for the most simple of cases.

That says allot about you packing hygiene, and you don't dump it..that you can do with tar alone. You convert it to a deb.

That says a lot about your packaging hygine. With alien, you're essentially dumping the contents of an RPM on hosts it wasn't meant for. It doen't matter that you converted it to deb format along the way, it's the same if you actually install it.

MyMy, you have no clue how packages and a package-manager works right?

Did you ever read a single line of alien source code? It extracts the metadata and files from a package, and re-archives it. That’s about it. It doesn’t handle package version differences, path differences, distro-specific rules, nothing. If your package has dependencies, forget about it because alien won’t retain dependency information it all. In other words, it only “works” for the most simple of cases.

I suggest you read the source code too, because otherwise you can cause real damage if you use it without understanding how it works.


Took me a minute to figure out where that came from, it was in the GGP comment.

But surely if you depend on those packages, you can maybe track (or just collect) their versions, then get the same versions in another package manager (possibly via pinning).

Though my experience with package manager is that dependency management is hell (rife with potential conflicts), so I do see the problem a bit better.

Still, it shouldn't take that long to fix? Like a few days of sprint to setup Nix or something like that?

I think it depends on the number of packages that you maintain. If it’s only a handful of packages, the entire process of repackaging, testing, and deployment might be doable in a matter of days. If it’s tens or hundreds of packages that depend on each other, I think that’s a whole different story. If those packages are maintained by different teams, it could take more than an year to complete the transition.

As for pinning packages, that’s only practical if you’re using Nix. As much as I prefer Nix over RPMs, not all of us have the pleasure of using Nix at work. It’s kind of a bummer because Nix packages are so much easier to work with and maintain compared to the competition.

An existing support contract?

If you run a for-profit operation, and downtime is costly, you (or your VP of eng) want a way to pay for immediate assistance from the OS's maker / distributor, when (not if) things go wrong.

> If you run a for-profit operation, and downtime is costly, you (or your VP of eng) want a way to pay for immediate assistance from the OS's maker / distributor, when (not if) things go wrong.

There is a difference between asking "how do I ensure there is one throat to choke when things go wrong?", and "how do I minimize the potential for things to go wrong?"

RHEL is a decent solution if you are trying to answer the former question (or both), but many budget conscious orgs focused on the latter and chose CentOS. Red Hat has now pulled the rug out from under them by trimming eight years off the EOL previously committed to with little notice.

EOL in this case means "no security updates" so even if your org was prepared, technically, to deal with a zero-day for example by rolling out an update in a timely manner without relying on paying a vendor for hand-holding, that option has now been eliminated.

Essentially, you now only get the stability and problem-minimization if you also pay the vendor for support. Otherwise you're stuck with a (relatively) unstable rolling release that will keep your internal teams very busy with a constant stream of minor issues, or potentially trying to roll-your-own updates or backports after the EOL for anything serious.

It's difficult to see this as anything other than a naked, money-grabbing betrayal of users.

I’d recommend using Oracle Linux 8 if you still need 1 to 1 compatibility. At my company I’m recommending Amazon Linux 2.

Not sure why I got downvoted. I’m trying to offer the poster a viable alternative to CentOS 8 with a minimum of changes needed. I get that a lot of people don’t like Oracle (and for good reasons) but really they are the best alternative (and free) EL8 disto now.

Consider downgrading to CentOS 7. It should work for 4 more years.

Can we take RedHat's word for it, though? CentOS 8 was supposed to be supported until 2029 before this bombshell announcement.


The dates on that page were always dependent on Red Hat. In retrospect, obviously we should have made that clearer, and we will endeavor to do so in the future.

Running modern versions of software on CentOS 7 can be a giant pain due to the old version of glibc that’s baked in.

I was supporting CentOS 6 until about a couple weeks ago.

Then surely you understand what a pain it is.

An easy way around this problem is to run the software in a docker container with a newer glibc.

Docker "works" but has huge performance costs (disk IO) on CentOS7.

Java and Postgres will work.

I go and drink some Corretto :)

Out of curiosity why did you decide to deploy your application as a VM image? Is there a reason you didn't go with a Helm chart or other container native deployment?

You're asking why someone who uses CentOS hasn't gone with Helm charts?

That's like polar opposites of stability and bleeding edge.

I was curious what drove the decision to deploy that way. I was under the impression most new applications being developed today would choose a more modern deployment method. There’s a lot to maintain in a VM image like that, containers just seem easier to me. Helm chart or otherwise

The amount of maintainance with virtual machines images or container images are mostly the same nowadays.

I will argue that there is less maintenance when handling virtual machines images, because it uses less bandwidth and need less tooling around it comparing with container image based infrastructure.

But in general both are nothing more than the golden image concept.

>a Helm chart

As a sysadmin i have never heard of this and es tun 800 VMs.

It's a bundle of jinja2 templates for YAML files defining Kubernetes API resources. There's an accompanying application, Helm, which applies values to the templates and handles the interaction with the Kubernetes API.

Helm uses Golangs built-in template language which is definitely not jinja2 compatible.

Basically the same thing: templating.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact