CentOS Becoming Rolling(-ish?) Release

Sorry if this sounds like it’s addressed to you, it’s meant as a reply to the link you provided (unless you are the author).

While there were complaints about change and communication, the most complaints were about axing support of CentOS 8 before even CentOS 7. By far, the most numerous (at least on Reddit).

I’ve never even seen a CentOS user file a RHEL issue (aka Bugzilla) or ping me on Twitter for CRI-O, Podman, Buildah, Skopeo, or anything else I work on at Red Hat. These users weren’t even on my radar.

Why in the world would a CentOS user file a bug report @ Red Hat and not @ CentOS? Red Hat’s Bugzilla is for Red Hat customers, ie RHEL users, not CentOS users. This is a non sequitur. Look at https://bugs.centos.org/view_all_bug_page.php for reported bugs.

At point #3, the author mentions the problem, the axing of support, but instead of explaining why (aside from a generic “we need the community to use CentOS Stream”), he briefly admits Red Hat could have been waiting until CentOS 9, then speedily proceeds to beat a strawman of CentOS competing with RHEL. CentOS is not and was not a RHEL competitor. RHEL was there for people who wanted support. CentOS was “you’re on your own and you have no guarantee of the software working.” There may have been a little overlap between them, but there was no competition between CentOS and RHEL, unlike RHEL vs Oracle Linux vs SUSE Linux Enterprise vs heck, even Ubuntu. CentOS was to Red Hat just like Oracle Linux w/o support, Open SUSE and Ubuntu w/o support is for each of these distros. Now that I addressed this, back to the main issue: axing of support. I would understand if CentOS was a community project and not under RHEL (even w/o funding, like the author claims) and if the developers decided to end the project. It would have been understandable, really, like take for example Debian or Gentoo, you have no assurance that the devs will keep working on them, just historical commitment. CentOS on the other hand…

Oh, I just realized. First, the author makes a point about CentOS users being lazy f$%^wits (my words, not his), then he says that Red Hat needs those lazy f$%^wits to contribute to CentOS Stream. Kinda contradictory, if I say, but I’m willing to forgive. I 100% agree with him that we are all humans, including Red Hat employees. Everyone makes mistakes.

In point #4, he argues that buying CentOS meant Red Hat had to give out more money from their pockets. You don’t say? I mean, did they expect to buy it and just leave it to rot? It’s something Red Hat thought deeply about before moving forward with the decision to buy CentOS. They knew they would spend more, but he argues as if CentOS users are at fault for the increased costs, but Red Hat took this upon themselves. This is not an argument.

What you might not realize is that whatever workload you are running on Downstream CentOS is creating value for your business, especially in years 6-10. Moreover, you’re likely charging for the service you run on downstream CentOS and capturing value. This is a leaky value chain that isn’t sustainable.

It’s the other way around. Our company is developing our software on CentOS, because our customers want to purchase support for the OS they are using. We literally don’t need CentOS or RHEL in our production environments, we could use Debian, Gentoo, pretty much anything for our prod (by prod, I mean things like GitLab, internal DNS, samba, public websites, proxies and reverse proxies etc.). Which is why we are migrating to Oracle Linux (already did a few). And because we are lazy and the migration script from oracle works wonders on CentOS 8 (we won’t upgrade CentOS 7 where we have it). But our product’s development environment is CentOS. Our customers are buying Red Hat support, so they are using our product on RHEL. So basically, it’s because of us that Red Hat is making money, we are not making money because of Red Hat. Oh, and our customers also use Oracle DBs, so we could easily move to Oracle Linux and say our software is only supported on Oracle Linux and we won’t offer support if they use RHEL, then deny Red Hat some revenue. But we don’t. We don’t care what our customers want to use (my personal opinion is that I’d rather Red Hat makes money from our customers than Oracle, so keep this in mind).

To be honest, 5 years of CentOS Stream is plenty. We live in a fast moving world, so I would rather upgrade every 3 years or so. And I believe CentOS Stream would be solid even for production use. But as Greg Kurtzer put it, not RHEL solid, which is what CentOS is.

There’s a perception that CentOS Stream will be less “stable” than downstream CentOS. This could be rephrased as, “this pisses me off because I always relied on paying RHEL users to absorb these changes, test them, and file bugs for me!”
Yet another strawman. As mentioned, people using CentOS who had problems, reported to CentOS. And it's true that Stream is less stable than RHEL, what is the author even trying to say? Sure, CentOS Stream won't be unstable like Ubuntu daily or Debian sid (which are also stable themselves, just not LTS or stable releases rock solid). But again, it won't be "RHEL solid." Otherwise, why would Stream even be open to the public and not for paying customers only? Red Hat is gaining value from Stream users, because the later will contribute to improving the next minor release of RHEL with not just bug reports, but also potentially contributing fixes themselves. Yet the author is arguing that CentOS stream will be "just the next version of Red Hat for free," without mentioning the benefits Red Hat receives from a widely used CentOS Stream. and users get to add "the next cool things in RHEL." This is fishy, especially because he even mentioned the development process in Red Hat. Bugs used to escape from testing to staging and now Stream will be the new staging, where the most eyes will contribute and have an even better RHEL downstream.
#7 You’re Not Looking For Opportunities to Improve This Ecosystem
True. I don't claim some moral highground, I use what is available to me for the lowest price _and also live with the consequences of such actions._ If there are bugs or breakages, I await for a fix (otherwise, I would be a paying Red Hat customer). There is a reason why FOSS support is profitable. There is a market for security (as in, "assurance," not anti-hacking).

(edit: I pressed CTRL+Enter instead of simply enter and posted the comment before I finished it)

RHEL and CentOS have become tied at the hip from an ecosystem perspective
True, I mentioned it above. CentOS has always been like an Ubuntu w/o support, but for RHEL (and a lot better than Ubuntu).

Finally, I will mention this again: CentOS Stream should be pretty good for prod environments. I don’t doubt Red Hat’s engineering. But not as good as RHEL. As I mentioned, I would probably even use it in some places, just not the mission-critical stuff. For those, I’d rather have a RHEL Clone. If no clone would be available for free, I would probably be using Stream. Sure, I am angry with the axing of support for CentOS 8, which is why I’m migrating to Oracle Linux. If things weren’t developing like this, who knows, maybe I would have used Stream in mission-critical stuff.

Another thing to mention, in my company’s position with CentOS development environments, where our clients use Red Hat, I believe we could probably receive free developer licenses from Red Hat, but I would bet it would be some bs thing like “up to 10 licenses.” As I mentioned, we aren’t making money from Red Hat, Red Hat is making money from us, through our clients (and I honestly believe that’s a good thing). But making us pay them for helping them make money is ludicrous. We have around 150 VMs running CentOS, a different environment for each of our clients. We have to have the same environments as our clients do to offer them support for our software. Our clients are not rushing at all to move to CentOS 8 (some are still even on CentOS 5, and they are multinational corportation and just last year started upgrading, it’s ridiculous!). We will keep going with CentOS 7 (and the rare instances of 5 and 6) for now, but in the future, our entire development infrastructure will either have to move to containers, or all our VMs will move to Oracle Linux or Rocky Linux and just pretend it’s the old CentOS. And just to add the cherry on top, if we had to manage RHEL Licenses, that would be an even harder pill to swallow.

This comment is a little grim, but I wholeheartedly hope RHEL and CentOS Stream keep making Red Hat money. Their commitment to Open Source is genuine and it would be devastating to see them fall.

Edit 2: only fixed grammar mistakes.

4 Likes

It’s technically upstream, so maybe you’d file a bug there, but in my experience, the bug was either already fixed in current RHEL or upstreams to a package or maybe all the way to Fedora.

Agreed, and really RH should just release a free RHEL version with no support. I totally understand why they don’t want to deal with supporting CentOS anymore. It’s completely redundant and a waste of time. Just add a free tier to the subscription manager and let the CentOS community use RHEL without support… and then they’ll definitely start filing bug reports and contributing to RHEL.

5 Likes

:point_up:t2: :point_up:t2: :point_up:t2:

5 Likes

Yeah, but that’s Oracles jam.

Oracle can only do it because Linux Support ain’t their only cash cow.

RH can’t really afford that.

The reason this is problematic is that companies like Dessault Systems, and Siemens are very bad at supporting their programs on the Linux platforms they support.

Instead of saying “our program depends on the following system libraries at these versions” they say “it works on RHEL 7, or SUSE, and if you use any other distribution we won’t offer you any support”

So, is you’re like me, you install CentOS 7 and are able to convince the support team that it’s equivalent to RHEL.

I expect that this will become an issue sometime in the next few years when the software vendors bother qualifying RHEL 8 as a supported platform, and will really bite when RHEL 7 support ends sometime later.

Why do we not consider this to be broken software?

If software cannot properly run on a modern system, it is broken.

If software is broken, it cannot be used in an enterprise environment.

“we have no other options” is not an acceptable answer. What did we do in the '30s?

2 Likes

It would arguably be cheaper than maintaining CentOS as they have for the past 6 years. In addition to support, RHEL could still paywall fancy things like RHV, RedShift, idM, etc (yes I know all branded versions of FOSS things, but more stable and with support).

I obviously don’t know how it will all play out, but I think they’re going to lose more business to Oracle than they’re going to gain if they kill mainline CentOS without a free RHEL.

3 Likes

If they just leave CentOS to languish without proper alternative that leaves a pretty big hole in the market that someone enterprising could fill, potentially really hurting Red Hat in the long run.

Oracle seems like a reasonable candidate here, if they throw their weight behind a community fork of CentOS (like Rocky Linux), or create one themselves, that could shift public perception rather in their favour, I imagine.

Still, big investment, somewhat risky, and Oracle and risk…

Interesting times, to be sure…

1 Like

They do have a free rebuild of RHEL available, that’s what people are mostly talking about in thread when they mention Oracle.

https://www.oracle.com/linux/

2 Likes

That’s literally what Oracle Linux is. I mean I guess it’s technically based on RHEL, but it’s close enough that you can easily convert CentOS to Oracle linux.

I’m aware of Oracle Linux, and what it is (among others, one of three distros that still run on UltraSPARC).

It’s not community maintained though, is it? If it is they’ve managed to hide it well at least.

In the past Oracle could just point to CentOS if people wanted to “see the code”* (barring the kernel modifications, I guess), but that won’t really be the case anymore, hence the “void” I mention.

*by which I refer to the the way the packages are built, which was CentOS’ entire claim to fame, not the actual application source code

I’m aware it’s a RHEL fork. CentOS got big by being a community project that attempted to stay compatible with RHEL (at least until after RH annexed the project).

Being open leads to quite a bit more trust, certainly when companies like Oracle are involved, not to mention community goodwill. Corporations probably don’t care about the latter too much, but being pushed by employees is how most big FLOSS projects get a foot in the door in the business world, including Linux itself, and being blatantly anti-community tends to be how they die, or at least fade into insignificance (Oracle knows a thing or two about that :wink: ). Hell CentOS just being there probably helped RHEL get accepted in places it other wouldn’t have been.

If Oracle wants to capture mindshare then working with the community would be to their benefit, certainly given Oracle’s reputation, and likely to Red Hat’s detriment, since they’ve been doing “everything but” especially when it comes to the entire CentOS thing.

1 Like

Yeah definitely agree with that. Oracle could just as easily pull the rug out at some point.

I’m going to wait to see which way the appliances go. oVirt in particular. I’m not sure if they can pull off a complex hypervisor on stream. I tried to install it manually the other day and I couldn’t get it to work. I think they rely on a lot of consistency in package versions.

2 Likes

Not sure what features oVirt has, since I never used it and was never curious enough to look beyond what it is in a few minutes. But if all else fails, you could try OpenNebula (it’s also built on top of KVM with libvirt). We used it in my company in the past. It was quite quirky (if you weren’t careful, you could delete a VM you did not want to, but we found a workaround by clicking the VM itself and not do a quick action from the selection menu) and it had too many features that didn’t help us in particular, like cost control, but for a DIY private cloud, it basically competes with OpenStack (to some degree) and it’s easier to implement. But depending on the size of your cluster, it could take a little to migrate everything; it was easy for us to migrate from OpenNebula to Proxmox, we could have automated it, but it only took 1 person, ie me, just under a week to recreate the VM configs and move the disk image files to the new locations (which were on the same NASes).

I still have nightmares with OpenNebula, but now that I think about it, I probably didn’t give it a fair chance back then (and neither did my colleagues), not to mention it had lots of updates since. Our implementation was also kinda bad, since the Orchestrator was running on a virtualization host, instead of a separate server (and if the host would ever go down, it would take the orchestrator with it, however, the VMs on the other hosts would keep running and even be accessed via Virt-Man).

1 Like

I’d probably go with Proxmox, although maybe xcp-ng depending on how they divorce themselves from centos (although they’re still on 7, so probably won’t make that decision for a few years).

1 Like

Woo

7 Likes

I wonder how this fares for the cPanel project. They were quite upset about the switch to stream. Crossing fingers they will support RHEL going forward.

Oh hell yes

Idk if there are any legal issues with using oracle in an appliance but that still seems to be the best solution for them.

Curious what this means for FreeIPA, oVirt, OpenShift, etc since RHEL already has them in rebranded form. I guess they will just be unstable versions in stream.

Also, why didn’t they announce this at the same time as the CentOS news? Would have saved them a lot of bad press.

2 Likes

Yep, just like RedHat intended CentOS Stream to be. Well, again, I believe “unstable” is a bit of a misnomer, it should be pretty solid, but again, not “Red Hat solid.”

Apparently Mattermost and AWS became Rocky Linux sponsors:

2 Likes

There are so many moving parts in ovirt, openshift and FreeIPA, they really rely on locked package versions or things will break. CentOS stream should be stable as an OS but these big services might not be.

1 Like