Need help choosing a hypervisor/k8s platform

High-Performance Computing Workstations & Servers
1

Hello all, I’m currently looking into choosing a hypervisor/k8s platform to deploy in a business environment. I already have pricing for servers, storage, etc. The problem is software. The original plan was to roll with VMware, but because of Broadcom’s purchase of VMware, the pricing is a joke.

Now, the requirement is to run several VMs with lots of a vCPUs for mostly compiling. We currently have 8 workstations each with a TRPro 5975WX & 128GB RAM used for this task. I can’t go into great detail about what the project is that we’re working on, but let’s just say it takes a lot of CPU.

We’re looking to move off these workstation class machines into a freshly built server room. The servers on the plan are powered by two AMD EPYC 9534 CPUs w/512GB RAM to deal with the workload. We’re going to start with 4 or 6 nodes depending on budget availability. We will likely add more nodes in the future as we grow.

We’re already pretty settled with hardware, but the problem is now software. VMware is prohibitively expensive. So that leaves us with Hyper-V, OpenStack and XEN as the other typical corporate approved solutions. That said, we’re willing to look into other options such as Proxmox.

The plan is to build our own “cloud” (this word makes me shudder, but here we are) where we can spin up both VMs and containers. A lot of the VM and container deployment will be handled by CI/CD pipelines, terraform, and other devops tools – an API is a must for these things.

The VMs will be a mix of both Windows and Linux.

The priority is CPU performance. VMware has a good reputation in this area, but having not used Xen or OpenStack for a while, I can’t appreciate the performance they may offer.

I’d like to keep the platform as simple as possible. The platform will be used by multiple teams.

So, here I am asking the L1 hive mind what hypervisor & k8s platform we should consider for our in-house cloud? What has your experience been with VMware alternatives? Do you have any horror stories? What about upgrades & maintenance?

2

I see OpenStack being more and more visible when I look for products regarding hypervisor/orchestration. The modular approach is very interesting indeed and it looks like a sophisticated open source product with active and professional development.

I’d just test drive this with available hardware and compare. OpenStack, running Linux, is just KVM under the hood (which is great) as with all of Linux distributions around virtualization. If you’re into object store, OpenStack has an own API called Swift that is different from the usual S3 but fully integrated into Ceph (which is the favored storage for an OpenStack cloud with Cinder block storage.)

Cloud is mostly about scale-out rather than scale-up, so lots of smaller nodes are usually the more efficient setups.

OpenStack is certainly on the rise and worth a look. But it covers many if not all bases with all the modules. The sheer amount of configurations is far from being simple, but it has much to offer.

I dug into into it and made a basic deployment, but that’s more tinkering and homelab stuff than actual production. But for vendor independent private or hybrid cloud powered by open source…that’s probably what you’re looking for.

3

If you need full on support you could go Nutanix AHV. (Yes on your own hardware)

Otherwise i would test ProxMox vs XCPng and see how your build compares on them.