Which Hypervisor would you suggest for my situation?

[Janos]

I recently built a 2 node Proxmox cluster with an virtual witness host, which I put on another Proxmox system, which also works as a Proxmox backup server. This is the smallest possible setup, if you want to have a suitable backup solution too.
The backup server runs as a VM with the HBA passed through. I connected the Storage Network directly without a switch with 40GB/s Mellanox nics. You have to separate the use of the networks as with VMware depending on the role, storage, client and cluster network. vMotion works fine and also HA, but it’s not like with vmware and vsan crash consistent at least not without additional effort.

https://pve.proxmox.com/pve-docs/chapter-pvecm.html#pvecm_adding_nodes_with_separated_cluster_network

[risk]

Hi @Jrx1216 , how’s everything going.

@Jrx1216

That’s the part that always confused me. I’ve never really seen enough benefit to using the host’s kernel over just installing those services on a host machine and configuring an out of memory daemon to restrain anything that gets out of control. Maybe I’ll play around with them on some of my more disposable hosts, but I’ve got enough storage and RAM that I’d rather just run full VMs for now, I think, especially if management of updates works the same between the two.

It’s all different tradeoffs along the same spectrum of tradeoffs as always (sharing vs isolation).

I wonder, with your prior experience, what you’d think of kubevirt and ceph iscsi gateways.

[Jrx1216]

@risk

Hi Jrx1216 , how’s everything going.

Pretty well, thanks for asking! I did end up settling on Proxmox with a few slight modifications, though I ended up completely restructuring my lab in the process.

My old storage server VM now runs on bare metal (an R430) and I’ve transitioned all of my other workloads to a cluster of old Optiplex devices that I’ve added 2x10Gb/s NICs to, with the exception of a couple of virtual “AI” workstation VMs that I have running on an old Gaming PC I bought of my roomate for $400 (not bad for a Ryzen 3600 and a 1080ti, I think?)

I ended up just using NFS shares for most of my storage for… Well, everything, haha, as it seems to preform well enough with 10Gbe, and it’s way easier than bothering with setting up ISCSi. I’d started looking into ceph, but I think I need to upgrade the local storage in all of the boxes before I go too deep down that rabbit hole.

The failover works well enough that I trust my lab workloads on it, and the difference in noise is amazing, haha! After replacing the fans on my MD1200, the whole lab is virtually silent at the moment.
I’ve obviously lost a bit of portability and DR potential with my storage server, but I’m now backing it up entirely to backblaze B2, and If needed, I could always throw it’s 2 HBAs in another server I have lying around, re-install TrueNAS and just import my config to get back up and running well enough.

At this point, I don’t think I’ll be looking back at ESXi anytime soon, but I do wish I’d given XCP-NG a bit more time to properly evaluate it. I couldn’t be bothered to compile from sources when I first eval’ed it, but if I had, I might have went that way instead. From what I’ve seen since deciding to go with Proxmox, at least, it’s not necessarily better for any of my current use cases, but I do think it would be a great tech to learn and I feel like I’d be more likely to implement that for a client over Proxmox at the moment… If I were doing any client work right now, haha! If I’d given it even a few more hours of time, I think I’d probably have realized that, but I certainly don’t have any (major) issues with Proxmox for my current workloads.

Thanks again to everyone in this thread for the assistance!