Your only few options are Proxmox, XCP-ng, Virt-Manager, oVirt and maybe some plugin for cockpit. That’s for a data center admin. If you want a self-hosted cloud, in which you can have multiple tenants running stuff in the background, OpenStack is really have, not really recommend it, but there’s also OpenNebula, its competitor, much more lightweight and more tightly integrated and somewhat (?) easier to setup. Except for virt-manager, none of these require anything be installed on workstations, other than a somewhat modern web browser.
Additional notes: except XCP-ng, all of those run on QEMU+KVM, with Proxmox using its own tooling called qm
, while all the rest use libvirt
. The tooling doesn’t matter, it’s just CLI voodoo, they are compatible with each other, just a pain to migrate between qm and libvirt (ask me how I know).
Proxmox
By far the easiest to use and easiest to get into. By a long shot! You’ll probably already be familiar with the panels layout in the web interface. Easy to install, easy to configure, pretty intuitive, pretty powerful.
That is wrong.
No. In my home cluster, I had a Pentium G4560, a Lynfield Xeon x3450 and a Celeron J3455 all with different specs. All you need is similarly configured servers if you want HA and fencing, so that for example, a VM having 32 threads doesn’t land on a machine with just 8 cores 16 threads total. Or one that has 64GB of RAM landing on a server with just 32GB. It’s not that it could happen, it’s just that it would fail. I think it would even tell you that you cannot do that, but I never tried it, as all my VMs I had in a production environment were small, with the biggest one having 8 threads and 32GB of RAM and that was a rarity.
If you want a HA cluster, you don’t even need 3 servers, just 2 similarly configured servers and a witness / tiebreaker that can be even a RPi running corosync. Keep in mind that if you only have 2 servers, you need to only use about 40% resources on both, so that if all VMs live migrate to the other, you’ll have enough resources for them. Don’t over-provision CPU and RAM.
Proxmox 6 was kinda buggy, 5 and 7 were better. But still, out of around 10 or so servers, only 1 had weird network card bugs, renaming its interfaces to “renamed6” and “renamed7” instead of something sane like “enp4s1” - and it was changing after each reboot.
XCP-ng
Pretty ballin’, uses Xen instead of KVM, still very compatible and fast, you won’t notice a difference between the two. I haven’t used Xen in years though and I never had a HA cluster or tried live migrating, so I don’t know how well that works. On KVM, live migration or HA failover is instant and you don’t even lose ping packets. I don’t know about Xen, but I will assume it’s similar, give that I think AWS runs on Xen if I’m not mistaken.
I can’t really say much about XCP-ng in general, as I haven’t used it, I only used XenServer and XenCenter on a cluster made out of desktops back in 2016-2018. Worked for what I needed. XCP-ng should be on-par feature wise with Proxmox.
oVirt
Can say even less about oVirt. It’s also using KVM as Proxmox, has lots of feature, I don’t know how clustering works on it. Do your own research, I guess? The only problem with oVirt is that Red Hat kinda abandoned it? At least that’s what I remember reading on this forum. But it is still getting updates, last one was in April. Big projects like this don’t get massive updates, mostly patches. But now it’s on life support, the community has to pick up the slack.
Virt-Manager
Not much to say. Easy to get into, hard to master. Has some advanced features via XML files. Can migrate VMs from a box to another, as long as you don’t run too different operating systems, like we used to. Couldn’t migrate between CentOS 6 and CentOS 7 (and that was when 6 was still supported). I don’t know if there are any HA and fencing options in it, but it is KVM, so the groundwork is there underneath… Just that setting it up would be a royal PITA. It’s mostly a replacement for the horrible Virtualbox on desktops, but you can run headless hosts and connect to them via Virt-Manager, as it uses SSH in the backend.
You need to install a program if you want to run it. The program is Virt-Manager itself. Runs on any Linux distro, can run on Windows under WSL2, I don’t know about macOS. Doesn’t run on Android AFAIK. Mostly used on desktops for VFIO / GPU passthrough builds for Windows VMs.
Cockpit
It’s a management interface for Linux servers. It has a plugin for KVM, so you can use this instead of Virt-Manager, but it still a bit of a DIY. But it gives you some control over the box you administer. Probably doesn’t have a lot of options, but may get the job done. Same deal with the HA probably applies, need to DIY.
OpenStack
It’s a big, clunky, resource hungry, DIY cloud infrastructure software. Most likely overkill for a homelab, probably still overkill for a datacenter that doesn’t have multiple tenants, or other businesses administering VMs. But it can do it. It’s the most widespread cloud infrastructure, with many corporations just rebranding it and adding a few touches here and there. Vendors include IBM and HPE, but openstack itself is open source, you don’t need to buy servers from those guys to use it. Just that you can’t use their own optimizations.
OpenNebula
I have used OpenNebula, it is pretty cool. It is a DIY cloud option, but the only real thing you have to install is opennebula itself and either a DB server, or use sqlite. The guy who implemented it before me used sqlite, the interface was fine for 3 users. Has a nice user administration and many options. It can get tricky to setup, but it doesn’t have to be overly complicated for a data center environment. It even has payment / credit options for VMs, in case you want to do calculations on what VMs would cost if you had tenants in them. Never used that feature.
OpenNebula only needs to be installed on a single host or VM, preferably something that doesn’t run VMs on, but you can run VMs on it too. The way it keeps track of VMs is pretty fascinating. Unfortunately, the disks in the backend can be hard to keep track of. I highly suggest using Postgres instead of SQLite, because you may need that if you detach a disk and don’t know from which VM it came from. The DB contain all that information. Well, normally you wouldn’t do that, but we had a problem on the way it was set up in our infrastructure and all VMs had their disks in one folder basically. Probably could have been fixed, but nobody knew how it was set up (the guy who did left the company 3 months after I came in and the only other IT guys were my colleague who came 1 month after me, and an outsourced dude).
I remember OpenNebula being very quirky and buggy, giving us a lot of headaches. For example, the web filter for VMs didn’t actually filter the VMs, when you would “Select All” you would not select just what was shown, but actually the whole page that had hidden items in it, risking to delete many VMs at once. Such a pain… We had to always go inside the VM properties and then delete it or move it or modify it.
But I do think we didn’t give it enough of a chance, but it did need a lot of rework to make it work for us. Moving to proxmox was a good move.
I wouldn’t really recommend it for your needs, but it’s there if you need it. Better than OpenStack because of integration.
Honorable mention
While not exactly VMs, although it can run VMs, LXD is an orchestrator for LXC, Linux containers. Not OCI containers (docker, k8s, podman), but just LXC. If your workflow allows it, like if you have tons of databases, web servers, mail servers, dns servers and so on, LXD can do wonders. It is CLI-only though, but it is very easy to get into.
For compatibility reasons, it can launch images in VM mode, but there is no much point. I find LXD to be way better than what Proxmox offers with their LXC controls (just like with qm
, they use pct
instead of lxc-
commands - lxd confusingly enough uses lxc
without -
commands, probably standing for LXD client). Also, Proxmox can’t launch containers that it doesn’t know the OS of, so you are stuck with their offerings. They are not too bad, you have CentOS / Stream / Rocky / Alma / RHEL, Ubuntu, Debian, Alpine, Gentoo, Arch and many Turnkey Linux images, but it is a bit limited.
Now for an even more opinionated view, I personally do not plan to run Proxmox again, due to how hard it is to recover a cluster’s heath in a home environment. I had 2 servers die, so I couldn’t delete the cluster on the last one standing. I can put it manually in standalone mode, but it is still in a failed cluster state. I even tried creating proxmox VMs and adding them to the cluster to get it back in health, but that didn’t work out, because you can modify the cluster if it’s failed. And if it’s a standalone, you have some restrictions and you always have to run some commands after each reboot to allow you to start VMs and stuff.
And it’s a royal PITA. With more than 5 servers, Proxmox is fine. Nothing against Proxmox when it comes to having many servers, in fact, it makes things very easy. But it’s just not for me anymore, at least not for my homelab.
Moving to LXD on multiple ARM SBCs and Virt-Manager on a single x86 PC. bhyve
sounds fun, but I don’t have much experience with BSDs outside of pfSense, OPNsense, a bit of FreeNAS before it changed to TrueNAS Core, and an OpenBSD VM from time to time. Planning an ARM OpenBSD router, because network devices shouldn’t run the same OS as the other ones.
I still recommend Proxmox to people, especially to VMWare refugees. If you can buy a subscription from them to support their efforts, all the better, but you’re not required to.
Again, above are opinions. Uninformed opinions at that, I don’t know much about oVirt, Cockpit and XCP-ng, I should have not talked as much about them.
Regarding user administration, I think both Proxmox and XCP-ng offers some advanced options. OpenNebula definitely does. Proxmox should be easier to get into. You have Admin, PVEAdmin, PVESDNAdmin, PVEPoolAdmin, PVEUsersAdmin, PVEUser and more roles to pick from.