Missing in Proxmox: scale-out management, grouping, nesting, trees..
A frequent nuisance with HCI: management clusters and storage clusters are ill defined and often somewhat antagonistic.
Please note: I am recording this, while my memories with regards to vSphere, oVirt and Xcp-ng are quickly fading… I could be wrong on the details!
Problem: IT infrastructure needs more or less constant rebuilding, new pieced of hardware are moved in, old ones get removed, things are being upgraded etc., but you want to maintain continuity for your customer base, be they family or paying the rent.
So you may just want to be able to have something like an ‘old farm’ and a ‘new farm’, or a ‘production farm’ and a ‘DR/QA/Transition farm, distinct, if you can afford it’ and then be able to move live VMs or container between them, to prepare for major infrastructure upgrades, physical or logical. In other words: management domains and failure domains are distinct and failure domains might correlate more with storage.
That’s where I think Proxmox falls seriously short in that it only knows servers and clusters of servers, not collections of servers, nor collections of servers and clusters, nor does it understand storage clustering.
Since the only way to group servers in Proxmox is to cluster them, that’s what you do. But clusters need to be constantly up in order to work (share the Corosync ‘brain’), partitioning a cluster is a no-no, but that’s exactly what you may want to operate, groups of clusters where availability dependencies are constained to the members of a group, while visibility is between all e.g. to support migrations.
Perhaps put more simply, you may want clusters A and B and the ability to shift workloads and then flip roles.
In my case, I maintain a primary HCI Proxmox triple cluster with CEPH, with another such triple in cold standby, that are using very modest hardware (NUCs and Atoms) to run critical admin services like Univention IAS. The big home-lab work VMs run on workstations, that aren’t actually running 24x7, while most of them actually boot serveral distinct operating systems, Proxmox is just one of them.
Proxmox doesn’t like such volatility, but I trick it into acceptance by giving the core cluster higher numbers of votes, so a quorum is maintained, even if all those workstations are doing something else or shut down.
But that fails when it comes to managing two distinct clusters to differentiate failure or operations domains, i.e. to live migrate workloads between them.
BTW. oVirt wasn’t much better there, not sure about vSphere.
So why do I mention this? Because that’s the one area, where Xcp-ng did shine, I could manage distinct cells or availability groups from a single management pane and live migrate VMs beween them.
Given the Proxmox design based on corosync, I don’t see an easy way out of this and I’m sure the Proxmox guys are painfully aware of this: in the mean-time I’d like to know how relevant this sort of thing is for you and how you manage
I currently use a very manual approach, where I shut down and backup all VMs I want to move between my two clusters and then restore them on the other. Since the critical VMs are HA at the applicaton level, it doesn’t even cause service interruptions.
On the primary cluster I also maintain a ‘transition’ node with local storage, that allows me to temporarily hold a VM I want to keep in operation, even while I do work on the CEPH cluster. Here the abiltiy to move VM disks from CEPH to local storage on the transition node helps me de-couple that dependency.
There I wish I could run ZFS replication on top of CEPH or have CEPH support async replication itself… it’s so easy to want everthing without paying or contributing 
