Proxmox: Everything You Wish You'd Known Sooner (especially for VMWare Refugees)

[starshipeleven]

@ThatGuyB

I was not talking about shared block storage, but single LUN (I explicitly said 1 LUN per host, not shared LUNs) and I would think you would figure that out, when I was talking about NFS instant live-migration (aside from RAM) vs having to transfer data between LUNs. Yea, with lvm thinpool you’d have to move the vdisks from 1 host to another, for live migration.

yeah ok it works technically, but this objectively a retarded way to use a SAN, ie you use it as if it was local storage. So you get the worst of both worlds.

@ThatGuyB

GFS2 works better at vm-level than hypervisor-level. At the hypervisor (leave aside the fact that proxmox doesn’t support it, even though debian would), you’d still need to do fencing on gfs2, to prevent the faulted host from accessing the shared resource.

So the VMs using GFS2 don’t need fencing? It’s a shared disk filesystem, it needs fencing period.

If the fencing for Proxmox cluster is set up correctly, it will cover the GFS2 or any other shared disk filesystem.

@ThatGuyB

But Proxmox has some idiosyncrasies you should be aware of

Afaik Proxmox’s (linux clustering in general, yes I’ve worked with linux clusters made with Pacemaker/corosync) relies on you configuring a STONITH to fence and reboot a node if it misbehaves. There are horror stories of packemaker clusters running the same resource on two nodes and causing issues with the shared storage/db/whatever they are accessing.

With Proxmox they seem to have moved to a watchdog-based system that is supposed to reboot the node if it cannot contact the others in the cluster, but I’m not a huge fan of it.

@ThatGuyB

I don’t find hypervisor HA that great anymore

It has always been a method that relies on good locking/fencing methods to work, and while it’s increasingly getting obsoleted by application-level clustering, it’s still going to stay with us for a long time.
The key difference here is that vmware uses storage based locking (so the SAN’s iscsi LUN becomes the “arbiter of truth” instead of relying on hypervisor clustering) so if a VM is up on some node the other nodes can’t access its disks. At least in my experience it’s more robust than Linux/proxmox’s method. When this fails (and yes I’ve seen this on vmware 5 and 6 especially after a crash of the Acronis backup agent, fuck Acronis btw) you get a perma-lock that prevents vm migration. It must be cleared by rebooting the host or by connecting in ssh on the host and doing some wizardry from CLI and restart some services. So it’s a more “fail safe” kind of failure where you can’t suddenly find multiple instances of the same VM running.

vSphere VMFS Locking Mechanisms

[Pav86]

Thanks for all the feedback

Sadly, creating 3 LUNs does not work for us, given how the VMs’ virtual disks make use of the 2 LUNs. One is for VM storage (all the VMs’ OS disks). Then the other is for File Storage, and the entire LUN is used by a single virtual disk for the data drive of our Windows file share server.

The shared LUNs in VM worked because of the VMware file system it puts on top of them.

For now, it sounds like the best option is just to use them as LVM so they can be shared among 3 nodes of the cluster. It’s good to know that LVM will now support snapshots, as that will help in the meantime. Most of the VMs are already thick provisioned anyway because of a mistake in a previous migration, so we will just have to convert them later when we get new storage.

As for new storage from the feedback above, it seems like my plan for qcow2 on NFS is the way to go. I like having the shared storage for the reasons that ThatGuyB mentioned. Being able to live-migrate a VM without moving the storage makes it very easy and fast to shuffle VMs around to update or upgrade a node and is the process we have been using.