Recommended Proxmox configuration for a 3-node cluster with 2 nodes connected to a DAS ?

Software & Operating Systems Linux
, ,
1

I am aiming to get a 3 nodes proxmox cluster with 2 nodes connected to a DAS with this topology :
image
The idea behind this configuration is to have an HA cluster and a HA storage. What I had in mind was to have a zfs pool accessible in an active/passive configuration between nodes and active/active betwen links of a node.
This way, only one node writes to the DAS maximizing the use of the two avaible links and if it fails, the other node takes over using its two links in the same way as the first node.

I have read about dm-multipath from my understanding this solution works for a single node connected to a storage array as you can see in the red hat manual : https://access.redhat.com/documenta…se_linux/7/html/dm_multipath/mpio_description
And I can’t find examples with dm-multipath with two nodes attached to the same storage array. Neither, I can’t seem to deduce from the examples I have seen a configuration to run a two-node setup.

During my research, I have came across this post https://github.com/ewwhite/zfs-ha/wiki which matches my goal plus one cluster node not connected to the DAS. Howewer, I can’t wrap my head around on how to do this in Proxmox. Could you point me the right direction ?

Thank you for your help !

2

You are mixing up the roles in your cluster …
You want:

  • a three node proxmox cluster
  • to reuse two of the nodes to provide highly available SAN functionality

Proxmox uses corosync to run it’s own cluster, and corosync is one of the components of the HA ISCSI target solutions, you will be on your own on making sure the cluster config for the storage cluster doesn’t clash with the cluster config for the proxmox clusters

Ideally your cluster would be the proxmox one, where you create a different group/limit the SAN functionality to only two nodes, and on top of that you implement the ISCSI target functionality.
I would suggest planning on having two different networks, one for the proxmox cluster /VM traffic, the other dedicated to the backend traffic to the ISCSI target nodes, ideally using separate network interfaces

The article you linked (HA ZFS) does apply, as you want to configure an highly availabale ISCSI target service, that will provide shared LUNs to Proxmox that will appear on the cluster nodes as different devices from the directly attached ones and it wil use ZFS as the storage layer for the ISCSI LUNs … I am not sure how efficient that will be in the end, and it will be a hell of a mess to sort out which LUN is coming from where … Also nested ZFS on the same server … ugh!

As for multipath, there’s two places where it will be used, Proxmox cluster will use multipath (if possible) to create multiple ISCSI sessions to the storage cluster. The storage cluster will use multipath to access the Direct attached storage through the multiple HBAS. So the storage nodes will need to have a dual multipath config, one for accessing the DAS storage, managed by you, one for accessing the ISCSI LUNs, managed by proxmox …

Once Proxmox is able to access the ISCSI storage, you will be able to choose whether to use ZFS as the block file system…

This is another article that explains how to set up a pacemaker/corosync cluster and an iscsi target:
https://netbergtw.com/top-support/articles/zfs-cib/

Do you really need the third node? because if you could make do without it proxmox directly supports direct attached storage (albeit without snapshots)
And more important … why Iscsi? the article you linked would provide all you need (an highly available NFS mount accessing shared storage with ZFS as the underlying media) and would be directly accessible by the Proxmox cluster … you would even get snapshot capabilities if you used the qcow disk format …

1 Like