I am planning on setting up a 3 Node Proxmox cluster with 40G NICs for the cluster network. I would like to set them up as seen here, using either the routing with fallback or the RSTP configuration. https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server
I would like to be able to use the high speed network to connect to the VMs from outside of the cluster as well. I was thinking of adding a 10G NIC (in addition to the 40G) on of of the servers and connecting that to my switch to allow high speed connections to VMs on the cluster. From what I’ve read it seems like this would be achievable using software defined networking but I haven’t seen much on how to configure it for this use case.
My main goals are having resilient fail-over on the cluster and having high speed conectivity to the VMs from outside the cluster. Is this a good solution for my use case, or is there a better option? How would I go about setting this up? Are there any good guides to using SDN on Proxmox? Is this something that is possible? Would mixing 10G with 40G cause issues?
You can set separate networks for cluster network, corosync, public network, Proxmox Web UI,etc. and use like 5 connections going out of every node.
But you don’t have to. Without a 40Gbit switch this is problematic, so you need an additional NIC in each node to communicate with the outside world. But you can run corosync and ceph cluster network via that 40G direct link.
So you need either a 40G switch or an additional NIC for Proxmox web UI, Ceph public network and everything that’s going out of the cluster.
If 10Gbit is fine to you…get 10Gbit NICs, one for each node. If you’re ok with e.g. on-board 2.5G, that works as well. But if Ceph is pushing out 40-120Gbit, it is limited to that 10Gbit on the public network. So that 40Gbit cluster network bandwidth is mostly there for VMs, balancing and recovery and otherwise limited to the weakest (slowest) link.
If you do some fancy routing and bridging and only use 1x 10Gbit NIC in a single node, this works, but you are introducing a single point of failure and lose HA which is the entire point of a cluster.
They are separate NICs and networks. No problem.
I’m personally going for 25Gbit networking. Better latency and you can get 100Gbit switches with 4x25Gbit breakout. And bonded 25Gbit connections have 5GB/s bandwidth, so that certainly is more than enough for me.
The nodes will all have a 1G connections for webui and public network. The 40G would mostly be for cluster replication traffic. I had considered a 10G NIC on each node but I really only have a spare pcie port on one node and I only have 4 10g ports on my switch.
I maybe misunderstanding how SDN works, but I was thinking of having a Zone on top of the 40G network that also connects to the single 10G NIC. That way the 10G only manages the connection to outside the cluster and doesn’t carry any cluster traffic. I am ok with the connection out having a single point of failure as it isn’t crucial.