Direct network link between proxmox hosts as backbone

Networking
, ,
1

Hello everyone,

I am currently experimenting with 10GB/s networking in my homelab and have a question regarding the setup.

I have two Proxmox hosts, each equipped with dual 10GB/s NICs. I am interested in connecting these two Proxmox hosts directly and utilizing that link for the server VLAN (20), allowing my VMs to utilize the bandwidth without going through the switch. My rationale behind this is to ensure that if the switch ever goes down, my services would remain unaffected.

There are other devices connected to the switch which is not shown in the diagram. Its the core switch of my network.

image
image1141Ă—417 30.7 KB

Can this be done on the Proxmox hosts or do I need additional virtual routers? Can this be done at all?

Cheers!

2

This will work just fine. Give them each an IP address in the same subnet on both of these interfaces or bridges on these interfaces, make sure they have the appropriate firewall rules, and ping each of them to confirm they can see each other over those interfaces.

If they are in a cluster and you want them to be able to communicate even when the switch is down, make sure you setup these IPs for use with corosync and configure them to perform migrations over this network.

4 Likes
3

Hello there!

So I could bridge the two NIC interfaces on the host and give each host a single IP. Would VLANs work properly considering the NIC - Switch Link is a trunk while the Host to Host is a single network?

Is it possible to bridge NIC2 to a VLAN on NIC1? instead of the entire interfaces?

4

No, I am saying to leave your existing switch connected interfaces alone.

On each of the proxmox hosts you assign an IP address to either the interface that you want to direct connect or a bridge that is associated with just that interface you want to direct connect. The IP addresses for both should be in the same subnet, and not be in the same subnet as any of your other interfaces. The goal is to create a network with just these two devices on it, and since they are in the subnet they do not need a router to communicate.

5

Understood.

Lets say the host-to-host is 192.168.99.x On each host I have a server VLAN 192.168.20.x which i’d like to be accessible over the host-to-host link. Is this possible?

I’m hoping to use this link to move large files without the switch. If the switch died, the core network should keep working over the link. Get my idea?

I suspect I need a router on the other both hosts. (already have one on host A)

6

If the switch died, the core network should keep working over the link

This is more complex. Every VM/container would need access to both interfaces, and you would be doing switching on a NIC rather than a switch which typically isn’t as fast. It also sort of begs the question of how useful the network would even be without the switch in the first place? Sure your two nodes can communicate, but what’s the point if everything else on the switch cant communicate with them? This is a problem that would be much better addressed by multiple redundant switches. Redundancy and high availability is a very expensive rabbit hole to fall down if you want to do it right.

7

Hello there,

Indeed redundancy is expensive. I’m trying to add redundancy by cutting some corners where I can.

8

Each host needs to choose where to send each packet.

@DockerOtter , can you answer a few questions?

Q1: When you unplug a green cable or any of the two blue cables, what do you want to happen with packets that were previously going over green and/or blue cables, … for example, is it OK if the two hosts can’t reach each other in that case?

Q2: Do you want the two hosts to have the same IP and/or same MAC address that’s reachable irrespective of what cables are unplugged, or not.

Q3: Do you want to aggregate bandwidth between the two hosts, when all of the links are up?

if the answer to all 3 is “no”. you don’t need any bridges, or daemons, and you don’t need VLANs on the green interface, just an IP on interfaces you want to talk.

If the answer is yes, you might need or want, OVS or FRR…

Can you clarify?

1 Like
9

Hello there,

You hit the nail on the head. Answer is Yes to all the above. Going to research the terms you mentioned.

Cheers!

10

So it might be complicated.

Basically, on the blue link, you do whatever you do now.

On the green link you don’t technically need VLANs or IPs, but you’ll get some anyway thanks to IPv6, and that’s ok.

You add an additional IP for the two proxmox hosts, ideally onto a bridge. These will become the IPs you should use for almost everything related to hosts, as opposed to any IPs on the hardware interfaces.

Install FRRouting / frr.

And configure EBGP, and say that each host is a separate AS.

And configure peers over green link using interface name, and over blue link using IPs, unless you go and setup a separate VLAN, then you can just use the interface too.

BGP will converge, and you’ll get two paths of same distance, they’ll be installed by FRR into Linux as ECMP and you’ll get load balancing of connections by hash, and when links fail BGP will detect it and update the routes.

The alternative instead of BGP is to use IS-IS or apparently OpenFabric, I haven’t used IS-IS in a few years, and I’ve yet to try OpenFabric … but it looks nice according to the proxmox wiki page, maybe even simpler than basic BGP, but I can’t see any mentions of ECMP.

This is the page Full Mesh Network for Ceph Server - Proxmox VE just ignore node 3, and add a second interface.

In the end see what iperf3 -P10 ends up transferring across hosts, e.g. if you get 20Gbps, and try unplugging cables.

Now, I see internet is connected to one of the hosts, you could stuff the internet into a VLAN on your switch and make a migrate-able pfSense VM, so that you don’t lose Internet when you want to upgrade or reboot one of your proxmox hosts.

Or, you could run one pfSense per proxmox hosts and VRRP/CARP between them, or OpenWRT with VRRP/keepalived in between.

That way you don’t lose Internet when either one of your proxmox hosts goes away.

edit:

have a look, bgp starts at this timestamp, but maybe look at the whole video multiple times if you want to go down bgp route, … or at least, you’ll be able to see how ECMP shows up in frr and linux.

1 Like