Can I use pfsense/OPNsense to bridge between 100 Gbps Infiniband and 100 Gbps Ethernet via a 100 Gbps IB switch?

Hardware Hub Networking
1

I’m using a Mellanox MSB-7890 36-port 100 Gbps externally managed Infiniband switch.

It will only run/use the IB protocol; cannot do ETH at all.

And I don’t want to have to go out and buy a 100 GbE switch.

I’m also using Proxmox on three systems, and they all have a Mellanox ConnectX-4 dual port VPI 100 Gbps IB cards.

In Proxmox, I was trying to create a Linux networking bridge so that VMs/CTs would be able to use the 100 Gbps network between VMs/CTs.

However, I am learning that the Linux network bridge that you can create via the Proxmox GUI is an ETHERNET network bridge, not an IB network bridge (or a network bridge that’s agnostic to whether it’s ETH nor IB).

So…given this – can I use something like pfSense or OPNsense to bridge these two protocols together, so that my VMs/CTs would be able to use it?

Your help is greatly appreciated.

Thank you.

2

I bet you can probably do it in Proxmox to set up a gateway that does the conversion. And since your ConnectX-4 cards are dual port models, you can configure one port as IB and one port as ETH. You can use one NIC/VPI assigned to a router VM on the Proxmox server and offload the switching onto the card itself to make the gateway fast if you cant just set up the vNICs and bridge directly on Proxmox settings. This bridging and using one NIC with dual ports is how the official Mellanox/Nvidia gateways have their hardware configured, with one port IB and the other ETH, since that way you can keep the traffic all within the NIC hardware itself and offload it onto the built in vRouter on newer gen NICs. The MCX623106AC-CDAT ($750 at ServerSupply) has the built in router hardware offloading that can do the gateway at wire speed, as well as offload a ton of other protocols and crypto acceleration.

To set up the gateway on your hardware, find out your GIDs and create a vNIC corresponding to the proper GID port and assign it a virtual IP. Create a VLAN and associate the Infiniband partition key (like an ethernet vlan but for infiniband side) with the vlan ID, and allow that vlan traffic into the ethernet side of things. Create the bridge between the Ethernet and Infiniband vNICs, and add the static routes to know what the gateway is to send to the Infiniband network.

This may help you get started even if it isnt for Proxmox specifically:
https://docs.oracle.com/cd/E18476_01/doc.220/e18478/GUID-88D64F6F-EF3A-4C89-9E8D-F6B7F3B4B1C3.htm#ELMOG76472

Also you may want to make sure jumbo frames are on and the MTU for Ethernet part of the gateway is configured for 4096 packet size as this is the default packet that Infiniband uses. You may run into communication problems if the Ethernet side is configured as 1500 MTU and IB is still 4096.

Beyond any of that I cant really help you. I havent ever worked in Proxmox myself and it has been years since I used Infiniband let alone set it up. Maybe someone else here on the forums has some additional knowledge.

1 Like
3

Thank you.

I appreciate this.

I think that my second problem is going to be that my Mellanox MSB-7890 is an externally managed IB switch.

So, I don’t know if there is a way to like “encapsulate” the ETH frames into an IB frame, and send it on the IB network, to have the receiving target then unpack the ETH frame out of said IB frame, and then do what it needs to do from there.

I REALLY don’t want to have to buy a 100 GbE switch if I can avoid it.

Funds are really available for that right now, so if I can work around it, that would be nice.

Thank you.

edit
I am currently in the process of deploying and testing out SR-IOV with my Infiniband NIC.

I am not 100% sure that this will work just yet, but I am in the process of testing it.

Two of my compute nodes (both of them are AMD Ryzen 9 5950X, uses Asus X570 motherboards, and I don’t have a GPU installed in it anymore as said Mellanox ConnectX-4 card takes up the primary PCIe slot now, so I can only remote into it over ssh and/or administer the system via the Proxmox web GUI).

Having said that, I don’t know if the Asus X570 motherboard has an explicit option to enable SR-IOV like my Supermicro dual Intel Xeon motherboard does (X10DRi-T4+), but so far, it looks like that with IOMMU enabled, it APPEARS to be working (at least as far as I can tell with lspci | grep Mell and ip link.

It’s almost 4 AM here, so I need to get to bed, so I’ll test out deploying containers and/or VMs tomorrow to see whether I will be successful in passing the virtual functions through (or not).

We shall see.

(Thinking about my “IB switch problem” was what led me to remember reading about SR-IOV from the Mellanox driver guide for the MLNX_OFED driver for Infiniband. Previously, I didn’t really pay much attention to it because I didn’t really use nor see the need for it, but now I understand what it was talking about better, so I am going to test it out.)

Thank you.