Pfsense, sr-iov and infiniband - no ipoib adapater in vm

Hello all,

Been reading from time to time for a while, but this is the first time I thought I’d try and find a little guidance.

Until a few days ago I had zero experience with infiniband, SR-IOV and pfsense, so it’s been a bit of a learning curve.

Home server setup is as follows, cobbled together from ebay and second hand server stuff:

  • Supermicro X10DRH-iT board, supports SR-IOV and it is enabled in the bios. it’s running ubuntu 20.10 at the moment. Probably switch back to debian soon as recent ubuntu developments have been somewhat annoying.

  • Mellanox ConnectX-3Pro (mcx354a-qcbt, flashed to standard mellanox cards with the most recent qcbt firmware). Card has been setup for SR-IOV and it shows five adapters, as configured, the base one and 4 virtual ones.

  • On the host I have assigned one of these adapters to a pfsense VM, this is the load balancer for two internet connections, that all works fine. The basic adapter has a network configured on it,.

  • I can’t add the ipoib adapter on the host to the bridge that pfsense is on, something basic within ipoib makes that impossible as I understand it, it has to do with 6bit headers for ethernet and 20bit headers for ipoib.

  • The infiniband works fine, nfs-rdma is fully accessible over the 40gb link and working on my workstation, I just donn’t get the internet connection from the pfsense over to my infiniband network, which is why I’d like the virtual adapter on the pfsense so it can manage the infiniband network aswell. So at the moment the funcionality is fine, I just need both an internet cable and an infiniband cable to my PC’s which is possibly I guess, but lacks elegance.

  • there are alternatives like using iptables to route between the infiniband network and the ethernet bridge on the host, but that feels like a bit of an ugly solution. Can’t imagine it’s conducive to my gaming performance either.

  • PFsense 2.5+ should support eh ConnectX-3, or so a post on the forum says. Within the pfsense VM I can get as far as:

    [2.5.2-RC][[email protected]]/root: pciconf -lv

    mlx4_core0@pci0:10:0:0: class=0x028000 card=0x61b015b3 chip=0x100415b3 rev=0x00 hdr=0x00

    vendor = ‘Mellanox Technologies’

    device = ‘MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]’

    class = network

But near as I can tell, the pfsense is missing some modules like ibcore for it to work.

Now the mystifying bit to me is that it’s listed as supported but it appears as if its missing those modules and pfsense seems to have no way to activate them without getting or building the modules on a different freebsd machine and then copying them over.

Before I resort to that, I was wondering whether anyone knows if I am missing anything stupidly obvious.

So, I since built the relevant modules on a second sandnbox freebsd12 VM and copied them over to the pfsense guest. Infiniband now works on pfsense, though it still isn’t routing internet. I can ssh to the pfsensemachine over the host using the infiniband adress so that’s over the virtual sr-iov adapter and I can ssh from the pfsense machine to my workstation using the infiniband ipoib adresses. I can’t go from the workstation to the pfsense machine, however. No idea why.

Anyone who has some ideas, would love to hear it.