Did a bit of searching and maybe I haven’t dug deep enough to find the answer but I wanted to have the gurus here give this a once-over.
This is an expansion to my home network/lab. I’m building a second Proxmox node (actually third if you count my pfSense box). Both big Proxmox nodes will each have a TrueNAS VM (not sure Core or Scale).
I’ll have a 10Gbit switch as the core. I’m thinking the Mikrotik CRS309-1G-8S+. My pfSense box has a dual 10Gbit SFP+ card in it (SFN7122F). I’d put that in a LAGG as the uplink from the switch.
Then for the two Proxmox nodes, I was thinking of using a dual QSFP card (Mellanox ConnectX3-Pro) in each of them. One of the ports would be used to directly connect (via DAC) to the other Proxmox node. This would be used for VM/Container migration and backups between the two. The second QSFP port on each card is where my question lies…
I’ve read about a “simple” QSFP to SFP+ adapter that I could use to effectively convert the card’s port to SFP+ and then use a single SFP+ DAC cable to the core switch.
The alternative would be to use a QSFP to 4x SFP+ breakout cable. Then, I’d use two of the SFP+ DAC ends of that breakout and put them into a LAGG/bond and connect them to the core switch.
Seems like that should work but it begs a few quesitons:
Is it worth it?
Would it actually work? Proxmox bond…Mellanox config … Mikrotik LAG
Would it really be a 20Gbit connection or rather act like extra “lanes” rather than extra speed for a given client?
Then as the bonus question… to pass through or not to pass through the NIC port(s). My instinct says not to pass through.
Interesting option. But with that, I’d still have to deal with getting things knocked down to 10Gbit for the rest of my backbone. I won’t have QSFP everywhere.
I did some more reading and it seems like it’s a crapshoot as to whether or not I’ll be able to get the bond working, but it does seem that it’s at least cheaper to get a breakout cable and just use one SFP+ instead of getting the QSFP to SFP+ adapter. I think I’ll go that way and try it out. It’s only money, right?
I know the breakout cable works from the (QSFP) switch to 10gb devices (including switches). Although on larger switches only specific ports allow breakout to 4x10gb.
I’m about to find out in a few days. LOL
I may end up with that Mellanox switch after all, if it doesn’t work.
Thanks for the info, by the way!
EDIT–
This post I found suggests that I may just end up with a single 10G port exposed with the other three inactive. That’s an acceptable outcome for me.
The gist seems to be this:
“Well, more specifically, each port on the NIC is only capable of bringing up one link. So if you break it out and connect it to a 10G switch, only the first link will come up as a 10G link, the other three will not come up at all. Breakout cables are mainly meant for use with switches as many switches can break a 40G or 100G port into four separate 10G or 25G ports.”
I just need to figure out how I’d configure it. And also decode what they are talking about with the probed ports. I thought I understood modprobe in a rudimentary way but not sure I understand what they are talking about in the article.
This article has nothing to do with port splitting. This article explains how you configure the driver to expose multiple devices so that you can attach virtual machines, etc. to use the card. SR-IOV.
So is that is a single port split into 4 virtual devices via SR-IOV? Each as a 10G connection? That seems like what I want to do at the very least to plug a single one of those into my switch. Now if I could bond them, we’d be cooking with gas!
Not quite.
You see a single 40gb device exposed 4 more times as virtual 40gb devices.
This allows virtual machines and processes to access the NIC device in parallel and share its capabilities in software. I.e. you don’t get the error message “this device is already in use”.
What you try to do is something different. You try to split the bandwidth of the 40gb NIC into 4x 10gb. While the underlying technology supports it, this is (IMHO) not enabled in your NIC hardware/firmware.
