Fastlinq Fight

Hardware Hub Networking
1

Putting this out there for anyone struggling to passthru a Marvell/QLogic/Cavium Fastlinq 41xxx/45xxx in an ESXi host:

esxcli system module set -m=qedi -e=false
esxcli system module set -m=qedentv -e=false

Disabling these modules effectively “removes” the device and makes it a blank slate from the ESXi perspective. It’s now just a mindless PCI device to be shoved at the VM.

This shouldn’t be necessary and I don’t understand what’s happening. The VMware and its VM guests should be capable of sharing the physical interface IMHO.

If you still have issues try using the new “Dynamic DirectPath” feature to effect PCIe passthru in the VM. I don’t understand why this works when classic passthru won’t let my VM power on.

If assigning an SRIOV VF to the VM doesn’t work (i.e. can’t talk), doesn’t let the VM boot, or underperforms – I’ve experienced all three over here – forget SR-IOV and instead pass thru one or more of the base device NPARs. SRIOV has been a bag of hurt for me.

In the VMs vmx file this is helpful:

pciPassthru.use64bitMMIO

This advanced host setting has been helpful:

VMkernel.Boot.disableACSCheck = true

Not sure about this one:

VMkernel.Boot.enableACSCheckForRP

Compared to a “normal” basic setup (ie. vSwitch, vmnic uplink, and vmxnet3 in the VM), random 4k Q1T1 IOPs are up roughly 20% thanks to PCIe passthru. Tested using iSCSI from a Windows host to a TrueNAS VM and using diskspd for testing. No multipathing or MCS.

Long sequential I/O saturates a 10Gb link regardless of what I do so I’ve got that going for me which is nice.

In the Windows client, disabling interrupt moderation, LSO, and recv seg. coalescing was worth another ~5% bump in IOPs.

My next steps are to try disabling these NIC offloads on the server side, try to get iSER running somehow, and try to get full iSCSI hardware offload working on the server side (assuming I can somehow engage target mode).

Hope this helps someone. The people hacking on Fastlinq CNAs in their homelabs are few and far between and I feel like I’m all alone over here hating on this interface…

Edit: re-worded much of the above. I write like a schizo sometimes…

1 Like
2

4k random r/w Q1T1 IOPs up another 5-7% by setting the TrueNAS VM’s latency sensitivity to “high.”

No time to work on the hardware offloads yet.

3

Replaced Windows 11 on the client with Windows Server 2025 GA.

Old driver kit was 8.72.15.0 from HPE. This new Server-turned-workstation uses the in-box driver for this converged network adapter.

Max 4k random read/write Q1T1 IOPs was a touch over 15,000 on Win11.

Now we’re a touch over 17,000 after re-applying the tweaks I mentioned earlier. No idea if this reduction in latency is some kind of OS optimization or due to the in-box drivers. But I’ll take it!