100 GbE setup not performing as expected

Hardware Hub Networking
, , ,
1

I’m setting up my new DIY NAS (TrueNAS CE) and the network is not performing as expected. The NAS has one ConnectX-6 and the client a ConnectX-7 (only running at 64 GT/s). They are connected over Duplex CWDM4 fiber. I’m expecting about 50GBit/s taking the degraded PCIe link into acount. Here us what iperf measures:

Connecting to host 192.168.2.20, port 5201
[  5] local 192.168.2.21 port 39804 connected to 192.168.2.20 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.65 GBytes  14.2 Gbits/sec   41    943 KBytes       
[  5]   1.00-2.00   sec  1.64 GBytes  14.1 Gbits/sec  287    814 KBytes       
[  5]   2.00-3.00   sec  1.67 GBytes  14.4 Gbits/sec    0    954 KBytes       
[  5]   3.00-4.00   sec  1.67 GBytes  14.3 Gbits/sec  129    788 KBytes       
[  5]   4.00-5.00   sec  1.67 GBytes  14.3 Gbits/sec   45    906 KBytes       
[  5]   5.00-6.00   sec  1.67 GBytes  14.3 Gbits/sec   42    716 KBytes       
[  5]   6.00-7.00   sec  1.67 GBytes  14.3 Gbits/sec   16    943 KBytes       
[  5]   7.00-8.00   sec  1.66 GBytes  14.3 Gbits/sec   14    716 KBytes       
[  5]   8.00-9.00   sec  1.66 GBytes  14.3 Gbits/sec    7    843 KBytes       
[  5]   9.00-10.00  sec  1.66 GBytes  14.3 Gbits/sec    2    915 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  16.6 GBytes  14.3 Gbits/sec  583             sender
[  5]   0.00-10.00  sec  16.6 GBytes  14.3 Gbits/sec                  receiver

iperf Done.
Here are a few info's about the client 
lspci -vvv
02:00.1 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
	Subsystem: Mellanox Technologies MT2910 Family [ConnectX-7]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 18
	IOMMU group: 1
	Region 0: Memory at 2004000000 (64-bit, prefetchable) [size=32M]
	Expansion ROM at df100000 [disabled] [size=1M]
	Capabilities: [60] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75.000W
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 32GT/s, Width x16, ASPM not supported
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s (downgraded), Width x8 (downgraded)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABC, TimeoutDis+ NROPrPrP- LTR-
			 10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- TPHComp- ExtTPHComp-
			 AtomicOpsCap: 32bit+ 64bit+ 128bitCAS+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis+ LTR- OBFF Disabled,
			 AtomicOpsCtl: ReqEn+
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [48] Vital Product Data
		Product Name: NVIDIA ConnectX-7 Adapter Card, 200GbE / NDR200, Dual-port QSFP112, PCIe 5.0x16 with x16 PCIe extension option, Crypto Disabled, Secure Boot Enabled                                  
		Read-only fields:
			[PN] Part number: MCX755106AS-HEAT         
			[EC] Engineering changes: A8
			[V2] Vendor specific: MCX755106AS-HEAT         
			[SN] Serial number: xxxxxxxxxxx   
			[V3] Vendor specific: xxxxxxxxx
			[VA] Vendor specific: MLX:MN=MLNX:CSKU=V2:UUID=V3:PCI=V0:MODL=CX755106A      
			[V0] Vendor specific: PCIeGen5 x16 
			[VU] Vendor specific: xxxxxxxxxxx
			[RV] Reserved: checksum good, 1 byte(s) reserved
		End
	Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [c0] Vendor Specific Information: Len=18 <?>
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot-,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 08, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy-
		IOVSta:	Migration-
		Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency Link: 01
		VF offset: 17, stride: 1, Device ID: 101e
		Supported Page Size: 000007ff, System Page Size: 00000001
		Region 0: Memory at 0000002008000000 (64-bit, prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Capabilities: [230 v1] Access Control Services
		ACSCap:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [420 v1] Data Link Feature <?>
	Kernel driver in use: mlx5_core
	Kernel modules: mlx5_core
ethtool enp2s0f1np1
Settings for enp2s0f1np1:
	Supported ports: [ FIBRE ]
	Supported link modes:   1000baseT/Full
	                        10000baseT/Full
	                        1000baseKX/Full
	                        10000baseKR/Full
	                        10000baseR_FEC
	                        40000baseKR4/Full
	                        40000baseCR4/Full
	                        40000baseSR4/Full
	                        40000baseLR4/Full
	                        25000baseCR/Full
	                        25000baseKR/Full
	                        25000baseSR/Full
	                        50000baseCR2/Full
	                        50000baseKR2/Full
	                        100000baseKR4/Full
	                        100000baseSR4/Full
	                        100000baseCR4/Full
	                        100000baseLR4_ER4/Full
	                        50000baseSR2/Full
	                        1000baseX/Full
	                        10000baseCR/Full
	                        10000baseSR/Full
	                        10000baseLR/Full
	                        10000baseER/Full
	                        50000baseKR/Full
	                        50000baseSR/Full
	                        50000baseCR/Full
	                        50000baseLR_ER_FR/Full
	                        50000baseDR/Full
	                        100000baseKR2/Full
	                        100000baseSR2/Full
	                        100000baseCR2/Full
	                        100000baseLR2_ER2_FR2/Full
	                        100000baseDR2/Full
	                        200000baseKR4/Full
	                        200000baseSR4/Full
	                        200000baseLR4_ER4_FR4/Full
	                        200000baseDR4/Full
	                        200000baseCR4/Full
	                        100000baseKR/Full
	                        100000baseSR/Full
	                        100000baseLR_ER_FR/Full
	                        100000baseCR/Full
	                        100000baseDR/Full
	                        200000baseKR2/Full
	                        200000baseSR2/Full
	                        200000baseLR2_ER2_FR2/Full
	                        200000baseDR2/Full
	                        200000baseCR2/Full
	Supported pause frame use: Symmetric
	Supports auto-negotiation: Yes
	Supported FEC modes: None	 RS	 BASER
	Advertised link modes:  1000baseT/Full
	                        10000baseT/Full
	                        1000baseKX/Full
	                        10000baseKR/Full
	                        10000baseR_FEC
	                        40000baseKR4/Full
	                        40000baseCR4/Full
	                        40000baseSR4/Full
	                        40000baseLR4/Full
	                        25000baseCR/Full
	                        25000baseKR/Full
	                        25000baseSR/Full
	                        50000baseCR2/Full
	                        50000baseKR2/Full
	                        100000baseKR4/Full
	                        100000baseSR4/Full
	                        100000baseCR4/Full
	                        100000baseLR4_ER4/Full
	                        50000baseSR2/Full
	                        1000baseX/Full
	                        10000baseCR/Full
	                        10000baseSR/Full
	                        10000baseLR/Full
	                        10000baseER/Full
	                        50000baseKR/Full
	                        50000baseSR/Full
	                        50000baseCR/Full
	                        50000baseLR_ER_FR/Full
	                        50000baseDR/Full
	                        100000baseKR2/Full
	                        100000baseSR2/Full
	                        100000baseCR2/Full
	                        100000baseLR2_ER2_FR2/Full
	                        100000baseDR2/Full
	                        200000baseKR4/Full
	                        200000baseSR4/Full
	                        200000baseLR4_ER4_FR4/Full
	                        200000baseDR4/Full
	                        200000baseCR4/Full
	                        100000baseKR/Full
	                        100000baseSR/Full
	                        100000baseLR_ER_FR/Full
	                        100000baseCR/Full
	                        100000baseDR/Full
	                        200000baseKR2/Full
	                        200000baseSR2/Full
	                        200000baseLR2_ER2_FR2/Full
	                        200000baseDR2/Full
	                        200000baseCR2/Full
	Advertised pause frame use: Symmetric
	Advertised auto-negotiation: Yes
	Advertised FEC modes: None
	Link partner advertised link modes:  Not reported
	Link partner advertised pause frame use: No
	Link partner advertised auto-negotiation: Yes
	Link partner advertised FEC modes: Not reported
	Speed: 100000Mb/s
	Duplex: Full
	Auto-negotiation: on
	Port: FIBRE
	PHYAD: 0
	Transceiver: internal
	Supports Wake-on: d
	Wake-on: d
        Current message level: 0x00000004 (4)
                               link
	Link detected: yes
mlxlink -d 02:00.1

Operational Info
----------------
State                              : Active 
Physical state                     : ETH_AN_FSM_ENABLE 
Speed                              : 100G 
Width                              : 4x 
FEC                                : No FEC 
Loopback Mode                      : No Loopback 
Auto Negotiation                   : ON 

Supported Info
--------------
Enabled Link Speed (Ext.)          : 0x00003ff2 (200G_2X,200G_4X,100G_1X,100G_2X,100G_4X,50G_1X,50G_2X,40G,25G,10G,1G) 
Supported Cable Speed (Ext.)       : 0x00000200 (100G_4X) 

Troubleshooting Info
--------------------
Status Opcode                      : 0 
Group Opcode                       : N/A 
Recommendation                     : No issue was observed 

Tool Information
----------------
Firmware Version                   : 28.43.1014 
amBER Version                      : 4.5 
MFT Version                        : mft 4.31.0-149
And here are details about the NAS 
lspci -s 41:00.0 -vvv
41:00.0 Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6]
        Subsystem: Mellanox Technologies MT28908 Family [ConnectX-6]
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 278
        NUMA node: 0
        IOMMU group: 51
        Region 0: Memory at 9007e000000 (64-bit, prefetchable) [size=32M]
        Expansion ROM at b0100000 [disabled] [size=1M]
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 512 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM not supported
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 16GT/s, Width x16
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABC, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: ReqEn+
                LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
                LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
                         EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [48] Vital Product Data
                Product Name: ConnectX-6 VPI adapter card, HDR IB (200Gb/s) and 200GbE, dual-port QSFP56                                                                                                            
                Read-only fields:
                        [PN] Part number: MCX653106A-HDAT          
                        [EC] Engineering changes: AC
                        [V2] Vendor specific: MCX653106A-HDAT          
                        [SN] Serial number: xxxxxxx   
                        [V3] Vendor specific: xxxxxxxx
                        [VA] Vendor specific: MLX:MN=MLNX:CSKU=V2:UUID=V3:PCI=V0:MODL=CX653106A      
                        [V0] Vendor specific: PCIeGen4 x16 
                        [VU] Vendor specific: xxxxxxxx
                        [RV] Reserved: checksum good, 1 byte(s) reserved
                End
        Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
                Vector table: BAR=0 offset=00002000
                PBA: BAR=0 offset=00003000
        Capabilities: [c0] Vendor Specific Information: Len=18 <?>
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot-,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
                UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO+ CmpltAbrt- UnxCmplt+ RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                AERCap: First Error Pointer: 08, ECRCGenCap+ ECRCGenEn+ ECRCChkCap+ ECRCChkEn+
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 1
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [1c0 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Capabilities: [230 v1] Access Control Services
                ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Capabilities: [320 v1] Lane Margining at the Receiver <?>
        Capabilities: [370 v1] Physical Layer 16.0 GT/s <?>
        Capabilities: [420 v1] Data Link Feature <?>
        Kernel driver in use: mlx5_core
        Kernel modules: mlx5_core
ethtool enp65s0f1np1   
Settings for enp65s0f1np1:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full
                                10000baseT/Full
                                1000baseKX/Full
                                10000baseKR/Full
                                10000baseR_FEC
                                40000baseKR4/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                40000baseLR4/Full
                                25000baseCR/Full
                                25000baseKR/Full
                                25000baseSR/Full
                                50000baseCR2/Full
                                50000baseKR2/Full
                                100000baseKR4/Full
                                100000baseSR4/Full
                                100000baseCR4/Full
                                100000baseLR4_ER4/Full
                                50000baseSR2/Full
                                1000baseX/Full
                                10000baseCR/Full
                                10000baseSR/Full
                                10000baseLR/Full
                                10000baseER/Full
                                50000baseKR/Full
                                50000baseSR/Full
                                50000baseCR/Full
                                50000baseLR_ER_FR/Full
                                50000baseDR/Full
                                100000baseKR2/Full
                                100000baseSR2/Full
                                100000baseCR2/Full
                                100000baseLR2_ER2_FR2/Full
                                100000baseDR2/Full
                                200000baseKR4/Full
                                200000baseSR4/Full
                                200000baseLR4_ER4_FR4/Full
                                200000baseDR4/Full
                                200000baseCR4/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: None        RS      BASER
        Advertised link modes:  1000baseT/Full
                                10000baseT/Full
                                1000baseKX/Full
                                10000baseKR/Full
                                10000baseR_FEC
                                40000baseKR4/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                40000baseLR4/Full
                                25000baseCR/Full
                                25000baseKR/Full
                                25000baseSR/Full
                                50000baseCR2/Full
                                50000baseKR2/Full
                                100000baseKR4/Full
                                100000baseSR4/Full
                                100000baseCR4/Full
                                100000baseLR4_ER4/Full
                                50000baseSR2/Full
                                1000baseX/Full
                                10000baseCR/Full
                                10000baseSR/Full
                                10000baseLR/Full
                                10000baseER/Full
                                50000baseKR/Full
                                50000baseSR/Full
                                50000baseCR/Full
                                50000baseLR_ER_FR/Full
                                50000baseDR/Full
                                100000baseKR2/Full
                                100000baseSR2/Full
                                100000baseCR2/Full
                                100000baseLR2_ER2_FR2/Full
                                100000baseDR2/Full
                                200000baseKR4/Full
                                200000baseSR4/Full
                                200000baseLR4_ER4_FR4/Full
                                200000baseDR4/Full
                                200000baseCR4/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: None
        Speed: 100000Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d
        Link detected: yes
2

I see three levels of changes that will impact the results from your test.

  1. Physical change. e.g. network performance is quite latency sensitive. Moving NICs to CPU connected PCIe lanes will have a measurable impact on performance,
  2. Network stack. Typical Linux default settings are not likely to yield top performance results. Changing settings like mtu, tx queue length, etc. have noticeable impact, and lastly
  3. iperf3 parameters. A single CPU core is unlikely to saturate a 100GbE link. Try using multiple threads in parallel using the -P switch.
6 Likes
3

What is MTU ?

Are you running iperf3 with -P 8 or so ?

Why TrueNAS CE ? CE is dead end. Install SCALE.

Use dctcp congestion control as well

4

32GT/s on 16 lanes is >4x what you need for 100GbE. You’re fine with that. (Yes the 32 is the per-lane speed, nothing seems downgraded to me.)

But like others said you’re not going to saturate it with a single CPU thread. Try 8, it should get you close to 100.

5

PCie 3.0 x8 is very much downgraded for lspci, myself and the iperf3 stats.

2 Likes
6

Lol derp, it does say so next to it. I was looking at LnkCap.

1 Like
7

Thanks for all your replies!

Both On the NAS and on the client the NICs are connected directly to the CPU

On the client side everything is default.
On the NAS side i have implemented @wardtj optimisations found here: 100 Gbe & TrueNAS Scale 23.10 iSCSI - Performance Unleashed

Yes I have of course tried that. The results are within the margin of error identical.

Here a run with 4 threads 
iperf3 -c 192.168.2.20 -P4
Connecting to host 192.168.2.20, port 5201
[  5] local 192.168.2.21 port 50108 connected to 192.168.2.20 port 5201
[  7] local 192.168.2.21 port 50120 connected to 192.168.2.20 port 5201
[  9] local 192.168.2.21 port 50136 connected to 192.168.2.20 port 5201
[ 11] local 192.168.2.21 port 50146 connected to 192.168.2.20 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   438 MBytes  3.67 Gbits/sec    0    211 KBytes       
[  7]   0.00-1.00   sec   440 MBytes  3.69 Gbits/sec    0    519 KBytes       
[  9]   0.00-1.00   sec   438 MBytes  3.67 Gbits/sec    0    274 KBytes       
[ 11]   0.00-1.00   sec   439 MBytes  3.68 Gbits/sec    0    409 KBytes       
[SUM]   0.00-1.00   sec  1.71 GBytes  14.7 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   439 MBytes  3.68 Gbits/sec    0    211 KBytes       
[  7]   1.00-2.00   sec   440 MBytes  3.69 Gbits/sec    0    519 KBytes       
[  9]   1.00-2.00   sec   439 MBytes  3.68 Gbits/sec    0    274 KBytes       
[ 11]   1.00-2.00   sec   439 MBytes  3.68 Gbits/sec    0    409 KBytes       
[SUM]   1.00-2.00   sec  1.72 GBytes  14.7 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   439 MBytes  3.68 Gbits/sec    0    211 KBytes       
[  7]   2.00-3.00   sec   439 MBytes  3.68 Gbits/sec    0    519 KBytes       
[  9]   2.00-3.00   sec   439 MBytes  3.69 Gbits/sec    0    274 KBytes       
[ 11]   2.00-3.00   sec   440 MBytes  3.69 Gbits/sec    0    409 KBytes       
[SUM]   2.00-3.00   sec  1.72 GBytes  14.7 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec   439 MBytes  3.68 Gbits/sec    0    211 KBytes       
[  7]   3.00-4.00   sec   440 MBytes  3.69 Gbits/sec    0    519 KBytes       
[  9]   3.00-4.00   sec   439 MBytes  3.68 Gbits/sec    0    274 KBytes       
[ 11]   3.00-4.00   sec   438 MBytes  3.68 Gbits/sec    0    409 KBytes       
[SUM]   3.00-4.00   sec  1.71 GBytes  14.7 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec   438 MBytes  3.68 Gbits/sec    0    211 KBytes       
[  7]   4.00-5.00   sec   439 MBytes  3.68 Gbits/sec    0    519 KBytes       
[  9]   4.00-5.00   sec   439 MBytes  3.68 Gbits/sec    0    274 KBytes       
[ 11]   4.00-5.00   sec   439 MBytes  3.68 Gbits/sec    0    409 KBytes       
[SUM]   4.00-5.00   sec  1.71 GBytes  14.7 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec   439 MBytes  3.68 Gbits/sec    0    211 KBytes       
[  7]   5.00-6.00   sec   439 MBytes  3.68 Gbits/sec    0    519 KBytes       
[  9]   5.00-6.00   sec   438 MBytes  3.68 Gbits/sec    0    274 KBytes       
[ 11]   5.00-6.00   sec   438 MBytes  3.68 Gbits/sec    0    409 KBytes       
[SUM]   5.00-6.00   sec  1.71 GBytes  14.7 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec   439 MBytes  3.68 Gbits/sec    0    211 KBytes       
[  7]   6.00-7.00   sec   439 MBytes  3.68 Gbits/sec    0    519 KBytes       
[  9]   6.00-7.00   sec   439 MBytes  3.68 Gbits/sec    0    274 KBytes       
[ 11]   6.00-7.00   sec   439 MBytes  3.68 Gbits/sec    0    409 KBytes       
[SUM]   6.00-7.00   sec  1.71 GBytes  14.7 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec   439 MBytes  3.68 Gbits/sec    0    211 KBytes       
[  7]   7.00-8.00   sec   439 MBytes  3.68 Gbits/sec    0    519 KBytes       
[  9]   7.00-8.00   sec   439 MBytes  3.68 Gbits/sec    0    274 KBytes       
[ 11]   7.00-8.00   sec   439 MBytes  3.68 Gbits/sec    0    409 KBytes       
[SUM]   7.00-8.00   sec  1.71 GBytes  14.7 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec   439 MBytes  3.68 Gbits/sec    0    211 KBytes       
[  7]   8.00-9.00   sec   439 MBytes  3.68 Gbits/sec    0    519 KBytes       
[  9]   8.00-9.00   sec   439 MBytes  3.68 Gbits/sec    0    274 KBytes       
[ 11]   8.00-9.00   sec   439 MBytes  3.68 Gbits/sec    0    409 KBytes       
[SUM]   8.00-9.00   sec  1.71 GBytes  14.7 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec   439 MBytes  3.68 Gbits/sec    0    211 KBytes       
[  7]   9.00-10.00  sec   439 MBytes  3.68 Gbits/sec    0    519 KBytes       
[  9]   9.00-10.00  sec   439 MBytes  3.68 Gbits/sec    0    274 KBytes       
[ 11]   9.00-10.00  sec   439 MBytes  3.68 Gbits/sec    0    409 KBytes       
[SUM]   9.00-10.00  sec  1.71 GBytes  14.7 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.28 GBytes  3.68 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  4.28 GBytes  3.68 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  4.29 GBytes  3.68 Gbits/sec    0             sender
[  7]   0.00-10.00  sec  4.28 GBytes  3.68 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  4.28 GBytes  3.68 Gbits/sec    0             sender
[  9]   0.00-10.00  sec  4.28 GBytes  3.68 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  4.29 GBytes  3.68 Gbits/sec    0             sender
[ 11]   0.00-10.00  sec  4.28 GBytes  3.68 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  17.1 GBytes  14.7 Gbits/sec    0             sender
[SUM]   0.00-10.00  sec  17.1 GBytes  14.7 Gbits/sec                  receiver

TrueNAS SCALE is now called TrueNAS Community Edition (CE).
I’m using 25.04 (latest at to time I’m writing this.)

8

Just clarifying: server and client are directly connected, right? No switches in the middle?

Check on both client and server what speed the NICs are connecting ethtool <adapter> | grep Speed. It should say
Speed: 100000Mb/s

If it doesn’t make sure to set the highest supported speed on both endpoints (e.g. 100000). Use ethtool -s <adapter> speed 100000 to set manually.

Make sure that flow control is enabled ethtool -a <adapter>. If not, turn flow control on with ethtool -A <adapter> rx on tx on.

2 Likes
9

I see you are using 25.04. It does not enable RDMA in the CE build by default. It will just connect via regular iscsi and it will not be accerlerated via RDMA to use iser. Workarounds from 23.10 will reenable RDMA for NFS. Using NFS you should be able to achieve almost 100gbe if RDMA is enabled on client and TrueNAS via NFS.

1 Like
10

Yes they are directly connected with a 2m long OS1 fiber patch cable. I don’t have a fast enough switch, yet.

Yes on both devices it says Speed: 100000Mb/s. The Original post has a full output for both devices of ethtool in the hidden text blocks.

Both devices show the same information:

sudo ethtool -a enp2s0f1np1
Pause parameters for enp2s0f1np1:
Autonegotiate:	off
RX:		on
TX:		on

While it is my plan to one day enable RDMA, at the moment I’m trying to get the connection speed that I expect. Once that is done, I’ll come back to your great tutorials/information to enable those features.

11

Didn’t know that, thanks!

12

What MTU are you using? It has to be 9000 to get any meaningful speeds.

Some with congestion protocol that I mentioned above

13

If you can’t get proper speeds with iperf3, looking at RDMA is waste of time

1 Like
14

As I said in my first post :

  1. set MTU to 9000 on both sides
  2. enable dctcp congestion on TrueNAS
  3. make sure to run iperf with -P 8 (4 is not enough for 100)
  4. if you run stuff under Proxmox - make sure that CPUs are passed as host and not some virtualized crap.

Below is a chinese $40 cheap Connect4-x Lx via Mikrotik CRS510

Its rock solid 25gbps

$ iperf3 -c nas  -P 8
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.76 GBytes  3.23 Gbits/sec  626            sender
[  5]   0.00-10.00  sec  3.76 GBytes  3.22 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  3.50 GBytes  3.00 Gbits/sec  1961            sender
[  7]   0.00-10.00  sec  3.48 GBytes  2.99 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  3.24 GBytes  2.78 Gbits/sec  1686            sender
[  9]   0.00-10.00  sec  3.23 GBytes  2.77 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  3.61 GBytes  3.10 Gbits/sec  1716            sender
[ 11]   0.00-10.00  sec  3.60 GBytes  3.09 Gbits/sec                  receiver
[ 13]   0.00-10.00  sec  3.46 GBytes  2.97 Gbits/sec  1341            sender
[ 13]   0.00-10.00  sec  3.46 GBytes  2.97 Gbits/sec                  receiver
[ 15]   0.00-10.00  sec  3.59 GBytes  3.08 Gbits/sec  1024            sender
[ 15]   0.00-10.00  sec  3.58 GBytes  3.08 Gbits/sec                  receiver
[ 17]   0.00-10.00  sec  3.77 GBytes  3.24 Gbits/sec  982            sender
[ 17]   0.00-10.00  sec  3.76 GBytes  3.23 Gbits/sec                  receiver
[ 19]   0.00-10.00  sec  3.73 GBytes  3.20 Gbits/sec  910            sender
[ 19]   0.00-10.00  sec  3.72 GBytes  3.20 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  28.7 GBytes  24.6 Gbits/sec  10246             sender
[SUM]   0.00-10.00  sec  28.6 GBytes  24.5 Gbits/sec                  receiver
2 Likes
15

Thanks, that helped a bit.
I changed the MTU to 9000 on both sides. dctcp had already been enabled.

Here is the ifconfig on the NAS side:

enp65s0f1np1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
        inet 192.168.2.20  netmask 255.255.255.0  broadcast 192.168.2.255
        inet6 fe80::63f:72ff:fed9:35a3  prefixlen 64  scopeid 0x20<link>
        ether 04:3f:72:d9:35:a3  txqueuelen 1000  (Ethernet)
        RX packets 18428071  bytes 76108391803 (70.8 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 571758  bytes 37750146 (36.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

And here on the client side:

enp2s0f1np1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
        inet 192.168.2.21  netmask 255.255.255.0  broadcast 192.168.2.255
        inet6 fe80::39bd:47b:6087:b994  prefixlen 64  scopeid 0x20<link>
        ether c4:70:bd:4f:86:39  txqueuelen 1000  (Ethernet)
        RX packets 701212  bytes 46302247 (46.3 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 34779331  bytes 100837308878 (100.8 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
And here are the latest iperf results (around 24 Gbit/s)

Btw I also ran a -P8 one and it was a bit slower.

iperf3 -c 192.168.2.20 -P16
Connecting to host 192.168.2.20, port 5201
[  5] local 192.168.2.21 port 34284 connected to 192.168.2.20 port 5201
[  7] local 192.168.2.21 port 34292 connected to 192.168.2.20 port 5201
[  9] local 192.168.2.21 port 34298 connected to 192.168.2.20 port 5201
[ 11] local 192.168.2.21 port 34314 connected to 192.168.2.20 port 5201
[ 13] local 192.168.2.21 port 34324 connected to 192.168.2.20 port 5201
[ 15] local 192.168.2.21 port 34332 connected to 192.168.2.20 port 5201
[ 17] local 192.168.2.21 port 34334 connected to 192.168.2.20 port 5201
[ 19] local 192.168.2.21 port 34338 connected to 192.168.2.20 port 5201
[ 21] local 192.168.2.21 port 34340 connected to 192.168.2.20 port 5201
[ 23] local 192.168.2.21 port 34356 connected to 192.168.2.20 port 5201
[ 25] local 192.168.2.21 port 34358 connected to 192.168.2.20 port 5201
[ 27] local 192.168.2.21 port 34360 connected to 192.168.2.20 port 5201
[ 29] local 192.168.2.21 port 34364 connected to 192.168.2.20 port 5201
[ 31] local 192.168.2.21 port 34368 connected to 192.168.2.20 port 5201
[ 33] local 192.168.2.21 port 34384 connected to 192.168.2.20 port 5201
[ 35] local 192.168.2.21 port 34390 connected to 192.168.2.20 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   192 MBytes  1.61 Gbits/sec    0   1.25 MBytes       
[  7]   0.00-1.00   sec   191 MBytes  1.60 Gbits/sec    0   1.25 MBytes       
[  9]   0.00-1.00   sec   191 MBytes  1.60 Gbits/sec    0   1.31 MBytes       
[ 11]   0.00-1.00   sec   191 MBytes  1.60 Gbits/sec    0   1.25 MBytes       
[ 13]   0.00-1.00   sec   192 MBytes  1.61 Gbits/sec    0   1.22 MBytes       
[ 15]   0.00-1.00   sec   192 MBytes  1.61 Gbits/sec    0   2.15 MBytes       
[ 17]   0.00-1.00   sec   192 MBytes  1.61 Gbits/sec    0   1.34 MBytes       
[ 19]   0.00-1.00   sec   192 MBytes  1.61 Gbits/sec    0   1.24 MBytes       
[ 21]   0.00-1.00   sec   191 MBytes  1.60 Gbits/sec    0   1.19 MBytes       
[ 23]   0.00-1.00   sec   192 MBytes  1.61 Gbits/sec    0   1.24 MBytes       
[ 25]   0.00-1.00   sec   192 MBytes  1.61 Gbits/sec    0   1.45 MBytes       
[ 27]   0.00-1.00   sec   192 MBytes  1.61 Gbits/sec    0   1.32 MBytes       
[ 29]   0.00-1.00   sec   191 MBytes  1.60 Gbits/sec    0   1.32 MBytes       
[ 31]   0.00-1.00   sec   192 MBytes  1.61 Gbits/sec    0   1.39 MBytes       
[ 33]   0.00-1.00   sec   191 MBytes  1.60 Gbits/sec    0    874 KBytes       
[ 35]   0.00-1.00   sec   192 MBytes  1.61 Gbits/sec    0   1.21 MBytes       
[SUM]   0.00-1.00   sec  3.00 GBytes  25.8 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   184 MBytes  1.54 Gbits/sec    0   1.25 MBytes       
[  7]   1.00-2.00   sec   182 MBytes  1.53 Gbits/sec    0   1.25 MBytes       
[  9]   1.00-2.00   sec   182 MBytes  1.53 Gbits/sec    0   1.31 MBytes       
[ 11]   1.00-2.00   sec   182 MBytes  1.53 Gbits/sec    0   1.25 MBytes       
[ 13]   1.00-2.00   sec   182 MBytes  1.53 Gbits/sec    0   1.22 MBytes       
[ 15]   1.00-2.00   sec   182 MBytes  1.53 Gbits/sec    0   2.15 MBytes       
[ 17]   1.00-2.00   sec   182 MBytes  1.53 Gbits/sec    0   1.34 MBytes       
[ 19]   1.00-2.00   sec   182 MBytes  1.53 Gbits/sec    0   1.24 MBytes       
[ 21]   1.00-2.00   sec   182 MBytes  1.53 Gbits/sec    0   1.19 MBytes       
[ 23]   1.00-2.00   sec   182 MBytes  1.53 Gbits/sec    0   1.24 MBytes       
[ 25]   1.00-2.00   sec   182 MBytes  1.53 Gbits/sec    0   1.45 MBytes       
[ 27]   1.00-2.00   sec   182 MBytes  1.53 Gbits/sec    0   1.32 MBytes       
[ 29]   1.00-2.00   sec   182 MBytes  1.53 Gbits/sec    0   1.32 MBytes       
[ 31]   1.00-2.00   sec   182 MBytes  1.53 Gbits/sec    0   1.39 MBytes       
[ 33]   1.00-2.00   sec   183 MBytes  1.53 Gbits/sec    0    874 KBytes       
[ 35]   1.00-2.00   sec   182 MBytes  1.53 Gbits/sec    0   1.21 MBytes       
[SUM]   1.00-2.00   sec  2.85 GBytes  24.5 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   179 MBytes  1.50 Gbits/sec    0   1.25 MBytes       
[  7]   2.00-3.00   sec   180 MBytes  1.51 Gbits/sec    0   1.25 MBytes       
[  9]   2.00-3.00   sec   180 MBytes  1.51 Gbits/sec    0   1.31 MBytes       
[ 11]   2.00-3.00   sec   180 MBytes  1.51 Gbits/sec    0   1.25 MBytes       
[ 13]   2.00-3.00   sec   180 MBytes  1.51 Gbits/sec    0   1.22 MBytes       
[ 15]   2.00-3.00   sec   180 MBytes  1.51 Gbits/sec    0   2.15 MBytes       
[ 17]   2.00-3.00   sec   180 MBytes  1.51 Gbits/sec    0   1.34 MBytes       
[ 19]   2.00-3.00   sec   180 MBytes  1.51 Gbits/sec    0   1.24 MBytes       
[ 21]   2.00-3.00   sec   180 MBytes  1.51 Gbits/sec    0   1.19 MBytes       
[ 23]   2.00-3.00   sec   180 MBytes  1.51 Gbits/sec    0   1.24 MBytes       
[ 25]   2.00-3.00   sec   180 MBytes  1.51 Gbits/sec    0   1.45 MBytes       
[ 27]   2.00-3.00   sec   179 MBytes  1.50 Gbits/sec    0   1.32 MBytes       
[ 29]   2.00-3.00   sec   180 MBytes  1.51 Gbits/sec    0   1.32 MBytes       
[ 31]   2.00-3.00   sec   180 MBytes  1.51 Gbits/sec    0   1.39 MBytes       
[ 33]   2.00-3.00   sec   179 MBytes  1.50 Gbits/sec    0    874 KBytes       
[ 35]   2.00-3.00   sec   180 MBytes  1.51 Gbits/sec    0   1.21 MBytes       
[SUM]   2.00-3.00   sec  2.81 GBytes  24.1 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec   179 MBytes  1.50 Gbits/sec    0   1.25 MBytes       
[  7]   3.00-4.00   sec   179 MBytes  1.50 Gbits/sec    0   1.25 MBytes       
[  9]   3.00-4.00   sec   179 MBytes  1.50 Gbits/sec    0   1.31 MBytes       
[ 11]   3.00-4.00   sec   179 MBytes  1.50 Gbits/sec    0   1.25 MBytes       
[ 13]   3.00-4.00   sec   178 MBytes  1.49 Gbits/sec    0   1.22 MBytes       
[ 15]   3.00-4.00   sec   178 MBytes  1.49 Gbits/sec    0   2.15 MBytes       
[ 17]   3.00-4.00   sec   178 MBytes  1.49 Gbits/sec    0   1.34 MBytes       
[ 19]   3.00-4.00   sec   178 MBytes  1.49 Gbits/sec    0   1.24 MBytes       
[ 21]   3.00-4.00   sec   179 MBytes  1.50 Gbits/sec    0   1.19 MBytes       
[ 23]   3.00-4.00   sec   178 MBytes  1.49 Gbits/sec    0   1.24 MBytes       
[ 25]   3.00-4.00   sec   178 MBytes  1.49 Gbits/sec    0   1.45 MBytes       
[ 27]   3.00-4.00   sec   179 MBytes  1.50 Gbits/sec    0   1.32 MBytes       
[ 29]   3.00-4.00   sec   178 MBytes  1.49 Gbits/sec    0   1.32 MBytes       
[ 31]   3.00-4.00   sec   178 MBytes  1.49 Gbits/sec    0   1.39 MBytes       
[ 33]   3.00-4.00   sec   178 MBytes  1.49 Gbits/sec    0    874 KBytes       
[ 35]   3.00-4.00   sec   178 MBytes  1.49 Gbits/sec    0   1.21 MBytes       
[SUM]   3.00-4.00   sec  2.78 GBytes  23.9 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec   176 MBytes  1.48 Gbits/sec    0   1.25 MBytes       
[  7]   4.00-5.00   sec   176 MBytes  1.48 Gbits/sec    0   1.25 MBytes       
[  9]   4.00-5.00   sec   176 MBytes  1.48 Gbits/sec    0   1.31 MBytes       
[ 11]   4.00-5.00   sec   176 MBytes  1.48 Gbits/sec    0   1.25 MBytes       
[ 13]   4.00-5.00   sec   176 MBytes  1.48 Gbits/sec    0   1.22 MBytes       
[ 15]   4.00-5.00   sec   178 MBytes  1.49 Gbits/sec    0   2.15 MBytes       
[ 17]   4.00-5.00   sec   178 MBytes  1.49 Gbits/sec    0   1.34 MBytes       
[ 19]   4.00-5.00   sec   178 MBytes  1.49 Gbits/sec    0   1.24 MBytes       
[ 21]   4.00-5.00   sec   176 MBytes  1.48 Gbits/sec    0   1.19 MBytes       
[ 23]   4.00-5.00   sec   176 MBytes  1.48 Gbits/sec    0   1.24 MBytes       
[ 25]   4.00-5.00   sec   178 MBytes  1.49 Gbits/sec    0   1.45 MBytes       
[ 27]   4.00-5.00   sec   176 MBytes  1.48 Gbits/sec    0   1.32 MBytes       
[ 29]   4.00-5.00   sec   178 MBytes  1.49 Gbits/sec    0   1.32 MBytes       
[ 31]   4.00-5.00   sec   176 MBytes  1.48 Gbits/sec    0   1.39 MBytes       
[ 33]   4.00-5.00   sec   177 MBytes  1.48 Gbits/sec    0    874 KBytes       
[ 35]   4.00-5.00   sec   176 MBytes  1.48 Gbits/sec    0   1.21 MBytes       
[SUM]   4.00-5.00   sec  2.76 GBytes  23.7 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec   174 MBytes  1.46 Gbits/sec    0   1.25 MBytes       
[  7]   5.00-6.00   sec   174 MBytes  1.46 Gbits/sec    0   1.25 MBytes       
[  9]   5.00-6.00   sec   174 MBytes  1.46 Gbits/sec    0   1.31 MBytes       
[ 11]   5.00-6.00   sec   174 MBytes  1.46 Gbits/sec    0   1.25 MBytes       
[ 13]   5.00-6.00   sec   174 MBytes  1.46 Gbits/sec    0   1.22 MBytes       
[ 15]   5.00-6.00   sec   172 MBytes  1.45 Gbits/sec    0   2.15 MBytes       
[ 17]   5.00-6.00   sec   172 MBytes  1.45 Gbits/sec    0   1.34 MBytes       
[ 19]   5.00-6.00   sec   172 MBytes  1.45 Gbits/sec    0   1.24 MBytes       
[ 21]   5.00-6.00   sec   174 MBytes  1.46 Gbits/sec    0   1.19 MBytes       
[ 23]   5.00-6.00   sec   174 MBytes  1.46 Gbits/sec    0   1.24 MBytes       
[ 25]   5.00-6.00   sec   172 MBytes  1.45 Gbits/sec    0   1.45 MBytes       
[ 27]   5.00-6.00   sec   174 MBytes  1.46 Gbits/sec    0   1.32 MBytes       
[ 29]   5.00-6.00   sec   174 MBytes  1.46 Gbits/sec    0   1.32 MBytes       
[ 31]   5.00-6.00   sec   174 MBytes  1.46 Gbits/sec    0   1.39 MBytes       
[ 33]   5.00-6.00   sec   173 MBytes  1.45 Gbits/sec    0    874 KBytes       
[ 35]   5.00-6.00   sec   174 MBytes  1.46 Gbits/sec    0   1.21 MBytes       
[SUM]   5.00-6.00   sec  2.71 GBytes  23.3 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec   174 MBytes  1.46 Gbits/sec    0   1.25 MBytes       
[  7]   6.00-7.00   sec   174 MBytes  1.46 Gbits/sec    0   1.25 MBytes       
[  9]   6.00-7.00   sec   174 MBytes  1.46 Gbits/sec    0   1.31 MBytes       
[ 11]   6.00-7.00   sec   174 MBytes  1.46 Gbits/sec    0   1.25 MBytes       
[ 13]   6.00-7.00   sec   175 MBytes  1.47 Gbits/sec    0   1.22 MBytes       
[ 15]   6.00-7.00   sec   175 MBytes  1.47 Gbits/sec    0   2.15 MBytes       
[ 17]   6.00-7.00   sec   175 MBytes  1.47 Gbits/sec    0   1.34 MBytes       
[ 19]   6.00-7.00   sec   175 MBytes  1.47 Gbits/sec    0   1.24 MBytes       
[ 21]   6.00-7.00   sec   174 MBytes  1.46 Gbits/sec    0   1.19 MBytes       
[ 23]   6.00-7.00   sec   174 MBytes  1.46 Gbits/sec    0   1.24 MBytes       
[ 25]   6.00-7.00   sec   175 MBytes  1.47 Gbits/sec    0   1.45 MBytes       
[ 27]   6.00-7.00   sec   174 MBytes  1.46 Gbits/sec    0   1.32 MBytes       
[ 29]   6.00-7.00   sec   174 MBytes  1.46 Gbits/sec    0   1.32 MBytes       
[ 31]   6.00-7.00   sec   175 MBytes  1.47 Gbits/sec    0   1.39 MBytes       
[ 33]   6.00-7.00   sec   175 MBytes  1.46 Gbits/sec    0    874 KBytes       
[ 35]   6.00-7.00   sec   174 MBytes  1.46 Gbits/sec    0   1.21 MBytes       
[SUM]   6.00-7.00   sec  2.72 GBytes  23.4 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec   178 MBytes  1.49 Gbits/sec    0   1.25 MBytes       
[  7]   7.00-8.00   sec   178 MBytes  1.49 Gbits/sec    0   1.25 MBytes       
[  9]   7.00-8.00   sec   178 MBytes  1.49 Gbits/sec    0   1.31 MBytes       
[ 11]   7.00-8.00   sec   178 MBytes  1.49 Gbits/sec    0   1.25 MBytes       
[ 13]   7.00-8.00   sec   176 MBytes  1.48 Gbits/sec    0   1.22 MBytes       
[ 15]   7.00-8.00   sec   176 MBytes  1.48 Gbits/sec    0   2.15 MBytes       
[ 17]   7.00-8.00   sec   176 MBytes  1.48 Gbits/sec    0   1.34 MBytes       
[ 19]   7.00-8.00   sec   176 MBytes  1.48 Gbits/sec    0   1.24 MBytes       
[ 21]   7.00-8.00   sec   178 MBytes  1.49 Gbits/sec    0   1.19 MBytes       
[ 23]   7.00-8.00   sec   178 MBytes  1.49 Gbits/sec    0   1.24 MBytes       
[ 25]   7.00-8.00   sec   176 MBytes  1.48 Gbits/sec    0   1.45 MBytes       
[ 27]   7.00-8.00   sec   178 MBytes  1.49 Gbits/sec    0   1.32 MBytes       
[ 29]   7.00-8.00   sec   178 MBytes  1.49 Gbits/sec    0   1.32 MBytes       
[ 31]   7.00-8.00   sec   176 MBytes  1.48 Gbits/sec    0   1.39 MBytes       
[ 33]   7.00-8.00   sec   177 MBytes  1.48 Gbits/sec    0    874 KBytes       
[ 35]   7.00-8.00   sec   178 MBytes  1.49 Gbits/sec    0   1.21 MBytes       
[SUM]   7.00-8.00   sec  2.77 GBytes  23.8 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec   174 MBytes  1.46 Gbits/sec    0   1.25 MBytes       
[  7]   8.00-9.00   sec   174 MBytes  1.46 Gbits/sec    0   1.25 MBytes       
[  9]   8.00-9.00   sec   174 MBytes  1.46 Gbits/sec    0   1.31 MBytes       
[ 11]   8.00-9.00   sec   174 MBytes  1.46 Gbits/sec    0   1.25 MBytes       
[ 13]   8.00-9.00   sec   174 MBytes  1.46 Gbits/sec    0   1.22 MBytes       
[ 15]   8.00-9.00   sec   175 MBytes  1.47 Gbits/sec    0   2.15 MBytes       
[ 17]   8.00-9.00   sec   175 MBytes  1.47 Gbits/sec    0   1.34 MBytes       
[ 19]   8.00-9.00   sec   175 MBytes  1.47 Gbits/sec    0   1.24 MBytes       
[ 21]   8.00-9.00   sec   174 MBytes  1.46 Gbits/sec    0   1.19 MBytes       
[ 23]   8.00-9.00   sec   174 MBytes  1.46 Gbits/sec    0   1.24 MBytes       
[ 25]   8.00-9.00   sec   175 MBytes  1.47 Gbits/sec    0   1.45 MBytes       
[ 27]   8.00-9.00   sec   174 MBytes  1.46 Gbits/sec    0   1.32 MBytes       
[ 29]   8.00-9.00   sec   174 MBytes  1.46 Gbits/sec    0   1.32 MBytes       
[ 31]   8.00-9.00   sec   174 MBytes  1.46 Gbits/sec    0   1.39 MBytes       
[ 33]   8.00-9.00   sec   175 MBytes  1.46 Gbits/sec    0    874 KBytes       
[ 35]   8.00-9.00   sec   174 MBytes  1.46 Gbits/sec    0   1.21 MBytes       
[SUM]   8.00-9.00   sec  2.72 GBytes  23.4 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec   170 MBytes  1.43 Gbits/sec    0   1.25 MBytes       
[  7]   9.00-10.00  sec   170 MBytes  1.43 Gbits/sec    0   1.25 MBytes       
[  9]   9.00-10.00  sec   170 MBytes  1.43 Gbits/sec    0   1.31 MBytes       
[ 11]   9.00-10.00  sec   170 MBytes  1.43 Gbits/sec    0   1.25 MBytes       
[ 13]   9.00-10.00  sec   170 MBytes  1.43 Gbits/sec    0   1.22 MBytes       
[ 15]   9.00-10.00  sec   170 MBytes  1.43 Gbits/sec    0   2.15 MBytes       
[ 17]   9.00-10.00  sec   170 MBytes  1.43 Gbits/sec    0   1.34 MBytes       
[ 19]   9.00-10.00  sec   170 MBytes  1.43 Gbits/sec    0   1.24 MBytes       
[ 21]   9.00-10.00  sec   170 MBytes  1.43 Gbits/sec    0   1.19 MBytes       
[ 23]   9.00-10.00  sec   170 MBytes  1.43 Gbits/sec    0   1.24 MBytes       
[ 25]   9.00-10.00  sec   170 MBytes  1.43 Gbits/sec    0   1.45 MBytes       
[ 27]   9.00-10.00  sec   170 MBytes  1.43 Gbits/sec    0   1.32 MBytes       
[ 29]   9.00-10.00  sec   170 MBytes  1.43 Gbits/sec    0   1.32 MBytes       
[ 31]   9.00-10.00  sec   171 MBytes  1.44 Gbits/sec    0   1.39 MBytes       
[ 33]   9.00-10.00  sec   170 MBytes  1.42 Gbits/sec    0    874 KBytes       
[ 35]   9.00-10.00  sec   170 MBytes  1.43 Gbits/sec    0   1.21 MBytes       
[SUM]   9.00-10.00  sec  2.66 GBytes  22.8 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.74 GBytes  1.49 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  1.73 GBytes  1.49 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  1.74 GBytes  1.49 Gbits/sec    0             sender
[  7]   0.00-10.00  sec  1.73 GBytes  1.49 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  1.74 GBytes  1.49 Gbits/sec    0             sender
[  9]   0.00-10.00  sec  1.73 GBytes  1.49 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  1.74 GBytes  1.49 Gbits/sec    0             sender
[ 11]   0.00-10.00  sec  1.73 GBytes  1.49 Gbits/sec                  receiver
[ 13]   0.00-10.00  sec  1.74 GBytes  1.49 Gbits/sec    0             sender
[ 13]   0.00-10.00  sec  1.73 GBytes  1.49 Gbits/sec                  receiver
[ 15]   0.00-10.00  sec  1.74 GBytes  1.49 Gbits/sec    0             sender
[ 15]   0.00-10.00  sec  1.73 GBytes  1.49 Gbits/sec                  receiver
[ 17]   0.00-10.00  sec  1.74 GBytes  1.49 Gbits/sec    0             sender
[ 17]   0.00-10.00  sec  1.73 GBytes  1.49 Gbits/sec                  receiver
[ 19]   0.00-10.00  sec  1.74 GBytes  1.49 Gbits/sec    0             sender
[ 19]   0.00-10.00  sec  1.73 GBytes  1.49 Gbits/sec                  receiver
[ 21]   0.00-10.00  sec  1.74 GBytes  1.49 Gbits/sec    0             sender
[ 21]   0.00-10.00  sec  1.73 GBytes  1.49 Gbits/sec                  receiver
[ 23]   0.00-10.00  sec  1.74 GBytes  1.49 Gbits/sec    0             sender
[ 23]   0.00-10.00  sec  1.73 GBytes  1.49 Gbits/sec                  receiver
[ 25]   0.00-10.00  sec  1.74 GBytes  1.49 Gbits/sec    0             sender
[ 25]   0.00-10.00  sec  1.73 GBytes  1.49 Gbits/sec                  receiver
[ 27]   0.00-10.00  sec  1.74 GBytes  1.49 Gbits/sec    0             sender
[ 27]   0.00-10.00  sec  1.73 GBytes  1.49 Gbits/sec                  receiver
[ 29]   0.00-10.00  sec  1.74 GBytes  1.49 Gbits/sec    0             sender
[ 29]   0.00-10.00  sec  1.73 GBytes  1.49 Gbits/sec                  receiver
[ 31]   0.00-10.00  sec  1.74 GBytes  1.49 Gbits/sec    0             sender
[ 31]   0.00-10.00  sec  1.73 GBytes  1.49 Gbits/sec                  receiver
[ 33]   0.00-10.00  sec  1.74 GBytes  1.49 Gbits/sec    0             sender
[ 33]   0.00-10.00  sec  1.73 GBytes  1.49 Gbits/sec                  receiver
[ 35]   0.00-10.00  sec  1.74 GBytes  1.49 Gbits/sec    0             sender
[ 35]   0.00-10.00  sec  1.73 GBytes  1.49 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  27.8 GBytes  23.9 Gbits/sec    0             sender
[SUM]   0.00-10.00  sec  27.7 GBytes  23.8 Gbits/sec                  receiver

I’m running on bare metal (Epyc 9334 QS)

1 Like
16

Increase your tx queue length (ip link set txqueuelen 20000 ${mlx_dev})
Try different values.

1 Like
17

That didn’t change a thing. I tried many values between the default 1000 and 100000.

Here is the startup script (pre init) modified from wardtj's original 
#!/bin/sh

PATH=“/bin:/sbin:/usr/bin:/usr/sbin:${PATH}”
export PATH

echo 128 > /sys/module/zfs/parameters/zfs_vdev_def_queue_depth
echo 0 > /sys/module/zfs/parameters/zfs_dmu_offset_next_sync
echo 12 > /sys/module/zfs/parameters/zfs_vdev_async_read_max_active
echo 4096 > /sys/module/zfs/parameters/zfs_vdev_max_active

for i in `ls /sys/block | grep -i nvme`
do
echo 512 > /sys/block/$i/queue/read_ahead_kb
echo mq-deadline > /sys/block/$i/queue/scheduler
done

for i in `ls /sys/block | grep -i sd`
do
echo 512 > /sys/block/$i/queue/read_ahead_kb
echo mq-deadline > /sys/block/$i/queue/scheduler
done

/usr/sbin/modprobe tcp_dctcp
/usr/sbin/sysctl -w net.ipv4.tcp_congestion_control=dctcp
/usr/sbin/sysctl -w net.ipv4.tcp_ecn_fallback=0
/usr/sbin/sysctl -w net.core.netdev_max_backlog=8192
/usr/sbin/sysctl -w net.ipv4.tcp_max_syn_backlog=8192
/usr/sbin/sysctl -w net.core.rmem_max=16777216
/usr/sbin/sysctl -w net.core.wmem_max=16777216
1 Like
18

I recommend watching Raid Owls 100gb network series. Heres the second in the series.

https://youtu.be/T6hHyJQATyc?si=SjiLdPNota0PRmW_

His series covers the problems in the home lab and whats needed to get the perf up.

His 40gb video is good as it explains why above 25gb is a waste.

HtH

19

100gb is pointless without RDMA support. If using it for SMB about 25gb is all one can expect unless one resorts to using robocopy versus gui.

Based on this thread i suspect hardware is not capable. Raid Owl notes this in the video i posted. Pcie slots and lanes can cripple the card. Aka putting it into a 8x slot when it wants x16 will choke it below x8 perforance. Youll get 10gb levels. Which we see in his tests.

Also heat is a problem on these. They need airflow across them or they throttle. Anything above ConnectX-3 needs fan or perforance degrades as it gets hot.

HtH!

2 Likes
20

That video is waste if the guy claiming that.

40gbe easily gives you 3-4gb/s with SMBdirect/rocev2 on windows

Plus 40GBe setup is dirt cheap

2 Likes