Thanks for all your replies!
Both On the NAS and on the client the NICs are connected directly to the CPU
On the client side everything is default.
On the NAS side i have implemented @wardtj optimisations found here: 100 Gbe & TrueNAS Scale 23.10 iSCSI - Performance Unleashed
Yes I have of course tried that. The results are within the margin of error identical.
Here a run with 4 threads
iperf3 -c 192.168.2.20 -P4
Connecting to host 192.168.2.20, port 5201
[ 5] local 192.168.2.21 port 50108 connected to 192.168.2.20 port 5201
[ 7] local 192.168.2.21 port 50120 connected to 192.168.2.20 port 5201
[ 9] local 192.168.2.21 port 50136 connected to 192.168.2.20 port 5201
[ 11] local 192.168.2.21 port 50146 connected to 192.168.2.20 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 438 MBytes 3.67 Gbits/sec 0 211 KBytes
[ 7] 0.00-1.00 sec 440 MBytes 3.69 Gbits/sec 0 519 KBytes
[ 9] 0.00-1.00 sec 438 MBytes 3.67 Gbits/sec 0 274 KBytes
[ 11] 0.00-1.00 sec 439 MBytes 3.68 Gbits/sec 0 409 KBytes
[SUM] 0.00-1.00 sec 1.71 GBytes 14.7 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 439 MBytes 3.68 Gbits/sec 0 211 KBytes
[ 7] 1.00-2.00 sec 440 MBytes 3.69 Gbits/sec 0 519 KBytes
[ 9] 1.00-2.00 sec 439 MBytes 3.68 Gbits/sec 0 274 KBytes
[ 11] 1.00-2.00 sec 439 MBytes 3.68 Gbits/sec 0 409 KBytes
[SUM] 1.00-2.00 sec 1.72 GBytes 14.7 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 439 MBytes 3.68 Gbits/sec 0 211 KBytes
[ 7] 2.00-3.00 sec 439 MBytes 3.68 Gbits/sec 0 519 KBytes
[ 9] 2.00-3.00 sec 439 MBytes 3.69 Gbits/sec 0 274 KBytes
[ 11] 2.00-3.00 sec 440 MBytes 3.69 Gbits/sec 0 409 KBytes
[SUM] 2.00-3.00 sec 1.72 GBytes 14.7 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 439 MBytes 3.68 Gbits/sec 0 211 KBytes
[ 7] 3.00-4.00 sec 440 MBytes 3.69 Gbits/sec 0 519 KBytes
[ 9] 3.00-4.00 sec 439 MBytes 3.68 Gbits/sec 0 274 KBytes
[ 11] 3.00-4.00 sec 438 MBytes 3.68 Gbits/sec 0 409 KBytes
[SUM] 3.00-4.00 sec 1.71 GBytes 14.7 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 438 MBytes 3.68 Gbits/sec 0 211 KBytes
[ 7] 4.00-5.00 sec 439 MBytes 3.68 Gbits/sec 0 519 KBytes
[ 9] 4.00-5.00 sec 439 MBytes 3.68 Gbits/sec 0 274 KBytes
[ 11] 4.00-5.00 sec 439 MBytes 3.68 Gbits/sec 0 409 KBytes
[SUM] 4.00-5.00 sec 1.71 GBytes 14.7 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 439 MBytes 3.68 Gbits/sec 0 211 KBytes
[ 7] 5.00-6.00 sec 439 MBytes 3.68 Gbits/sec 0 519 KBytes
[ 9] 5.00-6.00 sec 438 MBytes 3.68 Gbits/sec 0 274 KBytes
[ 11] 5.00-6.00 sec 438 MBytes 3.68 Gbits/sec 0 409 KBytes
[SUM] 5.00-6.00 sec 1.71 GBytes 14.7 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.00-7.00 sec 439 MBytes 3.68 Gbits/sec 0 211 KBytes
[ 7] 6.00-7.00 sec 439 MBytes 3.68 Gbits/sec 0 519 KBytes
[ 9] 6.00-7.00 sec 439 MBytes 3.68 Gbits/sec 0 274 KBytes
[ 11] 6.00-7.00 sec 439 MBytes 3.68 Gbits/sec 0 409 KBytes
[SUM] 6.00-7.00 sec 1.71 GBytes 14.7 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.00 sec 439 MBytes 3.68 Gbits/sec 0 211 KBytes
[ 7] 7.00-8.00 sec 439 MBytes 3.68 Gbits/sec 0 519 KBytes
[ 9] 7.00-8.00 sec 439 MBytes 3.68 Gbits/sec 0 274 KBytes
[ 11] 7.00-8.00 sec 439 MBytes 3.68 Gbits/sec 0 409 KBytes
[SUM] 7.00-8.00 sec 1.71 GBytes 14.7 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.00-9.00 sec 439 MBytes 3.68 Gbits/sec 0 211 KBytes
[ 7] 8.00-9.00 sec 439 MBytes 3.68 Gbits/sec 0 519 KBytes
[ 9] 8.00-9.00 sec 439 MBytes 3.68 Gbits/sec 0 274 KBytes
[ 11] 8.00-9.00 sec 439 MBytes 3.68 Gbits/sec 0 409 KBytes
[SUM] 8.00-9.00 sec 1.71 GBytes 14.7 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.00-10.00 sec 439 MBytes 3.68 Gbits/sec 0 211 KBytes
[ 7] 9.00-10.00 sec 439 MBytes 3.68 Gbits/sec 0 519 KBytes
[ 9] 9.00-10.00 sec 439 MBytes 3.68 Gbits/sec 0 274 KBytes
[ 11] 9.00-10.00 sec 439 MBytes 3.68 Gbits/sec 0 409 KBytes
[SUM] 9.00-10.00 sec 1.71 GBytes 14.7 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 4.28 GBytes 3.68 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 4.28 GBytes 3.68 Gbits/sec receiver
[ 7] 0.00-10.00 sec 4.29 GBytes 3.68 Gbits/sec 0 sender
[ 7] 0.00-10.00 sec 4.28 GBytes 3.68 Gbits/sec receiver
[ 9] 0.00-10.00 sec 4.28 GBytes 3.68 Gbits/sec 0 sender
[ 9] 0.00-10.00 sec 4.28 GBytes 3.68 Gbits/sec receiver
[ 11] 0.00-10.00 sec 4.29 GBytes 3.68 Gbits/sec 0 sender
[ 11] 0.00-10.00 sec 4.28 GBytes 3.68 Gbits/sec receiver
[SUM] 0.00-10.00 sec 17.1 GBytes 14.7 Gbits/sec 0 sender
[SUM] 0.00-10.00 sec 17.1 GBytes 14.7 Gbits/sec receiver
TrueNAS SCALE is now called TrueNAS Community Edition (CE).
I’m using 25.04 (latest at to time I’m writing this.)