It’s been a long time since I posted an update.
After a lot of testing and changes, I’m still running into this issue, but not as bad as before kinda sorta.:
- I replaced the ConnectX-6 NIC in my PC with a smaller low-profile one and put it on the x16 slot under my GPU (making it PCIe 4.0 x8 speeds even though it’s only an x4 card). The previous ConnectX-6 would readily drop speeds and sometimes not even connect. That’s probably one reason the fiber couldn’t keep up.
- I upgraded the
iperf3version with an unofficial build. Windows hasn’t had a compiled build since 2016, but some folks are doing it themselves like this one: GitHub - ar51an/iperf3-win-builds: iperf3 binaries for Windows. Benchmark your network limits..
The issue
This is the issue I was running into:
The NAS is capable of much faster read speeds, and the NVMe drive is capable of 7GB/s writes. Both of these should be maxing out 25Gb Ethernet.
I can’t test this with 10Gb copper right now, but it was transferring 1GB/s in the past with the same file to the same drive.
I already enabled SMB Multichannel enabled on both fiber lines:
PS > Get-SmbMultichannelConnection
Server Name Selected Client IP Server IP Client Interface Index Server Interface Index Client RSS Capable Client
RDMA
Capabl
e
----------- -------- --------- --------- ---------------------- ---------------------- ------------------ ------
storeman.octen True 10.1.0.49 10.1.0.6 12 4 True False
storeman.octen True 192.168.6.3 192.168.6.2 11 5 True False
More transfers
I did more transfers tonight to see what was going on, and this file transferred a lot faster, but not fast enough:
And here’s the way back:
I unplugged and re-plugged in the fiber jacks and this is what I got from NAS to PC:
And PC to NAS (even faster):
The NAS is running 2 x 25Gb fiber just like my PC. They both go through a UniFi switch, but I tried going direct in the past and saw no difference.
iperf3 benchmarks
2.5Gb PC onboard NIC to 10Gb NAS onboard NIC
# iperf3.exe -c 192.168.6.10
Connecting to host 192.168.6.10, port 5201
[ 5] local 192.168.6.12 port 56533 connected to 192.168.6.10 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 301 MBytes 2.50 Gbits/sec
[ 5] 1.01-2.00 sec 292 MBytes 2.47 Gbits/sec
[ 5] 2.00-3.01 sec 295 MBytes 2.47 Gbits/sec
[ 5] 3.01-4.00 sec 286 MBytes 2.42 Gbits/sec
[ 5] 4.00-5.01 sec 298 MBytes 2.47 Gbits/sec
[ 5] 5.01-6.01 sec 291 MBytes 2.47 Gbits/sec
[ 5] 6.01-7.01 sec 296 MBytes 2.47 Gbits/sec
[ 5] 7.01-8.00 sec 290 MBytes 2.46 Gbits/sec
[ 5] 8.00-9.01 sec 295 MBytes 2.47 Gbits/sec
[ 5] 9.01-10.01 sec 293 MBytes 2.44 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 2.87 GBytes 2.46 Gbits/sec sender
[ 5] 0.00-10.02 sec 2.87 GBytes 2.46 Gbits/sec receiver
# iperf3.exe -c 192.168.6.10 -R
Connecting to host 192.168.6.10, port 5201
Reverse mode, remote host 192.168.6.10 is sending
[ 5] local 192.168.6.12 port 56605 connected to 192.168.6.10 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 294 MBytes 2.46 Gbits/sec
[ 5] 1.00-2.00 sec 293 MBytes 2.46 Gbits/sec
[ 5] 2.00-3.00 sec 291 MBytes 2.44 Gbits/sec
[ 5] 3.00-4.01 sec 295 MBytes 2.46 Gbits/sec
[ 5] 4.01-5.01 sec 293 MBytes 2.46 Gbits/sec
[ 5] 5.01-6.01 sec 294 MBytes 2.45 Gbits/sec
[ 5] 6.01-7.01 sec 285 MBytes 2.39 Gbits/sec
[ 5] 7.01-8.01 sec 284 MBytes 2.40 Gbits/sec
[ 5] 8.01-9.01 sec 294 MBytes 2.46 Gbits/sec
[ 5] 9.01-10.01 sec 290 MBytes 2.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 2.85 GBytes 2.44 Gbits/sec 0 sender
[ 5] 0.00-10.01 sec 2.85 GBytes 2.44 Gbits/sec receiver
25Gb fiber PC (Port 1) to 25Gb fiber NAS (Port 1)
# iperf3.exe -c 10.1.0.6
Connecting to host 10.1.0.6, port 5201
[ 5] local 10.1.0.49 port 56904 connected to 10.1.0.6 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 2.52 GBytes 21.5 Gbits/sec
[ 5] 1.01-2.01 sec 2.46 GBytes 21.0 Gbits/sec
[ 5] 2.01-3.01 sec 2.56 GBytes 22.0 Gbits/sec
[ 5] 3.01-4.00 sec 2.54 GBytes 22.1 Gbits/sec
[ 5] 4.00-5.01 sec 2.63 GBytes 22.3 Gbits/sec
[ 5] 5.01-6.01 sec 2.62 GBytes 22.6 Gbits/sec
[ 5] 6.01-7.01 sec 2.59 GBytes 22.2 Gbits/sec
[ 5] 7.01-8.01 sec 2.61 GBytes 22.6 Gbits/sec
[ 5] 8.01-9.01 sec 2.67 GBytes 22.8 Gbits/sec
[ 5] 9.01-10.01 sec 2.54 GBytes 21.7 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 25.7 GBytes 22.1 Gbits/sec sender
[ 5] 0.00-10.02 sec 25.7 GBytes 22.1 Gbits/sec receiver
# iperf3.exe -c 10.1.0.6 -R
Connecting to host 10.1.0.6, port 5201
Reverse mode, remote host 10.1.0.6 is sending
[ 5] local 10.1.0.49 port 56877 connected to 10.1.0.6 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 1.55 GBytes 13.2 Gbits/sec
[ 5] 1.01-2.01 sec 1.58 GBytes 13.5 Gbits/sec
[ 5] 2.01-3.00 sec 1.45 GBytes 12.5 Gbits/sec
[ 5] 3.00-4.00 sec 1.44 GBytes 12.4 Gbits/sec
[ 5] 4.00-5.01 sec 1.54 GBytes 13.2 Gbits/sec
[ 5] 5.01-6.01 sec 1.45 GBytes 12.5 Gbits/sec
[ 5] 6.01-7.01 sec 1.46 GBytes 12.5 Gbits/sec
[ 5] 7.01-8.02 sec 1.62 GBytes 13.6 Gbits/sec
[ 5] 8.02-9.01 sec 1.39 GBytes 12.1 Gbits/sec
[ 5] 9.01-10.01 sec 1.43 GBytes 12.2 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 14.9 GBytes 12.8 Gbits/sec 1 sender
[ 5] 0.00-10.01 sec 14.9 GBytes 12.8 Gbits/sec receiver
# iperf3.exe -c 10.1.0.6 -P 10
Connecting to host 10.1.0.6, port 5201
[ ID] Interval Transfer Bitrate
[SUM] 0.00-10.01 sec 26.8 GBytes 23.0 Gbits/sec sender
[SUM] 0.00-10.02 sec 26.8 GBytes 23.0 Gbits/sec receiver
# iperf3.exe -c 10.1.0.6 -P 10 -R
Connecting to host 10.1.0.6, port 5201
Reverse mode, remote host 10.1.0.6 is sending
[SUM] 0.00-10.01 sec 14.3 GBytes 12.3 Gbits/sec 0 sender
[SUM] 0.00-10.01 sec 14.2 GBytes 12.2 Gbits/sec receiver
25Gb fiber PC (Port 2) to 25Gb fiber NAS (Port 2)
# iperf3.exe -c 192.168.6.2
Connecting to host 192.168.6.2, port 5201
[ 5] local 192.168.6.3 port 56640 connected to 192.168.6.2 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 2.84 GBytes 24.3 Gbits/sec
[ 5] 1.00-2.01 sec 2.87 GBytes 24.4 Gbits/sec
[ 5] 2.01-3.00 sec 2.84 GBytes 24.6 Gbits/sec
[ 5] 3.00-4.01 sec 2.90 GBytes 24.8 Gbits/sec
[ 5] 4.01-5.00 sec 2.83 GBytes 24.6 Gbits/sec
[ 5] 5.00-6.01 sec 2.60 GBytes 22.0 Gbits/sec
[ 5] 6.01-7.01 sec 2.83 GBytes 24.5 Gbits/sec
[ 5] 7.01-8.01 sec 2.79 GBytes 23.8 Gbits/sec
[ 5] 8.01-9.01 sec 2.49 GBytes 21.4 Gbits/sec
[ 5] 9.01-10.00 sec 2.86 GBytes 24.8 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 27.8 GBytes 23.9 Gbits/sec sender
[ 5] 0.00-10.00 sec 27.8 GBytes 23.9 Gbits/sec receiver
# iperf3.exe -c 192.168.6.2 -R
Connecting to host 192.168.6.2, port 5201
Reverse mode, remote host 192.168.6.2 is sending
[ 5] local 192.168.6.3 port 56652 connected to 192.168.6.2 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 1.64 GBytes 14.0 Gbits/sec
[ 5] 1.01-2.00 sec 1.55 GBytes 13.3 Gbits/sec
[ 5] 2.00-3.00 sec 1.68 GBytes 14.4 Gbits/sec
[ 5] 3.00-4.01 sec 1.72 GBytes 14.6 Gbits/sec
[ 5] 4.01-5.01 sec 1.68 GBytes 14.4 Gbits/sec
[ 5] 5.01-6.00 sec 1.65 GBytes 14.3 Gbits/sec
[ 5] 6.00-7.00 sec 1.78 GBytes 15.3 Gbits/sec
[ 5] 7.00-8.00 sec 1.63 GBytes 14.0 Gbits/sec
[ 5] 8.00-9.00 sec 1.73 GBytes 14.8 Gbits/sec
[ 5] 9.00-10.00 sec 1.64 GBytes 14.1 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 16.7 GBytes 14.3 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 16.7 GBytes 14.3 Gbits/sec receiver
Conclusion
Something is definitely wonky for the NAS → PC transfers on BOTH fiber wires. I don’t think that’s a coincidence.
At this point, it’s definitely not the zpool of the drives in Windows, it’s something wonky with the network traffic from the NAS to Windows.