After a couple hours of debugging I have correctly (I think) identified the root cause and 2 potential solutions, 1 on either side of the link.
Presumptions:
My primary method of testing (and usage) was copying files to and from the network shares. Aka using SMB.
To start eliminating causes I ran an iperf test:
iperf -s -B 192.168.10.5 (nas)
iperf -c 192.168.10.5 -P 2 -i 5 -t 20 (windows)
= [SUM] 0.00-5.00 sec 1.45 GBytes 2.48 Gbits/sec
This at least proves it’s not a NETWORK LINK issue. The network itself is fully capable of tossing around data at that speed.
That leaves us with 2 other potential causes:
- Routing
- SMB itself
Like I said in OP, I tested manually adding a route, which had no effect.
I am not a not a networking whiz, so this is a little out of my depth and I lack the knowledge to verify how the route is working. But assuming nothing broke means it was working, that eliminates routing as the culprit.
This leaves SMB itself. I went through dozens of commands, but I’ll only post what ultimately yielded results.
SMB has a feature called Multichannel. It’s on by default in the versions of samba and windows I have. While I can’t use it to aggregate my connections (apparently a server-only feature), apparently it functions as failover / high reliability. (Hence it using the other connection if I disable the 1st one.)
Get-SmbMultichannelConnection
=
Server Name Selected Client IP Server IP Client Interface Index Server Interface Index Client RSS Capable Client RDMA Capable
----------- -------- --------- --------- ---------------------- ---------------------- ------------------ -------------------
192.168.10.5 True 192.168.0.2 192.168.0.5 4 3 False False
So even tho the share is mapped with the IP of the fast connection, SMB does its own path discovery and is choosing the wrong connection.
Why tho?
- Eventually we land on the following command providing the critical clue:
Get-SmbMultichannelConnection -ServerName "192.168.10.5" | Format-List *
=
SmbInstance : Default
ClientInterfaceFriendlyName : Realtek 2.5G
ClientInterfaceIndex : 9
ClientIpAddress : 192.168.10.2
ClientLinkSpeed : 100000000
ClientRdmaCapable : False
ClientRSSCapable : False
CurrentChannels : 0
Failed : False
FailureCount : 0
MaxChannels : 1
Selected : True
ServerInterfaceIndex : 2
ServerIpAddress : 192.168.10.5
ServerLinkSpeed : 100000000
ServerName : 192.168.10.5
ServerRdmaCapable : False
ServerRSSCapable : True
PSComputerName :
If you count the 0’s you’ll notice that’s actually 100M. So technically SMB is doing the correct thing and choosing the higher bandwidth connection - only based on faulty data. But why is the data faulty?
Well, google’s dumb AI overview thing actually made itself useful for once. It worded the answer well, so here’s the relevant bit:
The underlying problem is that Windows relies on the remote server’s SMB implementation (such as Samba on TrueNAS/Unraid or Synology DSM) to negotiate and declare the interface specs.
The original source is https://linustechtips.com/topic/864034-problem-setting-up-samba-multichannel-connection-and-achieving-2gbit-trasnfers/
This gives us 2 possible solutions on either end:
1. Client side (windows)
Force SMB to use the faster link manually, regardless of what it thinks by using a constraint:
New-SmbMultichannelConstraint -ServerName "192.168.10.5" -InterfaceIndex 9 -Force
2. Server side (linux/samba)
Have the server correctly advertise the link speed.
/etc/samba/smb.conf
↓
interfaces = "eno1;capability=RSS" , "enp4s0;speed=2500000000,capability=RSS"
