SMB traffic "stuck" at ~200MB/s no matter what I change. Q_Q

Software & Operating Systems Linux
, ,
1

OK, so I am SUPER confused, I have a “ubuntu” 20.04 system that I can not for the life of me get to do SMB transfers over ~200MB/s no matter what I change and I am starting to think I am taking crazy pills or sshed into /dev/null.

Test Setup

Windows 10 VM on latest VMware Tools, VMXNET3 NIC

TrueNAS box with a connectx-4 card, both 25G ports are in a LAGG/port-group/VPC whatever you wanna call it. 4 vdev ZFS pool

“Ubuntu 20.04” box also has a connectx-4 card with both 25G ports are in a LAGG/port-group/VPC. 3 vdev ZFS pool

end-to-end latency with a TCP ping is ~0.63ms for TrueNAS

end-to-end latency with a TCP ping is ~0.54ms for “Ubuntu”

TrueNAS gets speeds I would expect with a single thread/stream.

image

“Ubuntu” gets way less.

image

I have tried every samba tuneing option I have seen get thrown around, most of them are old and seem to be compiled into the defaults now.

I have tried SMB Multichannel, all it does is spread the load out across the NIC’s, not make it any faster, rss/no rss, same thing, what am I missing?

2

Have you tested via mounting a ramdisk to make sure it’s SMB and not the drives/zpool?

1 Like
3

iperf3 running at 25Gbps? What does your per core CPU load look like?

2 Likes
4

are driver modules loaded? i have a 2.5gbit nic and thats faster as your setup, but im not running any vms. but ive to install the realtek modules last time i checked this.

what does ethtool say about the speed?

sudo ethtool eth0/enp*s*
5

Oh, yeah, HA node to HA node I can get ~1.7GB/s over SMB and ~2.2GB/s over NFS. This is specifically something screwy from Windows VM > NSX > BGP > Array.

It also screws with me that Linux VMs get ~450MB/s on the same host/network path.

6

iperf3 I get ~22-23Gbps from ESXi host to the array on the same L2 network, but from the Windows VM > the array I get ~6.5Gbps. Percore is pretty low, best guess is something screwy in TCP window sizing + routing latency (~0.3-0.5ms), but it doesn’t make sense that enabling SMB MultiChannel doesn’t change the throughput at all, it just spreads the same throughput out more evenly.

7

Here go.

root@xxx-xxx-slot0:/home/xxx# ethtool enp176s0f0np0
Settings for enp176s0f0np0:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseKX/Full
                                10000baseKR/Full
                                25000baseCR/Full
                                25000baseKR/Full
                                25000baseSR/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: None BaseR RS
        Advertised link modes:  1000baseKX/Full
                                10000baseKR/Full
                                25000baseCR/Full
                                25000baseKR/Full
                                25000baseSR/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: BaseR
        Speed: 25000Mb/s
        Duplex: Full
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: g
        Wake-on: d
        Current message level: 0x00000004 (4)
                               link
        Link detected: yes

root@xxx-xxx-slot0:/home/xxx# ethtool enp176s0f1np1
Settings for enp176s0f1np1:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseKX/Full
                                10000baseKR/Full
                                25000baseCR/Full
                                25000baseKR/Full
                                25000baseSR/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: None BaseR RS
        Advertised link modes:  1000baseKX/Full
                                10000baseKR/Full
                                25000baseCR/Full
                                25000baseKR/Full
                                25000baseSR/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: BaseR
        Speed: 25000Mb/s
        Duplex: Full
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: g
        Wake-on: d
        Current message level: 0x00000004 (4)
                               link
        Link detected: yes

root@xxx-xxx-slot0:/home/xxx# ethtool -i enp176s0f0np0
driver: mlx5_core
version: 5.15.0-48-generic
firmware-version: 14.31.1014 (SM_2111000001034)
expansion-rom-version:
bus-info: 0000:b0:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

root@xxx-xxx-slot0:/home/xxx# modinfo mlx5_core
filename:       /lib/modules/5.15.0-48-generic/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko
license:        Dual BSD/GPL
description:    Mellanox 5th generation network adapters (ConnectX series) core driver
author:         Eli Cohen <[email protected]>
srcversion:     08FE399DACB3DEDCF430A9B
alias:          auxiliary:mlx5_core.eth
alias:          pci:v000015B3d0000A2DFsv*sd*bc*sc*i*
alias:          pci:v000015B3d0000A2DCsv*sd*bc*sc*i*
alias:          pci:v000015B3d0000A2D6sv*sd*bc*sc*i*
alias:          pci:v000015B3d0000A2D3sv*sd*bc*sc*i*
alias:          pci:v000015B3d0000A2D2sv*sd*bc*sc*i*
alias:          pci:v000015B3d00001023sv*sd*bc*sc*i*
alias:          pci:v000015B3d00001021sv*sd*bc*sc*i*
alias:          pci:v000015B3d0000101Fsv*sd*bc*sc*i*
alias:          pci:v000015B3d0000101Esv*sd*bc*sc*i*
alias:          pci:v000015B3d0000101Dsv*sd*bc*sc*i*
alias:          pci:v000015B3d0000101Csv*sd*bc*sc*i*
alias:          pci:v000015B3d0000101Bsv*sd*bc*sc*i*
alias:          pci:v000015B3d0000101Asv*sd*bc*sc*i*
alias:          pci:v000015B3d00001019sv*sd*bc*sc*i*
alias:          pci:v000015B3d00001018sv*sd*bc*sc*i*
alias:          pci:v000015B3d00001017sv*sd*bc*sc*i*
alias:          pci:v000015B3d00001016sv*sd*bc*sc*i*
alias:          pci:v000015B3d00001015sv*sd*bc*sc*i*
alias:          pci:v000015B3d00001014sv*sd*bc*sc*i*
alias:          pci:v000015B3d00001013sv*sd*bc*sc*i*
alias:          pci:v000015B3d00001012sv*sd*bc*sc*i*
alias:          pci:v000015B3d00001011sv*sd*bc*sc*i*
alias:          auxiliary:mlx5_core.eth-rep
alias:          auxiliary:mlx5_core.sf
depends:        tls,pci-hyperv-intf,mlxfw,psample
retpoline:      Y
intree:         Y
name:           mlx5_core
vermagic:       5.15.0-48-generic SMP mod_unload modversions
sig_id:         PKCS#7
signer:         Build time autogenerated kernel key
sig_key:        71:BE:D6:27:24:EE:C7:44:45:7A:53:20:37:5C:FD:A2:10:6B:68:79
sig_hashalgo:   sha512
signature:      42:97:55:E5:6A:CB:02:06:10:43:57:71:5A:D3:F4:C4:89:9E:A5:B0:
                B8:AD:F5:98:DC:97:FE:D3:15:BF:80:5D:BF:7F:09:BF:E6:46:C7:9C:
                4F:08:8C:AB:AF:5B:5C:10:C6:56:A4:80:74:C0:3D:74:3B:14:83:E0:
                D5:6E:26:18:90:99:5D:93:1D:BC:7B:E6:39:22:9A:A6:49:2C:06:2B:
                53:AB:D3:EB:74:08:1A:9A:42:0F:AC:E8:EC:2A:DD:10:C3:2D:AA:0C:
                C4:62:79:AA:C3:4A:59:E3:3B:70:E2:72:CB:AD:C8:1D:DD:C7:FB:44:
                72:D5:77:62:3D:5C:6C:07:36:3B:9A:48:FC:B7:35:D0:36:67:39:8E:
                13:A7:2C:7B:5A:E9:05:16:80:5E:9A:27:D0:9E:C0:E4:8F:D8:0A:EB:
                49:46:B4:BD:C6:CD:BE:0A:E0:05:15:C5:FD:01:83:90:64:94:53:84:
                B3:B2:1B:C0:E5:79:44:CE:19:EC:6C:C2:F1:91:7E:58:04:BE:1D:18:
                29:A1:14:6E:CB:49:4B:06:83:DE:3B:CC:26:B1:A5:D9:F1:F5:2E:0F:
                1B:BE:FA:8C:63:E8:ED:DB:F8:58:1C:3B:B7:B4:95:6B:01:E7:BE:5F:
                0D:18:9F:53:77:A5:0C:98:CE:45:0D:D3:16:FD:28:04:22:41:F3:EE:
                3C:F9:F5:69:AC:40:80:6B:E7:5A:49:6D:BC:33:F9:58:38:0B:26:C3:
                86:9F:73:EA:C8:76:74:CA:0C:8D:39:DF:03:D0:46:60:3D:C6:04:77:
                FF:C6:03:E4:29:17:8C:F1:10:1E:E1:65:FE:C9:A2:E1:F6:2A:F2:AF:
                CA:EA:67:57:00:F7:6E:10:5F:DB:DB:DF:00:C6:4A:BE:61:FC:B3:38:
                25:02:1E:08:06:E8:B5:83:AD:F0:08:F2:76:9A:93:84:F7:A5:0C:8C:
                9F:48:88:16:5F:2B:B6:8B:C3:35:4E:4C:DC:B2:72:90:6D:3A:74:F8:
                12:2E:82:06:E7:A2:F7:D8:C3:9F:D3:B1:02:35:C2:FF:05:BD:3A:D1:
                13:63:3D:05:E4:E1:32:BC:A5:8B:F9:37:AB:E2:D6:E6:07:DF:83:C3:
                AC:64:CB:02:E5:42:32:B0:F5:3E:30:35:78:62:0A:2B:72:F3:BB:2E:
                54:6A:29:6A:E0:6E:42:AE:C7:20:5C:90:E1:13:68:E1:FC:CA:89:60:
                9F:D9:E1:FD:BE:33:5E:2B:D9:8C:E7:14:6F:DE:E1:74:02:82:51:3E:
                86:9E:A9:D4:5C:66:54:6A:C8:F9:F6:6F:1A:57:6C:9A:36:9E:1C:7E:
                2C:A7:73:6A:5A:10:22:14:E3:AF:14:8B
parm:           debug_mask:debug mask: 1 = dump cmd data, 2 = dump cmd exec time, 3 = both. Default=0 (uint)
parm:           prof_sel:profile selector. Valid range 0 - 2 (uint)
8

MTU set to 9000 on the NICs?

9

Nooooooooo, unnecessary, not worth the headache.

10

What’s a little packet fragmentation between friends?

11

lol

12

Well, that is annoying.

TrueNAS

image
image2581×1406 169 KB

Ubuntu

image
image2550×1406 170 KB

13

There were some windows updates lately that crippled SMB traffic on the Windows 10 side.

Maybe google that and confirm which ones you do/don’t have installed

14

old testing from ~ February.

Ubuntu > Ubuntu :NFS: ~2200MB/s
Ubuntu > Ubuntu :SMB: ~1300MB/s
Ubuntu > WinTestVm :SMB: ~722MB/s
WinTestVM > Ubuntu :SMB: ~216MB/s
WinTestVM > Isilon :SMB: 400-600MB/s

New neat findings are below, highlights are shown below.

OricalLinux8 to Ubuntu

lat (msec) : 50=0.18%, 100=7.42%, 250=91.73%, 500=0.67%
WRITE: bw=162MiB/s (170MB/s), 162MiB/s-162MiB/s (170MB/s-170MB/s), io=9742MiB (10.2GB), run=60030-60030msec

OricalLinux8 to FreeBSD

lat (msec) : 10=0.04%, 20=0.07%, 50=4.52%, 100=54.64%, 250=40.74%
WRITE: bw=275MiB/s (289MB/s), 275MiB/s-275MiB/s (289MB/s-289MB/s), io=16.1GiB (17.3GB), run=60018-60018msec

Ubuntu-s1 to Ubuntu-s0

lat (msec) : 50=0.04%, 100=22.90%, 250=77.06%
WRITE: bw=204MiB/s (214MB/s), 204MiB/s-204MiB/s (214MB/s-214MB/s), io=11.0GiB (12.9GB), run=60125-60125msec

Ubuntu-s1 to Ubuntu-s0 NFS

lat (msec) : 20=0.05%, 50=38.77%, 100=58.51%, 250=2.67%
WRITE: bw=615MiB/s (645MB/s), 615MiB/s-615MiB/s (645MB/s-645MB/s), io=36.1GiB (38.7GB), run=60046-60046msec

Ubuntu-s1 to Ubuntu-s0 NFS sync-disabled

lat (msec) : 4=0.01%, 10=1.79%, 20=92.53%, 50=4.85%, 100=0.61%
lat (msec) : 250=0.13%, 500=0.07%
WRITE: bw=2165MiB/s (2270MB/s), 2165MiB/s-2165MiB/s (2270MB/s-2270MB/s), io=127GiB (136GB), run=60013-60013msec

OricalLinux8 to Ubuntu NFS

lat (msec) : 20=0.07%, 50=4.50%, 100=92.08%, 250=3.34%
WRITE: bw=477MiB/s (500MB/s), 477MiB/s-477MiB/s (500MB/s-500MB/s), io=27.0GiB (30.1GB), run=60143-60143msec

lab win to Ubuntu

lat (msec) : 10=0.01%, 20=1.13%, 50=15.12%, 100=60.23%, 250=23.09%
lat (msec) : 500=0.42%
WRITE: bw=294MiB/s (308MB/s), 294MiB/s-294MiB/s (308MB/s-308MB/s), io=17.3GiB (18.5GB), run=60135-60135msec

lab win to FreeBSD

lat (msec) : 2=0.01%, 4=11.72%, 10=78.19%, 20=5.61%, 50=1.50%
lat (msec) : 100=1.85%, 250=0.18%, 500=0.11%, 750=0.23%, 1000=0.11%
lat (msec) : 2000=0.32%, >=2000=0.18%
WRITE: bw=1438MiB/s (1508MB/s), 1438MiB/s-1438MiB/s (1508MB/s-1508MB/s), io=84.3GiB (90.5GB), run=60018-60018msec

1 Like
15

Ubuntu Flame Graph

image
image2998×857 260 KB

FreeBSD Flame Graph

image
image3007×1226 134 KB

I would upload the .svg but @wendell says that is not an approved file type.