This has been driving me nuts this week. My Terramaster F4-424 Max is 2x faster when reading from SMB share (Raid1 with NVMe cache) than TrueNAS Scale 24.10.2 Raid 1 NVMe dataset.
I get 1.15gb/s when coping from Terramaster SMB share and only get 470 mb/s from TrueNAS.
My setup is
VM TrueNAS Scale 24.10.2 in Proxmox (q35, VirtIO SCSI single)
8 cores of Epyc 7402
96GB RAM to TrueNAS
2 x NVMe PM983 PCIe passthrough
1 x Mellanox ConnectX 4-Lx 25gbpe PCIe passthrough
I have ZFS raid 1 mirror with 2 x PM983 exported as SMB Multichannel share
TrueNAS runs very modern Samba 4.20. It supports SMB multichannel and io_ring.
It’s perfectly capable to determine interface speed and RSS support, but I also tried explicit via cli
interfaces = "x.x.x.x;capabilities=RSS,speed=25....lots of zeros"
via service smb update smb_options=
with subsequent smbd restart
also tried (altho as I read, they are deprecated settings in modern Samba)
aio read size = 1 or 16 * 1024
use sendfile = yes
Raid1 NVMe is fast, fio for sequential reads reports this :
$ fio --name=fio_test --ioengine=libaio --iodepth=16 --direct=1 --thread --rw=write --size=100M --bs=4M --numjobs=1 --time_based --runtime=60
Run status group 0 (all jobs):
WRITE: bw=6334MiB/s (6642MB/s), 6334MiB/s-6334MiB/s (6642MB/s-6642MB/s), io=371GiB (399GB), run=60001-60001msec
Network - iperf3 - 10gbps easily
Accepted connection from 192.168.1.16, port 60898
[ 5] local 192.168.1.40 port 5201 connected to 192.168.1.16 port 60899
[ 8] local 192.168.1.40 port 5201 connected to 192.168.1.16 port 60900
[ 10] local 192.168.1.40 port 5201 connected to 192.168.1.16 port 60901
[ 12] local 192.168.1.40 port 5201 connected to 192.168.1.16 port 60902
[ 5] 0.00-1.00 sec 299 MBytes 2.51 Gbits/sec
[ 8] 0.00-1.00 sec 299 MBytes 2.51 Gbits/sec
[ 10] 0.00-1.00 sec 292 MBytes 2.45 Gbits/sec
[ 12] 0.00-1.00 sec 292 MBytes 2.45 Gbits/sec
[SUM] 0.00-1.00 sec 1.15 GBytes 9.91 Gbits/sec
Super basic setup as you can see, everything is fast and supposed to be fast for clients.
But it is not, the max I get copying files from SMB export share is 470mb/s :
Client - Windows 11 Pro Mellanox ConnectX 4-Lx
SMB multichannel - 100% enabled and used (I even wasted day looking at Wireshark negotiating packets)
PS C:\Windows\System32> Get-SmbConnection
ServerName ShareName UserName Credential Dialect NumOpens
---------- --------- -------- ---------- ------- --------
192.168.1.40 video_fast KORESH\admin KORESH\otec 3.1.1 2
PS C:\Windows\System32> Get-SmbMultichannelConnection
Server Name Selected Client IP Server IP Client Interface Index Server Interface Index Client RSS Capable Client RDMA Capable
----------- -------- --------- --------- ---------------------- ---------------------- ------------------ -------------------
192.168.1.40 True 192.168.1.16 192.168.1.40 6 2 True False
That’s it. I tried numerous sysctl changes - but no point as iperf3 shows perfect 10gbps speed.
I connect same Windows 11 client to Terramaster F4-424 Max (running 1x10gbps RJ45 Marvell AQtion) , I get immediate 1.15gb/s
Terramaster RAID1 X18 SATA disks + NVMe cache (so on paper its even worse)
It runs Samba 4.15 with same settings SMB multichannel settings.
I did try TrueNAS Scale version 22 and even nightly 25, no luck, same speed.
WHY TRUENAS IS SLOW ??