Linux to Linux Samba performance issue

Networking
1

Issue with samba transfer performance. it is slower than expected. any ideas?

  • I get around 95 MB /s

  • Ubuntu server 20.04

    • hpe dl80 gen9, 2x e5-2640v3 cpus, 64 gb ram, 2 x H240 HBA cards
    • ZFS pool
      • 3 mirror vdevs all 8tb seagate exos drives, 6 drives total in this pool
    • networking
      • hpe 560FLR-SFP+
      • connected to ubiquiti pro aggregation switch

  • Ubuntu desktop 20.04 client

    • workstation with nvme storage
    • networking
      • aquantia 10 base-t networking card
      • connects to ubituiti pro aggregation switch using a sfp to 10 base-t converter
        • runs at 5g due to the cable length from the switch but has no packet loss at 5g

  • iperf3 from client to server

    • [ 5] 0.00-10.00 sec 5.36 GBytes 4.61 Gbits/sec 10 sender
    • [ 5] 0.00-10.00 sec 5.36 GBytes 4.60 Gbits/sec receiver

  • /etc/samba/smb.conf

log level = 0
#socket options = TCP_NODELAY
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=65536 SO_SNDBUF=65536
#socket options = TCP_NODELAY SO_RCVBUF=131072 SO_SNDBUF=131072 SO_KEEPALIVE IP>
#socket options = TCP_NODELAY SO_RCVBUF=131072 SO_SNDBUF=131072 IPTOS_LOWDELAY
read raw = yes
write raw = yes
min receivefile size = 16384
#use sendfile = true
aio read size = 0
aio write size = 0
server signing = no
#strict locking = no
#oplocks = yes
#max xmit = 65535
#dead time = 15
#getwd cache = yes
#use sendfile = true
#client min protocol = SMB2
server multi channel support = yes
2

I’d try bigger socket buffer sizes (on both sides, client and/or server), aim for 1MiB or 16MiB if you can, you may need to tweak some sysctl maybe for OS to allow samba to do these larger buffers.

Stuff like these.

You can also try making a tmpfs share, just for testing (on the client you can maybe keep making a file, writing out 1GB of data, and deleting it, in a loop).

3

I may be wrong, but aren’t HBAs a no-no for ZFS?

4
  • What kind of Files are you transfering, e.g. Type and Size?
  • Can you give exact Read and Write Numbers, if they differ?
  • How are you measuring the Speed, just using the File Manager?

I’d mainly recommend you go ahead and test the Raw Pool Performance on the Server first so you can rule out Samba and possible Networking Bottlenecks.

The Adapter Cards from HP, Dell and the Likes can operate either in HBA or RAID Mode.

RAID Mode = Hardware Raid, Bad for ZFS
HBA Mode = Drives get passed through to the OS, Good for ZFS

IT Mode, which is often being referred to, is just a special Firmware from LSI without any Raid Functionaly, therefore the same as HBA Mode.

1 Like
5

Its is a good thought, these are HBA cards and do not do raid on their own. The P series controllers from HP are raid cards such as the P440 or P840. They support 12g/s sas or sata and are fully supported by the server. The ILO (IMPI) is picky about what cards are in the server. In the Dl80 gen9 the front bays are 3 different back planes each handling 4 drives. I wen with 2 controllers rather than a sata / sas expanded so each drive has full bandwidth. All the drives are presented individually to the OS. I do appreciate the response if you have any more ideas, im all ears.

6

I was getting 95MB/s transferring a folder of files which were between 2 and 8 gb. The server was reading and the client was writing. I really like the suggestion of testing the pool performance. I did a fair bit of testing at the pool level before setting on this system’s config. I was using Bonnie++ to benchmark a few different configs. Below is the bonnie++ command and result. I set the file size to 128gb as the server has 64gb of ram.

Sudo bonnie++ -u root -s 128552 -d /media/v -f -b -n 1

Using uid:0, gid:0.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
ieatbytes   128552M            493m  97  219m  59            454m  70 497.8  46
Latency                        7085us     632ms               200ms     158ms
Version  1.98       ------Sequential Create------ --------Random Create--------
ieatbytes           -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                  1     0   2 +++++ +++ 724249376   2     0   2 +++++ +++ 724249376   2
Latency             58922us      23us   82673us   58417us      20us   55180us
1.98,1.98,ieatbytes,1,1651829870,128552M,,8192,5,,,504915,97,224053,59,,,464731,70,497.8,46,1,,,,,110,2,+++++,+++,108,2,110,2,+++++,+++,114,2,,7085us,632ms,,200ms,158ms,58922us,23us,82673us,58417us,20us,55180us

I interoperate these results to not be the bottleneck in the transfer. I will play with the send receive buffers. i know how to do that on the server side but need to get a bit smarted on the client side. i will post more when i figure that out. as always thanks for the help. Anyone have any other tests, ideas, or disagree?

7

Some things to consider here.

  • Is it really samba? Is the disk performance better from the server OS?
  • Why is server signing disabled? Manual suggests that signing is required for SMB2 or later. You might be forcefully downgrading the connection to SMB1 mode. While transferring a file try running sudo smbstatus on the server to check details about the connection.
8

I believe that was a holdover from testing, i agree it should be on. I just reran it with signing on and off in both cases the protocol under smbstatus was SMB3_11. also i forgot to mention earlier that the transfer numbers all match from nautilus status, and network activity on the server monitored with nmon. Thanks for the thought

9

You are correct about that, seems like you should be getting multiples of 95MB/s.

Do you have “noatime” set on your Pool to turn off access time?
For those drives you should also make sure that “ashift=12” for sector size.

After that, I’d recommend trying out NFS since you’re both on a Linux Server and Client.
Use nfsstat -s for the server and nfsstat -c for the client to monitor performance and possible bottlenecks.

10

ashift is set to 12

name: 'vpool'
    state: 0
    txg: 122291
    pool_guid: 2799819019058270511
    errata: 0
    hostid: 2759428791
    hostname: 'ieatbytes'
    com.delphix:has_per_vdev_zaps
    vdev_children: 3
    vdev_tree:
        type: 'root'
        id: 0
        guid: 2799819019058270511
        create_txg: 4
        children[0]:
            type: 'mirror'
            id: 0
            guid: 7841265841370025588
            metaslab_array: 256
            metaslab_shift: 34
            ashift: 12
            asize: 8001548713984
            is_log: 0
            create_txg: 4
            com.delphix:vdev_zap_top: 129

atime was set to on. I turned it off and saw a small speed increase.

NFS testing is interesting. i had to hack some permission on a specific folder and setup a share to it. i copied a single large file and got the same performance numbers. i am now suspecting something on the network side of things i would expect NFS to be faster. the issue is the iperf3 numbers look fine. i just find it strange that samba and nfs are both bottle-necking at the same speed. I also tried a samba copy to a different machine in the same rack. it is an esxi host but uses 10k sas drives in a hardware raid 10. It managed to sustain 140 MB/s. its not a fair test for the server as i believe the vm storage was the limiting factor. i will poke around on the network. i dont see packet loss but i will look again.

11

Now we are getting some where. I changed out the sfp+ to 10base-T module and my speed jumped up. samba is now at 225 MB/s for the test file and NFS ran at 250 MB/s. This seems more reasonable for performance.

suspected converter = Wiitek SFP-10G-T 30m
- auto negoated down to 5gb/s

new converter = Fibay 10G-SFP-T80-AR
- works properly at 10g

I double checked in unifi and at the client and didnt see any packet loss. Unifi saw it connected at 10g while the client saw it at 5g. I am not sure Ubiquiti supports these converters auto negotiating down. has any has issues with these converters when operating at 5gb?

1 Like
12

No wonder your’e slow…10 base-t is 10 Mb Ethernet. (If your’e lucky you’re too young to know that was ever a thing. :grinning: )

Perhaps you meant the connector? In which case it’s RJ45. Although I vaguely recall seeing a video that said what is commonly called RJ45 really isn’t. :man_shrugging:

1 Like
13

I expect he just meant 10GBase-T or 10GBe on copper.

Yes, the connector is technically an 8P8C, as in 8 positions and all 8 conductors used. RJ (registered jack) standards are for equipment connected to the telephone network. Just an extra pair more than the common 6P4C / RJ11 analog telephone cords you’re familiar with. Ethernet really isn’t part of the RJ45 standard. However, calling it an RJ45 connector, even if it won’t be used on the telephone network, isn’t far off the mark.

1 Like
14

Doesn’t the RJ45 connector just have an extra ridge along it’s side? like a normal telephone jack, but thicc-er.

And most of the stuff sold as rj45, is more like 8p8c anyway?

15

RJ45 is quite similar to RJ11, but there are a few small differences. Deeper, wider, ridged, etc:

750Ă—950 95.9 KB

1 Like