Windows - Slow SMB while Iperf is ok.

Hello there !

First post here, bug long time lurker. I apologize in advance, my english is not very good, but i’m trying :o

So, I’m sorry to bother you, but I’m hitting a wall in my homelab, between 2 W11 enterprise computer.

The relevant specs are :
First pc : 10920x - Intel x520 - I’m doing my test in ram but It’s all ssd and nvme
Second pc : small i3 8010- Intel x710 - same thing, tests are done in ram but all nvme drives.

First PC havec a single 10gb link to the network, the second have 2*10g. No link aggregation, just smb multichannel doing his things when needed.

First PC <=10g=> Ubuiquity 16-XG <=10g=> Mikrotik CRS 309 <=2*10g=> second pc

Now, when Iperf is running on the i3-8100, I have 10g in both direction, even with a single connection, no problem. BUT, when I try to copy some test file (between 4 and 8g) between the pcs, It’s very irregular and between 100 and 300mb/sec. It doesn’t matter which pc is initializing the transfert. The smb protocol is smb 3.1.1… Funny thing is, I have a nas on the CRS 309, and I can get 10gb from and to all the computers, with smb transfers. So the problem seems only when a windows is talking to another windows, only in smb (iperf3 runs great).

I tried reseting the tcp stack with netsh on both computers, differents drivers, differents settings, same stuff. Funny thing (or not) If I up/down a card, then I can get full speed again, for a few seconds to few minutes, then it’s hell again. I don’t have packet loss or anything… I played with autotuning, rss, flow control, buffer size, etc, nothing seems to really make a difference. i disabled smb encryption and compression, no change. CPU, even the small i3, I maxed during this (not even one single core).

One other thing, Crystal Mark 8 between the two is running great… I’m even more lost thing the share is using smb too.

I really don’t know what to try anymore for now…

I’ll test with 2 Mellanox X4 LX in a few days, but for now, i’m lost :o

Any help would be appreciated :slight_smile:

Thx a lot :slight_smile:

You need to check :

make sure MTU is 9014 on both sides

make sure RSS (receive side scaling) is enabled on both NICs

make sure SMB Multichannel is used and verify that it is enabled via Get-SmbMultichannelConnection and Get-SmbConnection

1 Like

Thx you.

Already checked that, and it’s ok.

robocopy with /mt flag set to 4 and see what happens

standard drag and drop is single threaded

1 Like

Standard drag and drop can easily saturate 10gbps line

image

I would forget about robocopy

Same problem :confused:

1 Like

Was iperf run in TCP or UDP mode? Makes a huge difference since SMB uses TCP.

Could be TCP window or SMB request size limiting transfer speed.
Willing to share a short (10 second) packet capture to see if either are the cause of the slowdown?

1 Like

iperf is run in tcp mode.

I’ll try to provide a relevant wireshark capture tomorrow or the day after, thx (I already took one few days ago, but it was way to “messy” to find something useful imo).

How do you tune SMB request size in Samba ?

TCP window is automatically set based on MTU right ?

I can’t add attachement, but here is a link to a 5 seconds wireshark captures. Speed is tanked from the start, so …

(I can’t post link either since I’ve a new account)

puthttpshere://web.rootax.org/temp/smbslow192168.zip

I copy a file, test.mkv from 192.168.1.52&51 to the localhost 192.168.1.40 & 41. Capture is from the last one.

Funny thing is, if I up/down one of the card on the .40/41 computer, It will be fine again, for a (seemingly) random amount of time. Then transferts are all over the place.

Thanks for the packet captures!

tl;dr

Receiver is waiting 200ms between requests.

Packet loss is causing some slowness.


Captures show:

  • Capture was started after TCP handshake. Best to start capture and then initiate TCP session (for SMB may need to unmount, wait for TCP socket to drop, then start capture, then mount the share, then copy file). Without the TCP handshake it is hard to know exact TCP window.
  • Capture is likely on the receiver/client and not the sender/server. Best to capture at the source/sender of the traffic (on the SMB server).
  • MTU of 9000 (MSS of 8960) is working fine. (This is good.) (This is often not relevant also.)
  • RTT of ~1ms (typical for a LAN) (This is good.)
  • Bytes-in-Flight (BiF)(TCP Window-ish) grows to a healthy 3MB.
  • Using BiF and RTT can google (I use google 'cuz I’ll inevitably get the math wrong): 3MB / 1ms = ? Gbps
    • This shows the receiver is saying to send at a max of 24Gbps (overkill?)
  • LOTS of packets show “SLE=” and “SRE=”. This is TCP SACK on the receiver saying “Hey, I got packets X but did not get packets before X, resend please”.
  • Client packets are duplicated. This is probably a capture glitch. Have seen this when capturing on Windows vSwitch where Windows captures “both sides” of the vSwitch. This makes the capture messy and harder to read but likely a non-issue.
  • SMB Read Length (packet 196) is 1MB (1048576). This could limit the transfer to 8Gbps if only 1 chunk was requested at a time. But sometimes multiple SMB read requests go out (packet 5853).
  • The client is often waiting 200ms from getting one 1MB chunk (packet 5747) to requesting the next 1MB chunk (packet 5853). Unsure why this is. Maybe busy writing to local disk? This is very likely a good portion of the “slowness”.
  • Wireshark tcptrace graph / Statistics / TCP Stream Graphs / tcptrace shows a lot of 200ms “flat spots”. (This is bad.)
    • This graph shows throughput (data over time). Steeper is faster/good. Flatter is slower/bad.
    • This also shows lots of TCP SACK (red bars) which is packet loss.

Could be that packet loss (could be on switch, or on receiving pc, or CRCs ie; bad cable, or pause frames which are usually bad in practice).

Looking back at your topology is the sender or receiver the one with 2x10Gbps?
If the sender/server has 2x10Gbps, then maybe it is sending packets out both and 2x10Gbps can’t seamlessly fit into 1x10Gbps which during a micro-burst could cause packet loss on switch port buffers.
Does disabling one of the 2x10Gbps show a difference?

I’m curious to see a sample of the “fast transfer” to compare.
Are same 200ms flat-spots seen?
Are 372 (tcp.analysis.retransmission) of 30347 packets retransmitted?
(1% packet loss may not seem like a lot, but often why ISPs have horrible service.) (I’m looking at your TWC.)

Hope that helps.

2 Likes

Thx a lot for your input ! I’ll provide more data as asked this week-end, when I can do more tests and captures :slight_smile:

What I can already answer is, I doubt this is pause frame since I disabled flow control everywhere I could. This capture is from a transfert from my 2x10gb nas to a 2x10gb computer (the small i3-8100), plugged in the same crs309.
I doubt it’s a storage problem on the receiving end, since I copy the file into a 12go virtual disk in ram. I tested without the ram drive, directly onto a nvme ssd, same behaviour.
Stats on the switch seems fine, and I doubt it’s a physical problem (bad cable &co) since everything is fine with iperf. I’ll try without any smb multichannel, to test one path at a time…

I’ll be back soon :o

Thx again !

Ok, so, maybe I’m onto something, it has been 24h without slowdowns. If I’m right, the problem was mostly do to problems with intel drivers/rss, and smb multichannel. With more and more test, I noticed that sometime the “balance” between the 2 nics was really messy, with big drops between links.

So, first problem was that get-netadapterrss outpout was not reflecting the values put in the drivers configuration. Mostly, the “NumberofReceiveQueues” was stuck to 1, with empty indirectional table. Long story short, I fixed the situation with the help of this post : puthttpshere://community.intel.com/t5/Ethernet-Products/I225-V-INF-aren-t-creating-the-necessary-registry-keys-to/td-p/1524552 , even if the situation was not exactly the same.
Still digging into the rss stuff, I noticed that the profil was not the same between my 2 cards. One was “closest”, the other NUMAstatic. I set both to NUMAStatic, reboot everything, aaaaaand, it seems ok right now…

For sure the problem will come back few minutes after posting this :o I’ll keep you posted.

1 Like

One receive queue can absolutely cause that packet loss “in the network” (really in the network stack within the endpoint as the NIC hardware tries to hand off to the OS, but the OS hands are full).
Nice find!

1 Like