I am trying to upgrade to 10 Gb network but am getting completely lost. While an iperf3 test shows that I should be able to achieve up to 10Gb (e.g., https://fasterdata.es.net/performance-testing/network-troubleshooting-tools/iperf/disk-testing-using-iperf/), I am not able to go beyond 1Gb when copying a 7Gb Centos image. I setup a pair of machines running PopOS 18.04 and I am copying a CentOS image file via samba. One host has an nvme and the other has an SSD. One host has 64 GB ESS RAM and the other has 32 GB ECC RAM. The router is very old and probably needs to be replaced so I am open to suggestions for that.
Do I need to configure samba so it is bound to the 10Gb interfaces even though I defined a route from HOST1 to HOST2 and vice-versa and specified that traffic should go via the 10Gb interface
Did I need to define the above routes in the first place
Given that I am using a 10 Gb un-managed switch and a router that is only 1Gb capable, is there a way for me to make sure that the traffic never really reaches the router and just goes through the switch? Or is this just part of TCP? This is really, really where I am confused.
Not sure what else to include here in as far as configuration so sorry about that. Any help would be great.
Hi. Your setup is scarily similar to mine and I dont have any issues hitting multi-hundred MBps on file copies, so there is no logical reason why your setup shouldnât work.
Couple of things to check - CAT6 cables - have you tried swapping these out?
Smaller files - does it have this behaviour with a smaller file?
Hi @Airstripone, I am using CAT6A and CAT7 cable. Donât have any extra ones though. Which cables are you using? Have not tried smaller files. How small are we talking about? Did you end up binding samba to the 10gb interface?
My runs are Cat6 to the two 10Gig ports. It could be you have a faulty cable and it is falling back to a lower input speed when copying large files, hence the suggestion to test another cable.
Unless I have misunderstood your setup I believe you only have one active connection to each pc? If so there should be no reason to define special routes, the packet switching should simply route the data across the 10Gbps connections.
Could you run a network interface monitoring app (like gnome system monitor) whilst you copy the files (multiple, say 20GB mixed load of large and small) and take a screenshot of the load characteristics. Or run one in terminal and graph afterwards? If the load is not stable, it will likely be a cable issue or your NIC is faulty.
Finally have you tried a different data protocol, say FTP?
As long as what is connected to the switch are all on the same subnet, it shouldnât hit the router
For tuning. Iâd just load your appropriate drivers, set your interface IPs, and enable Jumbo frames for a start and then do tuning from there. The 10g network and 1g network should be on different subnets and the 10g switch doesnât really need to be connected to the internet, just to prevent access of the file share from reaching the internet. Something like this (sorry for transparency)
You shouldnât need to do anything special, it should âjust workâ
My gut feel is one of your cables is damaged. I had that with a run that worked fine at first but after a day or so built up some weird impedance and dropped to 1Gig or even 100Mb if it was feeling cruel.
Given you have ordered new cables try one of those. Make sure there are no coils in your wires and do a mixed load test⌠It may even be the file you are copying as test is funky.
Note even with âmax speedâ I only get 350MBps sustained from my thinkserver to my prod machine due to disk speed limits⌠Donât expect 10Gig in real world without NVMe end to end. But your problem sounds more like protocol falling back to 1Gig. Annoying when you have spent $$$ on 10Gig kit I know!
I canât wait for the cable to arrive now. I will definitely go through this one more time. So, iperf3 is not sufficient for diagnostics? I am looking at other tools https://www.tecmint.com/linux-network-bandwidth-monitoring-tools/ but they seem like the same thing or overkill with full fledged monitoring. Will have to check for the gnome app on the PopOS system as I do not see it on this Fedora 30 Xfce which I tend to run in lieu of gnome.
Yeah I wouldnât over engineer it. I just use gnome system monitor because it is in the repo and I know it works. You can try conky as well but it never liked my threadripper system for some reason.
Basically you are looking for a visualisation of your data transfer. If it is jumpy, like a saw tooth, then you have something causing either packet drops or a bottleneck (maybe the CPU on the server or a misconfig of jumbo frames). If it is stable but capped the issue is likely the network. If it starts high then drops to a sustained low rate then it is the disks.
Unfortunately the only way to be sure is to test various combinations. One thing to check is that you are using SMBv3 and it isnât falling back to legacy signalling. This can cause drama.
If it is a recent purchase consider returning it if you still can. The EMX version is only a few $ more and has a handy console. Again shouldnât impact the actual switching, but may give you more diagnostics.
I apologize if this is too rudimentary, but Iâve found it solves many problems. The difference between Gb and GB (bits and bytes) is a factor of 8. Your network card is 10Gb (1.2GB), but many programs like FTP and SMB report speed in bytes. if you are able to achieve 1GBps, that is very close to the max speed, I would be happy with this result.
If you are achieving 9+Gbps with iperf(3), but SMB is falling short, I would say your network card and driver are fine, itâs likely other systems, like hard drive access, etc, that are limiting your performance.
but copying the file starts at around 1.2GiB and drops to 120 and sometimes I see a dips below that. But generally it stays around 120 MiB. This is going from a Samsung 970 Pro NVMe to an Intel or Samsung SSD.
How large is the file. If the file is large enough, itâll fill up the target SSDâs dram buffer/caches and be stuck waiting until thereâs cells free to dump to the cells. Iâve noticed this when copying files over 4GB generally
Can you create a ramdisk? That behaviour sounds like the disk controller is throttling, maybe too full to use the non-tlc cache. Does it do the same going the other way?
Really dumb question, where are the SSDs plugged in? They arenât on Sata1 ports? That would cap out at 150MBps, approx 130MiB after overhead losses.
And yes SMB should autonegotiate but Ubuntu is ⌠Well Ubuntu. Try a proper distro. (Triggers many people )
Itâs not actually hitting the controller that hard, itâs just emulated load.
Try the ramdisk option and we can eliminate the network. Hopefully the cables werenât too expensive
it is unmanaged. I wanted easier lol. Learning a lesson here.
)