I have 2 computers directly connected to each other using 10gig SFP+ and I wanted to test that I could actually achieve 10 Gigabit speeds. However, I have the feeling that my testing is flawed as I keep running out of CPU after achieving just over 4 Gbps as shown below:
To perform the test, I am running 4 of the command below in parallel from the other server (to use all 4 threads)
rcp 10GB.img nas.programster.org:/dev/null
I created the file with the following command:
dd if=/dev/zero of=10GB.img bs=1M count=10000
I have already checked that both servers are running the link at a 9000 MTU which I thought would have helped.
Is there a way to transfer data over the link whilst requiring less CPU (the reason Iām using rcp instead of scp), or is my processor just not powerful enough being a Pentium G4560t which is actually dual core hyper-threaded? I had thought that being fairly new and at 2.9 Ghz, it would be pretty efficient at just handling network traffic.
Excuse my ignorance, but you are sure the test file can be read faster than 4Gb/s too? Just in case bottleneck is on the sending side?
[Edit: like the img file is held on a RAM disk I mean, rather than just in /home]
[Edit 2: dropped speed by almost an order of magnitude]
Always a good thing to look at. In this case, itās /dev/zero, so Iād presume it can be read pretty dern quickly.
Other things that can drive up the CPU load with high network load are encryption (not sure if rcp uses it), and sheer packet size, depending on the NIC. I would presume that a proper 10Gb NIC would handle creating the packets, but you never know.
Iād say run with iperf for right now, that will test to make sure that you can achieve line speed. Then you can do file transfers to find out how much your disk is bottlenecking your wonderful interface.
I used rcp instead of scp on the understanding that it doesnt use encryption.
I am not reading from /dev/zero but from a file that was generated from reading from there to make in the first place. However, I donāt think reading from that file is causing a limiting factor (yet) since the htop output is only showing green bars. If it was waiting on disk, Iām pretty sure it would be showing grey bars as I already switched on the āDetailed CPU timeā option.
Iāll have a whack with iperf and see what that outputs. Either way though, 4gbps is enough to saturate my ZFS RAID10 array of 6 drives so this is purely a benchmarking excercise than any actual need.
I only asked about the file because it looked to me like /dev/zero went to 10GB.img file, then the file 10GB was rcp ād to remote hostās /dev/zero.
Wasnāt sure if I mis-read is allā¦
No youāre absolutely right and reading from the file could legitimately have been a bottlneck. However Iām reading from a pretty fast SSD and my note about grey-bars/htop makes me think its not the read speed being a bottlneck just yet. However beyond 5 or 6 gigabit it probably would be.
Just being nosy, but what distro/OS are you using locally and Nas?
Looks like you got the networking down almost square (bar tweaking speeds,) and thereās growing interest in 10gbe if you felt like a blog/post on how you set up?
If iperf confirms that youāre only getting 4Gb/s, then you might need to do some tuning.
45Drives has an article here which shows some kernel tweaks in FreeBSD. I have no idea if/how those would translate to Linux, but itās a place to start.
I would also check and see if a firmware update is available for your card, and see if thereās a newer driver available from the manufacturerās site.
Are you using officially supported transceiver modules? Iāve never seen a module affect performance (either works or it doesnāt in my experience), but you could try swapping them out if nothing else works.
My nas build is pretty much like my NAS blog post except that I have bought a NIC from ebay based on advice I got from this forum. Funnily enough LTT made a video about doing it this way a few days/weeks later as it is the most cost effective way to get 10Gbe.
Distro: Ubuntu 16.04 because Iām lazy and really easy to set up with ZFS.
SFP+ is a great way to get 10Gbe as I believe its optical rather than copper based. However this does mean that its hard/impossible to wire up a building gracefully with it (canāt just cut it to length like Cat 5/6/7). I just have a lose wire hanging between my KVM and my NAS servers and I use a separate 1 gigabit lan for internet access.
Arenāt the cables for SFP+ a lot thinner than Ethernet? Just loop the excess cable (not too tightly) and Velcro/ tape out the way?
Unless itās like DAC runs, which seem a lot thicker
By DAC, I mean the cable with built in modules either end of the cable (which could be copper or fibre) and a.m. ready to be corrected.
Also thought fibre cables are a lot lighter too, for tucking away
Good to know. I doubt I would feel comfortable enough to do this myself, but happy to pay someone else to wire up my flat one day if they know how. Getting a 10 gig connection between my office and my server would be a huge bonus.
With regards to modules, Iām a bit lost. I am guessing there are ZFS modules? I just installed it using the zfsutils-linux package.
@Trooper_ish, yes the cables are quite thin and light and I believe it requires a lot less power which are all huge bonuses. I have a 3 meter cable going between two boxes that are 1 foot apart. I just have to get whatever length is cheap and available on ebay.
Transceiver modules are the things at the end of the fiber optic cable. They look like this:
Historically, some SFP+ cards were extremely picky about having only officially supported modules. This is less of an issue lately, but it can still crop up. I donāt think the ConnectX-2ās are especially picky, and if they were, you wouldnāt get any connection at all. But if you want to eliminate all variables, make sure you use modules officially supported by the card.
