I’m trying to set up NFSoRDMA in Ubuntu 18.04 LTS using the rdma-core, mlx5_ib, etc. packages.
nfs-kernel-server has already been installed on the server and nfs-common has already been installed on the client.
I’ve confirmed that my Mellanox ConnectX-4 card is up and running via lsmod, lspci, ibv_devinfo, and ibstat.
ifconfig shows that an IPv4 address has been assigned and ping and rping works.
opensm is working (confirmed via ps aux | grep opensm).
When I try to mount:
# mount -v -o rdma,port=20049 server:/export/home/cluster /export/home/cluster
It says that the connection is refused.
I have no idea how to verify which version NFS comes with nfs-kernel-server and/or nfs-common. (e.g. whether it’s NFS v3, v4, or v4.1)
I also tried looking up the guides on how to update /usr/lib/systemd/scripts/nfs-utils_env.sh, to no avail.
Yes, I know that I can install the MLNX_OFED driver, but until relatively recently, Mellanox, in their infinite wisdom, decided to take out NFSoRDMA support from their v4.6 version of the drivers. (v4.7 has it back in now, but I want to set it up with the “inbox” drivers in case Mellanox decides to pull it from their drivers again).
Any help and/or suggestions would be greatly appreciated.
Fingers crossed, digging up a two year old thread … @alpha754293 did you get this working? I just received two x3-pro cards and want to try the same nfs over rdma.
If you are, I read the thread and they gave, once again, another less-than-intelligent answer.
(I had previously busted Mellanox for false advertising because they removed NFSoRDMA on their ConnectX-4 cards despite their advertising material saying that it supported NFSoRDMA from Mellanox’s own MLNX_OFED_LINUX drivers.)
So, I’m not sure if you’re in a position where your cluster can use that specific version of Ubuntu with that specific version of the OFED LTS driver, but it would appear, at least on the surface, and according to the documentation, that that might be one of your possible solution paths.
To answer your original question though, I’m checking through my OneNote and it does NOT look like that I ever got NFSoRDMA on Ubuntu 18.04.1 LTS to work. But then again, I was also using Mellanox’s 4.5-1.0.1.0 driver instead of 4.9, so…that might also have something to do with it as well.
There should be a package mlnx-nfsrdma that’s installed with that version of the driver, so maybe that might work for you.
Unfortunately, I run CentOS pretty much exclusively (back when I was learning about how to setting all of this stuff up) and because it worked for me then with CentOS 7.6 (and now it’s worked for me up through and including CentOS 7.9, but I still stick with CentOS 7.7 for compatibility reasons), I haven’t tried it with Ubuntu at all since then.
I MIGHT play around with it within the next month or so, but being that I also have a slightly newer card (ConnectX-4 vs. your ConnectX-3), so there is also a real possibility that what might work for me may not necessarily work for you due to this difference as well.
Let me know if you have any more questions and I will do my best to try and help.
Thanks.
edit
Note: It should be noted that I ended up NOT using (or NOT trying to use) Ubuntu’s “inbox” Infiniband driver and was, instead, using Mellanox’s drivers instead.
BUT, having said that, you MIGHT try to see if the Ubuntu “inbox” Infiniband drivers might work for you because I use CentOS’s “inbox” Infiniband driver because Mellanox, in their infinite wisdom, at the time when I was learning about this, decided to remove NFSoRDMA inexplicably from their driver (which like I said, I called them out on false advertising on their website and I was using v4.5 back then, and they put it back in somewhere around v4.7, but by then, I was so used to using CentOS’ “inbox” drivers, that I didn’t really need Mellanox’s drivers).
So, SOMETIMES, you might potentially have more luck with Ubuntu’s drivers (if they have it) than with Mellanox’s drivers. And this is one of the reasons why I don’t use Mellanox drivers because they will take away features or support for features without rhyme nor reason, so I’ve categorised them as someone who’s not trusthworthy (that they won’t take away the feature again in a future version of the driver).
Nope, not me on the Mellanox forum… I had x-posted this to Reddit, as well as STH forums.
I am using the drivers supplied with the Linux kernel, which perhaps is part of my problem but I’m not certain. Is that the “inbox” driver?
I have been able to get NFSoRDMA working between certain nodes on my network, using Mellanox hardware, as well as SoftiWarp and RXE (Soft RoCE) on generic Intel 10g hardware. However the results are not repeatable, I mean, I can’t take the exact same steps to connect “client1” to “server1” apply those same steps to connect “client2” to “server1”, despite all running identical hardware and OS/software.
I’m curious to try hardware iWarp but hesitant to buy the cards, modern ones still being rather expensive, and older ones being rather unsupported.
I’ve been working with Fedora 35 as the OS on all my machines. I have also tried Rocky 8.5 as an NFS client, and had success with SoftiWarp and RXE.
Sorry for the delay. I don’t always see the email notifications that someone has responded to this thread.
Unfortunately, I don’t have any experience with Fedora 35, so I might not be able to give you specific or exact instructions on how to deploy this, step-by-step.
But, let me maybe ask a few probing questions:
(I am running CentOS and they have a package group called Infiniband Support which will install a bunch of Infiniband-related tools. Not sure what Fedora 35’s version of that is called, but you might be able to google it. If not, I’ll try and see if I can get my CentOS system to dump out what’s all installed in the Infiniband Support group so that I can give you a list of packages that you can then manually try and find if Fedora 35 didn’t package them together for you.)
Are you running an Infiniband switch? If so, what make/model? Is it a managed switch or is it an externally managed switch? (I ask this because I am running a Mellanox MSB-7890 36-port 100 Gbps Infiniband switch and that’s externally managed, meaning I need another system running Linux, running the OpenSM subnet manager to get my IB up and running.)
If you are running a managed switch, consult the switch’s owner’s manual/user’s guide on how to make sure that you have one, maybe upto two subnet managers running. (I only use one, but I know that Mellanox’s managed switches can run up to two subnet managers, but if you have more running, that can cause problems for the switch itself.)
If it is an externally managed switch, you can check that on your Linux systems.
If you were run sudo iblinkinfo, does it report back what you expect it to report back?
(i.e. all links to and from the switch are active and up?)
If (2) doesn’t work, then try running sudo ibstat and see if the State: of the port is Active and also the Physical state: when it says “LinkUp” or not.
I forget what it says when it is waiting for the subnet manager to come online but I’ve had it happen where all of the rest of my clients were booted up and ready before my cluster headnode was booted up and ready, and the headnode runs the opensm so all of the client nodes were in like a “standby” mode until the headnode comes up. And once the headnode boots up and opensm is running, then all of the clients come up on the IB network as well.
If your card is running, and you have no problems with opensm running, and your switch is running, then the next thing that you would need to do is to make sure that RDMA is up and running on your systems.
In CentOS, I would enable rdma by typing:
sudo chkconfig rdma on
And also you can check the status of it by typing:
sudo systemctl status rdma
Once you have confirmed that it is working, in CentOS, I would have to edit /etc/sysconfig/nfs where it says:
RPCNFSDARGS=
to
RPCNFSDARGS="--rdma=20049"
I would also have to edit /etc/rdma/rdma.conf on the host to read:
XPRTRDMA_LOAD=yes SVCRDMA_LOAD=yes
and on the client, confirm in /etc/rdma/rdma.conf that it says:
XPRTRDMA_LOAD=yes
Your NFS export tab would be a normal NFS export:
/etc/exports:
/path/to/export/folder * (rw,[options])
whatever options you normally use for your NFS exports.
And then I would restart NFS using the command (run the following as root):
service nfslock stop; service nfs stop; service portmap stop; umount /proc/fs/nfsd; service portmap start; service nfs start; service nfslock start
On the client side, NFS mounts in /etc/fstab looks like this:
Whatever other options you may have for the mounts.
And then in my notes for my CentOS deployment, I have the lines:
sudo chkconfig nfs on
and also
sudo chkconfig nfslock on as well.
You can then reboot your server/host first and wait for that to all come up before rebooting your client an then you can re-run some of the IB diagnostics to see if it has made a successful connection to the switch and from your client to server by either using regular ping or using ibping.
(Regular ping is sometimes more telling than ibping because if normal ping doesn’t work, then there might be something that has to do with incorrect and/or improper IPv4 address assignment.)
You’ll want to watch to make sure that the network services started up properly for you and that there were not failures nor error messages that popped up during boot for either your server and/or your clients.
If everything works well, then you should be able to create a dummy text file on your server to see if the clients can see it and then see if your client can rm that dummy text file and create a new, dummy text file.
If there are no access issues, then the next thing that you can do is to try and see if you can send a decently large file over (e.g. 10 GiB file if you have one) (and you want a decent large file) because you want there to be enough time to see if your GUI system monitor is going to see the network traffic or not. (With RDMA, you shouldn’t see the network traffic in system monitor because that’s kind of the whole point of RDMA (bypassing the network stack).
rsync can be useful because it can still report the transfer rate over RDMA. (Or if you are just using the GUI for a normal copy operation, that should still be able to tell you what the transfer rate is and system monitor still won’t show it despite that. Or at least this is true in CentOS 7.7.1908.)
Not sure how these steps will work for you in Fedora 35, but if you find the Fedora 35 “version” of these steps, then you should be able to get your NFSoRDMA up and running on Fedora 35.
(Sidebar: I’ve found for example, that I can get CentOS to be the NFSoRDMA server, whereas I couldn’t get CAE Linux 2018 (which was originally built on Ubuntu 16.04 LTS) to be the same, NFSoRDMA server.
One of errors that I’ve also encountered, I forget if it was with Ubuntu or some distro other than CentOS which complained that the RDMA port 20049 for the NFS mount option in /etc/fstab wasn’t a valid option. (Might have been on TrueNAS Core 12.0 U1.1? I forget.)
But either way, my point being is that you might run into a problem like that where it doesn’t like the mount port, and/or that your specific distro’s version of the NFS client does NOT have RDMA available as an option (if you are using the “inbox” NFS drivers and Infiniband drivers.)
(This is one of the reasons why I ended up picking CentOS 7.7 to be my OS of choice because it met these functional/operational criteria, despite it being an older OS. But you can always upgrade the kernel from elrepo-kernel or do other stuff to bring the OS up to date with other distros like Fedora 35 unless there is something that you need out of Fedora 35 that you can’t get (or it’s harder to get (working)) in CentOS. YMMV.)
The infiniband support package is the same under Fedora, with all the ib* tools in it. Most of the tools return an error indicating there is no infiniband network present. I found a posting on a network mailing list from 2019 that indicates most of the ib* tool stack does not work for RoCE.
I do not have an infiniband switch or a managed switch that supports infiniband or PFC / QoS. I am using a basic layer 3 managed switch, an Aruba 5406R ZL2. I guess that’s part of the unknown for me, are RoCE and Infiniband the same thing? I don’t have Infiniband cards either, or maybe I do? The cards I bought are ConnectX-3 Pro EN (which I believe are firmware locked to 10g Ethernet, instead of 40g infiniband?) I don’t have any Infiniband subnet managers running, that I’m aware of.
There’s no distinct rdma service any more as far as I can see. If udev and systemd detect an RDMA capable card installed, certain changes are automatically made (and that’s probably a bad thing in my case since I don’t know what’s going on.) This post on reddit links to several of the changes that aren’t reflected in documentation for RDMA https://www.reddit.com/r/HPC/comments/p46et0/centos_84_does_not_provide_rdmaservice_and/
Same goes for svcrdma and xprtrdma. Those modules have been replaced with a module named rpcrdma, and it’s loaded and active. The rdma port is also already setup in rpc.
I’ve kinda given up at this point. None of my storage at home is fast enough to challenge the CPU, where RDMA offers the most benefit. Even reading and writing from a consumer level nvme SSD is barely a blip for the CPU.
I do appreciate your responses, sooner or later I’ll revisit this, maybe when datacenter grade nvme shows up cheap again on ebay.
You will notice that there is an entry in there for LINK_TYPE_P1 which, in my case, says “IB(1)”.
I forget if your card has Mellanox’s virtual port interface (VPI) technology where you can change the port type from Infiniband to Ethernet or Ethernet back to Infiniband.
One of the ways that you could test it would be to issue the command:
mstconfig -d 04:00.0 set LINK_TYPE_P1=2
and then query your card again to make sure that the changes were successful.
If it worked, then you are now running your Infiniband card with the ports in Ethernet mode.
If not, then this might be one of the reasons why your attempts at trying to run RDMA over Converged Ethernet (RoCE) isn’t working for you.
(For me, I am running straight RDMA (over Infiniband), and so, I don’t have this problem with the whole Ethernet layer/side of things.)
Your cards should be ETH only then, which in that case, you are absolutely correct in that you don’t need to have a subnet manager running at all then.
Yeah…so it really depends.
Like I am using four HGST 6 TB SATA 6 Gbps 7200 rpm mechanically rotating hard disk drives in RAID0, so it’s not terribly fast either. (The best that I’ve been able to get is about 800 MB/s combined write speed or about 200 MB/s contribution per drive, which works out to be around 6.4 Gbps.)
So by that metric, even regular 10 GbE would work and it would be a LOT easier to set up than RoCE.
In my use case which is understandably, a lot different than what most other people use it for is I actually will use the 100 Gbps Infiniband upto around 80 Gbps or so for high performance computing/computer aided engineering applications.
That’s where it really comes into play.
All of this NFS over RDMA stuff is just an added extra bonus of “things that I can do with 100 Gbps Infiniband” that I couldn’t really do with normal gigabit ethernet.
I think that when I first got my hardware, I did run some tests with straight NFS (no RDMA) and it was maybe doing about 150 MB/s writes.
So, needless to say, the RDMA DOES makes things faster, but with mechanically rotating hard disk drives, it has an upper limit based on the physics of the medium itself. (Not going to be breaking any records anytime soon.)
Conversely, because I went straight to 100 Gbps Infiniband, I’ve kind of skipped over the 10 Gigabit layer/step entirely (along with also skipping 25G, 40G, 50G, and even 56G).
So I use it as much as possible, in as many ways as possible, in as many places/systems as possible.
So with it being a ConnectX-3 Pro EN card, the Mellanox part number that the dual 10 GbE SFP+ variant maps onto is the MCX312B-XCCT.
“Since the SM is not present, querying a path is impossible. Therefore, the path record structure must be filled with relevant values before establishing a connection. Hence, it is recommended working with RDMA-CM to establish a connection as it takes care of filling the path record structure”
Again, with your card being a ConnectX-3 Pro EN, you can use RoCE v2 if you want to.
I’ve never tried setting up RoCE on my Infiniband cards, but from reading the MLNX OFED driver documentation about (the documentation for the EN driver points to the “full” MLNX OFED driver documentation for instructions on how to enable RoCE and it seems like it would be quite the pain (in the absence of a subnet manager) because you have to maybe perform extra steps according to the documentation/instructions, so I have no idea how to do that.
The info in that KB article seems to be one of the steps I was missing. Enabling RoCE v2 via modprobe options for mlx4_core has gotten me a few steps farther! I can now do rping and ucmatose between the servers, in both directions.
I was able to get nfs to mount using version 3, which I think is OK. It says it’s using RDMA, but when I copy large files, I’m able to see the traffic on the interface counters, which means the kernel / os can see the traffic as well, so RDMA isn’t actually doing it’s magic? At least traffic is showing up on the correct interface!
Command: mount -o proto=rdma 10.19.0.14:/mnt/vmstore /mnt/nfs-ssd
Result:
10.19.0.14:/mnt/vmstore on /mnt/nfs-ssd type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,mountaddr=10.19.0.14,mountvers=3,mountproto=tcp,local_lock=none,addr=10.19.0.14)
and on the NFS server:
Jan 26 11:15:18 server kernel: mlx4_core 0000:0b:00.0: Have more references for index 1, no need to modify mac table
Jan 26 11:15:18 server rpc.mountd[4062]: authenticated mount request from 10.19.0.16:1006 for /mnt/vmstore (/mnt/vmstore)
Jan 26 11:15:18 server kernel: mlx4_core 0000:0b:00.0: Registering MAC: 0x98f2b3ce9b51 for port 2 without duplicate
Yeah…I’m not really sure how RDMA over Converged Ethernet works because in theory, RDMA shouldn’t affect the network counters, but the Ethernet should/does.
re: " mount -o proto=rdma 10.19.0.14:/mnt/vmstore /mnt/nfs-ssd"
I was reading the Linux man pages for nfs(5) and I don’t have a better understand about how proto interacts with mountproto when proto=rdma.
(edit#2 On the IB side, things work a little bit differently:
aes0:/home/cluster on /home/cluster type nfs4 (rw,noatime,nodiratime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=10.0.1.117,local_lock=none,addr=10.0.1.100)
I don’t have mountproto for my NFSoRDMA mounts.)
I guess that one of things that you might be able to check would be to see if you have any real appreciable speed differences (if any) and/or any real or noticable/appreciable differences in reducing any CPU load/utilisation (again, if any, at all).
I know that you said that even without RoCE, you were saying that the CPU utlisation was very low, so I’m not sure if you might really see nor notice a difference.
But if rping and ucmatose is working, so that suggests that it is working for you, and I assume that this also means that prior to that, it meant that said rping and ucmatose WASN’T working for you previously.
On my Infiniband side of things, it RDMA works differently because with IB, I am LITERALLY and COMPLETELY skipping over the Ethernet layer and I assign a IPv4 address (which works with IPoIB) so none of my “normal” system monitoring tools for the network will read or be able to pick up on the IB/RDMA traffic.
As such, I have no idea how RoCE works because I don’t have it deployed on my cluster.
(If 100 GbE switches weren’t still so damn expensive, I might look into potentially deploying 100 GbE as well (because my ConnectX-4 cards are dual port, VPI which means that on the same card, I can have one port running in IB mode and the other port running in ETH mode. But at that point, the PCIe 3.0 x16 interface will become my limiting factor because that can only support upto 128 Gbps and a single 100 Gbps IB port can already take up almost all of that bandwidth, which means for me to get more bandwidth, I would have to replace my cards and systems with something that supports at least PCIe 4.0 x16, which I’m not looking to do). But I digress.)
Sounds like it’s up and running for you.
Just make sure that your /etc/rdma/rdma.conf parameters:
on the host and clients: XPRTRDMA_LOAD=yes
And additionally, on the host: SVCRDMA_LOAD=yes
are set.
(If you want to test it, and if your systems have enough RAM available, you can try and create a RAM drive (tmpfs), write a file to it, and then send it over and see what kind of speeds you’re getting. That might be one way for you to test to see if RDMA is working for you or not.)
I have also found that in “normal” data transfers (i.e. for storage management), RDMA is really limited to the speeds of the storage devices. So for me, because I am using spinning rust hard drives, 800 MB/s write (~6.4 Gbps) is about the best that I can do.
Conversely, when I use an application that uses the message passing interface (MPI), the application can use upto around 80 Gbps during a solve process.
So, true to its name, remote direct memory access - the speeds are usually realised when it’s RAM-to-RAM transfers. I think that even with four Samsung 860 EVO 1 TB SATA 6 Gbps SSDs in RAID0, the best that I’ve been able to see is maybe around 4.69 GB/s (~37.52 Gbps), but that’s the exception rather than the norm. (And sometimes, this is possible due to RDMA to RAM to SSD cache so it’s measuring the interface-to-cache speed.)
So…that might be a way for you to test to check and verify that RoCE is working properly for you.
edit
I’m not sure if it matters or not, but I am using NFS version 4.1 on my CentOS cluster.
What a great thread, I’m learning a lot and appreciate your detailed responses.
Good point about ethernet frames incrementing the counters, that could be what’s going on, I was using glances to monitor CPU and network while copying a 100G file from one SSD to another over the network, so it could have been showing me ethernet utilization, rather than tcp/ip throughput. I will have to use other tools to see exactly what counters are incrementing.
As I understand it, after reading a kernel mailing list thread from 2012 (here), mountproto is the method used to connect and disconnect the mount, for nfs v2 and v3 connections, where as proto is the transport method used for carrying the data. So tcp was used to authenticate or whatever with the nfs server, and rdma is used to transport data, I hope?
I’m not certain when, but it looks like the xprtrdma and svcrdma modules are obsolete, and replaced by rpcrdma now… maybe something new in Linux 5.x?
modinfo xprtrdma
filename: /lib/modules/5.15.14-200.fc35.x86_64/kernel/net/sunrpc/xprtrdma/rpcrdma.ko.xz
alias: rpcrdma6
alias: xprtrdma
alias: svcrdma
license: Dual BSD/GPL
description: RPC/RDMA Transport
author: Open Grid Computing and Network Appliance, Inc.
depends: ib_core,sunrpc,rdma_cm
An oddity I noticed, one of my virtual machines seems to have established an NFSv4 RDMA connection to the host server. The VM is using an SR_IOV virtual function passed through from the Mellanox card. Not sure why it works, but I won’t complain.
the-ripper:/mnt/storage/movies on /mnt/storage/movies type nfs4 (rw,relatime,vers=4.2,rsize=8192,wsize=8192,namlen=255,soft,proto=rdma,port=20049,timeo=14,retrans=2,sec=sys,clientaddr=192.168.2.12,local_lock=none,addr=192.168.2.14)
However, if I force my test environment to use nfs vers=4, it complains with a protocol error.
Hey at least this is enough progress to keep me interested.
Testing with cached reads on the server, and a ramdisk on the client, I was seeing about 8 out of 10gbit at times, just showing up as packets on the interface. the tcp and udp transport numbers were barely moving, probably from other things the server is doing.
The cp command was showing between 70 and 99% single thread CPU usage. But maybe that’s just how cp works, rather than cpu load from transport overhead. I didn’t see nfsd spike on cpu usage for example.
So I guess that means RDMA is actually skipping some of the middle-men along the way!
Now I just need to figure out why my ubuntu virtual machine will talk NFS v4.2 over RDMA to my server, but my Fedora physical machine will only talk NFS v3 over RDMA
EDIT:
A bit more, not very scientific testing., Copying from a ramdisk, to a ramdisk,
RDMA direct cable connection: Peak of 8gbit sustained, nfsd cpu usage 6 to 9%.
RDMA through basic L3 switch (no DCB/QoS support): Peak of 8gbit sustained, nfsd cpu usage 9-11% (more retries maybe?)
TCP direct connection: Peak of 6gbit sustained, nfsd cpu usage 16-20%
TCP through the switch: same as above
Yeah, so for IB, because RDMA completely bypasses Ethernet and the network stack, I HAVE to use IB specific tools to be able to read the NIC’s packet counters, etc.
(I haven’t ran my IB card in ETH mode in probably over 3-5 years now, so my memory on it is a little bit fuzzy. I think that I VAGUELY remember that when I do run it in ETH mode though, that the GNOME System Monitor will pick up on the ethernet traffic (and show/plot it). But I don’t remember if that was when I JUST got my cards so I was just using a DAC cable between two systems to test it out and to make sure that the cards were working, and LONG before I learned about everything else that I have learned since then.)
Yeah…not sure.
My CentOS cluster headnode is still running the 3.10.0-1062 kernel that ships with CentOS 7.7.1908. My principle thought process with running such an “old” kernel is that for my 4930K that’s in my cluster headnode, it doesn’t have NVMe slots, it doesn’t have a lot of things. So, if it isn’t broken, there is no real reason for me to update it. (If it works, don’t break it.)
I also, further, don’t know if those modules are a RedHat/CentOS thing or if it is something else. shrug
For the record though, svcrdma doesn’t appear to be a module because when I type in lsmod | grep rdma, it doesn’t show up in there.
As I mentioned, it’s in /etc/rdma/rdma.conf. Again, shrug. YMMV.
Like I said earlier, “if it works, don’t break it!”
lol…
(Sidenote: I would recommend you keeping a OneNote or some other note taking tool/app/whatever so that you can document your deployment notes from the lessons learned here. This is how I was able to provide the instructions for how I deploy my NFSoRDMA setup so if you have stumbled upon what is working for you, now would be going back through the command history and jotting that down so that in the event that you have to redeploy your server/clients/both – you will have all of the commands that you need for said deployment.)
I’m glad that this appears to be working for you now.
Yay!
That’s pretty good. 80% of the theorectical/rated capacity is usually what I would observe, max, in service/in practice.
Yeah…so, that depends.
Like I know that if I use rsync, rsync has/does its own thing to make sure the sources and targets are the same, so it’s doing some kind of checking with that, so I know that takes CPU load to do that kind of processing.
For nfsd, I would see it show up in the load averages in top, but in the GNOME System Monitor, it doesn’t show up as CPU load. shrug
Yeah…that’s weird.
And what makes that even weirder is that my older CentOS 7.7.1908 is able to negotiate the NFS connection using nfs4 version 4.1 automatically whilst the newer Fedora didn’t/couldn’t.
So, there might be a bit more research needed into that, if you are really that interested in it. (I don’t have Fedora set up/deployed, so unfortuantely, I don’t have a way to test that, plus again, the fact that I am using IB instead of ETH also makes a difference.)
re: your results
Looks like the NFSoRDMA between your server and Fedora and Ubuntu clients are working (although, like you said though, it might appear that the Ubuntu, with SR-IOV, might be working a LITTLE bit better than your physical Fedora client). But it still appears to be working, which is the important piece.
(Sidenote: I have struggled to get Ubuntu to be the NFSoRDMA SERVER. I don’t have it in my OneNote notes that shows that I ever got that working properly because it would complain about proto=rdma nor would it take port=20049. So…it seems that Linux distros that are derived from RedHat works better for NFSoRDMA servers than Ubuntu servers/systems. I might revisit that in the future, but again, right now, CentOS is working for me as my NFSoRDMA server, so I’m not going to spend much time messing with it when it (already) works. Ubuntu clients seems to work better than Ubuntu servers for this. shrug Go figures.)
You get higher transfer rates and a lower CPU usage with NFSoRDMA vs. without, so that gives some data/confidence that it appears to be doing what it is supposed to.
I just googled “RDMA copy” and this is a project that came up:
So…might be worthwhile for you to look into to see if that might help reduce your CPU load even further, if you really want/need/or otherwise interested in trying to do so.
But 8 Gbps on a 10 GbE link is about where I would expect it to max out unless you start spending quite a bit of time, tuning the network performance parameters.
For an untuned network, it’s still pretty good, I think.
For my 100 Gbps IB network, given that in practice, my slowest storage devices are mechanically rotating hard drives, so I don’t even bother trying to tune my network because the best write speeds that I can get is about 800 MB/s (or ~6.4 Gbps out of 100 Gbps that’s theorectically possible).
It’s good enough. Faster than my GbE network, and I try not to use SSDs of any kind (if I can avoid it, as much as possible) because if I managed to “unlock” writing to say NVMe SSDs at the full ~12.5 GB/s (100 Gbps), I would burn through the write endurance limit on said SSDs, even enterprise grade ones, through repeated usage of the blazing fast speed (which would mean the SSD would die, and I would have to try and RMA it for a new one).
So I don’t bother with it anymore.
Yes, it’s nice to see those speeds, but if/when you handle or process enough data where you can kill a SSD in a little over a year-and-a-half, it just isn’t worth it to me anymore.
I’ll check that out. I have a few bookmarks saved for doing zfs snapshots send/recv over RDMA, that’s probably the bulk of what I’d be sending over the network between the two machines.
I wonder if this has something to do with the bottleneck? I’m not sure if the 16.000Gb/s the kernel is talking about is gigabytes, or gigabits, but it seems my network card is only linking at PCIe 1.0 speed 2.5 GT/s, vs 5.0 GT/s for PCIe2 and 8.0 for PCIe3, IIRC
threadripper machine:
mlx4_core 0000:0b:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x8 link at 0000:00:03.2 (capable of 63.008 Gb/s with 8.0 GT/s PCIe x8 link)
Unfortunately, I don’t have any direct experience related to this, but my thoughts is that depending on the mechanism that it is using for the data transfer (whehter it’s “straight” cp or if it is rsync or if it is RDMA copy, there might be ways to make the transfers go faster).
As a part of my google search for “RDMA copy” yesterday, I also found this (that might be of interest to you) if your transfers are primarily unidirectional:
Gigabits.
A PCIe 1.0 2.5 GT/s x8 link is capable of 2.000 gigabytes per second transfers (16 gigabits/s).
Whereas a PCIe 3.0 8 GT/s x8 link is capable of 7.877 GB/s (64 Gb/s) transfers.
But it is interesting and strange that your card is only connecting at PCIe 1.0 x8, so I would check to make sure that the BIOS settings are correct and that you might need to power down the system so that you can unseat and reseat the card to make sure that it is making proper, electrical contact.
That seems strange that your Threadripper machine is doing that.
But to your question if it might have something to do with it - it is entirely possible.
I don’t remember if the PCI Express specification denotes the bit rates or the transfer rates (in transfer per second) as being unidirectional or bidirectional.
If it is bidirectional, then it would make sense for the card to be limited by 8 Gbps transfer rate (half of 16 Gbps). But if it is unidirectional, then even with a 8 Gbps transfer rate, there should be still another 8 Gbps of headroom available, so not really sure.
But I would definitely look into that and try and figure out why it is doing that because your Threadripper system should have no shortage of PCIe 4.0 lanes to support this even if the card itself is only a PCIe 3.0 card.