Linux mount ignores options?!

Software & Operating Systems Linux
1

Continuing to tilt at windmills and get RoCE working at home, I’ve cabled two Fedora servers together, no switch in between. Each has a Mellanox CX-3 Pro EN network card (supports RoCE v1). Both servers have all the rdma stuffs installed already.

  • 10.19.0.14: Server machine RDMA interface
  • 10.19.0.16: Client machine RDMA interface
  • 192.168.2.14: Server machine regular interface
  • 192.168.2.16: Client machine regular interface

Try to mount an nfs share on the client machine

mount -o proto=rdma,clientaddr=10.19.0.16 10.19.0.14:/mnt/vmstore /mnt/nfs-ssd

Results in the following mount be registered with the system

10.19.0.14:/mnt/vmstore on /mnt/nfs-ssd type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.16,local_lock=none,addr=192.168.2.14)

So mount totally ignored my mount options, using proto=tcp and using clientaddr=192.168.2.16 which is the non RDMA interface. It did use port 20049 which is the port for RDMA

I don’t want to type a novel here, what am I doing wrong?

Oh, here’s /etc/exports on the server

# cat /etc/exports
/mnt/vmstore 10.19.0.0/24(rw,async,insecure,no_root_squash)
2

Look at the nfsd log and the journalctl/dmes logs to see what is being thrown. I assume that it may not be able to connect with the RDMA IPs so it is falling back to the regular interfaces. Also, what does your hosts file look like? You may want to swap the order of the acceptable routes by putting your 10. IPs first.

1 Like
3

Thanks for the suggestions!

Turns out I was missing an option on a kernel module that enabled RoCE for my particular card, newer cards have it turned on by default.

I posted up the NFS mount details here, the mount claims to be using RDMA now, but I’m still seeing traffic on the interface counters, which I shouldn’t be able to see. My quest continues! :laughing:

1 Like