Continuing to tilt at windmills and get RoCE working at home, I’ve cabled two Fedora servers together, no switch in between. Each has a Mellanox CX-3 Pro EN network card (supports RoCE v1). Both servers have all the rdma stuffs installed already.
- 10.19.0.14: Server machine RDMA interface
- 10.19.0.16: Client machine RDMA interface
- 192.168.2.14: Server machine regular interface
- 192.168.2.16: Client machine regular interface
Try to mount an nfs share on the client machine
mount -o proto=rdma,clientaddr=10.19.0.16 10.19.0.14:/mnt/vmstore /mnt/nfs-ssd
Results in the following mount be registered with the system
10.19.0.14:/mnt/vmstore on /mnt/nfs-ssd type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.16,local_lock=none,addr=192.168.2.14)
So mount totally ignored my mount options, using proto=tcp and using clientaddr=192.168.2.16 which is the non RDMA interface. It did use port 20049 which is the port for RDMA
I don’t want to type a novel here, what am I doing wrong?
Oh, here’s /etc/exports on the server
# cat /etc/exports /mnt/vmstore 10.19.0.0/24(rw,async,insecure,no_root_squash)