The info in that KB article seems to be one of the steps I was missing. Enabling RoCE v2 via modprobe options for mlx4_core has gotten me a few steps farther! I can now do rping and ucmatose between the servers, in both directions.
I was able to get nfs to mount using version 3, which I think is OK. It says it’s using RDMA, but when I copy large files, I’m able to see the traffic on the interface counters, which means the kernel / os can see the traffic as well, so RDMA isn’t actually doing it’s magic? At least traffic is showing up on the correct interface!
Command: mount -o proto=rdma 10.19.0.14:/mnt/vmstore /mnt/nfs-ssd
Result:
10.19.0.14:/mnt/vmstore on /mnt/nfs-ssd type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,mountaddr=10.19.0.14,mountvers=3,mountproto=tcp,local_lock=none,addr=10.19.0.14)
and on the NFS server:
Jan 26 11:15:18 server kernel: mlx4_core 0000:0b:00.0: Have more references for index 1, no need to modify mac table
Jan 26 11:15:18 server rpc.mountd[4062]: authenticated mount request from 10.19.0.16:1006 for /mnt/vmstore (/mnt/vmstore)
Jan 26 11:15:18 server kernel: mlx4_core 0000:0b:00.0: Registering MAC: 0x98f2b3ce9b51 for port 2 without duplicate