TrueNAS Scale - SMB over RDMA

Operating Systems & Open Source Linux
1

I can’t find good information on weather SMB over RDMA or RDMA in general will work in TrueNAS scale. Did anyone get this to work?

2

SMB-Direct (the RDMA extension of SMB) is not yet fully supported by Samba (the open-source implementation of SMB used on TrueNAS SCALE and CORE), so it’s currently not possible unless you compile a custom Samba binary with the prototype RDMA code. More info is on the Samba wiki: Samba3/SMB2 - SambaWiki

5 Likes
3

Thank you for your reply!
„not yet fully supported“ sounds like it is coming? Will NFS over RDME work with TrueNAS Scale?

4

TrueNAS Scale has no official Support for NFS over RDMA.

You could set it up manually on the underlying Debian, but keep in mind that it’s likely to break during Updates and possibly conflict with settings from the UI.

5

RDMA support is on the distant “nice-to-have” roadmap for TrueNAS and (at least as far as I understand it) for Samba. It’ll come along eventually, but it’s impossible to say when. Same for NFS over RDMA and iSER (iSCSI Extensions for RDMA).

In the meantime, if you want to play with SMB over RDMA (SMB-Direct), you can set up a Windows server and even run ZFS on Windows: Releases · openzfsonwindows/openzfs · GitHub

6

It depends on the specific implementation (and also what hardware you are using (NIC(s), switch(es), etc.).

if you’re using IB, you can download the Debian MLNX_OFED driver and it is my understanding, from their release notes, that their version of NFS should or does support NFSoRDMA.

But, out of the box, the “inbox” Debian Infiniband driver does NOT support NFSoRDMA (the last time that I looked at it, which was probably about a year ago by now).

Conversely though, CentOS 7.x supports it, out of the box. (set the NFSoRDMA port to 20049, remember to enable RDMA on the client, and you’re good to go.)

I tried to pass virtio-fs (in Proxmox) to a CentOS VM so that I would be able to let CentOS take over NFSoRDMA, and then use virtio-fs on the backend, but that ended up with NFS contention/lock issues that were never resolved, so I abandoned that.