Hello everyone! I bought a bunch of $200 Intel E810 xxv-da2 cards on provantage when the price hit $215 and I am currently trying to start some personal programming projects that will require parallel file io to network attached storage. Currently I have a truenas scale with an E810 card in it … but truenas does not support RDMA … and since the two machines that will be doing this parallel file IO will be communicating with each other using the E810’s and rdma over ethernet, I thought it would be really nice if I could get rdma working on the server hosting my nfs share as well.
My question, how difficult is it to setup and administer a RHEL8/9 (or possibly another distro) file server that will support nfs with RDMA? Would it even be worth the effort if I am only serving up spinning rust, what about nvme, or what about optane?
I presently have the two compute machines communicating sucessfully with RDMA and it’s very fast, I wonder if it is worth uprooting my storage appliance to get rdma working there as well.
Thansk
If the storage is rust I don’t know if I’d expect it’s properly useful, but it seems a short document to configure and might be interesting.
As for RHEL as a storage `appliance’, if it is some way short of Truenas there is cockpit, which has storage, file share & even zfs applications.
Thanks,
If you look at how cockpit has changed in RHEL 9 it absolutely could replace a fair portion of the trueNAS gui.
Also I found this page TrueNAS Scale – Infiniband – XtremeOwnage which leads me to believe that forcing rdma support into trueNAS will not turn out very well.
2 Likes
I used to run only CentOS 7 and 8 (RIP) NAS’es. Depends on what you mean my appliance. I enabled NFS, set up the shares and that was about it. Not much to do after setting up, other than update on schedule. Not having much installed means less things to go wrong after an update.
I also had a separate CentOS VM running Samba for the Windows stuff in the infrastructure, for which I used groups and Linux file permissions extensively.
What are you using now in TrueNAS that you want to use in RHEL?
1 Like
Currently my trueNAS runs some docker appliances mostly importantly jellyfin and pi hole… but it is also an nfs share for a 2 node compute cluster and I have spent a pretty penny on all those E810 cards… so I am really wondering whether or not it makes sense to separate my docker host from the storage.
From my perspective, if I ditch trueNAS and switch to RHEL I can enable RDMA on the storage server and offload parallel file io to rdma, and at the same time supply iscsi or nfs shares to a separate server to run the docker containers.
You can do all of them on your NAS, especially if you don’t have a high load. You should be fine with RHEL and run docker or podman. I see no reason for it to not work.
Generally in data centers, it makes sense to split them up, but if you have a lot of computing power available to you on one box and it’s not being used, then either downclock and undervolt (or in more modern systems, lower the power limit and the CPU takes care of it), or make your hardware run more stuff.
Pi-Hole is basically insignificant (unless you have thousands of clients). Jellyfin will use most of the CPU if you don’t have hardware encoding. The storage, unless you have some slightly more exotic RAID configs (like, idk, 3 stripped RAID-Z2 or something), should not use a lot of your CPU cycles.
The process of running docker and NFS is trivial. Not sure about the RDMA stuff. But you should find tutorials for Rocky / Alma, if the RHEL ones are behind paywalls.
Thanks! I agree with what you said about pihole and nfs, and yes Jellyfin will crush your CPU with software transcodes. One of the reasons I was leaning towards a separate storage and docker host is due to my extensive use of consumer hardware… Each motherboard has two x8 PCIe lanes to the cpu and so the prospect of attaching the intel E810 nic, an HBA, and a GPU to help with transcoding and or rendering in paraview is just not possible. Where if I have two machines … one gets the GPU and one gets the HBA … both can have the 25gig network cards. I don’t know the more I think about it the more just using redhat or another server distro makes sense.