Hi everyone, I am putting this thread here. Please move if it should be in another category.
I have been battling a FreeNAS server for a couple of days now.
We have deployed said FreeNAS server on a Dell R730xd with these specs:
2x E5-2630L v3 CPUs, 4x 16GB of 1866MHz DDR4 ECC RAM, Intel X540 rNDC 10Gb NIC and 2x Dell PM1725a NVMe SSDs in a mirror, running FreeNAS-11.3-U4.1
I am using an HP DL360e Gen8 and a Dell R620 as clients.
I wanted to do a comparison between XCP-NG and Proxmox for VM performance but I stumbled on low NFS speed problems.
The servers are connected through a Buffalo BS-MP2008 10Gb 10Gbase-T switch.
The R730xd and R620 have the X540 rNDC cards and the HP has a Broadcom BCM57810 NIC. I have also tested with a direct connection from FreeNAS to Proxmox on the R620 without the switch in the middle, no change.
Whatever I do, I cannot get the NFS sequential read speeds to go above 600-700MB/s, even when reading from ARC. NFS writes and Samba reads are at wire speed when using high iodepths. I am benchmarking with fio.
The NVMe pool itself can do >3GB/s sequential reads and >1.5GB/s sequential writes. But NFS performance is lacking, both at sequential and random tests.
I have posted all my benchmarks at the TrueNAS forums, I am linking it here to avoid having to repaste everything: SOLVED - NVMe and NFS performance issues | TrueNAS Community
Help please!