NFS Tuning (for VM storage)?

Hi everyone, I am putting this thread here. Please move if it should be in another category.

I have been battling a FreeNAS server for a couple of days now.

We have deployed said FreeNAS server on a Dell R730xd with these specs:
2x E5-2630L v3 CPUs, 4x 16GB of 1866MHz DDR4 ECC RAM, Intel X540 rNDC 10Gb NIC and 2x Dell PM1725a NVMe SSDs in a mirror, running FreeNAS-11.3-U4.1

I am using an HP DL360e Gen8 and a Dell R620 as clients.

I wanted to do a comparison between XCP-NG and Proxmox for VM performance but I stumbled on low NFS speed problems.

The servers are connected through a Buffalo BS-MP2008 10Gb 10Gbase-T switch.
The R730xd and R620 have the X540 rNDC cards and the HP has a Broadcom BCM57810 NIC. I have also tested with a direct connection from FreeNAS to Proxmox on the R620 without the switch in the middle, no change.

Whatever I do, I cannot get the NFS sequential read speeds to go above 600-700MB/s, even when reading from ARC. NFS writes and Samba reads are at wire speed when using high iodepths. I am benchmarking with fio.

The NVMe pool itself can do >3GB/s sequential reads and >1.5GB/s sequential writes. But NFS performance is lacking, both at sequential and random tests.

I have posted all my benchmarks at the TrueNAS forums, I am linking it here to avoid having to repaste everything: NVMe and NFS performance issues | TrueNAS Community

Help please!

1 Like

Do you have compression, encryption or deduplication enabled on the pool?

Compression is on, dedup and encryption off.

You can see the pool and dataset properties on the TrueNAS forum post.

I was getting 600Mb with compression turned on and 980Mb with it off, try making a new dataset with compression disabled.

1 Like

Same thing with compression set to off on the dataset. Hard limit on 700MB/s, 650MB/s average read.