Epyc Milan Slow Transfers When Using ZFS

[dizzydre21]

Hello all,

I’ve been pulling my hair out with a presumed issue I’m having with an Epyc Milan rig.

I’m getting slow transfer speeds when using U.2 NVMe SSDs in a ZFS stripe. I can destroy the pool and get basically max speed per disk if they are formatted for EXT4. All of my testing is being done with fio . Please note that I’ve tried multiple SSDs in both a 2-way stripe and a 4-way stripe with results being slower than a single disk on EXT4. The 2-way PCIe gen4 disks and the 4-way PCIe gen3 disks were a little faster, but something makes me thing I’ve got a BIOS setting wrong or something.

Hardware:
AMD Epyc 7443P
Asrock ROMED8-2T (newest BIOS)
192GB 3200MHz ECC RDIMM (6x32GB)
OS Drive WD Black 1TB (Tried above tests with TrueNAS, Proxmox, Deb13, and EndeavourOS as bare-metal OS)
Test SSDs (All Striped and tested separately) - 2x1TB Intel P4510, 2x960GB Samsung PMA93, 4x2TB Intel P4510

2x1TB P4510 - max sequential r/w of 2,000MBps
2x960GB PM9A3 - max sequential r/w of 2,8000MBps
4x2TB P4510 - max sequential r/w of 3,300MBps

The 4x2TB P4510 stripe is now in a prosumer LGA1700 based server and gets about 6,500MBps, which is what I would expect in the Epyc machine.

Lastly, I posted on the Proxmox forums a while back with a slightly different setup, but the same issue that got me testing with fio instead of iperf3. I got no replies, so figured I’d try my luck at Level1Techs.

That post can be found here:

Epyc Milan Vm to VM Communication Slow

[twin_savage]

I’m kind of just throwing spaghetti at the wall here, but it’s possible the unbalanced memory configuration on the EPYC is causing performance issues.

[Janos]

@dizzydre21

but the same issue that got me testing with fio instead of iperf3

I don’t get it, what does iperf3 have to do with storage benchmark?
But regarding your problem, I had or still have the same issue with a Xeon Skylake system and TrueNAS, no matter how many NVMes I combine into a stripe, the performance is always better with just one.
I should test that again, a few updates have passed since then.

DC3000-ME is with ZFS default settings

For DC3000ME-2 I used this settings

zfs set compression=off             tank04/bench
zfs set primarycache=metadata       tank04/bench
zfs set secondarycache=none         tank04/bench
zfs set logbias=throughput          tank04/bench
zfs set sync=disabled               tank04/bench
zfs set dedup=off                   tank04/bench
zfs set atime=off                   tank04/bench

Here’s a comparison of Ext4 vs ZFS, but it’s via NVMe-OF.