Terrible ZFS performance on striped NVMe drives


I’ve began to troubleshoot an issue which I thought to be a VM issue, but one that turns out to be a ZFS related problem.
I notice incredibly low write speed on 2 1TB striped intel 665p NVMe drives. Where one would expect at least a gigabyte a second, it barely reaches 80MB/s writes.
I’ve tested it with the below command:

fio --name=seqwrite --rw=write --direct=1 --ioengine=libaio --bs=1M --numjobs=4 --size=5G --runtime=600 --group_reporting

Its able to reach 1.4GB/s for a good 2 seconds, before it right away goes down to around 200MB/s and settles between 50 to 80 MB/s in 10 seconds.

If I run the same command, but on 2 striped MX500 sata ssds, Im able to reach a sustained speeds of well over 300MB/s, even though thats over NFS!

Some details, I can confirm that the drives are not limited by their PCIe bus, they both have 4 PCIe 3.0 lanes.
They are also not overheating, while running the test, they do not reach temperatures over 40 degrees.
The ZFS pool is setup with a recordsize of 16K and an ASHIFT value of 13.

What’s going wrong with my NVMe pool here? Is it because those intel drives are QLC? Is it because they do not have a DRAM cache? Even then I would still expect much more than 80MB/s.

Before someone gives a better answer, surprisingly this Linus video may be a helpful watch?

Benchmark the raw performance of a single disk with a tool like gnome-disk-utility with a large file size, then you have the answer if it is because of QLC. I would think that is the reason.

Thanks, thats a video ill definitely be watching!

1 Like

I’m guessing that performing such benchmarks is not possible without formatting the drive(s)?

Would you have any recommendations for replacement drives? They’re relatively new so I could sell them secondhand and get proper ones because this is not usable.

You can benchmark write performance only by writing, so yes you would need to reformat/overwrite it.

I have both PNY XLR8 CS3030 and Transcend MTE220S SSDs. Both work fine.

Those seem decent, thank you!

1 Like

This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.