Hi,
I’ve began to troubleshoot an issue which I thought to be a VM issue, but one that turns out to be a ZFS related problem.
I notice incredibly low write speed on 2 1TB striped intel 665p NVMe drives. Where one would expect at least a gigabyte a second, it barely reaches 80MB/s writes.
I’ve tested it with the below command:
fio --name=seqwrite --rw=write --direct=1 --ioengine=libaio --bs=1M --numjobs=4 --size=5G --runtime=600 --group_reporting
Its able to reach 1.4GB/s for a good 2 seconds, before it right away goes down to around 200MB/s and settles between 50 to 80 MB/s in 10 seconds.
If I run the same command, but on 2 striped MX500 sata ssds, Im able to reach a sustained speeds of well over 300MB/s, even though thats over NFS!
Some details, I can confirm that the drives are not limited by their PCIe bus, they both have 4 PCIe 3.0 lanes.
They are also not overheating, while running the test, they do not reach temperatures over 40 degrees.
The ZFS pool is setup with a recordsize of 16K and an ASHIFT value of 13.
What’s going wrong with my NVMe pool here? Is it because those intel drives are QLC? Is it because they do not have a DRAM cache? Even then I would still expect much more than 80MB/s.
