I have a bit of an elaborate setup, but I’m wondering if anyone has some insights into my storage box’s performance. I have a 6 drive RAIDZ2 ZFS pool on CentOS ZFS on Linux that is mainly used as NFS storage for my VMware cluster. The pool has an Optane 800p SSD for a SLOG device, and the VMware datastore is just a plain dataset accessed via NFS.
I’ve made one discovery, which is that it appears a smaller recordsize on the dataset works better for VMware - ZFS uses 128 KB by default, and if I use 4 KB that seems to reduce the wait I/O on the storage side. However, one problem I run into is that when I hit the storage box with a lot of I/O (even sequential) it hits the CPU pretty hard. I probably need a faster CPU, but I’d like to understand what exactly is using it. Right now this box has a Pentium G4400, and when I direct a lot of I/O to the system, it usually shows next to no %us but instead over 50% of the CPU is %sy. Top shows a ZFS process using most of the CPU, with several NFS processes using a chunk each. Is there a way to tell what part of ZFS is using the CPU? Is this the parity calculations, or something else?
Toggling compression on the dataset has an effect so I’ve tested that but there’s more to it.
