Building a dual Xeon 6442Y system with 8 x DC5800X Optane NVME drives and ZFS IOPS/throughput is horrendous. RAID 10 and getting ~9K IOPS and ~2GB/s sequential read across all 8 drives. Can someone possibly point me to a resource for getting this tuned?
thanks!
-W
9k IOPS is actually really great. Depends on where and how you measure it.
And if you are reading from the pool via a bonded 10Gbit Link or the drive is running via PCIe 2.0, 2GB is the maximum you will get.
Without any further details, numbers mean nothing.
a single drive (ext4) gets nearly the full 7GB/s sequential reads + ~500K IOPS on a random read/write fio
the 500K IOPS single drive and 9K IOPS test (ZFS) were using the same fio args
LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM not supported
Also depends on your pool settings. record/blocksize, compression, atime,etc.
and especially ashift. I don’t know what ashift is best for optane? 12 always works.
ZFS has deficits in capitalizing on fast storage, because it relies so heavily on main memory doing the job. Pure NVMe pools sacrifice a lot of performance with ZFS regardless. But I’ve seen 12-13GB/sec sequential in ZFS, so something is odd with those 2GB/s.
In any case you will leave a lot of performance on the table. That’s why people don’t use Optane and (to a lesser degree) NAND as main storage.
That seems too slow even for ZFS, can you verify that your processor clocks are ramping during the fio run? I know sapphire rapids had some weird clock ramping behavior early on.
ZFS pool changes :
zfs set atime=off tank
zfs set compress=lz4 tank
zfs set primarycache=metadata tank
zfs set sync=off tank
zfs set sync=disabled tank
zfs set relatime=off tank
zfs set recordsize=16K tank
Installed tuned and changed the profile to “througput-performance”
fio --filename=./testfile --size=128GB --rw=randrw --bs=16k --iodepth=8 --ioengine=libaio --runtime=120 --time_based --name=testiops16
[m(1)][38.3%][r=72.5MiB/s,w=71.5MiB/s][r=4641,w=4576 IOPS][eta 01m:14s]
Ugh. I’ll just change a setting at a time until I see an improvement. This will eventually be used as a MariaDB store, hopefully.
Have you gone to intel’s site then used their tool to update the firmware on all of your drives?
I have not, however, I wouldn’t expect such a massive delta between a single drive (ext4) and RAID 10 with 8 drives (ZFS). Could probably get really good performance with md, but ZFS snapshots are so useful for database backups.
Be careful with the logbias setting. This results into serious fragmentation over time and there is nothing worse for performance than a framented pool. May not be that problematic with large blocks, but 16k and less, it’s nightmare even for NVMe/Optane. Writing the data orderly a second time keeps the pool healthy, but requires more work. I never found benefits outperforming the resulting framentation.
What are you running? FreeBSD, Linux, something else? I’m going to guess Linux since you mentioned ext4 though. Since you seem to have this as a test box, do you see the same behaving trying to use a newer kernel and/or FreeBSD 14.0? Are you sure lz4 compression don’t cause a bottleneck?
It’s Debian 12, kernel 6.1.0 and compression does slow things down a bit, the IOPS test increases to 17K vs ~4.5K with it on LZ4. I might try to load FreeBSD on the box. The box is essentially a playground for now until I can figure out how to get the storage to not suck this badly.
Edit :
created a md RAID10 set with all 8 drives, sequential reads are between 30 and 53GB/s, IOPS are 300-500Kish (which still seems pretty low when each drive is rated for 1.5M).
try it as a raidz instead. You still get a single drive of safety, and the rest are in a raid. Small transactions will only go to 2 drives, so you potentially get 4x the performance.
Also can you “zpool status” so we have the right visual?
It would be interesting to see if the I/O subsystem would play a role between Linux and FreeBSD regarding your benchmarks.
raidz improved performance by about 2X over raid10 :
[m(1)][33.1%][r=150MiB/s,w=146MiB/s][r=9589,w=9338 IOPS][eta 01m:21s]
zpool status
pool: tank
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
nvme0n1 ONLINE 0 0 0
nvme1n1 ONLINE 0 0 0
nvme2n1 ONLINE 0 0 0
nvme3n1 ONLINE 0 0 0
nvme4n1 ONLINE 0 0 0
nvme5n1 ONLINE 0 0 0
nvme6n1 ONLINE 0 0 0
nvme7n1 ONLINE 0 0 0
errors: No known data errors
this is with compression disabled
edit: redoing the ZFS pool as an 8 drive stripe shows pretty much the same numbers as Z1
Day #2 on this adventure. Noticed LBAs were set for 512 instead of 4K on the drive also downloaded the Intel CLI tool and flashed new firmware on everything. We’ll see.
on the random readwrite fio test, IOPS were bouncing between 30 (!) and ~5K. I completely disabled primary/secondary cache and things stabilized around 5K for RAIDZ so looks similar to before.
Are all of the drives connected at PCIe 4.0 x 4? Any PCIe errors? AER?
Connectivity is all fine. Single drive (xfs and ext4) performance is as expected. md RAID10 is nearly the throughput cap at ~50GB/s and ~500K IOPS. This seems to be a ZFS thing entirely. The only thing I really need ZFS for are snapshots and the endurance on these Optanes is such that it would take many lifetimes of expected load to even start denting them so maybe use md and stick ZFS on the /dev/mdX? Just thinking out loud.
zfs set recordsize=16K tank
Why so low? What will you eventually use this pool for?
MariaDB. it matches the INNO page size.