Poor ZFS performance on Enterprise SSDs in Raid0 array

[mrpopo]

I’ve got a Dell R720 with a HBA running in IT mode passing 6 Samsung MZ-6ER400T/0C3 400GB SAS SSDs through to proxmox. The drives came in with 520 byte sector sizes and I successfully reformatted them to 512 so proxmox could utilize them. I had all 6 together in a Raid0 array using ZFS since this is purely for homelab experimentation and wont be housing anything critical. Originally I was getting poor performance but I noticed ZFS threw an error stating that the configured block size for the pool was 4096 but the native block size was 8192. I destroyed and remade the array with ashift=13 to align the configured and native block sizes but i’m still getting absolutely godawful performance. I just ran a fio benchmark on the pool and these are the results:

I made sure to set the fio block size to match the block size in the pool. I’m not sure why the performance is so poor. The server itself has plenty of RAM and cores. How is it a ZFS pool on 6 enterprise SSDs has performance on par with a USB 3 thumb drive? I’d like to stick with ZFS because of it’s snapshot capabilities which comes in handy when trying to break and fix things, but i’m starting to think I should just go to mdadm or something.

[xzpfzxds]

Random writes smaller than the recordsize are pretty much the worst case scenario for ZFS.

If you’re using the default 128k recordsize then that’s a 16x write multiplication, and implicit read on each write.

If you run zpool iostat -L -r while it’s running, I think you’ll see a full recordsize sync_read frequency second only to the async_write frequency, as ZFS needs to read full records in order to merge the 8k writes back to the full record.

You’ll get more IOPS and bandwidth with more threads (--numjobs), of course.

[mrpopo]

Wouldn’t me specifying ashift=13 during pool creation override the 128k record size to 8k?

[mrpopo]

[NicKF]

I think the problem may be your test. Have you run the same fio script against something else

8K random IO is going to nuke most things. Try a 1M block size

[mrpopo]

Changing the block size of the fio test to 1M seems to have made the results reflect more of what I expect, but i’ll have to run some tests to see if the actual performance of VMs stored on the pool reflect that benchmark