ZFS Performance Woes

Software & Operating Systems Linux
1

I just built a small home server (part list: https://pcpartpicker.com/list/7ZZZ9G).

I decided to use ZFS as my filesystem for data integrity reliability, the usual reasons. I have 2 pools, both mirrors, both with ashift=12: main which consists of the SSDs contains the OS and programs as well as any VMs I decide to create, and vault which consists of the 2 WD Reds, which holds my data.

Performance is horrible. I can’t get over 200MB/s on either pool in sequential reads and for encrypted datasets it’s even worse (55MB/s on both pools)! I’m testing with fio and caching disabled on the datasets. At these speeds, it won’t even keep up with a gigabit NIC. I have no idea what’s going wrong, I thought the SIMD issue was fixed with zfs-0.8.3. Anyone have any ideas? Thanks.

2

I may be wrong but it looks like the drives are just really slow,

https://ssd.userbenchmark.com/SpeedTest/229920/KINGSTON-SA400S37120G

coupled with zfs overhead this maybe the expected speeds of this setup.

of course with ssds for performance always get over 512 GB … 1TB is perferable

3

Even so, I expected sequential reads at least over 300MB/s from each drive let alone a mirror, this can’t even crack 100MB/s.

4

What is it like with real data loads, not simulation?

If it maxes out your network and is stable, isn’t that enough? Use another filesystem if you want mega speed. ZFS is for stability in my book.

5

Wendell did some work on this here:

Maybe their is something in their that can help.

You can get an inland professional SSD at the Micro Center for about $20-25 for your OS drive. They are good and reliable.

One thing to note is that ZFS does a lot of mem copy’s. Faster ram greatly improves performance. If you can’t get your home server running faster with other methods you may want to try a 3rd gen Ryzen and 3200 mhz ECC memory… Looks like you RAM is on the QVL. Double check your RAM settings are correct in the BIOS. That could make a big difference to and easily cause bad performance…

6

I’d think it’s more likely that AsMedia SATA controller is just shitting itself. I’d say try putting an HBA (LSI 9211-8i/9207-8i are cheap) into it and see if it fixes things.

1 Like
7

My RAM is currently running at DDR4-2866 with a CL of 19. According to memtest86 it has a bandwidth of about 12.2GB/s so I think I’m good there.

I already tried changing the scheduler to none and that didn’t have an effect.

This is the output from perf top, maybe whatever acpi_processor_ffh_cstate_enter has something to do with it.

perf-top
perf-top.png849×1015 244 KB

8

Thanks, if nothing else works I’ll definitely look into those!

9

what os are you using ?

10

I am using Arch Linux with kernel 4.19.101 (the LTS kernel).

11

I thought the simd fix in 0.8.3 was only if at kernel 5.x and up

3 Likes
12

and also, this should not not be an issue but just in case the bios should have the sata controller set as ahci .

im not sure if the arch kernel and zfs version is sane yet, im under the impression it is, but but there is room for doubt.

13

Might be worth loading the pool in FreeBSD and seeing if the problem is specific to Linux.

14

Unfortunately, the zfs 0.8.3 won’t build on the mainline Arch kernel (5.5.2) due to the following error:

FATAL: modpost: GPL-incompatible module zfs.ko uses GPL-only symbol ‘__rcu_read_unlock’

Besides, looking at perf top, zfs is using AESNI and AVX2 for fletcher 4 checksumming so I don’t think that’s the issue.

15

I was looking at bencmarks for the SSD drive, and the low queue depth 4kiB read/writes were slow. ashift=9 is likely more appropriate there

Also these drives don’t have any cache at all, if the workload is forcing a sync, the drive is going to struggle as it can’t cheat at all.

16

Aren’t newer drive like that better at a shift=13? For 8k blocks?

17

SSD-Z reports sector size as 512. https://jazztechin.wordpress.com/2017/05/10/kingston-a400-240gb-ssd-review/

Even if the flash has bigger read cells, I’m assuming some sort of firmware shenanigans like CoW, block remap, and write consolidation. That work s well enough for journalingfilesystems, but overlap/copy/conflict with what zfs is doing. For instance this controller is compressing data to speed up operations, but most zfs pools are going to do that anyways.

That and no cheap drive that’s 70% full will match the performance out of the box with almost nothing on it.