@wendell or anyone that can help.
Let me start by thanking wendell and anyone that has contributed to this post. I’ve been following this guide and have run into some issues that, and have no idea what else to do. (Some background about myself: I’m more or less tech literate, but when it comes to stuff outside of windows i’m pretty much a noob.)
Problem: zfs pool composed of two vdevs in raidz 2x(5x10TB) has abysmal performance as reported by fio reads and write around 10MiB/s with the '–sync=1" and around 40MiB/s without. fio output at the bottom.
System: Ryzen 7 1700, 16GB ECC (2x8GB), LSI HBA 9400-16i, GT710(pciex1), (MOBO SATA connectors not used), Melanox 10Gbps NIC, 10x 10TB WD HDD, WD black M.2 Nvme 512GB.
Previous setup: Windows 10 Pro Workstation, ReSF formated StorageSpaces dual parity with 10 column setup with LSI controller. The sq write performance was comparable to single disk native performance ~180MB/s, but it was constantly fluctuating from peak to as low as 28MB/s , and the fuller it got the peak write kept decreasing. Also tried, RAID 6 with HighPoint RocketRaid 2840 with more consistent transfer rates. Sustained sq write was about 376 MB/s. I did not like the performance of storage spaces as it decreased as the drive got fuller, and I didn’t like that I was limited to NTSF with the raid card and it didn’t have native file integrity features.
Expectation: from the guide, zpool of 1 vdev of raidz, was getting ~180MiB/s, comparable to single disk performance. Since i’m using 2 vdevs, I was under the impression that it would be a bit higher.
Usage case: Similar to what was covered in the video. Repository for media, and video project. Plex Server, steam/lan cache, etc. total expected users about 5 and expected simultaneous users about 2-3.
What I tried: I followed the guide to the best of my ability, I did catch some typos here and there that were covered in the comments. I had to use the fio-2.21 package because when I ran the fio-3.14, I was getting a missing file error.
I tried to flash the latest firmware/bios to LSI card, but was unsuccessful.
Hypotheses: (slow performance)
Not up to date firmware/bios in LSI card.
Not enough RAM.
Inefficient zpool set up.
faulty disk, but all show healthy in SMART and each one of them in diskmark have about same performance results from each other.
Questions:
-What is causing the horrendous performance, and how can I fix it?
-Would changing it to a zpool with 3 raidz vdevs be better?
-How do you mount the zpool to UnRaid? I’m kind of unsure if I need the shadow copies, but wanted to mount the zpool and test it as a share.
Thank you in advance.
fio output
:~# fio --direct=1 --name=test --bs=256k --filename=/BigBoii/test/whatever.tmp --thread --size=32G --iodepth=64 --readwrite=randrw
test: (g=0): rw=randrw, bs=® 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=64
fio-2.21
Starting 1 thread
Jobs: 1 (f=1): [m(1)][99.8%][r=41.0MiB/s,w=42.5MiB/s][r=164,w=170 IOPS][eta 00m:02s]
test: (groupid=0, jobs=1): err= 0: pid=12411: Sat Feb 15 00:26:41 2020
read: IOPS=81, BW=20.4MiB/s (21.4MB/s)(15.9GiB/797870msec)
clat (usec): min=68, max=294247, avg=12117.06, stdev=9721.38
lat (usec): min=69, max=294248, avg=12117.77, stdev=9721.38
clat percentiles (usec):
| 1.00th=[ 732], 5.00th=[ 5792], 10.00th=[ 7136], 20.00th=[ 8256],
| 30.00th=[ 9024], 40.00th=[ 9536], 50.00th=[ 9920], 60.00th=[10304],
| 70.00th=[10816], 80.00th=[12736], 90.00th=[17792], 95.00th=[27264],
| 99.00th=[58624], 99.50th=[65280], 99.90th=[98816], 99.95th=[108032],
| 99.99th=[201728]
bw ( KiB/s): min= 6144, max=46080, per=0.10%, avg=20897.58, stdev=4825.14
write: IOPS=82, BW=20.6MiB/s (21.6MB/s)(16.1GiB/797870msec)
clat (usec): min=40, max=105560, avg=108.13, stdev=663.48
lat (usec): min=43, max=105563, avg=111.80, stdev=663.57
clat percentiles (usec):
| 1.00th=[ 46], 5.00th=[ 56], 10.00th=[ 75], 20.00th=[ 78],
| 30.00th=[ 80], 40.00th=[ 82], 50.00th=[ 89], 60.00th=[ 97],
| 70.00th=[ 99], 80.00th=[ 103], 90.00th=[ 114], 95.00th=[ 124],
| 99.00th=[ 334], 99.50th=[ 438], 99.90th=[ 2288], 99.95th=[ 9536],
| 99.99th=[14144]
bw ( KiB/s): min= 2560, max=46592, per=0.10%, avg=21124.17, stdev=6594.64
lat (usec) : 50=1.21%, 100=34.00%, 250=14.23%, 500=0.74%, 750=0.66%
lat (usec) : 1000=0.40%
lat (msec) : 2=0.10%, 4=0.31%, 10=24.07%, 20=20.79%, 50=2.61%
lat (msec) : 100=0.83%, 250=0.05%, 500=0.01%
cpu : usr=0.14%, sys=2.17%, ctx=72474, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=65179,65893,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=20.4MiB/s (21.4MB/s), 20.4MiB/s-20.4MiB/s (21.4MB/s-21.4MB/s), io=15.9GiB (17.1GB), run=797870-797870msec
WRITE: bw=20.6MiB/s (21.6MB/s), 20.6MiB/s-20.6MiB/s (21.6MB/s-21.6MB/s), io=16.1GiB (17.3GB), run=797870-797870msec