Hi Wendell thanks for taking the time to respond,
So for some context I was planning on using these 14TB external MACH.2 drives to act as an acceleration pool for an existing 5x14TB RaidZ2 pool, I figured adding dual actuators would get me to be able to saturate 25GbE with reasonable IOPS.
I had tested all of these drives as 50/50 split using badblocks and was able to get 140-160MB/s sustained after 30+ hrs per read / write cycle, reads were faster by about 10%.
For testing Iām using the following command: fio --filename=test --direct=1 --rw=write --bs=1M --numjobs=4 --iodepth=32 --group_reporting --name=test --filesize=32G --ioengine=libaio --runtime=300
With striped mirrors of the first 4 disks Iām able to get the following results:
test: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
...
fio-3.33
Starting 4 processes
test: Laying out IO file (1 file / 32768MiB)
Jobs: 4 (f=4): [W(4)][100.0%][w=797MiB/s][w=797 IOPS][eta 00m:00s]
test: (groupid=0, jobs=4): err= 0: pid=1175433: Sun Dec 8 13:24:05 2024
write: IOPS=855, BW=855MiB/s (897MB/s)(128GiB/153264msec); 0 zone resets
slat (usec): min=73, max=12151, avg=4675.60, stdev=1592.14
clat (nsec): min=1147, max=257183k, avg=144988285.25, stdev=49241311.53
lat (usec): min=1427, max=265258, avg=149663.88, stdev=50826.90
clat percentiles (msec):
| 1.00th=[ 6], 5.00th=[ 6], 10.00th=[ 44], 20.00th=[ 150],
| 30.00th=[ 153], 40.00th=[ 155], 50.00th=[ 157], 60.00th=[ 161],
| 70.00th=[ 165], 80.00th=[ 169], 90.00th=[ 178], 95.00th=[ 188],
| 99.00th=[ 222], 99.50th=[ 230], 99.90th=[ 251], 99.95th=[ 255],
| 99.99th=[ 257]
bw ( KiB/s): min=522240, max=21290194, per=99.96%, avg=875350.02, stdev=301537.82, samples=1224
iops : min= 510, max=20790, avg=854.83, stdev=294.45, samples=1224
lat (usec) : 2=0.01%
lat (msec) : 2=0.01%, 4=0.17%, 10=7.85%, 20=0.94%, 50=1.30%
lat (msec) : 100=1.67%, 250=87.96%, 500=0.11%
cpu : usr=0.54%, sys=2.38%, ctx=186478, majf=0, minf=47
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,131072,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=855MiB/s (897MB/s), 855MiB/s-855MiB/s (897MB/s-897MB/s), io=128GiB (137GB), run=153264-153264msec
The topology is as follows:
So now Iāll try again with the raid-z2 topology:
test: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
...
fio-3.33
Starting 4 processes
test: Laying out IO file (1 file / 32768MiB)
Jobs: 4 (f=4): [W(4)][100.0%][w=568MiB/s][w=567 IOPS][eta 00m:00s]
test: (groupid=0, jobs=4): err= 0: pid=1177076: Sun Dec 8 13:30:31 2024
write: IOPS=648, BW=649MiB/s (680MB/s)(128GiB/202066msec); 0 zone resets
slat (usec): min=73, max=11680, avg=6164.73, stdev=2098.94
clat (nsec): min=1246, max=284738k, avg=191155342.06, stdev=64924401.41
lat (usec): min=1928, max=293574, avg=197320.07, stdev=67014.79
clat percentiles (msec):
| 1.00th=[ 6], 5.00th=[ 6], 10.00th=[ 48], 20.00th=[ 197],
| 30.00th=[ 201], 40.00th=[ 205], 50.00th=[ 209], 60.00th=[ 213],
| 70.00th=[ 220], 80.00th=[ 228], 90.00th=[ 236], 95.00th=[ 243],
| 99.00th=[ 253], 99.50th=[ 259], 99.90th=[ 275], 99.95th=[ 284],
| 99.99th=[ 284]
bw ( KiB/s): min=460800, max=21111436, per=99.96%, avg=663934.43, stdev=263195.49, samples=1614
iops : min= 450, max=20616, avg=648.37, stdev=257.02, samples=1614
lat (usec) : 2=0.01%, 4=0.01%
lat (msec) : 2=0.01%, 4=0.17%, 10=7.79%, 20=0.94%, 50=1.16%
lat (msec) : 100=1.21%, 250=87.19%, 500=1.54%
cpu : usr=0.40%, sys=1.93%, ctx=204834, majf=0, minf=44
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,131072,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=649MiB/s (680MB/s), 649MiB/s-649MiB/s (680MB/s-680MB/s), io=128GiB (137GB), run=202066-202066msec
And the topology:
This pool has the first RAIDZ2 being the A platters and the 2nd being the B platters (sorry didnāt feel like censoring serials).
Odd⦠The results are much more normal nowā¦
So now I add 5x18TB single spindle disks (to represent the original disks that are in the pool)
test: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
...
fio-3.33
Starting 4 processes
Jobs: 4 (f=4): [W(4)][100.0%][w=1088MiB/s][w=1088 IOPS][eta 00m:00s]
test: (groupid=0, jobs=4): err= 0: pid=1180111: Sun Dec 8 13:40:35 2024
write: IOPS=1208, BW=1208MiB/s (1267MB/s)(128GiB/108471msec); 0 zone resets
slat (usec): min=76, max=6119, avg=3308.72, stdev=1081.70
clat (usec): min=2, max=141948, avg=102610.17, stdev=33392.99
lat (msec): min=2, max=146, avg=105.92, stdev=34.47
clat percentiles (msec):
| 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 36], 20.00th=[ 107],
| 30.00th=[ 110], 40.00th=[ 111], 50.00th=[ 113], 60.00th=[ 114],
| 70.00th=[ 116], 80.00th=[ 118], 90.00th=[ 124], 95.00th=[ 129],
| 99.00th=[ 136], 99.50th=[ 138], 99.90th=[ 142], 99.95th=[ 142],
| 99.99th=[ 142]
bw ( MiB/s): min= 898, max=22152, per=100.00%, avg=1208.82, stdev=364.14, samples=864
iops : min= 898, max=22152, avg=1208.82, stdev=364.14, samples=864
lat (usec) : 4=0.01%
lat (msec) : 4=0.77%, 10=7.46%, 20=0.99%, 50=1.42%, 100=2.55%
lat (msec) : 250=86.81%
cpu : usr=0.73%, sys=3.45%, ctx=294117, majf=0, minf=45
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,131072,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=1208MiB/s (1267MB/s), 1208MiB/s-1208MiB/s (1267MB/s-1267MB/s), io=128GiB (137GB), run=108471-108471msec
So pretty good and in the ball park for what Iām expecting, but there are some configuration that I can make some really strange results.
If I decrease the number of threads or iodepth or block size I think it starts caching to ARC in RAM or something.
So sequential async performance looks good.
4k random writes are the real killer:
fio --filename=test --direct=1 --rw=randwrite --bs=4k --numjobs=4 --iodepth=32 --group_reporting --name=test --filesize=4G --ioengine=libaio
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
fio-3.33
Starting 4 processes
Jobs: 4 (f=4): [w(4)][99.7%][w=43.5MiB/s][w=11.1k IOPS][eta 00m:03s]
test: (groupid=0, jobs=4): err= 0: pid=1203579: Sun Dec 8 14:42:38 2024
write: IOPS=4104, BW=16.0MiB/s (16.8MB/s)(16.0GiB/1021793msec); 0 zone resets
slat (usec): min=2, max=22150, avg=973.45, stdev=306.65
clat (nsec): min=822, max=65892k, avg=30206855.11, stdev=9460331.30
lat (usec): min=92, max=67966, avg=31180.30, stdev=9765.12
clat percentiles (usec):
| 1.00th=[ 5669], 5.00th=[14222], 10.00th=[19268], 20.00th=[21890],
| 30.00th=[24511], 40.00th=[27395], 50.00th=[30540], 60.00th=[33424],
| 70.00th=[36439], 80.00th=[39584], 90.00th=[42206], 95.00th=[43779],
| 99.00th=[46924], 99.50th=[48497], 99.90th=[57410], 99.95th=[60031],
| 99.99th=[63177]
bw ( KiB/s): min= 8144, max=245752, per=99.98%, avg=16416.60, stdev=1992.75, samples=8172
iops : min= 2036, max=61438, avg=4104.12, stdev=498.19, samples=8172
lat (nsec) : 1000=0.01%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.17%, 750=0.23%, 1000=0.15%
lat (msec) : 2=0.05%, 4=0.03%, 10=2.12%, 20=9.37%, 50=87.56%
lat (msec) : 100=0.33%
cpu : usr=0.12%, sys=1.35%, ctx=4182164, majf=0, minf=44
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4194304,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=16.0MiB/s (16.8MB/s), 16.0MiB/s-16.0MiB/s (16.8MB/s-16.8MB/s), io=16.0GiB (17.2GB), run=1021793-1021793msec
Here is the performance of just 4 of the disks on their own in a striped mirror:
fio --filename=test --direct=1 --rw=randwrite --bs=4k --numjobs=4 --iodepth=32 --group_reporting --name=test --filesize=4G --ioengine=libaio
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
fio-3.33
Starting 4 processes
Jobs: 4 (f=4): [w(4)][99.7%][w=43.5MiB/s][w=11.1k IOPS][eta 00m:03s]
test: (groupid=0, jobs=4): err= 0: pid=1203579: Sun Dec 8 14:42:38 2024
write: IOPS=4104, BW=16.0MiB/s (16.8MB/s)(16.0GiB/1021793msec); 0 zone resets
slat (usec): min=2, max=22150, avg=973.45, stdev=306.65
clat (nsec): min=822, max=65892k, avg=30206855.11, stdev=9460331.30
lat (usec): min=92, max=67966, avg=31180.30, stdev=9765.12
clat percentiles (usec):
| 1.00th=[ 5669], 5.00th=[14222], 10.00th=[19268], 20.00th=[21890],
| 30.00th=[24511], 40.00th=[27395], 50.00th=[30540], 60.00th=[33424],
| 70.00th=[36439], 80.00th=[39584], 90.00th=[42206], 95.00th=[43779],
| 99.00th=[46924], 99.50th=[48497], 99.90th=[57410], 99.95th=[60031],
| 99.99th=[63177]
bw ( KiB/s): min= 8144, max=245752, per=99.98%, avg=16416.60, stdev=1992.75, samples=8172
iops : min= 2036, max=61438, avg=4104.12, stdev=498.19, samples=8172
lat (nsec) : 1000=0.01%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.17%, 750=0.23%, 1000=0.15%
lat (msec) : 2=0.05%, 4=0.03%, 10=2.12%, 20=9.37%, 50=87.56%
lat (msec) : 100=0.33%
cpu : usr=0.12%, sys=1.35%, ctx=4182164, majf=0, minf=44
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4194304,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=16.0MiB/s (16.8MB/s), 16.0MiB/s-16.0MiB/s (16.8MB/s-16.8MB/s), io=16.0GiB (17.2GB), run=1021793-1021793msec
Does this 4k performance seem reasonable?