The disks do all have very similar direct fio numbers, direct disk 4k-1024k numbers
test_seq_mix_rw: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
test_seq_mix_rw: (groupid=0, jobs=1): err= 0: pid=3782475: Wed Sep 22 10:11:37 2021
read: IOPS=99.7k, BW=389MiB/s (408MB/s)(512MiB/1315msec)
slat (nsec): min=1953, max=36419, avg=3153.28, stdev=1499.19
clat (usec): min=10, max=2272, avg=192.07, stdev=196.03
lat (usec): min=14, max=2274, avg=195.35, stdev=195.97
clat percentiles (usec):
| 1.00th=[ 52], 5.00th=[ 116], 10.00th=[ 137], 20.00th=[ 147],
| 30.00th=[ 151], 40.00th=[ 153], 50.00th=[ 155], 60.00th=[ 161],
| 70.00th=[ 178], 80.00th=[ 196], 90.00th=[ 249], 95.00th=[ 306],
| 99.00th=[ 1778], 99.50th=[ 2008], 99.90th=[ 2147], 99.95th=[ 2180],
| 99.99th=[ 2245]
bw ( KiB/s): min=376840, max=412136, per=98.97%, avg=394488.00, stdev=24958.04, samples=2
iops : min=94210, max=103034, avg=98622.00, stdev=6239.51, samples=2
write: IOPS=99.7k, BW=389MiB/s (408MB/s)(512MiB/1315msec); 0 zone resets
slat (nsec): min=2024, max=35157, avg=3555.56, stdev=1617.24
clat (usec): min=8, max=317, avg=120.72, stdev=34.81
lat (usec): min=12, max=322, avg=124.42, stdev=34.96
clat percentiles (usec):
| 1.00th=[ 17], 5.00th=[ 43], 10.00th=[ 65], 20.00th=[ 96],
| 30.00th=[ 117], 40.00th=[ 127], 50.00th=[ 133], 60.00th=[ 139],
| 70.00th=[ 143], 80.00th=[ 147], 90.00th=[ 153], 95.00th=[ 155],
| 99.00th=[ 165], 99.50th=[ 167], 99.90th=[ 202], 99.95th=[ 221],
| 99.99th=[ 277]
bw ( KiB/s): min=374392, max=413312, per=98.76%, avg=393852.00, stdev=27520.60, samples=2
iops : min=93598, max=103328, avg=98463.00, stdev=6880.15, samples=2
lat (usec) : 10=0.01%, 20=1.09%, 50=2.63%, 100=8.89%, 250=82.43%
lat (usec) : 500=4.25%, 750=0.12%, 1000=0.05%
lat (msec) : 2=0.28%, 4=0.25%
cpu : usr=33.87%, sys=61.72%, ctx=1054, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=131040,131104,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=389MiB/s (408MB/s), 389MiB/s-389MiB/s (408MB/s-408MB/s), io=512MiB (537MB), run=1315-1315msec
WRITE: bw=389MiB/s (408MB/s), 389MiB/s-389MiB/s (408MB/s-408MB/s), io=512MiB (537MB), run=1315-1315msec
Disk stats (read/write):
nvme0n1: ios=113936/113631, merge=0/0, ticks=9782/1291, in_queue=11073, util=92.78%
test_seq_mix_rw: (g=0): rw=rw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
test_seq_mix_rw: (groupid=0, jobs=1): err= 0: pid=3782616: Wed Sep 22 10:11:43 2021
read: IOPS=77.7k, BW=607MiB/s (636MB/s)(513MiB/846msec)
slat (nsec): min=2375, max=55906, avg=3743.51, stdev=1649.55
clat (usec): min=10, max=2424, avg=285.78, stdev=352.78
lat (usec): min=15, max=2428, avg=289.65, stdev=352.67
clat percentiles (usec):
| 1.00th=[ 37], 5.00th=[ 87], 10.00th=[ 129], 20.00th=[ 163],
| 30.00th=[ 169], 40.00th=[ 174], 50.00th=[ 182], 60.00th=[ 208],
| 70.00th=[ 255], 80.00th=[ 318], 90.00th=[ 420], 95.00th=[ 510],
| 99.00th=[ 2114], 99.50th=[ 2147], 99.90th=[ 2245], 99.95th=[ 2278],
| 99.99th=[ 2343]
bw ( KiB/s): min=580448, max=580448, per=93.41%, avg=580448.00, stdev= 0.00, samples=1
iops : min=72556, max=72556, avg=72556.00, stdev= 0.00, samples=1
write: IOPS=77.3k, BW=604MiB/s (633MB/s)(511MiB/846msec); 0 zone resets
slat (nsec): min=2474, max=35828, avg=4404.78, stdev=1764.10
clat (usec): min=9, max=595, avg=117.12, stdev=53.01
lat (usec): min=14, max=604, avg=121.66, stdev=53.17
clat percentiles (usec):
| 1.00th=[ 15], 5.00th=[ 23], 10.00th=[ 38], 20.00th=[ 62],
| 30.00th=[ 86], 40.00th=[ 108], 50.00th=[ 130], 60.00th=[ 149],
| 70.00th=[ 159], 80.00th=[ 169], 90.00th=[ 174], 95.00th=[ 176],
| 99.00th=[ 196], 99.50th=[ 233], 99.90th=[ 322], 99.95th=[ 375],
| 99.99th=[ 469]
bw ( KiB/s): min=581680, max=581680, per=94.12%, avg=581680.00, stdev= 0.00, samples=1
iops : min=72710, max=72710, avg=72710.00, stdev= 0.00, samples=1
lat (usec) : 10=0.01%, 20=2.23%, 50=7.00%, 100=12.03%, 250=63.20%
lat (usec) : 500=12.85%, 750=0.77%, 1000=0.06%
lat (msec) : 2=0.82%, 4=1.04%
cpu : usr=31.36%, sys=54.20%, ctx=1925, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=65713,65359,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=607MiB/s (636MB/s), 607MiB/s-607MiB/s (636MB/s-636MB/s), io=513MiB (538MB), run=846-846msec
WRITE: bw=604MiB/s (633MB/s), 604MiB/s-604MiB/s (633MB/s-633MB/s), io=511MiB (535MB), run=846-846msec
Disk stats (read/write):
nvme0n1: ios=48887/48555, merge=0/0, ticks=8772/788, in_queue=9560, util=87.94%
test_seq_mix_rw: (g=0): rw=rw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
test_seq_mix_rw: (groupid=0, jobs=1): err= 0: pid=3782757: Wed Sep 22 10:11:50 2021
read: IOPS=56.2k, BW=877MiB/s (920MB/s)(512MiB/583msec)
slat (nsec): min=2766, max=64623, avg=4264.59, stdev=1883.07
clat (usec): min=13, max=2772, avg=427.73, stdev=573.49
lat (usec): min=18, max=2775, avg=432.13, stdev=573.35
clat percentiles (usec):
| 1.00th=[ 23], 5.00th=[ 53], 10.00th=[ 90], 20.00th=[ 155],
| 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 198], 60.00th=[ 225],
| 70.00th=[ 302], 80.00th=[ 469], 90.00th=[ 1598], 95.00th=[ 2057],
| 99.00th=[ 2212], 99.50th=[ 2278], 99.90th=[ 2409], 99.95th=[ 2606],
| 99.99th=[ 2769]
bw ( KiB/s): min=926880, max=926880, per=100.00%, avg=926880.00, stdev= 0.00, samples=1
iops : min=57930, max=57930, avg=57930.00, stdev= 0.00, samples=1
write: IOPS=56.3k, BW=879MiB/s (922MB/s)(512MiB/583msec); 0 zone resets
slat (nsec): min=2835, max=43553, avg=5180.80, stdev=2009.20
clat (usec): min=5, max=1843, avg=130.07, stdev=75.49
lat (usec): min=17, max=1846, avg=135.39, stdev=75.55
clat percentiles (usec):
| 1.00th=[ 17], 5.00th=[ 22], 10.00th=[ 33], 20.00th=[ 59],
| 30.00th=[ 86], 40.00th=[ 113], 50.00th=[ 141], 60.00th=[ 159],
| 70.00th=[ 174], 80.00th=[ 186], 90.00th=[ 192], 95.00th=[ 204],
| 99.00th=[ 392], 99.50th=[ 465], 99.90th=[ 586], 99.95th=[ 627],
| 99.99th=[ 1156]
bw ( KiB/s): min=929024, max=929024, per=100.00%, avg=929024.00, stdev= 0.00, samples=1
iops : min=58064, max=58064, avg=58064.00, stdev= 0.00, samples=1
lat (usec) : 10=0.01%, 20=2.42%, 50=8.17%, 100=12.86%, 250=57.93%
lat (usec) : 500=9.36%, 750=3.28%, 1000=0.28%
lat (msec) : 2=2.38%, 4=3.31%
cpu : usr=27.15%, sys=42.44%, ctx=1748, majf=0, minf=14
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=32737,32799,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=877MiB/s (920MB/s), 877MiB/s-877MiB/s (920MB/s-920MB/s), io=512MiB (536MB), run=583-583msec
WRITE: bw=879MiB/s (922MB/s), 879MiB/s-879MiB/s (922MB/s-922MB/s), io=512MiB (537MB), run=583-583msec
Disk stats (read/write):
nvme0n1: ios=22036/21938, merge=0/0, ticks=6864/704, in_queue=7568, util=82.26%
test_seq_mix_rw: (g=0): rw=rw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
test_seq_mix_rw: (groupid=0, jobs=1): err= 0: pid=3782898: Wed Sep 22 10:11:57 2021
read: IOPS=38.8k, BW=1214MiB/s (1273MB/s)(512MiB/422msec)
slat (nsec): min=2534, max=73940, avg=4650.70, stdev=2095.33
clat (usec): min=20, max=2825, avg=690.39, stdev=711.76
lat (usec): min=25, max=2829, avg=695.18, stdev=711.60
clat percentiles (usec):
| 1.00th=[ 34], 5.00th=[ 75], 10.00th=[ 111], 20.00th=[ 182],
| 30.00th=[ 225], 40.00th=[ 281], 50.00th=[ 371], 60.00th=[ 519],
| 70.00th=[ 668], 80.00th=[ 1221], 90.00th=[ 2114], 95.00th=[ 2212],
| 99.00th=[ 2343], 99.50th=[ 2409], 99.90th=[ 2540], 99.95th=[ 2638],
| 99.99th=[ 2802]
write: IOPS=38.8k, BW=1213MiB/s (1272MB/s)(512MiB/422msec); 0 zone resets
slat (nsec): min=2615, max=42871, avg=5712.32, stdev=2388.70
clat (usec): min=16, max=2203, avg=120.68, stdev=133.91
lat (usec): min=21, max=2209, avg=126.51, stdev=133.95
clat percentiles (usec):
| 1.00th=[ 22], 5.00th=[ 24], 10.00th=[ 28], 20.00th=[ 38],
| 30.00th=[ 52], 40.00th=[ 68], 50.00th=[ 84], 60.00th=[ 102],
| 70.00th=[ 125], 80.00th=[ 155], 90.00th=[ 219], 95.00th=[ 429],
| 99.00th=[ 652], 99.50th=[ 709], 99.90th=[ 1045], 99.95th=[ 1123],
| 99.99th=[ 2180]
lat (usec) : 20=0.12%, 50=15.68%, 100=17.97%, 250=29.50%, 500=14.36%
lat (usec) : 750=9.06%, 1000=2.32%
lat (msec) : 2=4.05%, 4=6.94%
cpu : usr=17.81%, sys=34.92%, ctx=2933, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=16390,16378,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=1214MiB/s (1273MB/s), 1214MiB/s-1214MiB/s (1273MB/s-1273MB/s), io=512MiB (537MB), run=422-422msec
WRITE: bw=1213MiB/s (1272MB/s), 1213MiB/s-1213MiB/s (1272MB/s-1272MB/s), io=512MiB (537MB), run=422-422msec
Disk stats (read/write):
nvme0n1: ios=14858/14766, merge=0/0, ticks=9883/962, in_queue=10844, util=82.26%
test_seq_mix_rw: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
test_seq_mix_rw: (groupid=0, jobs=1): err= 0: pid=3783039: Wed Sep 22 10:12:07 2021
read: IOPS=23.8k, BW=1487MiB/s (1559MB/s)(510MiB/343msec)
slat (nsec): min=4528, max=37170, avg=6283.33, stdev=2047.06
clat (usec): min=37, max=3490, avg=1239.57, stdev=622.83
lat (usec): min=42, max=3495, avg=1246.00, stdev=622.68
clat percentiles (usec):
| 1.00th=[ 153], 5.00th=[ 424], 10.00th=[ 537], 20.00th=[ 644],
| 30.00th=[ 750], 40.00th=[ 898], 50.00th=[ 1123], 60.00th=[ 1385],
| 70.00th=[ 1696], 80.00th=[ 1926], 90.00th=[ 2114], 95.00th=[ 2212],
| 99.00th=[ 2474], 99.50th=[ 2573], 99.90th=[ 3261], 99.95th=[ 3359],
| 99.99th=[ 3490]
write: IOPS=23.0k, BW=1499MiB/s (1571MB/s)(514MiB/343msec); 0 zone resets
slat (nsec): min=4578, max=58120, avg=8480.63, stdev=2813.91
clat (usec): min=26, max=832, avg=84.97, stdev=67.67
lat (usec): min=34, max=840, avg=93.59, stdev=67.84
clat percentiles (usec):
| 1.00th=[ 31], 5.00th=[ 32], 10.00th=[ 34], 20.00th=[ 40],
| 30.00th=[ 48], 40.00th=[ 56], 50.00th=[ 67], 60.00th=[ 78],
| 70.00th=[ 94], 80.00th=[ 115], 90.00th=[ 147], 95.00th=[ 194],
| 99.00th=[ 412], 99.50th=[ 474], 99.90th=[ 529], 99.95th=[ 586],
| 99.99th=[ 832]
lat (usec) : 50=17.29%, 100=19.84%, 250=12.13%, 500=4.68%, 750=11.24%
lat (usec) : 1000=7.31%
lat (msec) : 2=19.54%, 4=7.97%
cpu : usr=16.96%, sys=28.95%, ctx=4714, majf=0, minf=15
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.8%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=8160,8224,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=1487MiB/s (1559MB/s), 1487MiB/s-1487MiB/s (1559MB/s-1559MB/s), io=510MiB (535MB), run=343-343msec
WRITE: bw=1499MiB/s (1571MB/s), 1499MiB/s-1499MiB/s (1571MB/s-1571MB/s), io=514MiB (539MB), run=343-343msec
Disk stats (read/write):
nvme0n1: ios=3808/3909, merge=0/0, ticks=4035/321, in_queue=4356, util=63.16%
test_seq_mix_rw: (g=0): rw=rw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
test_seq_mix_rw: (groupid=0, jobs=1): err= 0: pid=3783181: Wed Sep 22 10:12:15 2021
read: IOPS=14.7k, BW=1834MiB/s (1923MB/s)(506MiB/276msec)
slat (nsec): min=6162, max=45607, avg=8454.63, stdev=1975.36
clat (usec): min=62, max=5533, avg=1836.18, stdev=793.24
lat (usec): min=70, max=5541, avg=1844.76, stdev=793.06
clat percentiles (usec):
| 1.00th=[ 343], 5.00th=[ 685], 10.00th=[ 922], 20.00th=[ 1205],
| 30.00th=[ 1401], 40.00th=[ 1582], 50.00th=[ 1745], 60.00th=[ 1926],
| 70.00th=[ 2180], 80.00th=[ 2409], 90.00th=[ 2769], 95.00th=[ 3163],
| 99.00th=[ 4490], 99.50th=[ 4883], 99.90th=[ 5473], 99.95th=[ 5473],
| 99.99th=[ 5538]
write: IOPS=15.0k, BW=1876MiB/s (1967MB/s)(518MiB/276msec); 0 zone resets
slat (nsec): min=6172, max=50576, avg=11992.60, stdev=3027.74
clat (usec): min=44, max=2834, avg=308.68, stdev=300.41
lat (usec): min=56, max=2846, avg=320.81, stdev=300.25
clat percentiles (usec):
| 1.00th=[ 50], 5.00th=[ 52], 10.00th=[ 56], 20.00th=[ 72],
| 30.00th=[ 97], 40.00th=[ 125], 50.00th=[ 167], 60.00th=[ 249],
| 70.00th=[ 396], 80.00th=[ 570], 90.00th=[ 791], 95.00th=[ 922],
| 99.00th=[ 1156], 99.50th=[ 1237], 99.90th=[ 1696], 99.95th=[ 2278],
| 99.99th=[ 2835]
lat (usec) : 50=1.09%, 100=15.27%, 250=14.40%, 500=9.29%, 750=7.59%
lat (usec) : 1000=7.39%
lat (msec) : 2=26.40%, 4=17.70%, 10=0.87%
cpu : usr=14.18%, sys=26.91%, ctx=4497, majf=0, minf=14
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=4050,4142,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=1834MiB/s (1923MB/s), 1834MiB/s-1834MiB/s (1923MB/s-1923MB/s), io=506MiB (531MB), run=276-276msec
WRITE: bw=1876MiB/s (1967MB/s), 1876MiB/s-1876MiB/s (1967MB/s-1967MB/s), io=518MiB (543MB), run=276-276msec
Disk stats (read/write):
nvme0n1: ios=2344/2418, merge=0/0, ticks=3505/959, in_queue=4464, util=61.54%
test_seq_mix_rw: (g=0): rw=rw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
test_seq_mix_rw: (groupid=0, jobs=1): err= 0: pid=3783322: Wed Sep 22 10:12:23 2021
read: IOPS=7488, BW=1872MiB/s (1963MB/s)(498MiB/266msec)
slat (nsec): min=9638, max=77687, avg=12177.98, stdev=3124.11
clat (usec): min=115, max=7210, avg=2423.52, stdev=1209.42
lat (usec): min=125, max=7221, avg=2435.79, stdev=1209.14
clat percentiles (usec):
| 1.00th=[ 510], 5.00th=[ 922], 10.00th=[ 1123], 20.00th=[ 1401],
| 30.00th=[ 1663], 40.00th=[ 1926], 50.00th=[ 2180], 60.00th=[ 2442],
| 70.00th=[ 2802], 80.00th=[ 3458], 90.00th=[ 4146], 95.00th=[ 4883],
| 99.00th=[ 5800], 99.50th=[ 5866], 99.90th=[ 7111], 99.95th=[ 7242],
| 99.99th=[ 7242]
write: IOPS=7909, BW=1977MiB/s (2073MB/s)(526MiB/266msec); 0 zone resets
slat (nsec): min=9878, max=79491, avg=18815.35, stdev=4286.38
clat (usec): min=86, max=6551, avg=1710.45, stdev=1105.67
lat (usec): min=97, max=6571, avg=1729.37, stdev=1105.86
clat percentiles (usec):
| 1.00th=[ 89], 5.00th=[ 153], 10.00th=[ 498], 20.00th=[ 889],
| 30.00th=[ 1139], 40.00th=[ 1319], 50.00th=[ 1565], 60.00th=[ 1795],
| 70.00th=[ 1975], 80.00th=[ 2278], 90.00th=[ 3163], 95.00th=[ 3818],
| 99.00th=[ 5604], 99.50th=[ 6259], 99.90th=[ 6390], 99.95th=[ 6390],
| 99.99th=[ 6521]
lat (usec) : 100=1.61%, 250=2.00%, 500=1.98%, 750=4.20%, 1000=5.91%
lat (msec) : 2=42.21%, 4=34.28%, 10=7.81%
cpu : usr=6.79%, sys=24.15%, ctx=2923, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.4%, 32=99.2%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=1992,2104,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=1872MiB/s (1963MB/s), 1872MiB/s-1872MiB/s (1963MB/s-1963MB/s), io=498MiB (522MB), run=266-266msec
WRITE: bw=1977MiB/s (2073MB/s), 1977MiB/s-1977MiB/s (2073MB/s-2073MB/s), io=526MiB (552MB), run=266-266msec
Disk stats (read/write):
nvme0n1: ios=1255/1275, merge=0/0, ticks=2516/1846, in_queue=4362, util=61.54%
test_seq_mix_rw: (g=0): rw=rw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
test_seq_mix_rw: (groupid=0, jobs=1): err= 0: pid=3783464: Wed Sep 22 10:12:32 2021
read: IOPS=3913, BW=1957MiB/s (2052MB/s)(497MiB/254msec)
slat (nsec): min=16762, max=76575, avg=22144.55, stdev=4027.76
clat (usec): min=357, max=12940, avg=4116.81, stdev=2289.31
lat (usec): min=378, max=12960, avg=4139.08, stdev=2288.96
clat percentiles (usec):
| 1.00th=[ 996], 5.00th=[ 1483], 10.00th=[ 1844], 20.00th=[ 2180],
| 30.00th=[ 2442], 40.00th=[ 2704], 50.00th=[ 3195], 60.00th=[ 4424],
| 70.00th=[ 5407], 80.00th=[ 6325], 90.00th=[ 7308], 95.00th=[ 8225],
| 99.00th=[10552], 99.50th=[11207], 99.90th=[12911], 99.95th=[12911],
| 99.99th=[12911]
write: IOPS=4149, BW=2075MiB/s (2176MB/s)(527MiB/254msec); 0 zone resets
slat (usec): min=22, max=102, avg=37.40, stdev= 6.35
clat (usec): min=175, max=8553, avg=3721.76, stdev=1466.17
lat (usec): min=218, max=8585, avg=3759.28, stdev=1466.76
clat percentiles (usec):
| 1.00th=[ 578], 5.00th=[ 1319], 10.00th=[ 1991], 20.00th=[ 2573],
| 30.00th=[ 2900], 40.00th=[ 3228], 50.00th=[ 3621], 60.00th=[ 4015],
| 70.00th=[ 4359], 80.00th=[ 4752], 90.00th=[ 5800], 95.00th=[ 6652],
| 99.00th=[ 7308], 99.50th=[ 7570], 99.90th=[ 7898], 99.95th=[ 8586],
| 99.99th=[ 8586]
lat (usec) : 250=0.05%, 500=0.39%, 750=1.03%, 1000=0.73%
lat (msec) : 2=10.01%, 4=46.14%, 10=40.97%, 20=0.68%
cpu : usr=5.14%, sys=23.72%, ctx=1577, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=0.8%, 32=98.5%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=994,1054,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=1957MiB/s (2052MB/s), 1957MiB/s-1957MiB/s (2052MB/s-2052MB/s), io=497MiB (521MB), run=254-254msec
WRITE: bw=2075MiB/s (2176MB/s), 2075MiB/s-2075MiB/s (2176MB/s-2176MB/s), io=527MiB (553MB), run=254-254msec
Disk stats (read/write):
nvme0n1: ios=672/653, merge=0/0, ticks=1670/2654, in_queue=4323, util=58.30%
test_seq_mix_rw: (g=0): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
test_seq_mix_rw: (groupid=0, jobs=1): err= 0: pid=3783605: Wed Sep 22 10:12:39 2021
read: IOPS=1373, BW=1374MiB/s (1441MB/s)(496MiB/361msec)
slat (nsec): min=33874, max=91533, avg=40402.20, stdev=6347.61
clat (usec): min=2450, max=29871, avg=20530.36, stdev=4273.93
lat (usec): min=2543, max=29909, avg=20570.89, stdev=4271.85
clat percentiles (usec):
| 1.00th=[ 3687], 5.00th=[12125], 10.00th=[15270], 20.00th=[18220],
| 30.00th=[19268], 40.00th=[20317], 50.00th=[21103], 60.00th=[21890],
| 70.00th=[22938], 80.00th=[23987], 90.00th=[24773], 95.00th=[25297],
| 99.00th=[28705], 99.50th=[29492], 99.90th=[29754], 99.95th=[29754],
| 99.99th=[29754]
write: IOPS=1462, BW=1463MiB/s (1534MB/s)(528MiB/361msec); 0 zone resets
slat (usec): min=34, max=116, avg=65.60, stdev=13.26
clat (usec): min=298, max=21950, avg=2348.68, stdev=4184.52
lat (usec): min=348, max=22030, avg=2414.42, stdev=4188.55
clat percentiles (usec):
| 1.00th=[ 302], 5.00th=[ 310], 10.00th=[ 310], 20.00th=[ 314],
| 30.00th=[ 318], 40.00th=[ 322], 50.00th=[ 392], 60.00th=[ 603],
| 70.00th=[ 1029], 80.00th=[ 3785], 90.00th=[ 6980], 95.00th=[13304],
| 99.00th=[18744], 99.50th=[21103], 99.90th=[21890], 99.95th=[21890],
| 99.99th=[21890]
lat (usec) : 500=27.44%, 750=7.03%, 1000=1.37%
lat (msec) : 2=2.44%, 4=4.10%, 10=6.05%, 20=21.09%, 50=30.47%
cpu : usr=8.89%, sys=8.89%, ctx=910, majf=0, minf=14
IO depths : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=1.6%, 32=97.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=496,528,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=1374MiB/s (1441MB/s), 1374MiB/s-1374MiB/s (1441MB/s-1441MB/s), io=496MiB (520MB), run=361-361msec
WRITE: bw=1463MiB/s (1534MB/s), 1463MiB/s-1463MiB/s (1534MB/s-1534MB/s), io=528MiB (554MB), run=361-361msec
Disk stats (read/write):
nvme0n1: ios=292/298, merge=0/0, ticks=4039/1666, in_queue=5705, util=55.06%
As far as my postgres test, I was just restoring a database backup. I just did a time
to check how long the restores were.
mdraid
---------------------------
real 1112m16.401s
user 114m47.432s
sys 16m19.817s
zfs 8k
---------------------
real 1262m27.236s
user 116m14.397s
sys 13m49.149s
By comparison, on the old system with MDRAID+LUKS, and the 8x 850 EVO drives
md + luks
real 550m43.996s
user 151m13.206s
sys 26m21.459s