Yes. I just re-ran the AIDA64 disk benchmarks and the results still hold.
Unless noted, everything was connected via the CPU PCIe 5.0 lanes.
Even with Optane media’s ridiculous endurance, I’d rather not. 
Here are some more comprehensive random read performance numbers using fio. They cover four models:
- P1600X 118GB (SSDPEK1A118GA)
- 905P 380GB (SSDPEL1D380GA)
- P5800X 3.2TB (SSDPF21Q032TB)
- P4800X 1.5TB (MDTPE21K015TA)
The tests are run using combinations of two block sizes (512 or 4,096 bytes) and four I/O depths (1, 4, 16, or 64). All the NVMe namespaces are configured with 512-byte logical sector sizes with no metadata.
The command:
# for device in $(find /dev -maxdepth 1 -regextype posix-extended -regex '^/dev/nvme(0|[1-9]\d*)$' | sort); do data=$(nvme id-ctrl "$device"); mn=$(echo "$data" | awk 'match($0, "^mn\\s*:\\s*(.+?)$", m) {print m[1]}' | xargs); for namespace in $(find /dev -maxdepth 1 -regextype posix-extended -regex "^$device"'n[1-9]\d*$' | sort); do for blocksize in 512 4k; do for depth in 1 4 16 64; do echo -e "\033[0;32mRandom read of device \033[1;32m$device ($mn)\033[0;32m at I/O depth $depth with block size $blocksize\033[0m"; fio --filename="$namespace" --direct=1 --rw=randread --readonly --ioengine=pvsync2 --runtime=60 --bs="$blocksize" --iodepth="$depth" --numjobs=1 --time_based --group_reporting --name=iops-test-job; echo; done; done; done; done
Enjoy. 
Output
Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) at I/O depth 1 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=41.8MiB/s][r=85.5k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8106: Sat Feb 25 14:45:10 2023
read: IOPS=85.9k, BW=42.0MiB/s (44.0MB/s)(2518MiB/60001msec)
clat (usec): min=4, max=379, avg=11.40, stdev= 1.24
lat (usec): min=4, max=379, avg=11.43, stdev= 1.24
clat percentiles (nsec):
| 1.00th=[ 5472], 5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
| 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
| 70.00th=[11456], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
| 99.00th=[11968], 99.50th=[15168], 99.90th=[16768], 99.95th=[18816],
| 99.99th=[27776]
bw ( KiB/s): min=40647, max=67016, per=100.00%, avg=42985.08, stdev=2234.87, samples=119
iops : min=81294, max=134032, avg=85970.15, stdev=4469.74, samples=119
lat (usec) : 10=1.58%, 20=98.38%, 50=0.03%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%
cpu : usr=3.51%, sys=11.68%, ctx=5156668, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5156680,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=42.0MiB/s (44.0MB/s), 42.0MiB/s-42.0MiB/s (44.0MB/s-44.0MB/s), io=2518MiB (2640MB), run=60001-60001msec
Disk stats (read/write):
nvme0n1: ios=5147663/0, merge=0/0, ticks=52356/0, in_queue=52356, util=99.85%
Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) at I/O depth 4 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=4
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=41.7MiB/s][r=85.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8142: Sat Feb 25 14:46:10 2023
read: IOPS=85.5k, BW=41.8MiB/s (43.8MB/s)(2506MiB/60001msec)
clat (usec): min=4, max=657, avg=11.48, stdev= 1.07
lat (usec): min=4, max=657, avg=11.50, stdev= 1.07
clat percentiles (nsec):
| 1.00th=[10816], 5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
| 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
| 70.00th=[11456], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
| 99.00th=[11840], 99.50th=[15040], 99.90th=[16064], 99.95th=[18048],
| 99.99th=[24192]
bw ( KiB/s): min=42571, max=43785, per=100.00%, avg=42781.55, stdev=140.18, samples=119
iops : min=85142, max=87570, avg=85563.13, stdev=280.35, samples=119
lat (usec) : 10=0.56%, 20=99.41%, 50=0.03%, 100=0.01%, 250=0.01%
lat (usec) : 750=0.01%
cpu : usr=2.98%, sys=11.90%, ctx=5132399, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5132420,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=41.8MiB/s (43.8MB/s), 41.8MiB/s-41.8MiB/s (43.8MB/s-43.8MB/s), io=2506MiB (2628MB), run=60001-60001msec
Disk stats (read/write):
nvme0n1: ios=5123326/12, merge=0/2, ticks=52621/1, in_queue=52623, util=99.86%
Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) at I/O depth 16 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=16
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=41.8MiB/s][r=85.6k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8179: Sat Feb 25 14:47:10 2023
read: IOPS=85.6k, BW=41.8MiB/s (43.8MB/s)(2508MiB/60001msec)
clat (usec): min=4, max=895, avg=11.48, stdev= 1.14
lat (usec): min=4, max=895, avg=11.50, stdev= 1.14
clat percentiles (nsec):
| 1.00th=[10560], 5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
| 30.00th=[11456], 40.00th=[11456], 50.00th=[11584], 60.00th=[11584],
| 70.00th=[11584], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
| 99.00th=[11968], 99.50th=[14016], 99.90th=[16064], 99.95th=[16192],
| 99.99th=[23168]
bw ( KiB/s): min=42631, max=50069, per=100.00%, avg=42822.87, stdev=670.75, samples=119
iops : min=85262, max=100138, avg=85645.75, stdev=1341.50, samples=119
lat (usec) : 10=0.69%, 20=99.29%, 50=0.02%, 100=0.01%, 250=0.01%
lat (usec) : 1000=0.01%
cpu : usr=3.06%, sys=12.20%, ctx=5137331, majf=0, minf=14
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5137350,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=41.8MiB/s (43.8MB/s), 41.8MiB/s-41.8MiB/s (43.8MB/s-43.8MB/s), io=2508MiB (2630MB), run=60001-60001msec
Disk stats (read/write):
nvme0n1: ios=5128190/9, merge=0/23, ticks=52427/1, in_queue=52428, util=99.85%
Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) at I/O depth 64 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=41.8MiB/s][r=85.5k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8226: Sat Feb 25 14:48:10 2023
read: IOPS=85.5k, BW=41.8MiB/s (43.8MB/s)(2506MiB/60001msec)
clat (usec): min=4, max=280, avg=11.47, stdev= 1.01
lat (usec): min=4, max=280, avg=11.49, stdev= 1.01
clat percentiles (nsec):
| 1.00th=[10688], 5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
| 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
| 70.00th=[11456], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
| 99.00th=[11840], 99.50th=[15168], 99.90th=[16064], 99.95th=[18816],
| 99.99th=[28032]
bw ( KiB/s): min=42626, max=45012, per=100.00%, avg=42778.61, stdev=210.98, samples=119
iops : min=85252, max=90026, avg=85557.24, stdev=422.14, samples=119
lat (usec) : 10=0.57%, 20=99.38%, 50=0.04%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%
cpu : usr=3.15%, sys=11.77%, ctx=5132105, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5132129,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=41.8MiB/s (43.8MB/s), 41.8MiB/s-41.8MiB/s (43.8MB/s-43.8MB/s), io=2506MiB (2628MB), run=60001-60001msec
Disk stats (read/write):
nvme0n1: ios=5123033/12, merge=0/3, ticks=52562/1, in_queue=52563, util=99.85%
Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) at I/O depth 1 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=334MiB/s][r=85.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8261: Sat Feb 25 14:49:11 2023
read: IOPS=85.5k, BW=334MiB/s (350MB/s)(19.6GiB/60001msec)
clat (nsec): min=6161, max=234810, avg=11534.74, stdev=992.08
lat (nsec): min=6181, max=234850, avg=11551.71, stdev=992.18
clat percentiles (nsec):
| 1.00th=[10048], 5.00th=[11328], 10.00th=[11456], 20.00th=[11456],
| 30.00th=[11584], 40.00th=[11584], 50.00th=[11584], 60.00th=[11584],
| 70.00th=[11584], 80.00th=[11584], 90.00th=[11712], 95.00th=[11712],
| 99.00th=[12480], 99.50th=[15296], 99.90th=[16064], 99.95th=[18816],
| 99.99th=[27776]
bw ( KiB/s): min=340848, max=401144, per=100.00%, avg=342250.49, stdev=5536.04, samples=119
iops : min=85212, max=100286, avg=85562.64, stdev=1384.01, samples=119
lat (usec) : 10=0.99%, 20=98.97%, 50=0.04%, 100=0.01%, 250=0.01%
cpu : usr=2.91%, sys=11.57%, ctx=5132332, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5132358,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=334MiB/s (350MB/s), 334MiB/s-334MiB/s (350MB/s-350MB/s), io=19.6GiB (21.0GB), run=60001-60001msec
Disk stats (read/write):
nvme0n1: ios=5123652/3, merge=0/7, ticks=52849/0, in_queue=52850, util=99.87%
Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) at I/O depth 4 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=4
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=333MiB/s][r=85.3k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8297: Sat Feb 25 14:50:11 2023
read: IOPS=85.3k, BW=333MiB/s (350MB/s)(19.5GiB/60001msec)
clat (nsec): min=6251, max=292708, avg=11566.22, stdev=944.08
lat (nsec): min=6272, max=292728, avg=11583.96, stdev=944.35
clat percentiles (nsec):
| 1.00th=[11200], 5.00th=[11328], 10.00th=[11456], 20.00th=[11456],
| 30.00th=[11584], 40.00th=[11584], 50.00th=[11584], 60.00th=[11584],
| 70.00th=[11584], 80.00th=[11584], 90.00th=[11712], 95.00th=[11712],
| 99.00th=[12352], 99.50th=[14656], 99.90th=[16064], 99.95th=[16320],
| 99.99th=[24448]
bw ( KiB/s): min=340944, max=341840, per=100.00%, avg=341454.32, stdev=258.79, samples=119
iops : min=85236, max=85460, avg=85363.60, stdev=64.72, samples=119
lat (usec) : 10=0.33%, 20=99.65%, 50=0.02%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%
cpu : usr=2.78%, sys=12.10%, ctx=5120444, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5120488,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=333MiB/s (350MB/s), 333MiB/s-333MiB/s (350MB/s-350MB/s), io=19.5GiB (21.0GB), run=60001-60001msec
Disk stats (read/write):
nvme0n1: ios=5111744/17, merge=0/25, ticks=52637/1, in_queue=52638, util=99.86%
Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) at I/O depth 16 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=16
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=334MiB/s][r=85.5k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8334: Sat Feb 25 14:51:11 2023
read: IOPS=85.5k, BW=334MiB/s (350MB/s)(19.6GiB/60001msec)
clat (nsec): min=6152, max=292407, avg=11548.75, stdev=961.55
lat (nsec): min=6171, max=292417, avg=11565.66, stdev=961.63
clat percentiles (nsec):
| 1.00th=[11200], 5.00th=[11328], 10.00th=[11456], 20.00th=[11456],
| 30.00th=[11584], 40.00th=[11584], 50.00th=[11584], 60.00th=[11584],
| 70.00th=[11584], 80.00th=[11584], 90.00th=[11712], 95.00th=[11712],
| 99.00th=[12480], 99.50th=[15296], 99.90th=[18048], 99.95th=[19072],
| 99.99th=[23680]
bw ( KiB/s): min=340600, max=378496, per=100.00%, avg=341900.30, stdev=3412.33, samples=119
iops : min=85150, max=94624, avg=85475.09, stdev=853.08, samples=119
lat (usec) : 10=0.74%, 20=99.23%, 50=0.03%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%
cpu : usr=2.62%, sys=11.81%, ctx=5127095, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5127117,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=334MiB/s (350MB/s), 334MiB/s-334MiB/s (350MB/s-350MB/s), io=19.6GiB (21.0GB), run=60001-60001msec
Disk stats (read/write):
nvme0n1: ios=5118363/10, merge=0/0, ticks=52635/0, in_queue=52635, util=99.86%
Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) at I/O depth 64 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=334MiB/s][r=85.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8369: Sat Feb 25 14:52:11 2023
read: IOPS=85.4k, BW=334MiB/s (350MB/s)(19.5GiB/60001msec)
clat (usec): min=6, max=966, avg=11.56, stdev= 1.01
lat (usec): min=6, max=966, avg=11.58, stdev= 1.01
clat percentiles (nsec):
| 1.00th=[11200], 5.00th=[11328], 10.00th=[11456], 20.00th=[11456],
| 30.00th=[11584], 40.00th=[11584], 50.00th=[11584], 60.00th=[11584],
| 70.00th=[11584], 80.00th=[11584], 90.00th=[11712], 95.00th=[11712],
| 99.00th=[12480], 99.50th=[15040], 99.90th=[16192], 99.95th=[18816],
| 99.99th=[23936]
bw ( KiB/s): min=340112, max=344384, per=100.00%, avg=341638.52, stdev=447.66, samples=119
iops : min=85028, max=86096, avg=85409.61, stdev=111.93, samples=119
lat (usec) : 10=0.49%, 20=99.48%, 50=0.03%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%, 1000=0.01%
cpu : usr=2.85%, sys=11.67%, ctx=5123108, majf=0, minf=14
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5123138,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=334MiB/s (350MB/s), 334MiB/s-334MiB/s (350MB/s-350MB/s), io=19.5GiB (21.0GB), run=60001-60001msec
Disk stats (read/write):
nvme0n1: ios=5114416/4, merge=0/16, ticks=52719/0, in_queue=52719, util=99.87%
Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) at I/O depth 1 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=41.2MiB/s][r=84.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8417: Sat Feb 25 14:53:12 2023
read: IOPS=84.3k, BW=41.2MiB/s (43.2MB/s)(2471MiB/60001msec)
clat (usec): min=6, max=275, avg=11.63, stdev= 1.68
lat (usec): min=6, max=275, avg=11.65, stdev= 1.68
clat percentiles (nsec):
| 1.00th=[11072], 5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
| 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
| 70.00th=[11456], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
| 99.00th=[19840], 99.50th=[20864], 99.90th=[36096], 99.95th=[39168],
| 99.99th=[41216]
bw ( KiB/s): min=42113, max=42245, per=100.00%, avg=42178.82, stdev=38.22, samples=119
iops : min=84226, max=84490, avg=84357.65, stdev=76.49, samples=119
lat (usec) : 10=0.14%, 20=98.98%, 50=0.88%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%
cpu : usr=3.67%, sys=11.31%, ctx=5060065, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5060056,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=41.2MiB/s (43.2MB/s), 41.2MiB/s-41.2MiB/s (43.2MB/s-43.2MB/s), io=2471MiB (2591MB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=5049070/0, merge=0/0, ticks=52393/0, in_queue=52393, util=99.82%
Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) at I/O depth 4 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=4
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=40.9MiB/s][r=83.7k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8452: Sat Feb 25 14:54:12 2023
read: IOPS=84.3k, BW=41.1MiB/s (43.1MB/s)(2469MiB/60001msec)
clat (usec): min=6, max=1913, avg=11.63, stdev= 2.55
lat (usec): min=6, max=1913, avg=11.65, stdev= 2.55
clat percentiles (nsec):
| 1.00th=[10816], 5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
| 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
| 70.00th=[11456], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
| 99.00th=[19840], 99.50th=[21632], 99.90th=[36608], 99.95th=[39168],
| 99.99th=[44288]
bw ( KiB/s): min=41374, max=42386, per=100.00%, avg=42147.93, stdev=129.34, samples=119
iops : min=82748, max=84772, avg=84295.87, stdev=258.69, samples=119
lat (usec) : 10=0.54%, 20=98.56%, 50=0.90%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%
lat (msec) : 2=0.01%
cpu : usr=3.38%, sys=11.77%, ctx=5055956, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5055965,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=41.1MiB/s (43.1MB/s), 41.1MiB/s-41.1MiB/s (43.1MB/s-43.1MB/s), io=2469MiB (2589MB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=5045792/0, merge=0/0, ticks=52354/0, in_queue=52354, util=99.83%
Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) at I/O depth 16 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=16
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=41.1MiB/s][r=84.3k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8490: Sat Feb 25 14:55:12 2023
read: IOPS=84.2k, BW=41.1MiB/s (43.1MB/s)(2465MiB/60001msec)
clat (usec): min=6, max=385, avg=11.64, stdev= 2.70
lat (usec): min=6, max=386, avg=11.66, stdev= 2.70
clat percentiles (usec):
| 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 12],
| 30.00th=[ 12], 40.00th=[ 12], 50.00th=[ 12], 60.00th=[ 12],
| 70.00th=[ 12], 80.00th=[ 12], 90.00th=[ 12], 95.00th=[ 12],
| 99.00th=[ 20], 99.50th=[ 22], 99.90th=[ 37], 99.95th=[ 40],
| 99.99th=[ 212]
bw ( KiB/s): min=41462, max=42352, per=100.00%, avg=42089.20, stdev=177.68, samples=119
iops : min=82924, max=84704, avg=84178.40, stdev=355.34, samples=119
lat (usec) : 10=0.40%, 20=98.69%, 50=0.89%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%
cpu : usr=3.42%, sys=11.38%, ctx=5049175, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5049177,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=41.1MiB/s (43.1MB/s), 41.1MiB/s-41.1MiB/s (43.1MB/s-43.1MB/s), io=2465MiB (2585MB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=5038924/0, merge=0/0, ticks=52528/0, in_queue=52528, util=99.84%
Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) at I/O depth 64 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=40.8MiB/s][r=83.6k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8525: Sat Feb 25 14:56:13 2023
read: IOPS=84.3k, BW=41.1MiB/s (43.1MB/s)(2468MiB/60001msec)
clat (usec): min=6, max=332, avg=11.62, stdev= 2.05
lat (usec): min=7, max=332, avg=11.64, stdev= 2.05
clat percentiles (nsec):
| 1.00th=[11072], 5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
| 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
| 70.00th=[11456], 80.00th=[11456], 90.00th=[11584], 95.00th=[11712],
| 99.00th=[19840], 99.50th=[21120], 99.90th=[36608], 99.95th=[39168],
| 99.99th=[41728]
bw ( KiB/s): min=41351, max=42234, per=100.00%, avg=42139.60, stdev=115.92, samples=119
iops : min=82702, max=84468, avg=84279.21, stdev=231.84, samples=119
lat (usec) : 10=0.21%, 20=98.90%, 50=0.89%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%
cpu : usr=3.69%, sys=11.58%, ctx=5055297, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5055300,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=41.1MiB/s (43.1MB/s), 41.1MiB/s-41.1MiB/s (43.1MB/s-43.1MB/s), io=2468MiB (2588MB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=5045050/0, merge=0/0, ticks=52271/0, in_queue=52271, util=99.84%
Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) at I/O depth 1 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=329MiB/s][r=84.2k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8560: Sat Feb 25 14:57:13 2023
read: IOPS=84.2k, BW=329MiB/s (345MB/s)(19.3GiB/60001msec)
clat (usec): min=8, max=1202, avg=11.66, stdev= 2.12
lat (usec): min=8, max=1202, avg=11.68, stdev= 2.12
clat percentiles (nsec):
| 1.00th=[11072], 5.00th=[11328], 10.00th=[11328], 20.00th=[11456],
| 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
| 70.00th=[11584], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
| 99.00th=[19840], 99.50th=[21888], 99.90th=[36608], 99.95th=[39168],
| 99.99th=[41728]
bw ( KiB/s): min=332344, max=337848, per=100.00%, avg=336867.40, stdev=1005.50, samples=119
iops : min=83086, max=84462, avg=84216.87, stdev=251.38, samples=119
lat (usec) : 10=0.17%, 20=98.90%, 50=0.93%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%
lat (msec) : 2=0.01%
cpu : usr=3.34%, sys=11.62%, ctx=5051521, majf=0, minf=14
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5051528,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=329MiB/s (345MB/s), 329MiB/s-329MiB/s (345MB/s-345MB/s), io=19.3GiB (20.7GB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=5042383/0, merge=0/0, ticks=52458/0, in_queue=52458, util=99.87%
Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) at I/O depth 4 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=4
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=328MiB/s][r=84.1k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8599: Sat Feb 25 14:58:13 2023
read: IOPS=84.1k, BW=329MiB/s (345MB/s)(19.3GiB/60001msec)
clat (usec): min=8, max=388, avg=11.66, stdev= 2.23
lat (usec): min=8, max=388, avg=11.68, stdev= 2.23
clat percentiles (nsec):
| 1.00th=[10944], 5.00th=[11328], 10.00th=[11328], 20.00th=[11456],
| 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
| 70.00th=[11584], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
| 99.00th=[19840], 99.50th=[21888], 99.90th=[36608], 99.95th=[39168],
| 99.99th=[43264]
bw ( KiB/s): min=328096, max=337816, per=100.00%, avg=336692.97, stdev=1433.86, samples=119
iops : min=82024, max=84454, avg=84173.24, stdev=358.50, samples=119
lat (usec) : 10=0.23%, 20=98.82%, 50=0.94%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%
cpu : usr=3.48%, sys=11.58%, ctx=5048664, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5048673,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=329MiB/s (345MB/s), 329MiB/s-329MiB/s (345MB/s-345MB/s), io=19.3GiB (20.7GB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=5039538/0, merge=0/0, ticks=52440/0, in_queue=52440, util=99.86%
Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) at I/O depth 16 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=16
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=329MiB/s][r=84.2k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8635: Sat Feb 25 14:59:13 2023
read: IOPS=84.2k, BW=329MiB/s (345MB/s)(19.3GiB/60001msec)
clat (usec): min=8, max=395, avg=11.66, stdev= 2.18
lat (usec): min=8, max=395, avg=11.68, stdev= 2.18
clat percentiles (nsec):
| 1.00th=[10944], 5.00th=[11328], 10.00th=[11328], 20.00th=[11456],
| 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
| 70.00th=[11584], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
| 99.00th=[19840], 99.50th=[21888], 99.90th=[36608], 99.95th=[39168],
| 99.99th=[42240]
bw ( KiB/s): min=331144, max=337832, per=100.00%, avg=336725.45, stdev=1154.46, samples=119
iops : min=82786, max=84458, avg=84181.38, stdev=288.63, samples=119
lat (usec) : 10=0.24%, 20=98.83%, 50=0.93%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%
cpu : usr=3.28%, sys=11.73%, ctx=5049526, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5049536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=329MiB/s (345MB/s), 329MiB/s-329MiB/s (345MB/s-345MB/s), io=19.3GiB (20.7GB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=5040365/0, merge=0/0, ticks=52441/0, in_queue=52441, util=99.86%
Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) at I/O depth 64 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=329MiB/s][r=84.3k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8671: Sat Feb 25 15:00:14 2023
read: IOPS=84.2k, BW=329MiB/s (345MB/s)(19.3GiB/60001msec)
clat (usec): min=8, max=318, avg=11.66, stdev= 2.03
lat (usec): min=8, max=318, avg=11.68, stdev= 2.04
clat percentiles (nsec):
| 1.00th=[11072], 5.00th=[11328], 10.00th=[11328], 20.00th=[11456],
| 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
| 70.00th=[11584], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
| 99.00th=[19840], 99.50th=[21888], 99.90th=[36608], 99.95th=[39168],
| 99.99th=[41728]
bw ( KiB/s): min=332704, max=337904, per=100.00%, avg=337005.24, stdev=899.24, samples=119
iops : min=83176, max=84476, avg=84251.33, stdev=224.80, samples=119
lat (usec) : 10=0.16%, 20=98.92%, 50=0.92%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%
cpu : usr=3.36%, sys=11.64%, ctx=5053532, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5053551,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=329MiB/s (345MB/s), 329MiB/s-329MiB/s (345MB/s-345MB/s), io=19.3GiB (20.7GB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=5044355/0, merge=0/0, ticks=52416/0, in_queue=52416, util=99.86%
Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) at I/O depth 1 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
Jobs: 1 (f=1): [r(1)][100.0%][r=41.7MiB/s][r=85.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8716: Sat Feb 25 15:01:14 2023
read: IOPS=85.3k, BW=41.7MiB/s (43.7MB/s)(2500MiB/60001msec)
clat (usec): min=4, max=354, avg=11.44, stdev= 2.10
lat (usec): min=4, max=354, avg=11.46, stdev= 2.10
clat percentiles (nsec):
| 1.00th=[10176], 5.00th=[11200], 10.00th=[11328], 20.00th=[11328],
| 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
| 70.00th=[11456], 80.00th=[11456], 90.00th=[11584], 95.00th=[11712],
| 99.00th=[12096], 99.50th=[16064], 99.90th=[20608], 99.95th=[20864],
| 99.99th=[53504]
bw ( KiB/s): min=40278, max=43410, per=100.00%, avg=42686.28, stdev=283.65, samples=119
iops : min=80556, max=86822, avg=85372.57, stdev=567.34, samples=119
lat (usec) : 10=0.96%, 20=98.69%, 50=0.34%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%
cpu : usr=4.12%, sys=11.94%, ctx=5120865, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5120860,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=41.7MiB/s (43.7MB/s), 41.7MiB/s-41.7MiB/s (43.7MB/s-43.7MB/s), io=2500MiB (2622MB), run=60001-60001msec
Disk stats (read/write):
nvme2n1: ios=5100271/0, merge=0/0, ticks=51705/0, in_queue=51705, util=99.61%
Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) at I/O depth 4 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=4
fio-3.28
Starting 1 process
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
Jobs: 1 (f=1): [r(1)][100.0%][r=42.0MiB/s][r=86.0k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8763: Sat Feb 25 15:02:15 2023
read: IOPS=85.2k, BW=41.6MiB/s (43.6MB/s)(2495MiB/60001msec)
clat (usec): min=4, max=1131, avg=11.45, stdev= 2.75
lat (usec): min=4, max=1131, avg=11.47, stdev= 2.75
clat percentiles (usec):
| 1.00th=[ 6], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 12],
| 30.00th=[ 12], 40.00th=[ 12], 50.00th=[ 12], 60.00th=[ 12],
| 70.00th=[ 12], 80.00th=[ 12], 90.00th=[ 12], 95.00th=[ 12],
| 99.00th=[ 16], 99.50th=[ 20], 99.90th=[ 22], 99.95th=[ 26],
| 99.99th=[ 215]
bw ( KiB/s): min=41126, max=44395, per=100.00%, avg=42602.17, stdev=436.29, samples=119
iops : min=82252, max=88790, avg=85204.34, stdev=872.57, samples=119
lat (usec) : 10=1.92%, 20=97.64%, 50=0.43%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%, 750=0.01%
lat (msec) : 2=0.01%
cpu : usr=4.11%, sys=12.38%, ctx=5110592, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5110592,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=41.6MiB/s (43.6MB/s), 41.6MiB/s-41.6MiB/s (43.6MB/s-43.6MB/s), io=2495MiB (2617MB), run=60001-60001msec
Disk stats (read/write):
nvme2n1: ios=5091009/0, merge=0/0, ticks=51783/0, in_queue=51783, util=99.64%
Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) at I/O depth 16 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=16
fio-3.28
Starting 1 process
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
Jobs: 1 (f=1): [r(1)][100.0%][r=41.7MiB/s][r=85.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8968: Sat Feb 25 15:03:15 2023
read: IOPS=85.3k, BW=41.6MiB/s (43.7MB/s)(2499MiB/60001msec)
clat (usec): min=4, max=473, avg=11.44, stdev= 3.68
lat (usec): min=4, max=473, avg=11.46, stdev= 3.68
clat percentiles (usec):
| 1.00th=[ 6], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 12],
| 30.00th=[ 12], 40.00th=[ 12], 50.00th=[ 12], 60.00th=[ 12],
| 70.00th=[ 12], 80.00th=[ 12], 90.00th=[ 12], 95.00th=[ 12],
| 99.00th=[ 16], 99.50th=[ 20], 99.90th=[ 23], 99.95th=[ 30],
| 99.99th=[ 221]
bw ( KiB/s): min=39652, max=43679, per=100.00%, avg=42663.71, stdev=452.34, samples=119
iops : min=79305, max=87358, avg=85327.45, stdev=904.63, samples=119
lat (usec) : 10=2.63%, 20=96.92%, 50=0.42%, 100=0.01%, 250=0.02%
lat (usec) : 500=0.01%
cpu : usr=4.28%, sys=12.13%, ctx=5117554, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5117557,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=41.6MiB/s (43.7MB/s), 41.6MiB/s-41.6MiB/s (43.7MB/s-43.7MB/s), io=2499MiB (2620MB), run=60001-60001msec
Disk stats (read/write):
nvme2n1: ios=5097707/0, merge=0/0, ticks=51954/0, in_queue=51954, util=99.63%
Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) at I/O depth 64 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=64
fio-3.28
Starting 1 process
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
Jobs: 1 (f=1): [r(1)][100.0%][r=41.7MiB/s][r=85.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=9051: Sat Feb 25 15:04:15 2023
read: IOPS=85.2k, BW=41.6MiB/s (43.6MB/s)(2497MiB/60001msec)
clat (usec): min=4, max=575, avg=11.42, stdev= 2.92
lat (usec): min=4, max=575, avg=11.44, stdev= 2.92
clat percentiles (usec):
| 1.00th=[ 6], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 12],
| 30.00th=[ 12], 40.00th=[ 12], 50.00th=[ 12], 60.00th=[ 12],
| 70.00th=[ 12], 80.00th=[ 12], 90.00th=[ 12], 95.00th=[ 12],
| 99.00th=[ 16], 99.50th=[ 20], 99.90th=[ 22], 99.95th=[ 25],
| 99.99th=[ 215]
bw ( KiB/s): min=41338, max=43458, per=100.00%, avg=42635.02, stdev=276.50, samples=119
iops : min=82676, max=86916, avg=85270.07, stdev=553.01, samples=119
lat (usec) : 10=1.88%, 20=97.70%, 50=0.40%, 100=0.01%, 250=0.02%
lat (usec) : 500=0.01%, 750=0.01%
cpu : usr=4.51%, sys=12.00%, ctx=5114450, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5114438,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=41.6MiB/s (43.6MB/s), 41.6MiB/s-41.6MiB/s (43.6MB/s-43.6MB/s), io=2497MiB (2619MB), run=60001-60001msec
Disk stats (read/write):
nvme2n1: ios=5094581/0, merge=0/0, ticks=51732/0, in_queue=51732, util=99.64%
Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) at I/O depth 1 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=327MiB/s][r=83.8k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=9234: Sat Feb 25 15:05:16 2023
read: IOPS=84.8k, BW=331MiB/s (347MB/s)(19.4GiB/60001msec)
clat (usec): min=6, max=393, avg=11.55, stdev= 1.94
lat (usec): min=6, max=393, avg=11.57, stdev= 1.94
clat percentiles (nsec):
| 1.00th=[10816], 5.00th=[11200], 10.00th=[11328], 20.00th=[11328],
| 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
| 70.00th=[11456], 80.00th=[11456], 90.00th=[11584], 95.00th=[11712],
| 99.00th=[15296], 99.50th=[20096], 99.90th=[21120], 99.95th=[22656],
| 99.99th=[33024]
bw ( KiB/s): min=335808, max=340968, per=100.00%, avg=339286.25, stdev=1022.91, samples=119
iops : min=83952, max=85242, avg=84821.56, stdev=255.73, samples=119
lat (usec) : 10=0.45%, 20=99.04%, 50=0.50%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%
cpu : usr=3.72%, sys=11.72%, ctx=5086854, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5086847,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=331MiB/s (347MB/s), 331MiB/s-331MiB/s (347MB/s-347MB/s), io=19.4GiB (20.8GB), run=60001-60001msec
Disk stats (read/write):
nvme2n1: ios=5076003/0, merge=0/0, ticks=52209/0, in_queue=52209, util=99.82%
Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) at I/O depth 4 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=4
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=331MiB/s][r=84.7k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=9275: Sat Feb 25 15:06:16 2023
read: IOPS=84.2k, BW=329MiB/s (345MB/s)(19.3GiB/60001msec)
clat (usec): min=6, max=1571, avg=11.61, stdev= 3.31
lat (usec): min=6, max=1571, avg=11.62, stdev= 3.31
clat percentiles (usec):
| 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 12],
| 30.00th=[ 12], 40.00th=[ 12], 50.00th=[ 12], 60.00th=[ 12],
| 70.00th=[ 12], 80.00th=[ 12], 90.00th=[ 12], 95.00th=[ 13],
| 99.00th=[ 19], 99.50th=[ 21], 99.90th=[ 24], 99.95th=[ 29],
| 99.99th=[ 217]
bw ( KiB/s): min=323704, max=340248, per=100.00%, avg=336765.18, stdev=3891.43, samples=119
iops : min=80926, max=85062, avg=84191.31, stdev=972.85, samples=119
lat (usec) : 10=1.19%, 20=98.15%, 50=0.63%, 100=0.01%, 250=0.02%
lat (usec) : 500=0.01%, 750=0.01%
lat (msec) : 2=0.01%
cpu : usr=4.14%, sys=12.40%, ctx=5049669, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5049668,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=329MiB/s (345MB/s), 329MiB/s-329MiB/s (345MB/s-345MB/s), io=19.3GiB (20.7GB), run=60001-60001msec
Disk stats (read/write):
nvme2n1: ios=5039313/0, merge=0/0, ticks=51934/0, in_queue=51934, util=99.85%
Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) at I/O depth 16 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=16
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=332MiB/s][r=84.9k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=9393: Sat Feb 25 15:07:16 2023
read: IOPS=84.7k, BW=331MiB/s (347MB/s)(19.4GiB/60001msec)
clat (usec): min=6, max=392, avg=11.56, stdev= 2.48
lat (usec): min=6, max=392, avg=11.58, stdev= 2.48
clat percentiles (usec):
| 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 12],
| 30.00th=[ 12], 40.00th=[ 12], 50.00th=[ 12], 60.00th=[ 12],
| 70.00th=[ 12], 80.00th=[ 12], 90.00th=[ 12], 95.00th=[ 12],
| 99.00th=[ 16], 99.50th=[ 20], 99.90th=[ 22], 99.95th=[ 24],
| 99.99th=[ 212]
bw ( KiB/s): min=334728, max=340728, per=100.00%, avg=338763.50, stdev=1178.48, samples=119
iops : min=83682, max=85182, avg=84690.87, stdev=294.62, samples=119
lat (usec) : 10=0.54%, 20=98.98%, 50=0.47%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%
cpu : usr=3.65%, sys=11.34%, ctx=5079832, majf=0, minf=14
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5079831,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=331MiB/s (347MB/s), 331MiB/s-331MiB/s (347MB/s-347MB/s), io=19.4GiB (20.8GB), run=60001-60001msec
Disk stats (read/write):
nvme2n1: ios=5069411/0, merge=0/0, ticks=52465/0, in_queue=52465, util=99.83%
Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) at I/O depth 64 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=329MiB/s][r=84.2k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=9560: Sat Feb 25 15:08:17 2023
read: IOPS=84.6k, BW=330MiB/s (346MB/s)(19.4GiB/60001msec)
clat (usec): min=6, max=406, avg=11.57, stdev= 2.62
lat (usec): min=6, max=406, avg=11.59, stdev= 2.62
clat percentiles (usec):
| 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 12],
| 30.00th=[ 12], 40.00th=[ 12], 50.00th=[ 12], 60.00th=[ 12],
| 70.00th=[ 12], 80.00th=[ 12], 90.00th=[ 12], 95.00th=[ 12],
| 99.00th=[ 17], 99.50th=[ 21], 99.90th=[ 23], 99.95th=[ 29],
| 99.99th=[ 212]
bw ( KiB/s): min=323560, max=341672, per=100.00%, avg=338359.13, stdev=2708.53, samples=119
iops : min=80890, max=85418, avg=84589.78, stdev=677.13, samples=119
lat (usec) : 10=0.80%, 20=98.59%, 50=0.59%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%
cpu : usr=3.88%, sys=11.90%, ctx=5073604, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5073624,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=330MiB/s (346MB/s), 330MiB/s-330MiB/s (346MB/s-346MB/s), io=19.4GiB (20.8GB), run=60001-60001msec
Disk stats (read/write):
nvme2n1: ios=5062697/0, merge=0/0, ticks=52126/0, in_queue=52126, util=99.82%
Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) at I/O depth 1 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=40.6MiB/s][r=83.1k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=9616: Sat Feb 25 15:09:17 2023
read: IOPS=82.3k, BW=40.2MiB/s (42.2MB/s)(2412MiB/60001msec)
clat (usec): min=6, max=1234, avg=11.84, stdev= 4.16
lat (usec): min=6, max=1234, avg=11.86, stdev= 4.16
clat percentiles (usec):
| 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 12],
| 30.00th=[ 12], 40.00th=[ 12], 50.00th=[ 12], 60.00th=[ 12],
| 70.00th=[ 12], 80.00th=[ 12], 90.00th=[ 12], 95.00th=[ 13],
| 99.00th=[ 24], 99.50th=[ 37], 99.90th=[ 42], 99.95th=[ 45],
| 99.99th=[ 225]
bw ( KiB/s): min=39535, max=41814, per=100.00%, avg=41186.59, stdev=568.35, samples=119
iops : min=79070, max=83628, avg=82373.19, stdev=1136.71, samples=119
lat (usec) : 10=1.30%, 20=96.75%, 50=1.91%, 100=0.01%, 250=0.02%
lat (usec) : 500=0.01%, 750=0.01%
lat (msec) : 2=0.01%
cpu : usr=3.90%, sys=12.43%, ctx=4940749, majf=0, minf=14
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4940775,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=40.2MiB/s (42.2MB/s), 40.2MiB/s-40.2MiB/s (42.2MB/s-42.2MB/s), io=2412MiB (2530MB), run=60001-60001msec
Disk stats (read/write):
nvme3n1: ios=4927054/0, merge=0/0, ticks=52222/0, in_queue=52222, util=99.76%
Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) at I/O depth 4 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=4
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=40.7MiB/s][r=83.3k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=9691: Sat Feb 25 15:10:17 2023
read: IOPS=82.4k, BW=40.2MiB/s (42.2MB/s)(2413MiB/60001msec)
clat (usec): min=6, max=533, avg=11.84, stdev= 4.36
lat (usec): min=6, max=533, avg=11.85, stdev= 4.36
clat percentiles (usec):
| 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 12],
| 30.00th=[ 12], 40.00th=[ 12], 50.00th=[ 12], 60.00th=[ 12],
| 70.00th=[ 12], 80.00th=[ 12], 90.00th=[ 12], 95.00th=[ 13],
| 99.00th=[ 24], 99.50th=[ 37], 99.90th=[ 42], 99.95th=[ 45],
| 99.99th=[ 225]
bw ( KiB/s): min=39615, max=41767, per=100.00%, avg=41198.88, stdev=499.84, samples=119
iops : min=79230, max=83534, avg=82397.78, stdev=999.66, samples=119
lat (usec) : 10=1.60%, 20=96.54%, 50=1.83%, 100=0.01%, 250=0.02%
lat (usec) : 500=0.01%, 750=0.01%
cpu : usr=4.15%, sys=12.56%, ctx=4942218, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4942241,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=40.2MiB/s (42.2MB/s), 40.2MiB/s-40.2MiB/s (42.2MB/s-42.2MB/s), io=2413MiB (2530MB), run=60001-60001msec
Disk stats (read/write):
nvme3n1: ios=4927366/0, merge=0/0, ticks=52047/0, in_queue=52047, util=99.75%
Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) at I/O depth 16 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=16
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=38.5MiB/s][r=78.8k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=9879: Sat Feb 25 15:11:18 2023
read: IOPS=81.3k, BW=39.7MiB/s (41.6MB/s)(2381MiB/60001msec)
clat (usec): min=6, max=2919, avg=11.97, stdev= 4.69
lat (usec): min=6, max=2919, avg=11.99, stdev= 4.69
clat percentiles (usec):
| 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 12],
| 30.00th=[ 12], 40.00th=[ 12], 50.00th=[ 12], 60.00th=[ 12],
| 70.00th=[ 12], 80.00th=[ 12], 90.00th=[ 13], 95.00th=[ 16],
| 99.00th=[ 24], 99.50th=[ 37], 99.90th=[ 43], 99.95th=[ 46],
| 99.99th=[ 225]
bw ( KiB/s): min=37191, max=41980, per=100.00%, avg=40648.64, stdev=1046.40, samples=119
iops : min=74382, max=83960, avg=81297.29, stdev=2092.82, samples=119
lat (usec) : 10=2.18%, 20=95.68%, 50=2.11%, 100=0.01%, 250=0.02%
lat (usec) : 500=0.01%, 750=0.01%
lat (msec) : 2=0.01%, 4=0.01%
cpu : usr=4.56%, sys=13.62%, ctx=4876034, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4876057,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=39.7MiB/s (41.6MB/s), 39.7MiB/s-39.7MiB/s (41.6MB/s-41.6MB/s), io=2381MiB (2497MB), run=60001-60001msec
Disk stats (read/write):
nvme3n1: ios=4862356/0, merge=0/0, ticks=52027/0, in_queue=52027, util=99.77%
Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) at I/O depth 64 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=40.4MiB/s][r=82.7k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=10045: Sat Feb 25 15:12:18 2023
read: IOPS=81.2k, BW=39.6MiB/s (41.6MB/s)(2379MiB/60001msec)
clat (usec): min=6, max=5477, avg=11.97, stdev= 4.87
lat (usec): min=6, max=5477, avg=11.99, stdev= 4.87
clat percentiles (usec):
| 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 12],
| 30.00th=[ 12], 40.00th=[ 12], 50.00th=[ 12], 60.00th=[ 12],
| 70.00th=[ 12], 80.00th=[ 12], 90.00th=[ 13], 95.00th=[ 16],
| 99.00th=[ 24], 99.50th=[ 37], 99.90th=[ 42], 99.95th=[ 45],
| 99.99th=[ 223]
bw ( KiB/s): min=38304, max=42250, per=100.00%, avg=40612.75, stdev=968.72, samples=119
iops : min=76608, max=84500, avg=81225.50, stdev=1937.46, samples=119
lat (usec) : 10=2.00%, 20=95.85%, 50=2.13%, 100=0.01%, 250=0.02%
lat (usec) : 500=0.01%, 750=0.01%
lat (msec) : 10=0.01%
cpu : usr=4.59%, sys=13.61%, ctx=4871350, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4871377,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=39.6MiB/s (41.6MB/s), 39.6MiB/s-39.6MiB/s (41.6MB/s-41.6MB/s), io=2379MiB (2494MB), run=60001-60001msec
Disk stats (read/write):
nvme3n1: ios=4857033/0, merge=0/0, ticks=51993/0, in_queue=51993, util=99.74%
Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) at I/O depth 1 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=317MiB/s][r=81.3k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=10346: Sat Feb 25 15:13:18 2023
read: IOPS=80.5k, BW=314MiB/s (330MB/s)(18.4GiB/60001msec)
clat (usec): min=8, max=503, avg=12.13, stdev= 4.13
lat (usec): min=8, max=503, avg=12.15, stdev= 4.13
clat percentiles (usec):
| 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 12],
| 30.00th=[ 12], 40.00th=[ 12], 50.00th=[ 12], 60.00th=[ 12],
| 70.00th=[ 12], 80.00th=[ 12], 90.00th=[ 14], 95.00th=[ 16],
| 99.00th=[ 24], 99.50th=[ 37], 99.90th=[ 42], 99.95th=[ 45],
| 99.99th=[ 221]
bw ( KiB/s): min=289368, max=334264, per=100.00%, avg=322181.45, stdev=11085.45, samples=119
iops : min=72342, max=83566, avg=80545.38, stdev=2771.36, samples=119
lat (usec) : 10=0.93%, 20=96.76%, 50=2.29%, 100=0.01%, 250=0.02%
lat (usec) : 500=0.01%, 750=0.01%
cpu : usr=4.37%, sys=13.09%, ctx=4830711, majf=0, minf=14
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4830746,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=314MiB/s (330MB/s), 314MiB/s-314MiB/s (330MB/s-330MB/s), io=18.4GiB (19.8GB), run=60001-60001msec
Disk stats (read/write):
nvme3n1: ios=4821800/0, merge=0/0, ticks=52322/0, in_queue=52322, util=99.87%
Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) at I/O depth 4 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=4
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=323MiB/s][r=82.8k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=10455: Sat Feb 25 15:14:18 2023
read: IOPS=82.5k, BW=322MiB/s (338MB/s)(18.9GiB/60001msec)
clat (usec): min=8, max=439, avg=11.89, stdev= 3.02
lat (usec): min=8, max=439, avg=11.91, stdev= 3.02
clat percentiles (nsec):
| 1.00th=[10688], 5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
| 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
| 70.00th=[11456], 80.00th=[11584], 90.00th=[11712], 95.00th=[13248],
| 99.00th=[23424], 99.50th=[33536], 99.90th=[40704], 99.95th=[41728],
| 99.99th=[48896]
bw ( KiB/s): min=301624, max=334352, per=100.00%, avg=329988.44, stdev=5462.50, samples=119
iops : min=75406, max=83588, avg=82497.14, stdev=1365.64, samples=119
lat (usec) : 10=0.36%, 20=97.76%, 50=1.87%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%
cpu : usr=3.21%, sys=12.09%, ctx=4948518, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4948560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=322MiB/s (338MB/s), 322MiB/s-322MiB/s (338MB/s-338MB/s), io=18.9GiB (20.3GB), run=60001-60001msec
Disk stats (read/write):
nvme3n1: ios=4939258/0, merge=0/0, ticks=52754/0, in_queue=52754, util=99.86%
Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) at I/O depth 16 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=16
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=325MiB/s][r=83.2k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=10494: Sat Feb 25 15:15:19 2023
read: IOPS=83.1k, BW=325MiB/s (341MB/s)(19.0GiB/60001msec)
clat (usec): min=8, max=470, avg=11.79, stdev= 2.77
lat (usec): min=8, max=470, avg=11.81, stdev= 2.77
clat percentiles (nsec):
| 1.00th=[11072], 5.00th=[11328], 10.00th=[11328], 20.00th=[11456],
| 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
| 70.00th=[11456], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
| 99.00th=[23424], 99.50th=[34048], 99.90th=[40704], 99.95th=[41728],
| 99.99th=[45312]
bw ( KiB/s): min=326664, max=333792, per=100.00%, avg=332638.99, stdev=1243.39, samples=119
iops : min=81666, max=83448, avg=83159.75, stdev=310.85, samples=119
lat (usec) : 10=0.12%, 20=98.10%, 50=1.78%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%
cpu : usr=3.45%, sys=11.92%, ctx=4987876, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4987909,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=325MiB/s (341MB/s), 325MiB/s-325MiB/s (341MB/s-341MB/s), io=19.0GiB (20.4GB), run=60001-60001msec
Disk stats (read/write):
nvme3n1: ios=4978615/0, merge=0/0, ticks=52329/0, in_queue=52329, util=99.86%
Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) at I/O depth 64 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=326MiB/s][r=83.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=10529: Sat Feb 25 15:16:19 2023
read: IOPS=82.0k, BW=320MiB/s (336MB/s)(18.8GiB/60001msec)
clat (usec): min=8, max=2079, avg=11.94, stdev= 3.52
lat (usec): min=8, max=2079, avg=11.96, stdev= 3.52
clat percentiles (nsec):
| 1.00th=[10816], 5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
| 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
| 70.00th=[11456], 80.00th=[11456], 90.00th=[11584], 95.00th=[13120],
| 99.00th=[23680], 99.50th=[36608], 99.90th=[47872], 99.95th=[50432],
| 99.99th=[58112]
bw ( KiB/s): min=307216, max=333824, per=100.00%, avg=327930.55, stdev=6598.48, samples=119
iops : min=76804, max=83456, avg=81982.64, stdev=1649.65, samples=119
lat (usec) : 10=0.17%, 20=97.73%, 50=2.04%, 100=0.05%, 250=0.01%
lat (usec) : 500=0.01%
lat (msec) : 2=0.01%, 4=0.01%
cpu : usr=3.74%, sys=11.88%, ctx=4917719, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=4917746,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=320MiB/s (336MB/s), 320MiB/s-320MiB/s (336MB/s-336MB/s), io=18.8GiB (20.1GB), run=60001-60001msec
Disk stats (read/write):
nvme3n1: ios=4908309/0, merge=0/0, ticks=52567/0, in_queue=52567, util=99.85%
tl;dr:
- P1600X 118GB: ~85,000 IOps; 42 MiB/s with 512-byte blocks and 334 MiB/s with 4 KiB blocks
- 905P 380GB: ~84,000 IOps; 41 MiB/s with 512-byte blocks and 329 MiB/s with 4 KiB blocks
- P5800X 3.2TB: ~85,000 IOps; 42 MiB/s with 512-byte blocks and 330 MiB/s with 4 KiB blocks
- P4800X 1.5TB: ~82,000 IOps; 40 MiB/s with 512-byte blocks and ~320 MiB/s with 4 KiB blocks