Intel Optane 905P Series 960GB (SSDPE21D960GAM3) vs Samsung 2TB 980 Pro as boot drive on windows 10?

Yes. I just re-ran the AIDA64 disk benchmarks and the results still hold.

Unless noted, everything was connected via the CPU PCIe 5.0 lanes.

Even with Optane media’s ridiculous endurance, I’d rather not. :sweat_smile:

Here are some more comprehensive random read performance numbers using fio. They cover four models:

The tests are run using combinations of two block sizes (512 or 4,096 bytes) and four I/O depths (1, 4, 16, or 64). All the NVMe namespaces are configured with 512-byte logical sector sizes with no metadata.

The command:

# for device in $(find /dev -maxdepth 1 -regextype posix-extended -regex '^/dev/nvme(0|[1-9]\d*)$' | sort); do data=$(nvme id-ctrl "$device"); mn=$(echo "$data" | awk 'match($0, "^mn\\s*:\\s*(.+?)$", m) {print m[1]}' | xargs); for namespace in $(find /dev -maxdepth 1 -regextype posix-extended -regex "^$device"'n[1-9]\d*$' | sort); do for blocksize in 512 4k; do for depth in 1 4 16 64; do echo -e "\033[0;32mRandom read of device \033[1;32m$device ($mn)\033[0;32m at I/O depth $depth with block size $blocksize\033[0m"; fio --filename="$namespace" --direct=1 --rw=randread --readonly --ioengine=pvsync2 --runtime=60 --bs="$blocksize" --iodepth="$depth" --numjobs=1 --time_based --group_reporting --name=iops-test-job; echo; done; done; done; done

Enjoy. :point_down::wink:

Output
Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) at I/O depth 1 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=41.8MiB/s][r=85.5k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8106: Sat Feb 25 14:45:10 2023
  read: IOPS=85.9k, BW=42.0MiB/s (44.0MB/s)(2518MiB/60001msec)
    clat (usec): min=4, max=379, avg=11.40, stdev= 1.24
     lat (usec): min=4, max=379, avg=11.43, stdev= 1.24
    clat percentiles (nsec):
     |  1.00th=[ 5472],  5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[11968], 99.50th=[15168], 99.90th=[16768], 99.95th=[18816],
     | 99.99th=[27776]
   bw (  KiB/s): min=40647, max=67016, per=100.00%, avg=42985.08, stdev=2234.87, samples=119
   iops        : min=81294, max=134032, avg=85970.15, stdev=4469.74, samples=119
  lat (usec)   : 10=1.58%, 20=98.38%, 50=0.03%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.51%, sys=11.68%, ctx=5156668, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5156680,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=42.0MiB/s (44.0MB/s), 42.0MiB/s-42.0MiB/s (44.0MB/s-44.0MB/s), io=2518MiB (2640MB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=5147663/0, merge=0/0, ticks=52356/0, in_queue=52356, util=99.85%

Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) at I/O depth 4 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=4
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=41.7MiB/s][r=85.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8142: Sat Feb 25 14:46:10 2023
  read: IOPS=85.5k, BW=41.8MiB/s (43.8MB/s)(2506MiB/60001msec)
    clat (usec): min=4, max=657, avg=11.48, stdev= 1.07
     lat (usec): min=4, max=657, avg=11.50, stdev= 1.07
    clat percentiles (nsec):
     |  1.00th=[10816],  5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[11840], 99.50th=[15040], 99.90th=[16064], 99.95th=[18048],
     | 99.99th=[24192]
   bw (  KiB/s): min=42571, max=43785, per=100.00%, avg=42781.55, stdev=140.18, samples=119
   iops        : min=85142, max=87570, avg=85563.13, stdev=280.35, samples=119
  lat (usec)   : 10=0.56%, 20=99.41%, 50=0.03%, 100=0.01%, 250=0.01%
  lat (usec)   : 750=0.01%
  cpu          : usr=2.98%, sys=11.90%, ctx=5132399, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5132420,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=41.8MiB/s (43.8MB/s), 41.8MiB/s-41.8MiB/s (43.8MB/s-43.8MB/s), io=2506MiB (2628MB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=5123326/12, merge=0/2, ticks=52621/1, in_queue=52623, util=99.86%

Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) at I/O depth 16 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=16
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=41.8MiB/s][r=85.6k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8179: Sat Feb 25 14:47:10 2023
  read: IOPS=85.6k, BW=41.8MiB/s (43.8MB/s)(2508MiB/60001msec)
    clat (usec): min=4, max=895, avg=11.48, stdev= 1.14
     lat (usec): min=4, max=895, avg=11.50, stdev= 1.14
    clat percentiles (nsec):
     |  1.00th=[10560],  5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11584], 60.00th=[11584],
     | 70.00th=[11584], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[11968], 99.50th=[14016], 99.90th=[16064], 99.95th=[16192],
     | 99.99th=[23168]
   bw (  KiB/s): min=42631, max=50069, per=100.00%, avg=42822.87, stdev=670.75, samples=119
   iops        : min=85262, max=100138, avg=85645.75, stdev=1341.50, samples=119
  lat (usec)   : 10=0.69%, 20=99.29%, 50=0.02%, 100=0.01%, 250=0.01%
  lat (usec)   : 1000=0.01%
  cpu          : usr=3.06%, sys=12.20%, ctx=5137331, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5137350,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=41.8MiB/s (43.8MB/s), 41.8MiB/s-41.8MiB/s (43.8MB/s-43.8MB/s), io=2508MiB (2630MB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=5128190/9, merge=0/23, ticks=52427/1, in_queue=52428, util=99.85%

Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) at I/O depth 64 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=41.8MiB/s][r=85.5k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8226: Sat Feb 25 14:48:10 2023
  read: IOPS=85.5k, BW=41.8MiB/s (43.8MB/s)(2506MiB/60001msec)
    clat (usec): min=4, max=280, avg=11.47, stdev= 1.01
     lat (usec): min=4, max=280, avg=11.49, stdev= 1.01
    clat percentiles (nsec):
     |  1.00th=[10688],  5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[11840], 99.50th=[15168], 99.90th=[16064], 99.95th=[18816],
     | 99.99th=[28032]
   bw (  KiB/s): min=42626, max=45012, per=100.00%, avg=42778.61, stdev=210.98, samples=119
   iops        : min=85252, max=90026, avg=85557.24, stdev=422.14, samples=119
  lat (usec)   : 10=0.57%, 20=99.38%, 50=0.04%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.15%, sys=11.77%, ctx=5132105, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5132129,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=41.8MiB/s (43.8MB/s), 41.8MiB/s-41.8MiB/s (43.8MB/s-43.8MB/s), io=2506MiB (2628MB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=5123033/12, merge=0/3, ticks=52562/1, in_queue=52563, util=99.85%

Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) at I/O depth 1 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=334MiB/s][r=85.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8261: Sat Feb 25 14:49:11 2023
  read: IOPS=85.5k, BW=334MiB/s (350MB/s)(19.6GiB/60001msec)
    clat (nsec): min=6161, max=234810, avg=11534.74, stdev=992.08
     lat (nsec): min=6181, max=234850, avg=11551.71, stdev=992.18
    clat percentiles (nsec):
     |  1.00th=[10048],  5.00th=[11328], 10.00th=[11456], 20.00th=[11456],
     | 30.00th=[11584], 40.00th=[11584], 50.00th=[11584], 60.00th=[11584],
     | 70.00th=[11584], 80.00th=[11584], 90.00th=[11712], 95.00th=[11712],
     | 99.00th=[12480], 99.50th=[15296], 99.90th=[16064], 99.95th=[18816],
     | 99.99th=[27776]
   bw (  KiB/s): min=340848, max=401144, per=100.00%, avg=342250.49, stdev=5536.04, samples=119
   iops        : min=85212, max=100286, avg=85562.64, stdev=1384.01, samples=119
  lat (usec)   : 10=0.99%, 20=98.97%, 50=0.04%, 100=0.01%, 250=0.01%
  cpu          : usr=2.91%, sys=11.57%, ctx=5132332, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5132358,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=334MiB/s (350MB/s), 334MiB/s-334MiB/s (350MB/s-350MB/s), io=19.6GiB (21.0GB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=5123652/3, merge=0/7, ticks=52849/0, in_queue=52850, util=99.87%

Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) at I/O depth 4 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=4
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=333MiB/s][r=85.3k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8297: Sat Feb 25 14:50:11 2023
  read: IOPS=85.3k, BW=333MiB/s (350MB/s)(19.5GiB/60001msec)
    clat (nsec): min=6251, max=292708, avg=11566.22, stdev=944.08
     lat (nsec): min=6272, max=292728, avg=11583.96, stdev=944.35
    clat percentiles (nsec):
     |  1.00th=[11200],  5.00th=[11328], 10.00th=[11456], 20.00th=[11456],
     | 30.00th=[11584], 40.00th=[11584], 50.00th=[11584], 60.00th=[11584],
     | 70.00th=[11584], 80.00th=[11584], 90.00th=[11712], 95.00th=[11712],
     | 99.00th=[12352], 99.50th=[14656], 99.90th=[16064], 99.95th=[16320],
     | 99.99th=[24448]
   bw (  KiB/s): min=340944, max=341840, per=100.00%, avg=341454.32, stdev=258.79, samples=119
   iops        : min=85236, max=85460, avg=85363.60, stdev=64.72, samples=119
  lat (usec)   : 10=0.33%, 20=99.65%, 50=0.02%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=2.78%, sys=12.10%, ctx=5120444, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5120488,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=333MiB/s (350MB/s), 333MiB/s-333MiB/s (350MB/s-350MB/s), io=19.5GiB (21.0GB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=5111744/17, merge=0/25, ticks=52637/1, in_queue=52638, util=99.86%

Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) at I/O depth 16 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=16
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=334MiB/s][r=85.5k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8334: Sat Feb 25 14:51:11 2023
  read: IOPS=85.5k, BW=334MiB/s (350MB/s)(19.6GiB/60001msec)
    clat (nsec): min=6152, max=292407, avg=11548.75, stdev=961.55
     lat (nsec): min=6171, max=292417, avg=11565.66, stdev=961.63
    clat percentiles (nsec):
     |  1.00th=[11200],  5.00th=[11328], 10.00th=[11456], 20.00th=[11456],
     | 30.00th=[11584], 40.00th=[11584], 50.00th=[11584], 60.00th=[11584],
     | 70.00th=[11584], 80.00th=[11584], 90.00th=[11712], 95.00th=[11712],
     | 99.00th=[12480], 99.50th=[15296], 99.90th=[18048], 99.95th=[19072],
     | 99.99th=[23680]
   bw (  KiB/s): min=340600, max=378496, per=100.00%, avg=341900.30, stdev=3412.33, samples=119
   iops        : min=85150, max=94624, avg=85475.09, stdev=853.08, samples=119
  lat (usec)   : 10=0.74%, 20=99.23%, 50=0.03%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=2.62%, sys=11.81%, ctx=5127095, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5127117,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=334MiB/s (350MB/s), 334MiB/s-334MiB/s (350MB/s-350MB/s), io=19.6GiB (21.0GB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=5118363/10, merge=0/0, ticks=52635/0, in_queue=52635, util=99.86%

Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) at I/O depth 64 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=334MiB/s][r=85.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8369: Sat Feb 25 14:52:11 2023
  read: IOPS=85.4k, BW=334MiB/s (350MB/s)(19.5GiB/60001msec)
    clat (usec): min=6, max=966, avg=11.56, stdev= 1.01
     lat (usec): min=6, max=966, avg=11.58, stdev= 1.01
    clat percentiles (nsec):
     |  1.00th=[11200],  5.00th=[11328], 10.00th=[11456], 20.00th=[11456],
     | 30.00th=[11584], 40.00th=[11584], 50.00th=[11584], 60.00th=[11584],
     | 70.00th=[11584], 80.00th=[11584], 90.00th=[11712], 95.00th=[11712],
     | 99.00th=[12480], 99.50th=[15040], 99.90th=[16192], 99.95th=[18816],
     | 99.99th=[23936]
   bw (  KiB/s): min=340112, max=344384, per=100.00%, avg=341638.52, stdev=447.66, samples=119
   iops        : min=85028, max=86096, avg=85409.61, stdev=111.93, samples=119
  lat (usec)   : 10=0.49%, 20=99.48%, 50=0.03%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%, 1000=0.01%
  cpu          : usr=2.85%, sys=11.67%, ctx=5123108, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5123138,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=334MiB/s (350MB/s), 334MiB/s-334MiB/s (350MB/s-350MB/s), io=19.5GiB (21.0GB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=5114416/4, merge=0/16, ticks=52719/0, in_queue=52719, util=99.87%

Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) at I/O depth 1 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=41.2MiB/s][r=84.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8417: Sat Feb 25 14:53:12 2023
  read: IOPS=84.3k, BW=41.2MiB/s (43.2MB/s)(2471MiB/60001msec)
    clat (usec): min=6, max=275, avg=11.63, stdev= 1.68
     lat (usec): min=6, max=275, avg=11.65, stdev= 1.68
    clat percentiles (nsec):
     |  1.00th=[11072],  5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[19840], 99.50th=[20864], 99.90th=[36096], 99.95th=[39168],
     | 99.99th=[41216]
   bw (  KiB/s): min=42113, max=42245, per=100.00%, avg=42178.82, stdev=38.22, samples=119
   iops        : min=84226, max=84490, avg=84357.65, stdev=76.49, samples=119
  lat (usec)   : 10=0.14%, 20=98.98%, 50=0.88%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.67%, sys=11.31%, ctx=5060065, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5060056,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=41.2MiB/s (43.2MB/s), 41.2MiB/s-41.2MiB/s (43.2MB/s-43.2MB/s), io=2471MiB (2591MB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=5049070/0, merge=0/0, ticks=52393/0, in_queue=52393, util=99.82%

Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) at I/O depth 4 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=4
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=40.9MiB/s][r=83.7k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8452: Sat Feb 25 14:54:12 2023
  read: IOPS=84.3k, BW=41.1MiB/s (43.1MB/s)(2469MiB/60001msec)
    clat (usec): min=6, max=1913, avg=11.63, stdev= 2.55
     lat (usec): min=6, max=1913, avg=11.65, stdev= 2.55
    clat percentiles (nsec):
     |  1.00th=[10816],  5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[19840], 99.50th=[21632], 99.90th=[36608], 99.95th=[39168],
     | 99.99th=[44288]
   bw (  KiB/s): min=41374, max=42386, per=100.00%, avg=42147.93, stdev=129.34, samples=119
   iops        : min=82748, max=84772, avg=84295.87, stdev=258.69, samples=119
  lat (usec)   : 10=0.54%, 20=98.56%, 50=0.90%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=3.38%, sys=11.77%, ctx=5055956, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5055965,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=41.1MiB/s (43.1MB/s), 41.1MiB/s-41.1MiB/s (43.1MB/s-43.1MB/s), io=2469MiB (2589MB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=5045792/0, merge=0/0, ticks=52354/0, in_queue=52354, util=99.83%

Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) at I/O depth 16 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=16
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=41.1MiB/s][r=84.3k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8490: Sat Feb 25 14:55:12 2023
  read: IOPS=84.2k, BW=41.1MiB/s (43.1MB/s)(2465MiB/60001msec)
    clat (usec): min=6, max=385, avg=11.64, stdev= 2.70
     lat (usec): min=6, max=386, avg=11.66, stdev= 2.70
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   12], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   12], 95.00th=[   12],
     | 99.00th=[   20], 99.50th=[   22], 99.90th=[   37], 99.95th=[   40],
     | 99.99th=[  212]
   bw (  KiB/s): min=41462, max=42352, per=100.00%, avg=42089.20, stdev=177.68, samples=119
   iops        : min=82924, max=84704, avg=84178.40, stdev=355.34, samples=119
  lat (usec)   : 10=0.40%, 20=98.69%, 50=0.89%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.42%, sys=11.38%, ctx=5049175, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5049177,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=41.1MiB/s (43.1MB/s), 41.1MiB/s-41.1MiB/s (43.1MB/s-43.1MB/s), io=2465MiB (2585MB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=5038924/0, merge=0/0, ticks=52528/0, in_queue=52528, util=99.84%

Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) at I/O depth 64 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=40.8MiB/s][r=83.6k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8525: Sat Feb 25 14:56:13 2023
  read: IOPS=84.3k, BW=41.1MiB/s (43.1MB/s)(2468MiB/60001msec)
    clat (usec): min=6, max=332, avg=11.62, stdev= 2.05
     lat (usec): min=7, max=332, avg=11.64, stdev= 2.05
    clat percentiles (nsec):
     |  1.00th=[11072],  5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11456], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[19840], 99.50th=[21120], 99.90th=[36608], 99.95th=[39168],
     | 99.99th=[41728]
   bw (  KiB/s): min=41351, max=42234, per=100.00%, avg=42139.60, stdev=115.92, samples=119
   iops        : min=82702, max=84468, avg=84279.21, stdev=231.84, samples=119
  lat (usec)   : 10=0.21%, 20=98.90%, 50=0.89%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.69%, sys=11.58%, ctx=5055297, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5055300,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=41.1MiB/s (43.1MB/s), 41.1MiB/s-41.1MiB/s (43.1MB/s-43.1MB/s), io=2468MiB (2588MB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=5045050/0, merge=0/0, ticks=52271/0, in_queue=52271, util=99.84%

Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) at I/O depth 1 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=329MiB/s][r=84.2k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8560: Sat Feb 25 14:57:13 2023
  read: IOPS=84.2k, BW=329MiB/s (345MB/s)(19.3GiB/60001msec)
    clat (usec): min=8, max=1202, avg=11.66, stdev= 2.12
     lat (usec): min=8, max=1202, avg=11.68, stdev= 2.12
    clat percentiles (nsec):
     |  1.00th=[11072],  5.00th=[11328], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11584], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[19840], 99.50th=[21888], 99.90th=[36608], 99.95th=[39168],
     | 99.99th=[41728]
   bw (  KiB/s): min=332344, max=337848, per=100.00%, avg=336867.40, stdev=1005.50, samples=119
   iops        : min=83086, max=84462, avg=84216.87, stdev=251.38, samples=119
  lat (usec)   : 10=0.17%, 20=98.90%, 50=0.93%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=3.34%, sys=11.62%, ctx=5051521, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5051528,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=329MiB/s (345MB/s), 329MiB/s-329MiB/s (345MB/s-345MB/s), io=19.3GiB (20.7GB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=5042383/0, merge=0/0, ticks=52458/0, in_queue=52458, util=99.87%

Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) at I/O depth 4 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=4
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=328MiB/s][r=84.1k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8599: Sat Feb 25 14:58:13 2023
  read: IOPS=84.1k, BW=329MiB/s (345MB/s)(19.3GiB/60001msec)
    clat (usec): min=8, max=388, avg=11.66, stdev= 2.23
     lat (usec): min=8, max=388, avg=11.68, stdev= 2.23
    clat percentiles (nsec):
     |  1.00th=[10944],  5.00th=[11328], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11584], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[19840], 99.50th=[21888], 99.90th=[36608], 99.95th=[39168],
     | 99.99th=[43264]
   bw (  KiB/s): min=328096, max=337816, per=100.00%, avg=336692.97, stdev=1433.86, samples=119
   iops        : min=82024, max=84454, avg=84173.24, stdev=358.50, samples=119
  lat (usec)   : 10=0.23%, 20=98.82%, 50=0.94%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.48%, sys=11.58%, ctx=5048664, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5048673,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=329MiB/s (345MB/s), 329MiB/s-329MiB/s (345MB/s-345MB/s), io=19.3GiB (20.7GB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=5039538/0, merge=0/0, ticks=52440/0, in_queue=52440, util=99.86%

Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) at I/O depth 16 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=16
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=329MiB/s][r=84.2k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8635: Sat Feb 25 14:59:13 2023
  read: IOPS=84.2k, BW=329MiB/s (345MB/s)(19.3GiB/60001msec)
    clat (usec): min=8, max=395, avg=11.66, stdev= 2.18
     lat (usec): min=8, max=395, avg=11.68, stdev= 2.18
    clat percentiles (nsec):
     |  1.00th=[10944],  5.00th=[11328], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11584], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[19840], 99.50th=[21888], 99.90th=[36608], 99.95th=[39168],
     | 99.99th=[42240]
   bw (  KiB/s): min=331144, max=337832, per=100.00%, avg=336725.45, stdev=1154.46, samples=119
   iops        : min=82786, max=84458, avg=84181.38, stdev=288.63, samples=119
  lat (usec)   : 10=0.24%, 20=98.83%, 50=0.93%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.28%, sys=11.73%, ctx=5049526, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5049536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=329MiB/s (345MB/s), 329MiB/s-329MiB/s (345MB/s-345MB/s), io=19.3GiB (20.7GB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=5040365/0, merge=0/0, ticks=52441/0, in_queue=52441, util=99.86%

Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) at I/O depth 64 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=329MiB/s][r=84.3k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8671: Sat Feb 25 15:00:14 2023
  read: IOPS=84.2k, BW=329MiB/s (345MB/s)(19.3GiB/60001msec)
    clat (usec): min=8, max=318, avg=11.66, stdev= 2.03
     lat (usec): min=8, max=318, avg=11.68, stdev= 2.04
    clat percentiles (nsec):
     |  1.00th=[11072],  5.00th=[11328], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11584], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[19840], 99.50th=[21888], 99.90th=[36608], 99.95th=[39168],
     | 99.99th=[41728]
   bw (  KiB/s): min=332704, max=337904, per=100.00%, avg=337005.24, stdev=899.24, samples=119
   iops        : min=83176, max=84476, avg=84251.33, stdev=224.80, samples=119
  lat (usec)   : 10=0.16%, 20=98.92%, 50=0.92%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.36%, sys=11.64%, ctx=5053532, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5053551,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=329MiB/s (345MB/s), 329MiB/s-329MiB/s (345MB/s-345MB/s), io=19.3GiB (20.7GB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=5044355/0, merge=0/0, ticks=52416/0, in_queue=52416, util=99.86%

Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) at I/O depth 1 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
Jobs: 1 (f=1): [r(1)][100.0%][r=41.7MiB/s][r=85.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8716: Sat Feb 25 15:01:14 2023
  read: IOPS=85.3k, BW=41.7MiB/s (43.7MB/s)(2500MiB/60001msec)
    clat (usec): min=4, max=354, avg=11.44, stdev= 2.10
     lat (usec): min=4, max=354, avg=11.46, stdev= 2.10
    clat percentiles (nsec):
     |  1.00th=[10176],  5.00th=[11200], 10.00th=[11328], 20.00th=[11328],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11456], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[12096], 99.50th=[16064], 99.90th=[20608], 99.95th=[20864],
     | 99.99th=[53504]
   bw (  KiB/s): min=40278, max=43410, per=100.00%, avg=42686.28, stdev=283.65, samples=119
   iops        : min=80556, max=86822, avg=85372.57, stdev=567.34, samples=119
  lat (usec)   : 10=0.96%, 20=98.69%, 50=0.34%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=4.12%, sys=11.94%, ctx=5120865, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5120860,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=41.7MiB/s (43.7MB/s), 41.7MiB/s-41.7MiB/s (43.7MB/s-43.7MB/s), io=2500MiB (2622MB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=5100271/0, merge=0/0, ticks=51705/0, in_queue=51705, util=99.61%

Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) at I/O depth 4 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=4
fio-3.28
Starting 1 process
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
Jobs: 1 (f=1): [r(1)][100.0%][r=42.0MiB/s][r=86.0k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8763: Sat Feb 25 15:02:15 2023
  read: IOPS=85.2k, BW=41.6MiB/s (43.6MB/s)(2495MiB/60001msec)
    clat (usec): min=4, max=1131, avg=11.45, stdev= 2.75
     lat (usec): min=4, max=1131, avg=11.47, stdev= 2.75
    clat percentiles (usec):
     |  1.00th=[    6],  5.00th=[   12], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   12], 95.00th=[   12],
     | 99.00th=[   16], 99.50th=[   20], 99.90th=[   22], 99.95th=[   26],
     | 99.99th=[  215]
   bw (  KiB/s): min=41126, max=44395, per=100.00%, avg=42602.17, stdev=436.29, samples=119
   iops        : min=82252, max=88790, avg=85204.34, stdev=872.57, samples=119
  lat (usec)   : 10=1.92%, 20=97.64%, 50=0.43%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%, 750=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=4.11%, sys=12.38%, ctx=5110592, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5110592,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=41.6MiB/s (43.6MB/s), 41.6MiB/s-41.6MiB/s (43.6MB/s-43.6MB/s), io=2495MiB (2617MB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=5091009/0, merge=0/0, ticks=51783/0, in_queue=51783, util=99.64%

Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) at I/O depth 16 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=16
fio-3.28
Starting 1 process
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
Jobs: 1 (f=1): [r(1)][100.0%][r=41.7MiB/s][r=85.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8968: Sat Feb 25 15:03:15 2023
  read: IOPS=85.3k, BW=41.6MiB/s (43.7MB/s)(2499MiB/60001msec)
    clat (usec): min=4, max=473, avg=11.44, stdev= 3.68
     lat (usec): min=4, max=473, avg=11.46, stdev= 3.68
    clat percentiles (usec):
     |  1.00th=[    6],  5.00th=[   11], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   12], 95.00th=[   12],
     | 99.00th=[   16], 99.50th=[   20], 99.90th=[   23], 99.95th=[   30],
     | 99.99th=[  221]
   bw (  KiB/s): min=39652, max=43679, per=100.00%, avg=42663.71, stdev=452.34, samples=119
   iops        : min=79305, max=87358, avg=85327.45, stdev=904.63, samples=119
  lat (usec)   : 10=2.63%, 20=96.92%, 50=0.42%, 100=0.01%, 250=0.02%
  lat (usec)   : 500=0.01%
  cpu          : usr=4.28%, sys=12.13%, ctx=5117554, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5117557,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=41.6MiB/s (43.7MB/s), 41.6MiB/s-41.6MiB/s (43.7MB/s-43.7MB/s), io=2499MiB (2620MB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=5097707/0, merge=0/0, ticks=51954/0, in_queue=51954, util=99.63%

Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) at I/O depth 64 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=64
fio-3.28
Starting 1 process
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
Jobs: 1 (f=1): [r(1)][100.0%][r=41.7MiB/s][r=85.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=9051: Sat Feb 25 15:04:15 2023
  read: IOPS=85.2k, BW=41.6MiB/s (43.6MB/s)(2497MiB/60001msec)
    clat (usec): min=4, max=575, avg=11.42, stdev= 2.92
     lat (usec): min=4, max=575, avg=11.44, stdev= 2.92
    clat percentiles (usec):
     |  1.00th=[    6],  5.00th=[   12], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   12], 95.00th=[   12],
     | 99.00th=[   16], 99.50th=[   20], 99.90th=[   22], 99.95th=[   25],
     | 99.99th=[  215]
   bw (  KiB/s): min=41338, max=43458, per=100.00%, avg=42635.02, stdev=276.50, samples=119
   iops        : min=82676, max=86916, avg=85270.07, stdev=553.01, samples=119
  lat (usec)   : 10=1.88%, 20=97.70%, 50=0.40%, 100=0.01%, 250=0.02%
  lat (usec)   : 500=0.01%, 750=0.01%
  cpu          : usr=4.51%, sys=12.00%, ctx=5114450, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5114438,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=41.6MiB/s (43.6MB/s), 41.6MiB/s-41.6MiB/s (43.6MB/s-43.6MB/s), io=2497MiB (2619MB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=5094581/0, merge=0/0, ticks=51732/0, in_queue=51732, util=99.64%

Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) at I/O depth 1 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=327MiB/s][r=83.8k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=9234: Sat Feb 25 15:05:16 2023
  read: IOPS=84.8k, BW=331MiB/s (347MB/s)(19.4GiB/60001msec)
    clat (usec): min=6, max=393, avg=11.55, stdev= 1.94
     lat (usec): min=6, max=393, avg=11.57, stdev= 1.94
    clat percentiles (nsec):
     |  1.00th=[10816],  5.00th=[11200], 10.00th=[11328], 20.00th=[11328],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11456], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[15296], 99.50th=[20096], 99.90th=[21120], 99.95th=[22656],
     | 99.99th=[33024]
   bw (  KiB/s): min=335808, max=340968, per=100.00%, avg=339286.25, stdev=1022.91, samples=119
   iops        : min=83952, max=85242, avg=84821.56, stdev=255.73, samples=119
  lat (usec)   : 10=0.45%, 20=99.04%, 50=0.50%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.72%, sys=11.72%, ctx=5086854, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5086847,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=331MiB/s (347MB/s), 331MiB/s-331MiB/s (347MB/s-347MB/s), io=19.4GiB (20.8GB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=5076003/0, merge=0/0, ticks=52209/0, in_queue=52209, util=99.82%

Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) at I/O depth 4 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=4
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=331MiB/s][r=84.7k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=9275: Sat Feb 25 15:06:16 2023
  read: IOPS=84.2k, BW=329MiB/s (345MB/s)(19.3GiB/60001msec)
    clat (usec): min=6, max=1571, avg=11.61, stdev= 3.31
     lat (usec): min=6, max=1571, avg=11.62, stdev= 3.31
    clat percentiles (usec):
     |  1.00th=[   10],  5.00th=[   12], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   12], 95.00th=[   13],
     | 99.00th=[   19], 99.50th=[   21], 99.90th=[   24], 99.95th=[   29],
     | 99.99th=[  217]
   bw (  KiB/s): min=323704, max=340248, per=100.00%, avg=336765.18, stdev=3891.43, samples=119
   iops        : min=80926, max=85062, avg=84191.31, stdev=972.85, samples=119
  lat (usec)   : 10=1.19%, 20=98.15%, 50=0.63%, 100=0.01%, 250=0.02%
  lat (usec)   : 500=0.01%, 750=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=4.14%, sys=12.40%, ctx=5049669, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5049668,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=329MiB/s (345MB/s), 329MiB/s-329MiB/s (345MB/s-345MB/s), io=19.3GiB (20.7GB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=5039313/0, merge=0/0, ticks=51934/0, in_queue=51934, util=99.85%

Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) at I/O depth 16 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=16
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=332MiB/s][r=84.9k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=9393: Sat Feb 25 15:07:16 2023
  read: IOPS=84.7k, BW=331MiB/s (347MB/s)(19.4GiB/60001msec)
    clat (usec): min=6, max=392, avg=11.56, stdev= 2.48
     lat (usec): min=6, max=392, avg=11.58, stdev= 2.48
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   12], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   12], 95.00th=[   12],
     | 99.00th=[   16], 99.50th=[   20], 99.90th=[   22], 99.95th=[   24],
     | 99.99th=[  212]
   bw (  KiB/s): min=334728, max=340728, per=100.00%, avg=338763.50, stdev=1178.48, samples=119
   iops        : min=83682, max=85182, avg=84690.87, stdev=294.62, samples=119
  lat (usec)   : 10=0.54%, 20=98.98%, 50=0.47%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.65%, sys=11.34%, ctx=5079832, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5079831,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=331MiB/s (347MB/s), 331MiB/s-331MiB/s (347MB/s-347MB/s), io=19.4GiB (20.8GB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=5069411/0, merge=0/0, ticks=52465/0, in_queue=52465, util=99.83%

Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) at I/O depth 64 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=329MiB/s][r=84.2k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=9560: Sat Feb 25 15:08:17 2023
  read: IOPS=84.6k, BW=330MiB/s (346MB/s)(19.4GiB/60001msec)
    clat (usec): min=6, max=406, avg=11.57, stdev= 2.62
     lat (usec): min=6, max=406, avg=11.59, stdev= 2.62
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   12], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   12], 95.00th=[   12],
     | 99.00th=[   17], 99.50th=[   21], 99.90th=[   23], 99.95th=[   29],
     | 99.99th=[  212]
   bw (  KiB/s): min=323560, max=341672, per=100.00%, avg=338359.13, stdev=2708.53, samples=119
   iops        : min=80890, max=85418, avg=84589.78, stdev=677.13, samples=119
  lat (usec)   : 10=0.80%, 20=98.59%, 50=0.59%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.88%, sys=11.90%, ctx=5073604, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5073624,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=330MiB/s (346MB/s), 330MiB/s-330MiB/s (346MB/s-346MB/s), io=19.4GiB (20.8GB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=5062697/0, merge=0/0, ticks=52126/0, in_queue=52126, util=99.82%

Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) at I/O depth 1 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=40.6MiB/s][r=83.1k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=9616: Sat Feb 25 15:09:17 2023
  read: IOPS=82.3k, BW=40.2MiB/s (42.2MB/s)(2412MiB/60001msec)
    clat (usec): min=6, max=1234, avg=11.84, stdev= 4.16
     lat (usec): min=6, max=1234, avg=11.86, stdev= 4.16
    clat percentiles (usec):
     |  1.00th=[   10],  5.00th=[   12], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   12], 95.00th=[   13],
     | 99.00th=[   24], 99.50th=[   37], 99.90th=[   42], 99.95th=[   45],
     | 99.99th=[  225]
   bw (  KiB/s): min=39535, max=41814, per=100.00%, avg=41186.59, stdev=568.35, samples=119
   iops        : min=79070, max=83628, avg=82373.19, stdev=1136.71, samples=119
  lat (usec)   : 10=1.30%, 20=96.75%, 50=1.91%, 100=0.01%, 250=0.02%
  lat (usec)   : 500=0.01%, 750=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=3.90%, sys=12.43%, ctx=4940749, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4940775,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=40.2MiB/s (42.2MB/s), 40.2MiB/s-40.2MiB/s (42.2MB/s-42.2MB/s), io=2412MiB (2530MB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=4927054/0, merge=0/0, ticks=52222/0, in_queue=52222, util=99.76%

Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) at I/O depth 4 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=4
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=40.7MiB/s][r=83.3k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=9691: Sat Feb 25 15:10:17 2023
  read: IOPS=82.4k, BW=40.2MiB/s (42.2MB/s)(2413MiB/60001msec)
    clat (usec): min=6, max=533, avg=11.84, stdev= 4.36
     lat (usec): min=6, max=533, avg=11.85, stdev= 4.36
    clat percentiles (usec):
     |  1.00th=[    9],  5.00th=[   11], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   12], 95.00th=[   13],
     | 99.00th=[   24], 99.50th=[   37], 99.90th=[   42], 99.95th=[   45],
     | 99.99th=[  225]
   bw (  KiB/s): min=39615, max=41767, per=100.00%, avg=41198.88, stdev=499.84, samples=119
   iops        : min=79230, max=83534, avg=82397.78, stdev=999.66, samples=119
  lat (usec)   : 10=1.60%, 20=96.54%, 50=1.83%, 100=0.01%, 250=0.02%
  lat (usec)   : 500=0.01%, 750=0.01%
  cpu          : usr=4.15%, sys=12.56%, ctx=4942218, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4942241,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=40.2MiB/s (42.2MB/s), 40.2MiB/s-40.2MiB/s (42.2MB/s-42.2MB/s), io=2413MiB (2530MB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=4927366/0, merge=0/0, ticks=52047/0, in_queue=52047, util=99.75%

Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) at I/O depth 16 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=16
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=38.5MiB/s][r=78.8k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=9879: Sat Feb 25 15:11:18 2023
  read: IOPS=81.3k, BW=39.7MiB/s (41.6MB/s)(2381MiB/60001msec)
    clat (usec): min=6, max=2919, avg=11.97, stdev= 4.69
     lat (usec): min=6, max=2919, avg=11.99, stdev= 4.69
    clat percentiles (usec):
     |  1.00th=[    9],  5.00th=[   11], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   13], 95.00th=[   16],
     | 99.00th=[   24], 99.50th=[   37], 99.90th=[   43], 99.95th=[   46],
     | 99.99th=[  225]
   bw (  KiB/s): min=37191, max=41980, per=100.00%, avg=40648.64, stdev=1046.40, samples=119
   iops        : min=74382, max=83960, avg=81297.29, stdev=2092.82, samples=119
  lat (usec)   : 10=2.18%, 20=95.68%, 50=2.11%, 100=0.01%, 250=0.02%
  lat (usec)   : 500=0.01%, 750=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=4.56%, sys=13.62%, ctx=4876034, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4876057,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=39.7MiB/s (41.6MB/s), 39.7MiB/s-39.7MiB/s (41.6MB/s-41.6MB/s), io=2381MiB (2497MB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=4862356/0, merge=0/0, ticks=52027/0, in_queue=52027, util=99.77%

Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) at I/O depth 64 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=40.4MiB/s][r=82.7k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=10045: Sat Feb 25 15:12:18 2023
  read: IOPS=81.2k, BW=39.6MiB/s (41.6MB/s)(2379MiB/60001msec)
    clat (usec): min=6, max=5477, avg=11.97, stdev= 4.87
     lat (usec): min=6, max=5477, avg=11.99, stdev= 4.87
    clat percentiles (usec):
     |  1.00th=[    9],  5.00th=[   11], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   13], 95.00th=[   16],
     | 99.00th=[   24], 99.50th=[   37], 99.90th=[   42], 99.95th=[   45],
     | 99.99th=[  223]
   bw (  KiB/s): min=38304, max=42250, per=100.00%, avg=40612.75, stdev=968.72, samples=119
   iops        : min=76608, max=84500, avg=81225.50, stdev=1937.46, samples=119
  lat (usec)   : 10=2.00%, 20=95.85%, 50=2.13%, 100=0.01%, 250=0.02%
  lat (usec)   : 500=0.01%, 750=0.01%
  lat (msec)   : 10=0.01%
  cpu          : usr=4.59%, sys=13.61%, ctx=4871350, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4871377,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=39.6MiB/s (41.6MB/s), 39.6MiB/s-39.6MiB/s (41.6MB/s-41.6MB/s), io=2379MiB (2494MB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=4857033/0, merge=0/0, ticks=51993/0, in_queue=51993, util=99.74%

Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) at I/O depth 1 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=317MiB/s][r=81.3k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=10346: Sat Feb 25 15:13:18 2023
  read: IOPS=80.5k, BW=314MiB/s (330MB/s)(18.4GiB/60001msec)
    clat (usec): min=8, max=503, avg=12.13, stdev= 4.13
     lat (usec): min=8, max=503, avg=12.15, stdev= 4.13
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   12], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   14], 95.00th=[   16],
     | 99.00th=[   24], 99.50th=[   37], 99.90th=[   42], 99.95th=[   45],
     | 99.99th=[  221]
   bw (  KiB/s): min=289368, max=334264, per=100.00%, avg=322181.45, stdev=11085.45, samples=119
   iops        : min=72342, max=83566, avg=80545.38, stdev=2771.36, samples=119
  lat (usec)   : 10=0.93%, 20=96.76%, 50=2.29%, 100=0.01%, 250=0.02%
  lat (usec)   : 500=0.01%, 750=0.01%
  cpu          : usr=4.37%, sys=13.09%, ctx=4830711, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4830746,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=314MiB/s (330MB/s), 314MiB/s-314MiB/s (330MB/s-330MB/s), io=18.4GiB (19.8GB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=4821800/0, merge=0/0, ticks=52322/0, in_queue=52322, util=99.87%

Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) at I/O depth 4 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=4
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=323MiB/s][r=82.8k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=10455: Sat Feb 25 15:14:18 2023
  read: IOPS=82.5k, BW=322MiB/s (338MB/s)(18.9GiB/60001msec)
    clat (usec): min=8, max=439, avg=11.89, stdev= 3.02
     lat (usec): min=8, max=439, avg=11.91, stdev= 3.02
    clat percentiles (nsec):
     |  1.00th=[10688],  5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11584], 90.00th=[11712], 95.00th=[13248],
     | 99.00th=[23424], 99.50th=[33536], 99.90th=[40704], 99.95th=[41728],
     | 99.99th=[48896]
   bw (  KiB/s): min=301624, max=334352, per=100.00%, avg=329988.44, stdev=5462.50, samples=119
   iops        : min=75406, max=83588, avg=82497.14, stdev=1365.64, samples=119
  lat (usec)   : 10=0.36%, 20=97.76%, 50=1.87%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.21%, sys=12.09%, ctx=4948518, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4948560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=322MiB/s (338MB/s), 322MiB/s-322MiB/s (338MB/s-338MB/s), io=18.9GiB (20.3GB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=4939258/0, merge=0/0, ticks=52754/0, in_queue=52754, util=99.86%

Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) at I/O depth 16 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=16
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=325MiB/s][r=83.2k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=10494: Sat Feb 25 15:15:19 2023
  read: IOPS=83.1k, BW=325MiB/s (341MB/s)(19.0GiB/60001msec)
    clat (usec): min=8, max=470, avg=11.79, stdev= 2.77
     lat (usec): min=8, max=470, avg=11.81, stdev= 2.77
    clat percentiles (nsec):
     |  1.00th=[11072],  5.00th=[11328], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[23424], 99.50th=[34048], 99.90th=[40704], 99.95th=[41728],
     | 99.99th=[45312]
   bw (  KiB/s): min=326664, max=333792, per=100.00%, avg=332638.99, stdev=1243.39, samples=119
   iops        : min=81666, max=83448, avg=83159.75, stdev=310.85, samples=119
  lat (usec)   : 10=0.12%, 20=98.10%, 50=1.78%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.45%, sys=11.92%, ctx=4987876, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4987909,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=325MiB/s (341MB/s), 325MiB/s-325MiB/s (341MB/s-341MB/s), io=19.0GiB (20.4GB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=4978615/0, merge=0/0, ticks=52329/0, in_queue=52329, util=99.86%

Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) at I/O depth 64 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=326MiB/s][r=83.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=10529: Sat Feb 25 15:16:19 2023
  read: IOPS=82.0k, BW=320MiB/s (336MB/s)(18.8GiB/60001msec)
    clat (usec): min=8, max=2079, avg=11.94, stdev= 3.52
     lat (usec): min=8, max=2079, avg=11.96, stdev= 3.52
    clat percentiles (nsec):
     |  1.00th=[10816],  5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11456], 90.00th=[11584], 95.00th=[13120],
     | 99.00th=[23680], 99.50th=[36608], 99.90th=[47872], 99.95th=[50432],
     | 99.99th=[58112]
   bw (  KiB/s): min=307216, max=333824, per=100.00%, avg=327930.55, stdev=6598.48, samples=119
   iops        : min=76804, max=83456, avg=81982.64, stdev=1649.65, samples=119
  lat (usec)   : 10=0.17%, 20=97.73%, 50=2.04%, 100=0.05%, 250=0.01%
  lat (usec)   : 500=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=3.74%, sys=11.88%, ctx=4917719, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4917746,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=320MiB/s (336MB/s), 320MiB/s-320MiB/s (336MB/s-336MB/s), io=18.8GiB (20.1GB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=4908309/0, merge=0/0, ticks=52567/0, in_queue=52567, util=99.85%

tl;dr:

  • P1600X 118GB: ~85,000 IOps; 42 MiB/s with 512-byte blocks and 334 MiB/s with 4 KiB blocks
  • 905P 380GB: ~84,000 IOps; 41 MiB/s with 512-byte blocks and 329 MiB/s with 4 KiB blocks
  • P5800X 3.2TB: ~85,000 IOps; 42 MiB/s with 512-byte blocks and 330 MiB/s with 4 KiB blocks
  • P4800X 1.5TB: ~82,000 IOps; 40 MiB/s with 512-byte blocks and ~320 MiB/s with 4 KiB blocks
1 Like