Intel Optane 905P Series 960GB (SSDPE21D960GAM3) vs Samsung 2TB 980 Pro as boot drive on windows 10?

jared5 · February 25, 2023, 12:00am

i’m building a new pc and trying to decide on my boot drive.

the optane has much lower latency, but it seems like the samsung has better random read/write iops plus better sequential read/write. the high write endurance is an obvious benefit of the optane though.

i have thought about trying to use the 905p with primocache for my other storage, but I have not used primocache before.

edit: looks like Wendell actually made a video about this. But i would be interested in hearing more from people if they’d tried it personally Optane Latency and Why I've Been Obsessed with Intel's Fire Sale - YouTube

LiKenun · February 25, 2023, 12:29am

If it’s purely used for booting and storing your software (i.e., /home or C:\Users is a mount point for your HDD or NAND SSD), sequential read/write doesn’t add much value.

What is your PC going to be used for anyway?

jared5 · February 25, 2023, 1:17am

yeah i knew that the sequential didn’t matter that much for booting.

my question was mainly about which is better for boot drive?

980 Pro’s 1million (up to) random read/write iops
or
Optane’s 575,000 (up to) random read/write iops plus much better latency

does the theoretical almost 2x random read/write iops of the Samsung make it faster or does the way better latency of the Optane make it faster in real world day to day operation.

gaming, some software dev, video archiving, etc

LiKenun · February 25, 2023, 2:42am

Pursue latency. The reason is that day to day activities hinge on responsiveness of low queue depth I/O. Putting the operating system and software files on it takes advantage of that.

I can’t speak for games as I have not played them, but downloading League of Legends puts 18.5 GiB of files on my drive distributed over 693 files coming out to about 27.36 MiB per file. That’s something I would put on the PCIe 4.0 SSD assuming these large files will be read sequentially.

The choice of storage for video archival is a no-brainer if you are doing what I think you are doing with it (dumping them somewhere never to be seen again). I’d buy an SMR HDD if money was tight, but QLC NAND has higher speed and density, and at a low enough price it could be a viable choice. (I snagged a brand new 30.72 TB PCIe 4.0 drive for $3,000 last month.)

Software development workloads benefit heavily from low latency—at least mine. If you develop with node.js and have to deal with the millions of tiny files in node_modules, Optane is your best friend.

EDIT: You might want to take a look at this: 1TB Performance Results - Samsung 980 Pro M.2 NVMe SSD Review: Redefining Gen4 Performance | Tom's Hardware

jared5 · February 25, 2023, 12:04pm

okay then i’ll give the optane a try as my boot drive for the faster latency. to clarify i have both the optane and 980 Pro already, i was just trying to figure out which to boot off of, not whether to purchase them.

my motherboard has 4 m.2 slots.

M.2_1 Source (From CPU) supports up to PCIe 4.0 x4
M.2_2 Source (From Chipset) supports up to PCIe 4.0 x4
M.2_3 Source (From Chipset) supports up to PCIe 4.0 x4 / SATA mode
M.2_4 Source (From Chipset) supports up to PCIe 4.0 x4

since the optane is pcie 3 x4, does it make more sense to put it on one of my chipset m.2 slots? so i can give the 980 pro the full pcie 4 x4 bandwidth from the cpu? Or does the added latency of the chipset negate the whole point of picking the optane as a boot drive?

also what does pcie 4 x4 / sata mode mean?

LiKenun · February 25, 2023, 3:54pm

Optane goes here.

You are right on the money. When I benchmarked the 905P (380GB M.2 model) using AIDA64, putting the device on chipset-connected PCIe slots decreased random 4K read performance by about 8%.

c.f.: Help selecting interface for optane - #2 by LiKenun

My CPU-connected PCIe slots are 5.0, and my chipset-connected PCIe slots are 4.0. All the devices (except for the SAMSUNG 780 QVO) are PCIe 3.0.

It means you can connect either an NVMe M.2 stick or an SATA M.2 stick to it.

Void_Null · February 25, 2023, 4:28pm

Did the P1600X 118GB really outperform the 905P 380GB? Did you have it in a PCIe 4 or PCIe 5 M.2 slot? Also, did you see the same with random writes?

I’m just curious because the spec sheet lists the 905P 380GB with about 575k and 550k IOPS for reads and writes, respectively, versus the P1600X with 410k and 243k IOPS.

LiKenun · February 25, 2023, 8:33pm

Yes. I just re-ran the AIDA64 disk benchmarks and the results still hold.

Unless noted, everything was connected via the CPU PCIe 5.0 lanes.

Even with Optane media’s ridiculous endurance, I’d rather not.

Here are some more comprehensive random read performance numbers using fio. They cover four models:

P1600X 118GB (SSDPEK1A118GA)
905P 380GB (SSDPEL1D380GA)
P5800X 3.2TB (SSDPF21Q032TB)
P4800X 1.5TB (MDTPE21K015TA)

The tests are run using combinations of two block sizes (512 or 4,096 bytes) and four I/O depths (1, 4, 16, or 64). All the NVMe namespaces are configured with 512-byte logical sector sizes with no metadata.

The command:

# for device in $(find /dev -maxdepth 1 -regextype posix-extended -regex '^/dev/nvme(0|[1-9]\d*)$' | sort); do data=$(nvme id-ctrl "$device"); mn=$(echo "$data" | awk 'match($0, "^mn\\s*:\\s*(.+?)$", m) {print m[1]}' | xargs); for namespace in $(find /dev -maxdepth 1 -regextype posix-extended -regex "^$device"'n[1-9]\d*$' | sort); do for blocksize in 512 4k; do for depth in 1 4 16 64; do echo -e "\033[0;32mRandom read of device \033[1;32m$device ($mn)\033[0;32m at I/O depth $depth with block size $blocksize\033[0m"; fio --filename="$namespace" --direct=1 --rw=randread --readonly --ioengine=pvsync2 --runtime=60 --bs="$blocksize" --iodepth="$depth" --numjobs=1 --time_based --group_reporting --name=iops-test-job; echo; done; done; done; done

Enjoy.

Output

Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) at I/O depth 1 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=41.8MiB/s][r=85.5k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8106: Sat Feb 25 14:45:10 2023
  read: IOPS=85.9k, BW=42.0MiB/s (44.0MB/s)(2518MiB/60001msec)
    clat (usec): min=4, max=379, avg=11.40, stdev= 1.24
     lat (usec): min=4, max=379, avg=11.43, stdev= 1.24
    clat percentiles (nsec):
     |  1.00th=[ 5472],  5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[11968], 99.50th=[15168], 99.90th=[16768], 99.95th=[18816],
     | 99.99th=[27776]
   bw (  KiB/s): min=40647, max=67016, per=100.00%, avg=42985.08, stdev=2234.87, samples=119
   iops        : min=81294, max=134032, avg=85970.15, stdev=4469.74, samples=119
  lat (usec)   : 10=1.58%, 20=98.38%, 50=0.03%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.51%, sys=11.68%, ctx=5156668, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5156680,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=42.0MiB/s (44.0MB/s), 42.0MiB/s-42.0MiB/s (44.0MB/s-44.0MB/s), io=2518MiB (2640MB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=5147663/0, merge=0/0, ticks=52356/0, in_queue=52356, util=99.85%

Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) at I/O depth 4 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=4
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=41.7MiB/s][r=85.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8142: Sat Feb 25 14:46:10 2023
  read: IOPS=85.5k, BW=41.8MiB/s (43.8MB/s)(2506MiB/60001msec)
    clat (usec): min=4, max=657, avg=11.48, stdev= 1.07
     lat (usec): min=4, max=657, avg=11.50, stdev= 1.07
    clat percentiles (nsec):
     |  1.00th=[10816],  5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[11840], 99.50th=[15040], 99.90th=[16064], 99.95th=[18048],
     | 99.99th=[24192]
   bw (  KiB/s): min=42571, max=43785, per=100.00%, avg=42781.55, stdev=140.18, samples=119
   iops        : min=85142, max=87570, avg=85563.13, stdev=280.35, samples=119
  lat (usec)   : 10=0.56%, 20=99.41%, 50=0.03%, 100=0.01%, 250=0.01%
  lat (usec)   : 750=0.01%
  cpu          : usr=2.98%, sys=11.90%, ctx=5132399, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5132420,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=41.8MiB/s (43.8MB/s), 41.8MiB/s-41.8MiB/s (43.8MB/s-43.8MB/s), io=2506MiB (2628MB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=5123326/12, merge=0/2, ticks=52621/1, in_queue=52623, util=99.86%

Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) at I/O depth 16 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=16
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=41.8MiB/s][r=85.6k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8179: Sat Feb 25 14:47:10 2023
  read: IOPS=85.6k, BW=41.8MiB/s (43.8MB/s)(2508MiB/60001msec)
    clat (usec): min=4, max=895, avg=11.48, stdev= 1.14
     lat (usec): min=4, max=895, avg=11.50, stdev= 1.14
    clat percentiles (nsec):
     |  1.00th=[10560],  5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11584], 60.00th=[11584],
     | 70.00th=[11584], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[11968], 99.50th=[14016], 99.90th=[16064], 99.95th=[16192],
     | 99.99th=[23168]
   bw (  KiB/s): min=42631, max=50069, per=100.00%, avg=42822.87, stdev=670.75, samples=119
   iops        : min=85262, max=100138, avg=85645.75, stdev=1341.50, samples=119
  lat (usec)   : 10=0.69%, 20=99.29%, 50=0.02%, 100=0.01%, 250=0.01%
  lat (usec)   : 1000=0.01%
  cpu          : usr=3.06%, sys=12.20%, ctx=5137331, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5137350,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=41.8MiB/s (43.8MB/s), 41.8MiB/s-41.8MiB/s (43.8MB/s-43.8MB/s), io=2508MiB (2630MB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=5128190/9, merge=0/23, ticks=52427/1, in_queue=52428, util=99.85%

Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) at I/O depth 64 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=41.8MiB/s][r=85.5k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8226: Sat Feb 25 14:48:10 2023
  read: IOPS=85.5k, BW=41.8MiB/s (43.8MB/s)(2506MiB/60001msec)
    clat (usec): min=4, max=280, avg=11.47, stdev= 1.01
     lat (usec): min=4, max=280, avg=11.49, stdev= 1.01
    clat percentiles (nsec):
     |  1.00th=[10688],  5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[11840], 99.50th=[15168], 99.90th=[16064], 99.95th=[18816],
     | 99.99th=[28032]
   bw (  KiB/s): min=42626, max=45012, per=100.00%, avg=42778.61, stdev=210.98, samples=119
   iops        : min=85252, max=90026, avg=85557.24, stdev=422.14, samples=119
  lat (usec)   : 10=0.57%, 20=99.38%, 50=0.04%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.15%, sys=11.77%, ctx=5132105, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5132129,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=41.8MiB/s (43.8MB/s), 41.8MiB/s-41.8MiB/s (43.8MB/s-43.8MB/s), io=2506MiB (2628MB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=5123033/12, merge=0/3, ticks=52562/1, in_queue=52563, util=99.85%

Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) at I/O depth 1 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=334MiB/s][r=85.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8261: Sat Feb 25 14:49:11 2023
  read: IOPS=85.5k, BW=334MiB/s (350MB/s)(19.6GiB/60001msec)
    clat (nsec): min=6161, max=234810, avg=11534.74, stdev=992.08
     lat (nsec): min=6181, max=234850, avg=11551.71, stdev=992.18
    clat percentiles (nsec):
     |  1.00th=[10048],  5.00th=[11328], 10.00th=[11456], 20.00th=[11456],
     | 30.00th=[11584], 40.00th=[11584], 50.00th=[11584], 60.00th=[11584],
     | 70.00th=[11584], 80.00th=[11584], 90.00th=[11712], 95.00th=[11712],
     | 99.00th=[12480], 99.50th=[15296], 99.90th=[16064], 99.95th=[18816],
     | 99.99th=[27776]
   bw (  KiB/s): min=340848, max=401144, per=100.00%, avg=342250.49, stdev=5536.04, samples=119
   iops        : min=85212, max=100286, avg=85562.64, stdev=1384.01, samples=119
  lat (usec)   : 10=0.99%, 20=98.97%, 50=0.04%, 100=0.01%, 250=0.01%
  cpu          : usr=2.91%, sys=11.57%, ctx=5132332, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5132358,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=334MiB/s (350MB/s), 334MiB/s-334MiB/s (350MB/s-350MB/s), io=19.6GiB (21.0GB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=5123652/3, merge=0/7, ticks=52849/0, in_queue=52850, util=99.87%

Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) at I/O depth 4 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=4
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=333MiB/s][r=85.3k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8297: Sat Feb 25 14:50:11 2023
  read: IOPS=85.3k, BW=333MiB/s (350MB/s)(19.5GiB/60001msec)
    clat (nsec): min=6251, max=292708, avg=11566.22, stdev=944.08
     lat (nsec): min=6272, max=292728, avg=11583.96, stdev=944.35
    clat percentiles (nsec):
     |  1.00th=[11200],  5.00th=[11328], 10.00th=[11456], 20.00th=[11456],
     | 30.00th=[11584], 40.00th=[11584], 50.00th=[11584], 60.00th=[11584],
     | 70.00th=[11584], 80.00th=[11584], 90.00th=[11712], 95.00th=[11712],
     | 99.00th=[12352], 99.50th=[14656], 99.90th=[16064], 99.95th=[16320],
     | 99.99th=[24448]
   bw (  KiB/s): min=340944, max=341840, per=100.00%, avg=341454.32, stdev=258.79, samples=119
   iops        : min=85236, max=85460, avg=85363.60, stdev=64.72, samples=119
  lat (usec)   : 10=0.33%, 20=99.65%, 50=0.02%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=2.78%, sys=12.10%, ctx=5120444, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5120488,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=333MiB/s (350MB/s), 333MiB/s-333MiB/s (350MB/s-350MB/s), io=19.5GiB (21.0GB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=5111744/17, merge=0/25, ticks=52637/1, in_queue=52638, util=99.86%

Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) at I/O depth 16 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=16
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=334MiB/s][r=85.5k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8334: Sat Feb 25 14:51:11 2023
  read: IOPS=85.5k, BW=334MiB/s (350MB/s)(19.6GiB/60001msec)
    clat (nsec): min=6152, max=292407, avg=11548.75, stdev=961.55
     lat (nsec): min=6171, max=292417, avg=11565.66, stdev=961.63
    clat percentiles (nsec):
     |  1.00th=[11200],  5.00th=[11328], 10.00th=[11456], 20.00th=[11456],
     | 30.00th=[11584], 40.00th=[11584], 50.00th=[11584], 60.00th=[11584],
     | 70.00th=[11584], 80.00th=[11584], 90.00th=[11712], 95.00th=[11712],
     | 99.00th=[12480], 99.50th=[15296], 99.90th=[18048], 99.95th=[19072],
     | 99.99th=[23680]
   bw (  KiB/s): min=340600, max=378496, per=100.00%, avg=341900.30, stdev=3412.33, samples=119
   iops        : min=85150, max=94624, avg=85475.09, stdev=853.08, samples=119
  lat (usec)   : 10=0.74%, 20=99.23%, 50=0.03%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=2.62%, sys=11.81%, ctx=5127095, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5127117,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=334MiB/s (350MB/s), 334MiB/s-334MiB/s (350MB/s-350MB/s), io=19.6GiB (21.0GB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=5118363/10, merge=0/0, ticks=52635/0, in_queue=52635, util=99.86%

Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) at I/O depth 64 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=334MiB/s][r=85.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8369: Sat Feb 25 14:52:11 2023
  read: IOPS=85.4k, BW=334MiB/s (350MB/s)(19.5GiB/60001msec)
    clat (usec): min=6, max=966, avg=11.56, stdev= 1.01
     lat (usec): min=6, max=966, avg=11.58, stdev= 1.01
    clat percentiles (nsec):
     |  1.00th=[11200],  5.00th=[11328], 10.00th=[11456], 20.00th=[11456],
     | 30.00th=[11584], 40.00th=[11584], 50.00th=[11584], 60.00th=[11584],
     | 70.00th=[11584], 80.00th=[11584], 90.00th=[11712], 95.00th=[11712],
     | 99.00th=[12480], 99.50th=[15040], 99.90th=[16192], 99.95th=[18816],
     | 99.99th=[23936]
   bw (  KiB/s): min=340112, max=344384, per=100.00%, avg=341638.52, stdev=447.66, samples=119
   iops        : min=85028, max=86096, avg=85409.61, stdev=111.93, samples=119
  lat (usec)   : 10=0.49%, 20=99.48%, 50=0.03%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%, 1000=0.01%
  cpu          : usr=2.85%, sys=11.67%, ctx=5123108, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5123138,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=334MiB/s (350MB/s), 334MiB/s-334MiB/s (350MB/s-350MB/s), io=19.5GiB (21.0GB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=5114416/4, merge=0/16, ticks=52719/0, in_queue=52719, util=99.87%

Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) at I/O depth 1 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=41.2MiB/s][r=84.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8417: Sat Feb 25 14:53:12 2023
  read: IOPS=84.3k, BW=41.2MiB/s (43.2MB/s)(2471MiB/60001msec)
    clat (usec): min=6, max=275, avg=11.63, stdev= 1.68
     lat (usec): min=6, max=275, avg=11.65, stdev= 1.68
    clat percentiles (nsec):
     |  1.00th=[11072],  5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[19840], 99.50th=[20864], 99.90th=[36096], 99.95th=[39168],
     | 99.99th=[41216]
   bw (  KiB/s): min=42113, max=42245, per=100.00%, avg=42178.82, stdev=38.22, samples=119
   iops        : min=84226, max=84490, avg=84357.65, stdev=76.49, samples=119
  lat (usec)   : 10=0.14%, 20=98.98%, 50=0.88%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.67%, sys=11.31%, ctx=5060065, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5060056,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=41.2MiB/s (43.2MB/s), 41.2MiB/s-41.2MiB/s (43.2MB/s-43.2MB/s), io=2471MiB (2591MB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=5049070/0, merge=0/0, ticks=52393/0, in_queue=52393, util=99.82%

Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) at I/O depth 4 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=4
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=40.9MiB/s][r=83.7k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8452: Sat Feb 25 14:54:12 2023
  read: IOPS=84.3k, BW=41.1MiB/s (43.1MB/s)(2469MiB/60001msec)
    clat (usec): min=6, max=1913, avg=11.63, stdev= 2.55
     lat (usec): min=6, max=1913, avg=11.65, stdev= 2.55
    clat percentiles (nsec):
     |  1.00th=[10816],  5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[19840], 99.50th=[21632], 99.90th=[36608], 99.95th=[39168],
     | 99.99th=[44288]
   bw (  KiB/s): min=41374, max=42386, per=100.00%, avg=42147.93, stdev=129.34, samples=119
   iops        : min=82748, max=84772, avg=84295.87, stdev=258.69, samples=119
  lat (usec)   : 10=0.54%, 20=98.56%, 50=0.90%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=3.38%, sys=11.77%, ctx=5055956, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5055965,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=41.1MiB/s (43.1MB/s), 41.1MiB/s-41.1MiB/s (43.1MB/s-43.1MB/s), io=2469MiB (2589MB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=5045792/0, merge=0/0, ticks=52354/0, in_queue=52354, util=99.83%

Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) at I/O depth 16 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=16
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=41.1MiB/s][r=84.3k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8490: Sat Feb 25 14:55:12 2023
  read: IOPS=84.2k, BW=41.1MiB/s (43.1MB/s)(2465MiB/60001msec)
    clat (usec): min=6, max=385, avg=11.64, stdev= 2.70
     lat (usec): min=6, max=386, avg=11.66, stdev= 2.70
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   12], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   12], 95.00th=[   12],
     | 99.00th=[   20], 99.50th=[   22], 99.90th=[   37], 99.95th=[   40],
     | 99.99th=[  212]
   bw (  KiB/s): min=41462, max=42352, per=100.00%, avg=42089.20, stdev=177.68, samples=119
   iops        : min=82924, max=84704, avg=84178.40, stdev=355.34, samples=119
  lat (usec)   : 10=0.40%, 20=98.69%, 50=0.89%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.42%, sys=11.38%, ctx=5049175, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5049177,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=41.1MiB/s (43.1MB/s), 41.1MiB/s-41.1MiB/s (43.1MB/s-43.1MB/s), io=2465MiB (2585MB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=5038924/0, merge=0/0, ticks=52528/0, in_queue=52528, util=99.84%

Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) at I/O depth 64 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=40.8MiB/s][r=83.6k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8525: Sat Feb 25 14:56:13 2023
  read: IOPS=84.3k, BW=41.1MiB/s (43.1MB/s)(2468MiB/60001msec)
    clat (usec): min=6, max=332, avg=11.62, stdev= 2.05
     lat (usec): min=7, max=332, avg=11.64, stdev= 2.05
    clat percentiles (nsec):
     |  1.00th=[11072],  5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11456], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[19840], 99.50th=[21120], 99.90th=[36608], 99.95th=[39168],
     | 99.99th=[41728]
   bw (  KiB/s): min=41351, max=42234, per=100.00%, avg=42139.60, stdev=115.92, samples=119
   iops        : min=82702, max=84468, avg=84279.21, stdev=231.84, samples=119
  lat (usec)   : 10=0.21%, 20=98.90%, 50=0.89%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.69%, sys=11.58%, ctx=5055297, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5055300,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=41.1MiB/s (43.1MB/s), 41.1MiB/s-41.1MiB/s (43.1MB/s-43.1MB/s), io=2468MiB (2588MB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=5045050/0, merge=0/0, ticks=52271/0, in_queue=52271, util=99.84%

Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) at I/O depth 1 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=329MiB/s][r=84.2k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8560: Sat Feb 25 14:57:13 2023
  read: IOPS=84.2k, BW=329MiB/s (345MB/s)(19.3GiB/60001msec)
    clat (usec): min=8, max=1202, avg=11.66, stdev= 2.12
     lat (usec): min=8, max=1202, avg=11.68, stdev= 2.12
    clat percentiles (nsec):
     |  1.00th=[11072],  5.00th=[11328], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11584], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[19840], 99.50th=[21888], 99.90th=[36608], 99.95th=[39168],
     | 99.99th=[41728]
   bw (  KiB/s): min=332344, max=337848, per=100.00%, avg=336867.40, stdev=1005.50, samples=119
   iops        : min=83086, max=84462, avg=84216.87, stdev=251.38, samples=119
  lat (usec)   : 10=0.17%, 20=98.90%, 50=0.93%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=3.34%, sys=11.62%, ctx=5051521, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5051528,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=329MiB/s (345MB/s), 329MiB/s-329MiB/s (345MB/s-345MB/s), io=19.3GiB (20.7GB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=5042383/0, merge=0/0, ticks=52458/0, in_queue=52458, util=99.87%

Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) at I/O depth 4 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=4
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=328MiB/s][r=84.1k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8599: Sat Feb 25 14:58:13 2023
  read: IOPS=84.1k, BW=329MiB/s (345MB/s)(19.3GiB/60001msec)
    clat (usec): min=8, max=388, avg=11.66, stdev= 2.23
     lat (usec): min=8, max=388, avg=11.68, stdev= 2.23
    clat percentiles (nsec):
     |  1.00th=[10944],  5.00th=[11328], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11584], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[19840], 99.50th=[21888], 99.90th=[36608], 99.95th=[39168],
     | 99.99th=[43264]
   bw (  KiB/s): min=328096, max=337816, per=100.00%, avg=336692.97, stdev=1433.86, samples=119
   iops        : min=82024, max=84454, avg=84173.24, stdev=358.50, samples=119
  lat (usec)   : 10=0.23%, 20=98.82%, 50=0.94%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.48%, sys=11.58%, ctx=5048664, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5048673,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=329MiB/s (345MB/s), 329MiB/s-329MiB/s (345MB/s-345MB/s), io=19.3GiB (20.7GB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=5039538/0, merge=0/0, ticks=52440/0, in_queue=52440, util=99.86%

Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) at I/O depth 16 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=16
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=329MiB/s][r=84.2k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8635: Sat Feb 25 14:59:13 2023
  read: IOPS=84.2k, BW=329MiB/s (345MB/s)(19.3GiB/60001msec)
    clat (usec): min=8, max=395, avg=11.66, stdev= 2.18
     lat (usec): min=8, max=395, avg=11.68, stdev= 2.18
    clat percentiles (nsec):
     |  1.00th=[10944],  5.00th=[11328], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11584], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[19840], 99.50th=[21888], 99.90th=[36608], 99.95th=[39168],
     | 99.99th=[42240]
   bw (  KiB/s): min=331144, max=337832, per=100.00%, avg=336725.45, stdev=1154.46, samples=119
   iops        : min=82786, max=84458, avg=84181.38, stdev=288.63, samples=119
  lat (usec)   : 10=0.24%, 20=98.83%, 50=0.93%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.28%, sys=11.73%, ctx=5049526, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5049536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=329MiB/s (345MB/s), 329MiB/s-329MiB/s (345MB/s-345MB/s), io=19.3GiB (20.7GB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=5040365/0, merge=0/0, ticks=52441/0, in_queue=52441, util=99.86%

Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) at I/O depth 64 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=329MiB/s][r=84.3k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8671: Sat Feb 25 15:00:14 2023
  read: IOPS=84.2k, BW=329MiB/s (345MB/s)(19.3GiB/60001msec)
    clat (usec): min=8, max=318, avg=11.66, stdev= 2.03
     lat (usec): min=8, max=318, avg=11.68, stdev= 2.04
    clat percentiles (nsec):
     |  1.00th=[11072],  5.00th=[11328], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11584], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[19840], 99.50th=[21888], 99.90th=[36608], 99.95th=[39168],
     | 99.99th=[41728]
   bw (  KiB/s): min=332704, max=337904, per=100.00%, avg=337005.24, stdev=899.24, samples=119
   iops        : min=83176, max=84476, avg=84251.33, stdev=224.80, samples=119
  lat (usec)   : 10=0.16%, 20=98.92%, 50=0.92%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.36%, sys=11.64%, ctx=5053532, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5053551,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=329MiB/s (345MB/s), 329MiB/s-329MiB/s (345MB/s-345MB/s), io=19.3GiB (20.7GB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=5044355/0, merge=0/0, ticks=52416/0, in_queue=52416, util=99.86%

Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) at I/O depth 1 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
Jobs: 1 (f=1): [r(1)][100.0%][r=41.7MiB/s][r=85.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8716: Sat Feb 25 15:01:14 2023
  read: IOPS=85.3k, BW=41.7MiB/s (43.7MB/s)(2500MiB/60001msec)
    clat (usec): min=4, max=354, avg=11.44, stdev= 2.10
     lat (usec): min=4, max=354, avg=11.46, stdev= 2.10
    clat percentiles (nsec):
     |  1.00th=[10176],  5.00th=[11200], 10.00th=[11328], 20.00th=[11328],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11456], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[12096], 99.50th=[16064], 99.90th=[20608], 99.95th=[20864],
     | 99.99th=[53504]
   bw (  KiB/s): min=40278, max=43410, per=100.00%, avg=42686.28, stdev=283.65, samples=119
   iops        : min=80556, max=86822, avg=85372.57, stdev=567.34, samples=119
  lat (usec)   : 10=0.96%, 20=98.69%, 50=0.34%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=4.12%, sys=11.94%, ctx=5120865, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5120860,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=41.7MiB/s (43.7MB/s), 41.7MiB/s-41.7MiB/s (43.7MB/s-43.7MB/s), io=2500MiB (2622MB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=5100271/0, merge=0/0, ticks=51705/0, in_queue=51705, util=99.61%

Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) at I/O depth 4 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=4
fio-3.28
Starting 1 process
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
Jobs: 1 (f=1): [r(1)][100.0%][r=42.0MiB/s][r=86.0k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8763: Sat Feb 25 15:02:15 2023
  read: IOPS=85.2k, BW=41.6MiB/s (43.6MB/s)(2495MiB/60001msec)
    clat (usec): min=4, max=1131, avg=11.45, stdev= 2.75
     lat (usec): min=4, max=1131, avg=11.47, stdev= 2.75
    clat percentiles (usec):
     |  1.00th=[    6],  5.00th=[   12], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   12], 95.00th=[   12],
     | 99.00th=[   16], 99.50th=[   20], 99.90th=[   22], 99.95th=[   26],
     | 99.99th=[  215]
   bw (  KiB/s): min=41126, max=44395, per=100.00%, avg=42602.17, stdev=436.29, samples=119
   iops        : min=82252, max=88790, avg=85204.34, stdev=872.57, samples=119
  lat (usec)   : 10=1.92%, 20=97.64%, 50=0.43%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%, 750=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=4.11%, sys=12.38%, ctx=5110592, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5110592,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=41.6MiB/s (43.6MB/s), 41.6MiB/s-41.6MiB/s (43.6MB/s-43.6MB/s), io=2495MiB (2617MB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=5091009/0, merge=0/0, ticks=51783/0, in_queue=51783, util=99.64%

Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) at I/O depth 16 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=16
fio-3.28
Starting 1 process
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
Jobs: 1 (f=1): [r(1)][100.0%][r=41.7MiB/s][r=85.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=8968: Sat Feb 25 15:03:15 2023
  read: IOPS=85.3k, BW=41.6MiB/s (43.7MB/s)(2499MiB/60001msec)
    clat (usec): min=4, max=473, avg=11.44, stdev= 3.68
     lat (usec): min=4, max=473, avg=11.46, stdev= 3.68
    clat percentiles (usec):
     |  1.00th=[    6],  5.00th=[   11], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   12], 95.00th=[   12],
     | 99.00th=[   16], 99.50th=[   20], 99.90th=[   23], 99.95th=[   30],
     | 99.99th=[  221]
   bw (  KiB/s): min=39652, max=43679, per=100.00%, avg=42663.71, stdev=452.34, samples=119
   iops        : min=79305, max=87358, avg=85327.45, stdev=904.63, samples=119
  lat (usec)   : 10=2.63%, 20=96.92%, 50=0.42%, 100=0.01%, 250=0.02%
  lat (usec)   : 500=0.01%
  cpu          : usr=4.28%, sys=12.13%, ctx=5117554, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5117557,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=41.6MiB/s (43.7MB/s), 41.6MiB/s-41.6MiB/s (43.7MB/s-43.7MB/s), io=2499MiB (2620MB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=5097707/0, merge=0/0, ticks=51954/0, in_queue=51954, util=99.63%

Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) at I/O depth 64 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=64
fio-3.28
Starting 1 process
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
Jobs: 1 (f=1): [r(1)][100.0%][r=41.7MiB/s][r=85.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=9051: Sat Feb 25 15:04:15 2023
  read: IOPS=85.2k, BW=41.6MiB/s (43.6MB/s)(2497MiB/60001msec)
    clat (usec): min=4, max=575, avg=11.42, stdev= 2.92
     lat (usec): min=4, max=575, avg=11.44, stdev= 2.92
    clat percentiles (usec):
     |  1.00th=[    6],  5.00th=[   12], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   12], 95.00th=[   12],
     | 99.00th=[   16], 99.50th=[   20], 99.90th=[   22], 99.95th=[   25],
     | 99.99th=[  215]
   bw (  KiB/s): min=41338, max=43458, per=100.00%, avg=42635.02, stdev=276.50, samples=119
   iops        : min=82676, max=86916, avg=85270.07, stdev=553.01, samples=119
  lat (usec)   : 10=1.88%, 20=97.70%, 50=0.40%, 100=0.01%, 250=0.02%
  lat (usec)   : 500=0.01%, 750=0.01%
  cpu          : usr=4.51%, sys=12.00%, ctx=5114450, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5114438,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=41.6MiB/s (43.6MB/s), 41.6MiB/s-41.6MiB/s (43.6MB/s-43.6MB/s), io=2497MiB (2619MB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=5094581/0, merge=0/0, ticks=51732/0, in_queue=51732, util=99.64%

Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) at I/O depth 1 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=327MiB/s][r=83.8k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=9234: Sat Feb 25 15:05:16 2023
  read: IOPS=84.8k, BW=331MiB/s (347MB/s)(19.4GiB/60001msec)
    clat (usec): min=6, max=393, avg=11.55, stdev= 1.94
     lat (usec): min=6, max=393, avg=11.57, stdev= 1.94
    clat percentiles (nsec):
     |  1.00th=[10816],  5.00th=[11200], 10.00th=[11328], 20.00th=[11328],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11456], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[15296], 99.50th=[20096], 99.90th=[21120], 99.95th=[22656],
     | 99.99th=[33024]
   bw (  KiB/s): min=335808, max=340968, per=100.00%, avg=339286.25, stdev=1022.91, samples=119
   iops        : min=83952, max=85242, avg=84821.56, stdev=255.73, samples=119
  lat (usec)   : 10=0.45%, 20=99.04%, 50=0.50%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.72%, sys=11.72%, ctx=5086854, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5086847,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=331MiB/s (347MB/s), 331MiB/s-331MiB/s (347MB/s-347MB/s), io=19.4GiB (20.8GB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=5076003/0, merge=0/0, ticks=52209/0, in_queue=52209, util=99.82%

Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) at I/O depth 4 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=4
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=331MiB/s][r=84.7k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=9275: Sat Feb 25 15:06:16 2023
  read: IOPS=84.2k, BW=329MiB/s (345MB/s)(19.3GiB/60001msec)
    clat (usec): min=6, max=1571, avg=11.61, stdev= 3.31
     lat (usec): min=6, max=1571, avg=11.62, stdev= 3.31
    clat percentiles (usec):
     |  1.00th=[   10],  5.00th=[   12], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   12], 95.00th=[   13],
     | 99.00th=[   19], 99.50th=[   21], 99.90th=[   24], 99.95th=[   29],
     | 99.99th=[  217]
   bw (  KiB/s): min=323704, max=340248, per=100.00%, avg=336765.18, stdev=3891.43, samples=119
   iops        : min=80926, max=85062, avg=84191.31, stdev=972.85, samples=119
  lat (usec)   : 10=1.19%, 20=98.15%, 50=0.63%, 100=0.01%, 250=0.02%
  lat (usec)   : 500=0.01%, 750=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=4.14%, sys=12.40%, ctx=5049669, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5049668,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=329MiB/s (345MB/s), 329MiB/s-329MiB/s (345MB/s-345MB/s), io=19.3GiB (20.7GB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=5039313/0, merge=0/0, ticks=51934/0, in_queue=51934, util=99.85%

Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) at I/O depth 16 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=16
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=332MiB/s][r=84.9k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=9393: Sat Feb 25 15:07:16 2023
  read: IOPS=84.7k, BW=331MiB/s (347MB/s)(19.4GiB/60001msec)
    clat (usec): min=6, max=392, avg=11.56, stdev= 2.48
     lat (usec): min=6, max=392, avg=11.58, stdev= 2.48
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   12], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   12], 95.00th=[   12],
     | 99.00th=[   16], 99.50th=[   20], 99.90th=[   22], 99.95th=[   24],
     | 99.99th=[  212]
   bw (  KiB/s): min=334728, max=340728, per=100.00%, avg=338763.50, stdev=1178.48, samples=119
   iops        : min=83682, max=85182, avg=84690.87, stdev=294.62, samples=119
  lat (usec)   : 10=0.54%, 20=98.98%, 50=0.47%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.65%, sys=11.34%, ctx=5079832, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5079831,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=331MiB/s (347MB/s), 331MiB/s-331MiB/s (347MB/s-347MB/s), io=19.4GiB (20.8GB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=5069411/0, merge=0/0, ticks=52465/0, in_queue=52465, util=99.83%

Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) at I/O depth 64 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=329MiB/s][r=84.2k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=9560: Sat Feb 25 15:08:17 2023
  read: IOPS=84.6k, BW=330MiB/s (346MB/s)(19.4GiB/60001msec)
    clat (usec): min=6, max=406, avg=11.57, stdev= 2.62
     lat (usec): min=6, max=406, avg=11.59, stdev= 2.62
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   12], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   12], 95.00th=[   12],
     | 99.00th=[   17], 99.50th=[   21], 99.90th=[   23], 99.95th=[   29],
     | 99.99th=[  212]
   bw (  KiB/s): min=323560, max=341672, per=100.00%, avg=338359.13, stdev=2708.53, samples=119
   iops        : min=80890, max=85418, avg=84589.78, stdev=677.13, samples=119
  lat (usec)   : 10=0.80%, 20=98.59%, 50=0.59%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.88%, sys=11.90%, ctx=5073604, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5073624,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=330MiB/s (346MB/s), 330MiB/s-330MiB/s (346MB/s-346MB/s), io=19.4GiB (20.8GB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=5062697/0, merge=0/0, ticks=52126/0, in_queue=52126, util=99.82%

Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) at I/O depth 1 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=40.6MiB/s][r=83.1k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=9616: Sat Feb 25 15:09:17 2023
  read: IOPS=82.3k, BW=40.2MiB/s (42.2MB/s)(2412MiB/60001msec)
    clat (usec): min=6, max=1234, avg=11.84, stdev= 4.16
     lat (usec): min=6, max=1234, avg=11.86, stdev= 4.16
    clat percentiles (usec):
     |  1.00th=[   10],  5.00th=[   12], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   12], 95.00th=[   13],
     | 99.00th=[   24], 99.50th=[   37], 99.90th=[   42], 99.95th=[   45],
     | 99.99th=[  225]
   bw (  KiB/s): min=39535, max=41814, per=100.00%, avg=41186.59, stdev=568.35, samples=119
   iops        : min=79070, max=83628, avg=82373.19, stdev=1136.71, samples=119
  lat (usec)   : 10=1.30%, 20=96.75%, 50=1.91%, 100=0.01%, 250=0.02%
  lat (usec)   : 500=0.01%, 750=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=3.90%, sys=12.43%, ctx=4940749, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4940775,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=40.2MiB/s (42.2MB/s), 40.2MiB/s-40.2MiB/s (42.2MB/s-42.2MB/s), io=2412MiB (2530MB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=4927054/0, merge=0/0, ticks=52222/0, in_queue=52222, util=99.76%

Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) at I/O depth 4 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=4
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=40.7MiB/s][r=83.3k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=9691: Sat Feb 25 15:10:17 2023
  read: IOPS=82.4k, BW=40.2MiB/s (42.2MB/s)(2413MiB/60001msec)
    clat (usec): min=6, max=533, avg=11.84, stdev= 4.36
     lat (usec): min=6, max=533, avg=11.85, stdev= 4.36
    clat percentiles (usec):
     |  1.00th=[    9],  5.00th=[   11], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   12], 95.00th=[   13],
     | 99.00th=[   24], 99.50th=[   37], 99.90th=[   42], 99.95th=[   45],
     | 99.99th=[  225]
   bw (  KiB/s): min=39615, max=41767, per=100.00%, avg=41198.88, stdev=499.84, samples=119
   iops        : min=79230, max=83534, avg=82397.78, stdev=999.66, samples=119
  lat (usec)   : 10=1.60%, 20=96.54%, 50=1.83%, 100=0.01%, 250=0.02%
  lat (usec)   : 500=0.01%, 750=0.01%
  cpu          : usr=4.15%, sys=12.56%, ctx=4942218, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4942241,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=40.2MiB/s (42.2MB/s), 40.2MiB/s-40.2MiB/s (42.2MB/s-42.2MB/s), io=2413MiB (2530MB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=4927366/0, merge=0/0, ticks=52047/0, in_queue=52047, util=99.75%

Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) at I/O depth 16 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=16
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=38.5MiB/s][r=78.8k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=9879: Sat Feb 25 15:11:18 2023
  read: IOPS=81.3k, BW=39.7MiB/s (41.6MB/s)(2381MiB/60001msec)
    clat (usec): min=6, max=2919, avg=11.97, stdev= 4.69
     lat (usec): min=6, max=2919, avg=11.99, stdev= 4.69
    clat percentiles (usec):
     |  1.00th=[    9],  5.00th=[   11], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   13], 95.00th=[   16],
     | 99.00th=[   24], 99.50th=[   37], 99.90th=[   43], 99.95th=[   46],
     | 99.99th=[  225]
   bw (  KiB/s): min=37191, max=41980, per=100.00%, avg=40648.64, stdev=1046.40, samples=119
   iops        : min=74382, max=83960, avg=81297.29, stdev=2092.82, samples=119
  lat (usec)   : 10=2.18%, 20=95.68%, 50=2.11%, 100=0.01%, 250=0.02%
  lat (usec)   : 500=0.01%, 750=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=4.56%, sys=13.62%, ctx=4876034, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4876057,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=39.7MiB/s (41.6MB/s), 39.7MiB/s-39.7MiB/s (41.6MB/s-41.6MB/s), io=2381MiB (2497MB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=4862356/0, merge=0/0, ticks=52027/0, in_queue=52027, util=99.77%

Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) at I/O depth 64 with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=40.4MiB/s][r=82.7k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=10045: Sat Feb 25 15:12:18 2023
  read: IOPS=81.2k, BW=39.6MiB/s (41.6MB/s)(2379MiB/60001msec)
    clat (usec): min=6, max=5477, avg=11.97, stdev= 4.87
     lat (usec): min=6, max=5477, avg=11.99, stdev= 4.87
    clat percentiles (usec):
     |  1.00th=[    9],  5.00th=[   11], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   13], 95.00th=[   16],
     | 99.00th=[   24], 99.50th=[   37], 99.90th=[   42], 99.95th=[   45],
     | 99.99th=[  223]
   bw (  KiB/s): min=38304, max=42250, per=100.00%, avg=40612.75, stdev=968.72, samples=119
   iops        : min=76608, max=84500, avg=81225.50, stdev=1937.46, samples=119
  lat (usec)   : 10=2.00%, 20=95.85%, 50=2.13%, 100=0.01%, 250=0.02%
  lat (usec)   : 500=0.01%, 750=0.01%
  lat (msec)   : 10=0.01%
  cpu          : usr=4.59%, sys=13.61%, ctx=4871350, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4871377,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=39.6MiB/s (41.6MB/s), 39.6MiB/s-39.6MiB/s (41.6MB/s-41.6MB/s), io=2379MiB (2494MB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=4857033/0, merge=0/0, ticks=51993/0, in_queue=51993, util=99.74%

Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) at I/O depth 1 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=317MiB/s][r=81.3k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=10346: Sat Feb 25 15:13:18 2023
  read: IOPS=80.5k, BW=314MiB/s (330MB/s)(18.4GiB/60001msec)
    clat (usec): min=8, max=503, avg=12.13, stdev= 4.13
     lat (usec): min=8, max=503, avg=12.15, stdev= 4.13
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   12], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   14], 95.00th=[   16],
     | 99.00th=[   24], 99.50th=[   37], 99.90th=[   42], 99.95th=[   45],
     | 99.99th=[  221]
   bw (  KiB/s): min=289368, max=334264, per=100.00%, avg=322181.45, stdev=11085.45, samples=119
   iops        : min=72342, max=83566, avg=80545.38, stdev=2771.36, samples=119
  lat (usec)   : 10=0.93%, 20=96.76%, 50=2.29%, 100=0.01%, 250=0.02%
  lat (usec)   : 500=0.01%, 750=0.01%
  cpu          : usr=4.37%, sys=13.09%, ctx=4830711, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4830746,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=314MiB/s (330MB/s), 314MiB/s-314MiB/s (330MB/s-330MB/s), io=18.4GiB (19.8GB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=4821800/0, merge=0/0, ticks=52322/0, in_queue=52322, util=99.87%

Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) at I/O depth 4 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=4
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=323MiB/s][r=82.8k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=10455: Sat Feb 25 15:14:18 2023
  read: IOPS=82.5k, BW=322MiB/s (338MB/s)(18.9GiB/60001msec)
    clat (usec): min=8, max=439, avg=11.89, stdev= 3.02
     lat (usec): min=8, max=439, avg=11.91, stdev= 3.02
    clat percentiles (nsec):
     |  1.00th=[10688],  5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11584], 90.00th=[11712], 95.00th=[13248],
     | 99.00th=[23424], 99.50th=[33536], 99.90th=[40704], 99.95th=[41728],
     | 99.99th=[48896]
   bw (  KiB/s): min=301624, max=334352, per=100.00%, avg=329988.44, stdev=5462.50, samples=119
   iops        : min=75406, max=83588, avg=82497.14, stdev=1365.64, samples=119
  lat (usec)   : 10=0.36%, 20=97.76%, 50=1.87%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.21%, sys=12.09%, ctx=4948518, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4948560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=322MiB/s (338MB/s), 322MiB/s-322MiB/s (338MB/s-338MB/s), io=18.9GiB (20.3GB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=4939258/0, merge=0/0, ticks=52754/0, in_queue=52754, util=99.86%

Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) at I/O depth 16 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=16
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=325MiB/s][r=83.2k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=10494: Sat Feb 25 15:15:19 2023
  read: IOPS=83.1k, BW=325MiB/s (341MB/s)(19.0GiB/60001msec)
    clat (usec): min=8, max=470, avg=11.79, stdev= 2.77
     lat (usec): min=8, max=470, avg=11.81, stdev= 2.77
    clat percentiles (nsec):
     |  1.00th=[11072],  5.00th=[11328], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[23424], 99.50th=[34048], 99.90th=[40704], 99.95th=[41728],
     | 99.99th=[45312]
   bw (  KiB/s): min=326664, max=333792, per=100.00%, avg=332638.99, stdev=1243.39, samples=119
   iops        : min=81666, max=83448, avg=83159.75, stdev=310.85, samples=119
  lat (usec)   : 10=0.12%, 20=98.10%, 50=1.78%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.45%, sys=11.92%, ctx=4987876, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4987909,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=325MiB/s (341MB/s), 325MiB/s-325MiB/s (341MB/s-341MB/s), io=19.0GiB (20.4GB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=4978615/0, merge=0/0, ticks=52329/0, in_queue=52329, util=99.86%

Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) at I/O depth 64 with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=64
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=326MiB/s][r=83.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=10529: Sat Feb 25 15:16:19 2023
  read: IOPS=82.0k, BW=320MiB/s (336MB/s)(18.8GiB/60001msec)
    clat (usec): min=8, max=2079, avg=11.94, stdev= 3.52
     lat (usec): min=8, max=2079, avg=11.96, stdev= 3.52
    clat percentiles (nsec):
     |  1.00th=[10816],  5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11456], 90.00th=[11584], 95.00th=[13120],
     | 99.00th=[23680], 99.50th=[36608], 99.90th=[47872], 99.95th=[50432],
     | 99.99th=[58112]
   bw (  KiB/s): min=307216, max=333824, per=100.00%, avg=327930.55, stdev=6598.48, samples=119
   iops        : min=76804, max=83456, avg=81982.64, stdev=1649.65, samples=119
  lat (usec)   : 10=0.17%, 20=97.73%, 50=2.04%, 100=0.05%, 250=0.01%
  lat (usec)   : 500=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=3.74%, sys=11.88%, ctx=4917719, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4917746,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=320MiB/s (336MB/s), 320MiB/s-320MiB/s (336MB/s-336MB/s), io=18.8GiB (20.1GB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=4908309/0, merge=0/0, ticks=52567/0, in_queue=52567, util=99.85%

tl;dr:

P1600X 118GB: ~85,000 IOps; 42 MiB/s with 512-byte blocks and 334 MiB/s with 4 KiB blocks
905P 380GB: ~84,000 IOps; 41 MiB/s with 512-byte blocks and 329 MiB/s with 4 KiB blocks
P5800X 3.2TB: ~85,000 IOps; 42 MiB/s with 512-byte blocks and 330 MiB/s with 4 KiB blocks
P4800X 1.5TB: ~82,000 IOps; 40 MiB/s with 512-byte blocks and ~320 MiB/s with 4 KiB blocks

Void_Null · February 25, 2023, 9:14pm

Thanks! These results are very useful.

LiKenun · February 25, 2023, 9:52pm

There is more:

So how do you hit those high advertised IOps? You have more processes hitting the devices! The next test varies the number of jobs instead of I/O depth.

The tests are run using combinations of two block sizes (512 or 4,096 bytes) and four job counts (1, 4, 16, or 64). All the NVMe namespaces are configured with 512-byte logical sector sizes with no metadata.

The command:

# for device in $(find /dev -maxdepth 1 -regextype posix-extended -regex '^/dev/nvme(0|[1-9]\d*)$' | sort); do data=$(nvme id-ctrl "$device"); mn=$(echo "$data" | awk 'match($0, "^mn\\s*:\\s*(.+?)$", m) {print m[1]}' | xargs); for namespace in $(find /dev -maxdepth 1 -regextype posix-extended -regex "^$device"'n[1-9]\d*$' | sort); do for blocksize in 512 4k; do for jobs in 1 4 16 64; do echo -e "\033[0;32mRandom read of device \033[1;32m$device ($mn)\033[0;32m using $jobs job(s) with block size $blocksize\033[0m"; fio --filename="$namespace" --direct=1 --rw=randread --readonly --ioengine=pvsync2 --runtime=60 --bs="$blocksize" --iodepth=1 --numjobs="$jobs" --time_based --group_reporting --name=iops-test-job; echo; done; done; done; done

Enjoy part 2.

Output

Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) using 1 job(s) with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=41.8MiB/s][r=85.5k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=5729: Sat Feb 25 16:03:09 2023
  read: IOPS=85.9k, BW=41.9MiB/s (44.0MB/s)(2516MiB/60001msec)
    clat (usec): min=4, max=393, avg=11.42, stdev= 1.35
     lat (usec): min=4, max=393, avg=11.44, stdev= 1.35
    clat percentiles (nsec):
     |  1.00th=[ 5536],  5.00th=[11328], 10.00th=[11456], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11584], 90.00th=[11584], 95.00th=[11584],
     | 99.00th=[11840], 99.50th=[13248], 99.90th=[15936], 99.95th=[15936],
     | 99.99th=[24192]
   bw (  KiB/s): min=42701, max=60722, per=100.00%, avg=42955.65, stdev=1673.93, samples=119
   iops        : min=85402, max=121444, avg=85911.28, stdev=3347.87, samples=119
  lat (usec)   : 10=1.31%, 20=98.68%, 50=0.01%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.40%, sys=12.26%, ctx=5153007, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5153027,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=41.9MiB/s (44.0MB/s), 41.9MiB/s-41.9MiB/s (44.0MB/s-44.0MB/s), io=2516MiB (2638MB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=5143860/87, merge=0/61, ticks=52159/5, in_queue=52164, util=99.86%

Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) using 4 job(s) with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [r(4)][100.0%][r=171MiB/s][r=350k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=5794: Sat Feb 25 16:04:09 2023
  read: IOPS=375k, BW=183MiB/s (192MB/s)(10.7GiB/60001msec)
    clat (usec): min=4, max=383, avg=10.43, stdev= 2.84
     lat (usec): min=4, max=383, avg=10.45, stdev= 2.84
    clat percentiles (nsec):
     |  1.00th=[ 4640],  5.00th=[ 5088], 10.00th=[ 5408], 20.00th=[ 6176],
     | 30.00th=[11584], 40.00th=[11712], 50.00th=[11712], 60.00th=[11712],
     | 70.00th=[11840], 80.00th=[11968], 90.00th=[12096], 95.00th=[12096],
     | 99.00th=[12608], 99.50th=[13632], 99.90th=[16768], 99.95th=[18304],
     | 99.99th=[22144]
   bw (  KiB/s): min=171233, max=258161, per=100.00%, avg=187724.43, stdev=4456.35, samples=476
   iops        : min=342466, max=516322, avg=375448.87, stdev=8912.71, samples=476
  lat (usec)   : 10=22.26%, 20=77.71%, 50=0.02%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.82%, sys=14.12%, ctx=22506710, majf=0, minf=54
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=22506774,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=183MiB/s (192MB/s), 183MiB/s-183MiB/s (192MB/s-192MB/s), io=10.7GiB (11.5GB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=22468146/72, merge=0/48, ticks=202900/3, in_queue=202903, util=99.86%

Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) using 16 job(s) with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
...
fio-3.28
Starting 16 processes
Jobs: 16 (f=16): [r(16)][100.0%][r=778MiB/s][r=1593k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=16): err= 0: pid=5862: Sat Feb 25 16:05:10 2023
  read: IOPS=1622k, BW=792MiB/s (831MB/s)(46.4GiB/60001msec)
    clat (usec): min=4, max=3527, avg= 9.52, stdev= 3.26
     lat (usec): min=4, max=3527, avg= 9.54, stdev= 3.27
    clat percentiles (usec):
     |  1.00th=[    6],  5.00th=[    7], 10.00th=[    8], 20.00th=[    8],
     | 30.00th=[    9], 40.00th=[    9], 50.00th=[   10], 60.00th=[   10],
     | 70.00th=[   11], 80.00th=[   11], 90.00th=[   12], 95.00th=[   13],
     | 99.00th=[   18], 99.50th=[   20], 99.90th=[   24], 99.95th=[   26],
     | 99.99th=[  215]
   bw (  KiB/s): min=721224, max=902568, per=100.00%, avg=811405.91, stdev=2111.24, samples=1904
   iops        : min=1442448, max=1805136, avg=1622811.87, stdev=4222.47, samples=1904
  lat (usec)   : 10=63.77%, 20=35.84%, 50=0.38%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=5.73%, sys=23.09%, ctx=97314547, majf=0, minf=222
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=97330368,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=792MiB/s (831MB/s), 792MiB/s-792MiB/s (831MB/s-831MB/s), io=46.4GiB (49.8GB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=97121596/333, merge=0/290, ticks=718924/33, in_queue=718956, util=99.84%

Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) using 64 job(s) with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
...
fio-3.28
Starting 64 processes
Jobs: 64 (f=64): [r(64)][100.0%][r=723MiB/s][r=1482k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=64): err= 0: pid=5959: Sat Feb 25 16:06:10 2023
  read: IOPS=1483k, BW=724MiB/s (759MB/s)(42.4GiB/60001msec)
    clat (usec): min=4, max=16112, avg=42.74, stdev=23.44
     lat (usec): min=4, max=16112, avg=42.77, stdev=23.44
    clat percentiles (usec):
     |  1.00th=[   14],  5.00th=[   16], 10.00th=[   19], 20.00th=[   23],
     | 30.00th=[   28], 40.00th=[   33], 50.00th=[   38], 60.00th=[   44],
     | 70.00th=[   52], 80.00th=[   60], 90.00th=[   74], 95.00th=[   85],
     | 99.00th=[  110], 99.50th=[  118], 99.90th=[  145], 99.95th=[  225],
     | 99.99th=[  293]
   bw (  KiB/s): min=355525, max=1505249, per=100.00%, avg=742253.96, stdev=3247.32, samples=7616
   iops        : min=711050, max=3010500, avg=1484508.62, stdev=6494.64, samples=7616
  lat (usec)   : 10=0.02%, 20=11.71%, 50=56.66%, 100=29.69%, 250=1.90%
  lat (usec)   : 500=0.03%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=1.76%, sys=4.97%, ctx=88984917, majf=0, minf=896
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=88980740,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=724MiB/s (759MB/s), 724MiB/s-724MiB/s (759MB/s-759MB/s), io=42.4GiB (45.6GB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=88639460/174, merge=0/145, ticks=3603324/22, in_queue=3603347, util=99.89%

Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) using 1 job(s) with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=333MiB/s][r=85.2k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=6072: Sat Feb 25 16:07:10 2023
  read: IOPS=85.4k, BW=334MiB/s (350MB/s)(19.6GiB/60001msec)
    clat (usec): min=6, max=538, avg=11.50, stdev= 2.83
     lat (usec): min=6, max=538, avg=11.51, stdev= 2.83
    clat percentiles (usec):
     |  1.00th=[    7],  5.00th=[   12], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   12], 95.00th=[   12],
     | 99.00th=[   14], 99.50th=[   16], 99.90th=[   19], 99.95th=[   23],
     | 99.99th=[  221]
   bw (  KiB/s): min=331472, max=346384, per=100.00%, avg=341908.10, stdev=1999.73, samples=119
   iops        : min=82868, max=86596, avg=85477.04, stdev=499.94, samples=119
  lat (usec)   : 10=2.10%, 20=97.84%, 50=0.04%, 100=0.01%, 250=0.02%
  lat (usec)   : 500=0.01%, 750=0.01%
  cpu          : usr=3.33%, sys=12.74%, ctx=5126320, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5126331,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=334MiB/s (350MB/s), 334MiB/s-334MiB/s (350MB/s-350MB/s), io=19.6GiB (21.0GB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=5117497/107, merge=0/86, ticks=52185/7, in_queue=52192, util=99.88%

Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) using 4 job(s) with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [r(4)][100.0%][r=723MiB/s][r=185k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=6108: Sat Feb 25 16:08:11 2023
  read: IOPS=218k, BW=850MiB/s (891MB/s)(49.8GiB/60001msec)
    clat (usec): min=5, max=1374, avg=18.14, stdev= 6.52
     lat (usec): min=5, max=1374, avg=18.15, stdev= 6.52
    clat percentiles (usec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   11], 20.00th=[   11],
     | 30.00th=[   14], 40.00th=[   21], 50.00th=[   22], 60.00th=[   22],
     | 70.00th=[   22], 80.00th=[   22], 90.00th=[   23], 95.00th=[   23],
     | 99.00th=[   24], 99.50th=[   27], 99.90th=[   35], 99.95th=[   42],
     | 99.99th=[  229]
   bw (  KiB/s): min=728680, max=1488904, per=100.00%, avg=871522.62, stdev=67250.53, samples=476
   iops        : min=182170, max=372226, avg=217880.64, stdev=16812.63, samples=476
  lat (usec)   : 10=2.46%, 20=32.13%, 50=65.35%, 100=0.01%, 250=0.04%
  lat (usec)   : 500=0.01%, 750=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=2.40%, sys=8.46%, ctx=13051056, majf=0, minf=64
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=13051084,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=850MiB/s (891MB/s), 850MiB/s-850MiB/s (891MB/s-891MB/s), io=49.8GiB (53.5GB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=13032283/88, merge=0/88, ticks=216684/15, in_queue=216699, util=99.88%

Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) using 16 job(s) with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
...
fio-3.28
Starting 16 processes
Jobs: 16 (f=16): [r(16)][100.0%][r=725MiB/s][r=186k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=16): err= 0: pid=6146: Sat Feb 25 16:09:11 2023
  read: IOPS=185k, BW=723MiB/s (758MB/s)(42.4GiB/60001msec)
    clat (usec): min=10, max=1584, avg=86.11, stdev=21.23
     lat (usec): min=10, max=1584, avg=86.13, stdev=21.23
    clat percentiles (usec):
     |  1.00th=[   64],  5.00th=[   65], 10.00th=[   67], 20.00th=[   71],
     | 30.00th=[   75], 40.00th=[   76], 50.00th=[   80], 60.00th=[   81],
     | 70.00th=[   96], 80.00th=[  102], 90.00th=[  115], 95.00th=[  129],
     | 99.00th=[  141], 99.50th=[  161], 99.90th=[  223], 99.95th=[  265],
     | 99.99th=[  318]
   bw (  KiB/s): min=481496, max=940456, per=100.00%, avg=740920.43, stdev=6924.17, samples=1904
   iops        : min=120374, max=235114, avg=185230.10, stdev=1731.04, samples=1904
  lat (usec)   : 20=0.01%, 50=0.01%, 100=75.87%, 250=24.06%, 500=0.07%
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=0.78%, sys=1.86%, ctx=11110039, majf=0, minf=255
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=11109959,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=723MiB/s (758MB/s), 723MiB/s-723MiB/s (758MB/s-758MB/s), io=42.4GiB (45.5GB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=11090160/98, merge=0/127, ticks=935213/37, in_queue=935251, util=99.86%

Random read of device /dev/nvme0 (INTEL SSDPEK1A118GA) using 64 job(s) with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
...
fio-3.28
Starting 64 processes
Jobs: 64 (f=64): [r(64)][100.0%][r=726MiB/s][r=186k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=64): err= 0: pid=6200: Sat Feb 25 16:10:11 2023
  read: IOPS=186k, BW=725MiB/s (761MB/s)(42.5GiB/60001msec)
    clat (usec): min=8, max=6322, avg=344.31, stdev=189.46
     lat (usec): min=9, max=6322, avg=344.33, stdev=189.46
    clat percentiles (usec):
     |  1.00th=[   77],  5.00th=[   86], 10.00th=[  129], 20.00th=[  174],
     | 30.00th=[  217], 40.00th=[  262], 50.00th=[  302], 60.00th=[  347],
     | 70.00th=[  416], 80.00th=[  486], 90.00th=[  644], 95.00th=[  734],
     | 99.00th=[  865], 99.50th=[  906], 99.90th=[  988], 99.95th=[ 1037],
     | 99.99th=[ 1254]
   bw (  KiB/s): min=296888, max=2381757, per=100.00%, avg=742942.49, stdev=5887.99, samples=7616
   iops        : min=74222, max=595439, avg=185735.55, stdev=1472.00, samples=7616
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.01%, 100=7.07%, 250=26.63%
  lat (usec)   : 500=48.03%, 750=14.17%, 1000=4.04%
  lat (msec)   : 2=0.06%, 4=0.01%, 10=0.01%
  cpu          : usr=0.22%, sys=0.51%, ctx=11141523, majf=0, minf=903
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=11141319,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=725MiB/s (761MB/s), 725MiB/s-725MiB/s (761MB/s-761MB/s), io=42.5GiB (45.6GB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=11116969/106, merge=0/123, ticks=3804352/67, in_queue=3804418, util=99.85%

Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) using 1 job(s) with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=41.2MiB/s][r=84.3k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=6309: Sat Feb 25 16:11:11 2023
  read: IOPS=84.1k, BW=41.0MiB/s (43.0MB/s)(2463MiB/60001msec)
    clat (usec): min=6, max=382, avg=11.64, stdev= 2.85
     lat (usec): min=6, max=382, avg=11.66, stdev= 2.85
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   12], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   12], 95.00th=[   12],
     | 99.00th=[   20], 99.50th=[   22], 99.90th=[   38], 99.95th=[   40],
     | 99.99th=[  215]
   bw (  KiB/s): min=41175, max=42328, per=100.00%, avg=42046.07, stdev=200.67, samples=119
   iops        : min=82350, max=84656, avg=84092.12, stdev=401.36, samples=119
  lat (usec)   : 10=0.73%, 20=98.33%, 50=0.93%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.84%, sys=11.99%, ctx=5043844, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5043852,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=41.0MiB/s (43.0MB/s), 41.0MiB/s-41.0MiB/s (43.0MB/s-43.0MB/s), io=2463MiB (2582MB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=5033296/0, merge=0/0, ticks=52127/0, in_queue=52127, util=99.84%

Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) using 4 job(s) with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [r(4)][100.0%][r=177MiB/s][r=362k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=6349: Sat Feb 25 16:12:12 2023
  read: IOPS=375k, BW=183MiB/s (192MB/s)(10.7GiB/60001msec)
    clat (usec): min=6, max=408, avg=10.39, stdev= 4.12
     lat (usec): min=6, max=408, avg=10.41, stdev= 4.12
    clat percentiles (nsec):
     |  1.00th=[ 7072],  5.00th=[ 7136], 10.00th=[ 7264], 20.00th=[ 7456],
     | 30.00th=[ 7776], 40.00th=[ 8032], 50.00th=[ 8640], 60.00th=[11584],
     | 70.00th=[11840], 80.00th=[12096], 90.00th=[15168], 95.00th=[15808],
     | 99.00th=[23424], 99.50th=[25728], 99.90th=[38144], 99.95th=[41216],
     | 99.99th=[63744]
   bw (  KiB/s): min=160510, max=214454, per=100.00%, avg=187578.82, stdev=2804.75, samples=476
   iops        : min=321020, max=428908, avg=375157.66, stdev=5609.50, samples=476
  lat (usec)   : 10=53.37%, 20=44.44%, 50=2.18%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=4.65%, sys=14.36%, ctx=22496851, majf=0, minf=54
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=22496826,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=183MiB/s (192MB/s), 183MiB/s-183MiB/s (192MB/s-192MB/s), io=10.7GiB (11.5GB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=22449711/0, merge=0/0, ticks=201837/0, in_queue=201837, util=99.82%

Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) using 16 job(s) with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
...
fio-3.28
Starting 16 processes
Jobs: 16 (f=16): [r(16)][100.0%][r=291MiB/s][r=595k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=16): err= 0: pid=6388: Sat Feb 25 16:13:12 2023
  read: IOPS=594k, BW=290MiB/s (304MB/s)(17.0GiB/60001msec)
    clat (usec): min=7, max=2538, avg=26.58, stdev= 4.84
     lat (usec): min=7, max=2538, avg=26.60, stdev= 4.84
    clat percentiles (usec):
     |  1.00th=[   23],  5.00th=[   25], 10.00th=[   25], 20.00th=[   26],
     | 30.00th=[   26], 40.00th=[   26], 50.00th=[   26], 60.00th=[   27],
     | 70.00th=[   27], 80.00th=[   28], 90.00th=[   28], 95.00th=[   30],
     | 99.00th=[   41], 99.50th=[   56], 99.90th=[   61], 99.95th=[   67],
     | 99.99th=[  231]
   bw (  KiB/s): min=277816, max=299850, per=100.00%, avg=296925.04, stdev=136.64, samples=1904
   iops        : min=555634, max=599700, avg=593850.13, stdev=273.26, samples=1904
  lat (usec)   : 10=0.01%, 20=0.12%, 50=99.29%, 100=0.56%, 250=0.03%
  lat (usec)   : 500=0.01%, 750=0.01%
  lat (msec)   : 4=0.01%
  cpu          : usr=2.39%, sys=6.35%, ctx=35618759, majf=0, minf=210
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=35618707,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=290MiB/s (304MB/s), 290MiB/s-290MiB/s (304MB/s-304MB/s), io=17.0GiB (18.2GB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=35507926/0, merge=0/0, ticks=888365/0, in_queue=888365, util=99.79%

Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) using 64 job(s) with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
...
fio-3.28
Starting 64 processes
Jobs: 64 (f=64): [r(64)][100.0%][r=288MiB/s][r=591k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=64): err= 0: pid=6442: Sat Feb 25 16:14:13 2023
  read: IOPS=597k, BW=291MiB/s (305MB/s)(17.1GiB/60001msec)
    clat (usec): min=7, max=21287, avg=106.79, stdev=50.18
     lat (usec): min=7, max=21287, avg=106.82, stdev=50.18
    clat percentiles (usec):
     |  1.00th=[   58],  5.00th=[   67], 10.00th=[   70], 20.00th=[   75],
     | 30.00th=[   79], 40.00th=[   83], 50.00th=[   89], 60.00th=[  105],
     | 70.00th=[  118], 80.00th=[  133], 90.00th=[  163], 95.00th=[  196],
     | 99.00th=[  289], 99.50th=[  322], 99.90th=[  469], 99.95th=[  494],
     | 99.99th=[  523]
   bw (  KiB/s): min=100757, max=483796, per=99.93%, avg=298073.18, stdev=1321.21, samples=7616
   iops        : min=201514, max=967593, avg=596147.35, stdev=2642.41, samples=7616
  lat (usec)   : 10=0.02%, 20=0.13%, 50=0.32%, 100=56.98%, 250=40.66%
  lat (usec)   : 500=1.85%, 750=0.04%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=0.81%, sys=1.64%, ctx=35834995, majf=0, minf=918
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=35795966,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=291MiB/s (305MB/s), 291MiB/s-291MiB/s (305MB/s-305MB/s), io=17.1GiB (18.3GB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=35775483/0, merge=0/0, ticks=3758692/0, in_queue=3758692, util=99.72%

Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) using 1 job(s) with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=329MiB/s][r=84.3k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=6544: Sat Feb 25 16:15:13 2023
  read: IOPS=83.7k, BW=327MiB/s (343MB/s)(19.2GiB/60001msec)
    clat (usec): min=8, max=404, avg=11.73, stdev= 3.89
     lat (usec): min=8, max=404, avg=11.75, stdev= 3.89
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   12], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   12], 95.00th=[   12],
     | 99.00th=[   21], 99.50th=[   23], 99.90th=[   39], 99.95th=[   41],
     | 99.99th=[  223]
   bw (  KiB/s): min=324440, max=338416, per=100.00%, avg=335069.24, stdev=2535.62, samples=119
   iops        : min=81110, max=84604, avg=83767.31, stdev=633.91, samples=119
  lat (usec)   : 10=0.61%, 20=98.36%, 50=0.99%, 100=0.01%, 250=0.03%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.29%, sys=12.02%, ctx=5024870, majf=0, minf=15
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5024894,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=327MiB/s (343MB/s), 327MiB/s-327MiB/s (343MB/s-343MB/s), io=19.2GiB (20.6GB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=5015685/0, merge=0/0, ticks=52402/0, in_queue=52402, util=99.86%

Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) using 4 job(s) with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [r(4)][100.0%][r=1646MiB/s][r=421k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=6591: Sat Feb 25 16:16:13 2023
  read: IOPS=416k, BW=1624MiB/s (1703MB/s)(95.1GiB/60001msec)
    clat (usec): min=8, max=389, avg= 9.40, stdev= 3.06
     lat (usec): min=8, max=389, avg= 9.41, stdev= 3.06
    clat percentiles (nsec):
     |  1.00th=[ 8256],  5.00th=[ 8256], 10.00th=[ 8384], 20.00th=[ 8384],
     | 30.00th=[ 8512], 40.00th=[ 8512], 50.00th=[ 8640], 60.00th=[ 8768],
     | 70.00th=[ 8896], 80.00th=[ 9152], 90.00th=[10688], 95.00th=[15168],
     | 99.00th=[22144], 99.50th=[24192], 99.90th=[38656], 99.95th=[41216],
     | 99.99th=[47360]
   bw (  MiB/s): min= 1360, max= 1662, per=100.00%, avg=1624.81, stdev=10.51, samples=476
   iops        : min=348286, max=425582, avg=415952.45, stdev=2691.16, samples=476
  lat (usec)   : 10=88.45%, 20=10.18%, 50=1.36%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=4.39%, sys=16.25%, ctx=24939940, majf=0, minf=57
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=24939918,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=1624MiB/s (1703MB/s), 1624MiB/s-1624MiB/s (1703MB/s-1703MB/s), io=95.1GiB (102GB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=24896634/0, merge=0/0, ticks=198954/0, in_queue=198954, util=99.90%

Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) using 16 job(s) with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
...
fio-3.28
Starting 16 processes
Jobs: 16 (f=16): [r(16)][100.0%][r=2323MiB/s][r=595k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=16): err= 0: pid=6633: Sat Feb 25 16:17:14 2023
  read: IOPS=594k, BW=2322MiB/s (2434MB/s)(136GiB/60001msec)
    clat (usec): min=8, max=1042, avg=26.62, stdev= 4.30
     lat (usec): min=8, max=1042, avg=26.64, stdev= 4.30
    clat percentiles (usec):
     |  1.00th=[   24],  5.00th=[   25], 10.00th=[   25], 20.00th=[   26],
     | 30.00th=[   26], 40.00th=[   26], 50.00th=[   26], 60.00th=[   27],
     | 70.00th=[   27], 80.00th=[   28], 90.00th=[   28], 95.00th=[   30],
     | 99.00th=[   40], 99.50th=[   56], 99.90th=[   60], 99.95th=[   63],
     | 99.99th=[  223]
   bw (  MiB/s): min= 2270, max= 2326, per=100.00%, avg=2322.29, stdev= 0.45, samples=1904
   iops        : min=581368, max=595644, avg=594505.87, stdev=115.89, samples=1904
  lat (usec)   : 10=0.01%, 20=0.07%, 50=99.34%, 100=0.57%, 250=0.01%
  lat (usec)   : 500=0.01%, 750=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=1.95%, sys=6.17%, ctx=35662141, majf=0, minf=223
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=35662068,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=2322MiB/s (2434MB/s), 2322MiB/s-2322MiB/s (2434MB/s-2434MB/s), io=136GiB (146GB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=35591473/0, merge=0/0, ticks=894855/0, in_queue=894855, util=99.84%

Random read of device /dev/nvme1 (INTEL SSDPEL1D380GA) using 64 job(s) with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
...
fio-3.28
Starting 64 processes
Jobs: 64 (f=64): [r(64)][100.0%][r=2324MiB/s][r=595k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=64): err= 0: pid=6686: Sat Feb 25 16:18:14 2023
  read: IOPS=595k, BW=2323MiB/s (2436MB/s)(136GiB/60001msec)
    clat (usec): min=8, max=17616, avg=107.24, stdev=49.01
     lat (usec): min=8, max=17616, avg=107.26, stdev=49.01
    clat percentiles (usec):
     |  1.00th=[   58],  5.00th=[   63], 10.00th=[   69], 20.00th=[   74],
     | 30.00th=[   78], 40.00th=[   83], 50.00th=[   89], 60.00th=[  104],
     | 70.00th=[  119], 80.00th=[  135], 90.00th=[  165], 95.00th=[  202],
     | 99.00th=[  289], 99.50th=[  330], 99.90th=[  404], 99.95th=[  429],
     | 99.99th=[  506]
   bw (  MiB/s): min=  902, max= 3786, per=100.00%, avg=2325.14, stdev= 9.49, samples=7616
   iops        : min=231034, max=969232, avg=595235.85, stdev=2429.38, samples=7616
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.07%, 100=57.96%, 250=39.72%
  lat (usec)   : 500=2.24%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=0.71%, sys=1.65%, ctx=35686734, majf=0, minf=834
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=35686339,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=2323MiB/s (2436MB/s), 2323MiB/s-2323MiB/s (2436MB/s-2436MB/s), io=136GiB (146GB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=35584482/0, merge=0/0, ticks=3754781/0, in_queue=3754781, util=99.92%

Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) using 1 job(s) with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
Jobs: 1 (f=1): [r(1)][100.0%][r=41.7MiB/s][r=85.5k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=6795: Sat Feb 25 16:19:14 2023
  read: IOPS=85.5k, BW=41.8MiB/s (43.8MB/s)(2506MiB/60001msec)
    clat (usec): min=4, max=438, avg=11.39, stdev= 2.62
     lat (usec): min=4, max=438, avg=11.41, stdev= 2.62
    clat percentiles (usec):
     |  1.00th=[    6],  5.00th=[   12], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   12], 95.00th=[   12],
     | 99.00th=[   14], 99.50th=[   16], 99.90th=[   21], 99.95th=[   21],
     | 99.99th=[  212]
   bw (  KiB/s): min=41468, max=43534, per=100.00%, avg=42787.08, stdev=269.77, samples=119
   iops        : min=82936, max=87068, avg=85574.18, stdev=539.52, samples=119
  lat (usec)   : 10=2.01%, 20=97.63%, 50=0.35%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=4.07%, sys=12.46%, ctx=5132530, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5132522,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=41.8MiB/s (43.8MB/s), 41.8MiB/s-41.8MiB/s (43.8MB/s-43.8MB/s), io=2506MiB (2628MB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=5112347/0, merge=0/0, ticks=51650/0, in_queue=51650, util=99.63%

Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) using 4 job(s) with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
...
fio-3.28
Starting 4 processes
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
Jobs: 4 (f=4): [r(4)][100.0%][r=177MiB/s][r=363k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=6830: Sat Feb 25 16:20:15 2023
  read: IOPS=360k, BW=176MiB/s (185MB/s)(10.3GiB/60001msec)
    clat (usec): min=4, max=515, avg=10.80, stdev= 2.45
     lat (usec): min=4, max=515, avg=10.82, stdev= 2.45
    clat percentiles (nsec):
     |  1.00th=[ 5152],  5.00th=[ 5472], 10.00th=[ 5792], 20.00th=[11328],
     | 30.00th=[11456], 40.00th=[11584], 50.00th=[11584], 60.00th=[11712],
     | 70.00th=[11840], 80.00th=[11840], 90.00th=[11968], 95.00th=[12224],
     | 99.00th=[12608], 99.50th=[13376], 99.90th=[22912], 99.95th=[24448],
     | 99.99th=[27008]
   bw (  KiB/s): min=160570, max=187029, per=100.00%, avg=180305.13, stdev=639.73, samples=476
   iops        : min=321140, max=374058, avg=360610.35, stdev=1279.47, samples=476
  lat (usec)   : 10=15.66%, 20=84.11%, 50=0.22%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%, 750=0.01%
  cpu          : usr=4.57%, sys=16.70%, ctx=21622130, majf=0, minf=48
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=21622113,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=176MiB/s (185MB/s), 176MiB/s-176MiB/s (185MB/s-185MB/s), io=10.3GiB (11.1GB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=21608482/0, merge=0/0, ticks=197420/0, in_queue=197420, util=99.56%

Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) using 16 job(s) with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
...
fio-3.28
Starting 16 processes
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
Jobs: 16 (f=16): [r(16)][100.0%][r=1174MiB/s][r=2404k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=16): err= 0: pid=6874: Sat Feb 25 16:21:16 2023
  read: IOPS=2359k, BW=1152MiB/s (1208MB/s)(67.5GiB/60001msec)
    clat (usec): min=4, max=1593, avg= 6.41, stdev= 1.16
     lat (usec): min=4, max=1593, avg= 6.43, stdev= 1.16
    clat percentiles (nsec):
     |  1.00th=[ 5280],  5.00th=[ 5472], 10.00th=[ 5600], 20.00th=[ 5792],
     | 30.00th=[ 5984], 40.00th=[ 6176], 50.00th=[ 6304], 60.00th=[ 6496],
     | 70.00th=[ 6624], 80.00th=[ 6816], 90.00th=[ 7200], 95.00th=[ 7584],
     | 99.00th=[ 9024], 99.50th=[10176], 99.90th=[15936], 99.95th=[18048],
     | 99.99th=[22400]
   bw (  MiB/s): min=  582, max= 1250, per=100.00%, avg=1152.15, stdev= 4.90, samples=1904
   iops        : min=1193723, max=2560682, avg=2359600.62, stdev=10033.88, samples=1904
  lat (usec)   : 10=99.46%, 20=0.51%, 50=0.03%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=9.46%, sys=30.53%, ctx=141515180, majf=0, minf=201
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=141514302,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=1152MiB/s (1208MB/s), 1152MiB/s-1152MiB/s (1208MB/s-1208MB/s), io=67.5GiB (72.5GB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=141503618/0, merge=0/0, ticks=639423/0, in_queue=639423, util=98.58%

Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) using 64 job(s) with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
...
fio-3.28
Starting 64 processes
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/nvme2n1 exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
Jobs: 62 (f=60): [r(1),_(1),r(5),f(1),r(25),f(1),r(6),_(1),r(23)][33.9%][r=2303MJobs: 44 (f=43): [_(6),r(1),_(1),r(6),_(1),r(10),f(1),r(4),_(8),r(2),_(1),r(3),_(3),r(17)][5.0%][r=2191MiB/s][r=4488k IOPS][eta 20m:00s]
iops-test-job: (groupid=0, jobs=64): err= 0: pid=6925: Sat Feb 25 16:22:20 2023
  read: IOPS=4703k, BW=2296MiB/s (2408MB/s)(135GiB/60001msec)
    clat (usec): min=4, max=22524, avg=12.80, stdev=12.31
     lat (usec): min=4, max=22524, avg=12.84, stdev=12.32
    clat percentiles (usec):
     |  1.00th=[    8],  5.00th=[   10], 10.00th=[   10], 20.00th=[   11],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   14], 80.00th=[   15], 90.00th=[   17], 95.00th=[   19],
     | 99.00th=[   29], 99.50th=[   36], 99.90th=[   52], 99.95th=[   65],
     | 99.99th=[  116]
   bw (  MiB/s): min=  803, max= 2900, per=100.00%, avg=2301.90, stdev= 3.76, samples=7577
   iops        : min=1644625, max=5940263, avg=4714292.67, stdev=7710.50, samples=7577
  lat (usec)   : 10=13.62%, 20=82.34%, 50=3.92%, 100=0.10%, 250=0.01%
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=7.65%, sys=35.91%, ctx=288455319, majf=0, minf=888
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=282158128,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=2296MiB/s (2408MB/s), 2296MiB/s-2296MiB/s (2408MB/s-2408MB/s), io=135GiB (144GB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=281987903/0, merge=0/0, ticks=1759645/0, in_queue=1759645, util=97.25%

Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) using 1 job(s) with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=332MiB/s][r=85.0k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=7024: Sat Feb 25 16:23:20 2023
  read: IOPS=84.9k, BW=332MiB/s (348MB/s)(19.4GiB/60001msec)
    clat (usec): min=6, max=1089, avg=11.55, stdev= 1.59
     lat (usec): min=6, max=1089, avg=11.57, stdev= 1.59
    clat percentiles (nsec):
     |  1.00th=[11200],  5.00th=[11328], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11584], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[14400], 99.50th=[19840], 99.90th=[20864], 99.95th=[21888],
     | 99.99th=[23424]
   bw (  KiB/s): min=338928, max=340200, per=100.00%, avg=339877.11, stdev=254.87, samples=119
   iops        : min=84732, max=85050, avg=84969.28, stdev=63.71, samples=119
  lat (usec)   : 10=0.21%, 20=99.36%, 50=0.42%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=3.20%, sys=11.43%, ctx=5096738, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5096737,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=332MiB/s (348MB/s), 332MiB/s-332MiB/s (348MB/s-348MB/s), io=19.4GiB (20.9GB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=5086209/0, merge=0/0, ticks=52529/0, in_queue=52529, util=99.83%

Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) using 4 job(s) with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [r(4)][100.0%][r=1665MiB/s][r=426k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=7059: Sat Feb 25 16:24:20 2023
  read: IOPS=414k, BW=1615MiB/s (1694MB/s)(94.7GiB/60001msec)
    clat (usec): min=6, max=527, avg= 9.40, stdev= 2.88
     lat (usec): min=6, max=527, avg= 9.42, stdev= 2.88
    clat percentiles (nsec):
     |  1.00th=[ 6752],  5.00th=[ 6816], 10.00th=[ 6816], 20.00th=[ 6880],
     | 30.00th=[ 7008], 40.00th=[ 7456], 50.00th=[ 7968], 60.00th=[11584],
     | 70.00th=[11712], 80.00th=[11840], 90.00th=[12096], 95.00th=[12224],
     | 99.00th=[15936], 99.50th=[22656], 99.90th=[24960], 99.95th=[25728],
     | 99.99th=[28032]
   bw (  MiB/s): min= 1499, max= 1709, per=100.00%, avg=1615.92, stdev=13.56, samples=476
   iops        : min=383926, max=437594, avg=413676.40, stdev=3471.69, samples=476
  lat (usec)   : 10=56.15%, 20=43.15%, 50=0.70%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%, 750=0.01%
  cpu          : usr=4.88%, sys=15.74%, ctx=24814000, majf=0, minf=61
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=24813993,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=1615MiB/s (1694MB/s), 1615MiB/s-1615MiB/s (1694MB/s-1694MB/s), io=94.7GiB (102GB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=24758787/0, merge=0/0, ticks=197914/0, in_queue=197914, util=99.82%

Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) using 16 job(s) with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
...
fio-3.28
Starting 16 processes
Jobs: 16 (f=16): [r(16)][100.0%][r=5323MiB/s][r=1363k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=16): err= 0: pid=7098: Sat Feb 25 16:25:21 2023
  read: IOPS=1362k, BW=5321MiB/s (5580MB/s)(312GiB/60001msec)
    clat (usec): min=6, max=522, avg=11.41, stdev= 2.47
     lat (usec): min=6, max=522, avg=11.43, stdev= 2.47
    clat percentiles (nsec):
     |  1.00th=[ 7776],  5.00th=[ 8640], 10.00th=[ 9152], 20.00th=[ 9792],
     | 30.00th=[10176], 40.00th=[10688], 50.00th=[11072], 60.00th=[11584],
     | 70.00th=[12096], 80.00th=[12864], 90.00th=[14016], 95.00th=[15040],
     | 99.00th=[17536], 99.50th=[19328], 99.90th=[25728], 99.95th=[27520],
     | 99.99th=[31360]
   bw (  MiB/s): min= 5066, max= 5590, per=100.00%, avg=5324.94, stdev= 7.32, samples=1904
   iops        : min=1297044, max=1431156, avg=1363185.02, stdev=1873.26, samples=1904
  lat (usec)   : 10=25.25%, 20=74.33%, 50=0.42%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%, 750=0.01%
  cpu          : usr=4.87%, sys=16.63%, ctx=81739468, majf=0, minf=225
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=81739224,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=5321MiB/s (5580MB/s), 5321MiB/s-5321MiB/s (5580MB/s-5580MB/s), io=312GiB (335GB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=81485948/0, merge=0/0, ticks=785790/0, in_queue=785790, util=99.73%

Random read of device /dev/nvme2 (INTEL SSDPF21Q032TB) using 64 job(s) with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
...
fio-3.28
Starting 64 processes
Jobs: 64 (f=64): [r(64)][100.0%][r=2702MiB/s][r=692k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=64): err= 0: pid=7151: Sat Feb 25 16:26:21 2023
  read: IOPS=1625k, BW=6349MiB/s (6657MB/s)(372GiB/60001msec)
    clat (usec): min=6, max=34613, avg=38.86, stdev=14.13
     lat (usec): min=6, max=34613, avg=38.90, stdev=14.13
    clat percentiles (usec):
     |  1.00th=[   29],  5.00th=[   32], 10.00th=[   33], 20.00th=[   35],
     | 30.00th=[   36], 40.00th=[   37], 50.00th=[   38], 60.00th=[   39],
     | 70.00th=[   40], 80.00th=[   42], 90.00th=[   44], 95.00th=[   48],
     | 99.00th=[   93], 99.50th=[   93], 99.90th=[   95], 99.95th=[   96],
     | 99.99th=[  245]
   bw (  MiB/s): min= 2702, max= 6742, per=100.00%, avg=6373.60, stdev=13.75, samples=7616
   iops        : min=691806, max=1726113, avg=1631640.78, stdev=3519.16, samples=7616
  lat (usec)   : 10=0.01%, 20=0.12%, 50=96.35%, 100=3.49%, 250=0.02%
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=2.41%, sys=6.11%, ctx=97560750, majf=0, minf=964
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=97520137,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=6349MiB/s (6657MB/s), 6349MiB/s-6349MiB/s (6657MB/s-6657MB/s), io=372GiB (399GB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=97466003/0, merge=0/0, ticks=3556360/0, in_queue=3556360, util=99.72%

Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) using 1 job(s) with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=40.7MiB/s][r=83.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=7261: Sat Feb 25 16:27:22 2023
  read: IOPS=83.1k, BW=40.6MiB/s (42.6MB/s)(2436MiB/60001msec)
    clat (usec): min=7, max=586, avg=11.76, stdev= 3.26
     lat (usec): min=7, max=586, avg=11.78, stdev= 3.26
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   12], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   12], 95.00th=[   12],
     | 99.00th=[   24], 99.50th=[   33], 99.90th=[   42], 99.95th=[   43],
     | 99.99th=[  204]
   bw (  KiB/s): min=40187, max=41781, per=100.00%, avg=41581.91, stdev=220.72, samples=119
   iops        : min=80374, max=83562, avg=83163.82, stdev=441.44, samples=119
  lat (usec)   : 10=0.73%, 20=97.64%, 50=1.61%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%, 750=0.01%
  cpu          : usr=3.77%, sys=12.25%, ctx=4988102, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4988130,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=40.6MiB/s (42.6MB/s), 40.6MiB/s-40.6MiB/s (42.6MB/s-42.6MB/s), io=2436MiB (2554MB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=4974263/0, merge=0/0, ticks=52127/0, in_queue=52127, util=99.75%

Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) using 4 job(s) with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [r(4)][100.0%][r=162MiB/s][r=332k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=7297: Sat Feb 25 16:28:22 2023
  read: IOPS=335k, BW=164MiB/s (171MB/s)(9811MiB/60001msec)
    clat (usec): min=7, max=1380, avg=11.66, stdev= 4.55
     lat (usec): min=7, max=1380, avg=11.68, stdev= 4.55
    clat percentiles (nsec):
     |  1.00th=[ 7392],  5.00th=[ 7584], 10.00th=[ 7712], 20.00th=[ 8032],
     | 30.00th=[ 8384], 40.00th=[ 9280], 50.00th=[11584], 60.00th=[11968],
     | 70.00th=[12480], 80.00th=[15424], 90.00th=[16512], 95.00th=[17024],
     | 99.00th=[24448], 99.50th=[37120], 99.90th=[42752], 99.95th=[44288],
     | 99.99th=[51968]
   bw (  KiB/s): min=157269, max=181451, per=100.00%, avg=167623.66, stdev=608.21, samples=476
   iops        : min=314538, max=362902, avg=335247.31, stdev=1216.41, samples=476
  lat (usec)   : 10=42.72%, 20=54.35%, 50=2.91%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%, 750=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=3.86%, sys=14.83%, ctx=20093702, majf=0, minf=60
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=20093705,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=164MiB/s (171MB/s), 164MiB/s-164MiB/s (171MB/s-171MB/s), io=9811MiB (10.3GB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=20032017/0, merge=0/0, ticks=202817/0, in_queue=202817, util=99.76%

Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) using 16 job(s) with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
...
fio-3.28
Starting 16 processes
Jobs: 16 (f=16): [r(16)][100.0%][r=272MiB/s][r=557k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=16): err= 0: pid=7338: Sat Feb 25 16:29:23 2023
  read: IOPS=558k, BW=273MiB/s (286MB/s)(16.0GiB/60001msec)
    clat (usec): min=7, max=3086, avg=28.22, stdev= 7.27
     lat (usec): min=7, max=3086, avg=28.24, stdev= 7.27
    clat percentiles (usec):
     |  1.00th=[   22],  5.00th=[   25], 10.00th=[   26], 20.00th=[   27],
     | 30.00th=[   27], 40.00th=[   27], 50.00th=[   28], 60.00th=[   28],
     | 70.00th=[   28], 80.00th=[   29], 90.00th=[   31], 95.00th=[   38],
     | 99.00th=[   58], 99.50th=[   61], 99.90th=[   72], 99.95th=[  221],
     | 99.99th=[  241]
   bw (  KiB/s): min=253311, max=282391, per=99.95%, avg=279025.29, stdev=187.59, samples=1904
   iops        : min=506623, max=564782, avg=558050.72, stdev=375.17, samples=1904
  lat (usec)   : 10=0.06%, 20=0.48%, 50=97.89%, 100=1.50%, 250=0.06%
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=2.45%, sys=6.42%, ctx=33501625, majf=0, minf=214
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=33501563,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=273MiB/s (286MB/s), 273MiB/s-273MiB/s (286MB/s-286MB/s), io=16.0GiB (17.2GB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=33383393/0, merge=0/0, ticks=889992/0, in_queue=889992, util=99.36%

Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) using 64 job(s) with block size 512
iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=pvsync2, iodepth=1
...
fio-3.28
Starting 64 processes
Jobs: 58 (f=58): [r(2),_(2),r(16),_(1),r(9),_(1),r(2),_(1),r(3),_(1),r(26)][14.5%][r=274MiB/s][r=561k IOPS][eta 06m:00s]
iops-test-job: (groupid=0, jobs=64): err= 0: pid=7394: Sat Feb 25 16:30:25 2023
  read: IOPS=564k, BW=275MiB/s (289MB/s)(16.1GiB/60001msec)
    clat (usec): min=7, max=23952, avg=112.45, stdev=55.31
     lat (usec): min=7, max=23953, avg=112.47, stdev=55.31
    clat percentiles (usec):
     |  1.00th=[   41],  5.00th=[   67], 10.00th=[   71], 20.00th=[   77],
     | 30.00th=[   82], 40.00th=[   87], 50.00th=[   95], 60.00th=[  111],
     | 70.00th=[  124], 80.00th=[  143], 90.00th=[  176], 95.00th=[  212],
     | 99.00th=[  310], 99.50th=[  347], 99.90th=[  424], 99.95th=[  478],
     | 99.99th=[  537]
   bw (  KiB/s): min=104123, max=496778, per=99.67%, avg=281079.17, stdev=1134.92, samples=7614
   iops        : min=208255, max=993567, avg=562167.21, stdev=2269.84, samples=7614
  lat (usec)   : 10=0.04%, 20=0.21%, 50=0.99%, 100=52.75%, 250=43.30%
  lat (usec)   : 500=2.67%, 750=0.03%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=1.12%, sys=1.63%, ctx=33979069, majf=0, minf=940
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=33841451,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=275MiB/s (289MB/s), 275MiB/s-275MiB/s (289MB/s-289MB/s), io=16.1GiB (17.3GB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=33841148/0, merge=0/0, ticks=3741223/0, in_queue=3741223, util=98.82%

Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) using 1 job(s) with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=325MiB/s][r=83.1k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=7499: Sat Feb 25 16:31:25 2023
  read: IOPS=83.1k, BW=325MiB/s (340MB/s)(19.0GiB/60001msec)
    clat (usec): min=8, max=413, avg=11.78, stdev= 3.12
     lat (usec): min=8, max=413, avg=11.80, stdev= 3.12
    clat percentiles (nsec):
     |  1.00th=[10432],  5.00th=[11200], 10.00th=[11328], 20.00th=[11456],
     | 30.00th=[11456], 40.00th=[11456], 50.00th=[11456], 60.00th=[11456],
     | 70.00th=[11456], 80.00th=[11456], 90.00th=[11584], 95.00th=[11712],
     | 99.00th=[23168], 99.50th=[32384], 99.90th=[41216], 99.95th=[41728],
     | 99.99th=[60160]
   bw (  KiB/s): min=318840, max=334056, per=100.00%, avg=332567.06, stdev=2414.81, samples=119
   iops        : min=79710, max=83514, avg=83141.78, stdev=603.71, samples=119
  lat (usec)   : 10=0.42%, 20=97.99%, 50=1.58%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=3.73%, sys=11.98%, ctx=4986642, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4986673,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=325MiB/s (340MB/s), 325MiB/s-325MiB/s (340MB/s-340MB/s), io=19.0GiB (20.4GB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=4977434/0, merge=0/0, ticks=52270/0, in_queue=52270, util=99.86%

Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) using 4 job(s) with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
...
fio-3.28
Starting 4 processes
Jobs: 4 (f=4): [r(4)][100.0%][r=1511MiB/s][r=387k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=7537: Sat Feb 25 16:32:25 2023
  read: IOPS=389k, BW=1518MiB/s (1592MB/s)(89.0GiB/60001msec)
    clat (usec): min=8, max=522, avg=10.02, stdev= 3.72
     lat (usec): min=8, max=522, avg=10.04, stdev= 3.72
    clat percentiles (nsec):
     |  1.00th=[ 8384],  5.00th=[ 8384], 10.00th=[ 8512], 20.00th=[ 8640],
     | 30.00th=[ 8768], 40.00th=[ 8896], 50.00th=[ 9024], 60.00th=[ 9280],
     | 70.00th=[ 9536], 80.00th=[10048], 90.00th=[12096], 95.00th=[15808],
     | 99.00th=[24192], 99.50th=[35584], 99.90th=[43776], 99.95th=[44288],
     | 99.99th=[50432]
   bw (  MiB/s): min= 1283, max= 1581, per=100.00%, avg=1518.91, stdev=13.08, samples=476
   iops        : min=328550, max=404940, avg=388840.46, stdev=3348.98, samples=476
  lat (usec)   : 10=79.97%, 20=17.83%, 50=2.19%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%, 750=0.01%
  cpu          : usr=4.75%, sys=14.63%, ctx=23321897, majf=0, minf=55
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=23321872,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=1518MiB/s (1592MB/s), 1518MiB/s-1518MiB/s (1592MB/s-1592MB/s), io=89.0GiB (95.5GB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=23277253/0, merge=0/0, ticks=201555/0, in_queue=201555, util=99.85%

Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) using 16 job(s) with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
...
fio-3.28
Starting 16 processes
Jobs: 16 (f=16): [r(16)][100.0%][r=2194MiB/s][r=562k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=16): err= 0: pid=7578: Sat Feb 25 16:33:25 2023
  read: IOPS=566k, BW=2210MiB/s (2318MB/s)(130GiB/60001msec)
    clat (usec): min=8, max=515, avg=27.93, stdev= 5.55
     lat (usec): min=8, max=515, avg=27.95, stdev= 5.55
    clat percentiles (usec):
     |  1.00th=[   24],  5.00th=[   26], 10.00th=[   26], 20.00th=[   27],
     | 30.00th=[   27], 40.00th=[   27], 50.00th=[   28], 60.00th=[   28],
     | 70.00th=[   28], 80.00th=[   28], 90.00th=[   29], 95.00th=[   37],
     | 99.00th=[   57], 99.50th=[   60], 99.90th=[   65], 99.95th=[   72],
     | 99.99th=[  233]
   bw (  MiB/s): min= 2154, max= 2218, per=100.00%, avg=2211.19, stdev= 0.46, samples=1904
   iops        : min=551618, max=567876, avg=566063.92, stdev=118.58, samples=1904
  lat (usec)   : 10=0.01%, 20=0.04%, 50=98.50%, 100=1.44%, 250=0.02%
  lat (usec)   : 500=0.01%, 750=0.01%
  cpu          : usr=2.14%, sys=6.15%, ctx=33952228, majf=0, minf=218
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=33952117,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=2210MiB/s (2318MB/s), 2210MiB/s-2210MiB/s (2318MB/s-2318MB/s), io=130GiB (139GB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=33872489/0, merge=0/0, ticks=892363/0, in_queue=892363, util=99.78%

Random read of device /dev/nvme3 (INTEL MDTPE21K015TA) using 64 job(s) with block size 4k
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=pvsync2, iodepth=1
...
fio-3.28
Starting 64 processes
Jobs: 64 (f=64): [r(64)][100.0%][r=2200MiB/s][r=563k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=64): err= 0: pid=7628: Sat Feb 25 16:34:26 2023
  read: IOPS=567k, BW=2214MiB/s (2321MB/s)(130GiB/60001msec)
    clat (usec): min=8, max=22639, avg=112.51, stdev=55.83
     lat (usec): min=8, max=22643, avg=112.53, stdev=55.83
    clat percentiles (usec):
     |  1.00th=[   61],  5.00th=[   68], 10.00th=[   71], 20.00th=[   77],
     | 30.00th=[   81], 40.00th=[   86], 50.00th=[   92], 60.00th=[  106],
     | 70.00th=[  122], 80.00th=[  139], 90.00th=[  178], 95.00th=[  217],
     | 99.00th=[  326], 99.50th=[  379], 99.90th=[  498], 99.95th=[  553],
     | 99.99th=[  586]
   bw (  MiB/s): min=  755, max= 3531, per=100.00%, avg=2213.72, stdev= 9.18, samples=7616
   iops        : min=193280, max=904172, avg=566712.17, stdev=2349.14, samples=7616
  lat (usec)   : 10=0.01%, 20=0.04%, 50=0.12%, 100=56.35%, 250=40.55%
  lat (usec)   : 500=2.84%, 750=0.10%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=0.76%, sys=1.56%, ctx=34007112, majf=0, minf=939
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=34003345,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=2214MiB/s (2321MB/s), 2214MiB/s-2214MiB/s (2321MB/s-2321MB/s), io=130GiB (139GB), run=60001-60001msec

Disk stats (read/write):
  nvme3n1: ios=33962317/0, merge=0/0, ticks=3763744/0, in_queue=3763744, util=99.82%

tl;dr:

P1600X 118GB:
- 512-byte blocks: 1,622,000 IOps at 792 MiB/s with 16 jobs (64 jobs performs worse)
- 4 KiB blocks: 218,000 IOps at 850 MiB/s with 4 jobs (16 jobs performs worse)
905P 380GB:
- 512-byte blocks: 597,000 IOps at 291 MiB/s with 64 jobs
- 4 KiB blocks: 595,000 IOps at 2,324 MiB/s with 64 jobs
P5800X 3.2TB:
- 512-byte blocks: 4,703,000 IOps at 2,296 MiB/s with 64 jobs
- 4 KiB blocks: 1,625,000 IOps at 6,349 MiB/s with 64 jobs
P4800X 1.5TB:
- 512-byte blocks: 564,000 IOps at 275 MiB/s with 64 jobs
- 4 KiB blocks: 567,000 IOps at 2,214 MiB/s with 64 jobs

Other than the P1600X, the others hit their stride at 16 jobs and don’t get much better at 64 jobs (only a thousand more IOps and few more MiB/s).

For the P5800X, I noticed the IOps above were not as high as I had seen claimed elsewhere (5 million IOps). I was able to hit that after changing the I/O engine to libaio:

The command:

# fio --filename=/dev/nvme2n1 --direct=1 --rw=randread --readonly --ioengine=libaio --runtime=60 --bs=512 --random_generator=tausworthe64 --iodepth=1 --numjobs=64 --time_based --group_reporting --name=iops-test-job

Sweet! Hit 4,971,000 IOps and 2,427 MiB/s:

iops-test-job: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=1
...
fio-3.28
Starting 64 processes
Jobs: 62 (f=61): [r(12),f(1),r(18),_(2),r(31)][33.9%][r=2449MiB/s][r=5016k IOPS]Jobs: 25 (f=22): [r(1),_(32),r(4),_(3),r(1),f(2),_(1),E(1),r(3),f(1),_(1),r(2),_(1),r(11)][2.7%][r=2096MiB/s][r=4293k IOPS][eta 38m:00s]
iops-test-job: (groupid=0, jobs=64): err= 0: pid=8089: Sat Feb 25 16:46:43 2023
  read: IOPS=4971k, BW=2427MiB/s (2545MB/s)(142GiB/60001msec)
    slat (nsec): min=741, max=15132k, avg=2101.10, stdev=2852.14
    clat (nsec): min=160, max=28361k, avg=9812.87, stdev=11544.73
     lat (usec): min=5, max=28364, avg=11.99, stdev=11.92
    clat percentiles (nsec):
     |  1.00th=[ 5664],  5.00th=[ 6816], 10.00th=[ 7392], 20.00th=[ 8096],
     | 30.00th=[ 8512], 40.00th=[ 8896], 50.00th=[ 9280], 60.00th=[ 9664],
     | 70.00th=[10432], 80.00th=[11200], 90.00th=[12736], 95.00th=[13888],
     | 99.00th=[19328], 99.50th=[22400], 99.90th=[31616], 99.95th=[37632],
     | 99.99th=[62720]
   bw (  MiB/s): min=  824, max= 2939, per=100.00%, avg=2438.76, stdev= 3.65, samples=7566
   iops        : min=1689102, max=6019264, avg=4994585.68, stdev=7482.69, samples=7566
  lat (nsec)   : 250=0.05%, 500=0.35%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.02%, 4=0.09%, 10=64.10%, 20=34.57%, 50=0.80%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=12.72%, sys=23.93%, ctx=304262689, majf=0, minf=989
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=298249219,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=2427MiB/s (2545MB/s), 2427MiB/s-2427MiB/s (2545MB/s-2545MB/s), io=142GiB (153GB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=298060905/0, merge=0/0, ticks=2204516/0, in_queue=2204516, util=97.40%

Tulnek · February 28, 2023, 1:51am

Excellent responses by LiKenun.
I have a 905p and 5800X. Optane for OS installs always. Optane is the best for low latency, but also doesn’t bog down over time, it stays fast at 100% utilization, whereas flash-based drives will fill their cache and slow down by a ton.
Also, Optane can load everything at once with low latency, it doesn’t slow down. One of the first things I noticed when getting a 905p a few years ago was how fast programs started up on OS load, everything could start simultaneously no problem. Optane can be hammered all day and night by I/O and ask for more, it’s designed for it, and they don’t even have a DRAM cache, it’s not needed.
Switching from a very fast drive (905p) to an incredibly fast drive (5800X) is not noticeable day-to-day unless constantly working with files, where the extra performance of the 5800X would matter. I’m rocking a 5800X now for Windows and while it’s faster sequentially, I’m also not loading huge files all day. Loading small files quickly matters more, for my use case which is daily driving an OS.
Huge files, or lots of small files, use the 5800X. Just OS stuff, a 905P is perfect.
I’m not going back to flash for a main OS drive for the foreseeable future, but do use flash for secondary storage.

LiKenun · March 1, 2023, 1:18am

These two series are pretty much the best of the Optanes. Surprisingly, the P480*X series perform slightly worse than the 905P. The 905Ps hold their performance very well in a wider range of I/O loads.

I’d put the 905Ps and P480*Xs at about one-third the performance of P5800Xs.

system · November 29, 2023, 7:18pm

This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.