Wow!
So this things seems stupid fast. 1.7 million iops – that’s leading Samsung! What’s the catch?
| Controller | Phison E18 (2021) |
| Controller | Phison E18 (2021) |
| Max Speed | 7400 Megabytes/sec (measured) 7150 Megabytes/sec (rated) |
| Max Write | 5600 Megabytes/sec |
| 4k random read | 380,000 iops * |
| 4k random write | 650,000 iops * |
| Endurance | 1.4 PBW |
| Media Type | 3d QLC |
| Protocol comp. | NVME 1.3 spec ** |
| Warranty | 5 year |
** Does not support reprogramming the nvme namespace size or “underpovisioning”
- See below for measured iops in different scenarios
Power Consumption : 6.6 watts
Smartctl -a
smartctl 7.2 2021-01-17 r5171 [x86_64-linux-5.11.17-200.fc33.x86_64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Number: MKNSSDGA2TB-D8
Serial Number:
Firmware Version: EIFM21.1
PCI Vendor/Subsystem ID: 0x1987
IEEE OUI Identifier: 0x6479a7
Total NVM Capacity: 2,000,398,934,016 [2.00 TB]
Unallocated NVM Capacity: 0
Controller ID: 1
NVMe Version: 1.4
Number of Namespaces: 1
Namespace 1 Size/Capacity: 2,000,398,934,016 [2.00 TB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: 6479a7 47c0201a79
Local Time is: Sat May 15 21:22:13 2021 EDT
Firmware Updates (0x12): 1 Slot, no Reset required
Optional Admin Commands (0x0017): Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005d): Comp DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Log Page Attributes (0x08): Telmtry_Lg
Maximum Data Transfer Size: 512 Pages
Warning Comp. Temp. Threshold: 70 Celsius
Critical Comp. Temp. Threshold: 110 Celsius
Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 8.80W - - 0 0 0 0 0 0
1 + 7.10W - - 1 1 1 1 0 0
2 + 5.20W - - 2 2 2 2 0 0
3 - 0.0620W - - 3 3 3 3 2000 2000
4 - 0.0440W - - 4 4 4 4 25000 25000
Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 2
1 - 4096 0 1
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 46 Celsius
Available Spare: 100%
Available Spare Threshold: 5%
Percentage Used: 0%
Data Units Read: 9,391,598 [4.80 TB]
Data Units Written: 8,197,856 [4.19 TB]
Host Read Commands: 628,979,210
Host Write Commands: 58,573,164
Controller Busy Time: 89
Power Cycles: 5
Power On Hours: 58
Unsafe Shutdowns: 1
Media and Data Integrity Errors: 0
Error Information Log Entries: 11
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Thermal Temp. 1 Total Time: 835
Error Information (NVMe Log 0x01, 16 of 63 entries)
No Errors Logged
FIO Results (Read)
Jobs: 4 (f=4): [r(4)][100.0%][r=1748MiB/s][r=447k IOPS][eta 00m:00s]
onessd: (groupid=0, jobs=4): err= 0: pid=212638: Sun May 9 18:48:16 2021
read: IOPS=1490k, BW=5822MiB/s (6105MB/s)(171GiB/30001msec)
bw ( MiB/s): min= 1735, max= 6964, per=100.00%, avg=5892.93, stdev=512.05, samples=236
iops : min=444348, max=1782876, avg=1508590.92, stdev=131084.96, samples=236
cpu : usr=13.03%, sys=82.53%, ctx=275, majf=0, minf=14
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=44713336,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=5822MiB/s (6105MB/s), 5822MiB/s-5822MiB/s (6105MB/s-6105MB/s), io=171GiB (183GB), run=30001-30001msec
Disk stats (read/write):
nvme19n1: ios=44645031/4, merge=0/0, ticks=3484415/1, in_queue=3484417, util=99.79%
Crystal Diskmark Results
Configuration: Size:1024m Loops:5 Write Only Zeroes:0
Running Benchmark, please wait…
Results:
Sequential Read: 5361MB/s IOPS=5
Sequential Write: 4259MB/s IOPS=4
512KB Read: 2439MB/s IOPS=4878
512KB Write: 4298MB/s IOPS=8597
Sequential Q32T1 Read: 6328MB/s IOPS=197
Sequential Q32T1 Write: 6328MB/s IOPS=197
4KB Read: 56MB/s IOPS=14570
4KB Write: 191MB/s IOPS=48907
4KB Q32T1 Read: 100MB/s IOPS=25763
4KB Q32T1 Write: 256MB/s IOPS=65660
4KB Q8T8 Read: 119MB/s IOPS=30531
4KB Q8T8 Write: 274MB/s IOPS=70243
Ah, the mystery is solved
So as the drive is empty, it operates in SLC mode. It is very fast until it is about 1/4 full, and then it becomes slower.
I created initially about 500 of gb of test data. This brought the average iops down from ~1.7m iops to about ~1.3 million.
Mushkin rates this drive for about ~800k iops; a number I suspect is for when the drive is 1/2 full.
500gb used:
[root@localhost 2]# fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=64k --size=256m --numjobs=16 --iodepth=16 --runtime=60 --time_based --end_fsync=1 ./a
random-write: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=posixaio, iodepth=16
...
fio-3.21
Starting 16 processes
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
random-write: Laying out IO file (1 file / 256MiB)
Jobs: 16 (f=16): [w(16)][100.0%][w=33.0MiB/s][w=528 IOPS][eta 00m:00s]
random-write: (groupid=0, jobs=1): err= 0: pid=18675: Sat May 15 20:52:47 2021
write: IOPS=4319, BW=270MiB/s (283MB/s)(16.0GiB/60682msec); 0 zone resets
slat (nsec): min=320, max=406593, avg=1680.67, stdev=1464.17
clat (usec): min=51, max=6938.5k, avg=2779.11, stdev=61228.48
lat (usec): min=73, max=6938.5k, avg=2780.79, stdev=61228.48
clat percentiles (usec):
| 1.00th=[ 355], 5.00th=[ 412], 10.00th=[ 490],
| 20.00th=[ 562], 30.00th=[ 627], 40.00th=[ 693],
| 50.00th=[ 734], 60.00th=[ 775], 70.00th=[ 840],
| 80.00th=[ 922], 90.00th=[ 1020], 95.00th=[ 1074],
| 99.00th=[ 1205], 99.50th=[ 2180], 99.90th=[ 501220],
| 99.95th=[ 599786], 99.99th=[2600469]
bw ( KiB/s): min=23680, max=1000448, per=9.92%, avg=424715.34, stdev=173933.78, samples=79
iops : min= 370, max=15632, avg=6636.18, stdev=2717.72, samples=79
lat (usec) : 100=0.01%, 250=0.03%, 500=11.03%, 750=42.13%, 1000=34.42%
lat (msec) : 2=11.79%, 4=0.28%, 20=0.01%, 250=0.05%, 500=0.14%
lat (msec) : 750=0.08%, 1000=0.01%, >=2000=0.03%
cpu : usr=1.15%, sys=9.87%, ctx=134944, majf=0, minf=32
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.2%, 16=49.7%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=95.7%, 8=1.7%, 16=2.6%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,262145,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=18676: Sat May 15 20:52:47 2021
write: IOPS=4328, BW=271MiB/s (284MB/s)(16.0GiB/60561msec); 0 zone resets
slat (nsec): min=311, max=522120, avg=1603.76, stdev=1630.85
clat (usec): min=54, max=5688.4k, avg=2535.31, stdev=56556.77
lat (usec): min=65, max=5688.4k, avg=2536.91, stdev=56556.77
clat percentiles (usec):
| 1.00th=[ 355], 5.00th=[ 416], 10.00th=[ 537],
| 20.00th=[ 603], 30.00th=[ 635], 40.00th=[ 668],
| 50.00th=[ 717], 60.00th=[ 775], 70.00th=[ 840],
| 80.00th=[ 947], 90.00th=[ 1020], 95.00th=[ 1074],
| 99.00th=[ 1254], 99.50th=[ 1975], 99.90th=[ 505414],
| 99.95th=[ 616563], 99.99th=[2969568]
bw ( KiB/s): min=83968, max=840832, per=9.92%, avg=424728.30, stdev=156025.89, samples=79
iops : min= 1312, max=13138, avg=6636.38, stdev=2437.90, samples=79
lat (usec) : 100=0.01%, 250=0.02%, 500=7.09%, 750=49.38%, 1000=30.09%
lat (msec) : 2=12.92%, 4=0.24%, 10=0.01%, 250=0.04%, 500=0.11%
lat (msec) : 750=0.08%, 1000=0.01%, 2000=0.01%, >=2000=0.02%
cpu : usr=1.09%, sys=9.78%, ctx=135016, majf=0, minf=32
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.2%, 16=49.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=95.7%, 8=1.5%, 16=2.8%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,262145,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=18677: Sat May 15 20:52:47 2021
write: IOPS=4049, BW=253MiB/s (265MB/s)(15.0GiB/60682msec); 0 zone resets
slat (nsec): min=261, max=436759, avg=1813.04, stdev=2208.00
clat (usec): min=54, max=7704.8k, avg=3076.53, stdev=67010.12
lat (usec): min=60, max=7704.8k, avg=3078.34, stdev=67010.11
clat percentiles (usec):
| 1.00th=[ 375], 5.00th=[ 482], 10.00th=[ 553],
| 20.00th=[ 611], 30.00th=[ 668], 40.00th=[ 717],
| 50.00th=[ 758], 60.00th=[ 807], 70.00th=[ 873],
| 80.00th=[ 963], 90.00th=[ 1037], 95.00th=[ 1074],
| 99.00th=[ 1303], 99.50th=[ 2376], 99.90th=[ 583009],
| 99.95th=[ 658506], 99.99th=[2701132]
bw ( KiB/s): min= 2048, max=901120, per=9.93%, avg=425077.62, stdev=174029.97, samples=74
iops : min= 32, max=14080, avg=6641.84, stdev=2719.22, samples=74
lat (usec) : 100=0.01%, 250=0.03%, 500=5.53%, 750=42.52%, 1000=37.23%
lat (msec) : 2=14.05%, 4=0.32%, 20=0.01%, 100=0.01%, 250=0.02%
lat (msec) : 500=0.12%, 750=0.13%, 1000=0.01%, >=2000=0.03%
cpu : usr=1.16%, sys=9.46%, ctx=124492, majf=0, minf=33
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.2%, 16=49.7%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=95.5%, 8=2.5%, 16=2.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,245761,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=18678: Sat May 15 20:52:47 2021
write: IOPS=4252, BW=266MiB/s (279MB/s)(15.8GiB/60682msec); 0 zone resets
slat (nsec): min=331, max=321272, avg=1927.45, stdev=1467.61
clat (usec): min=52, max=6985.0k, avg=3107.13, stdev=75609.95
lat (usec): min=59, max=6986.0k, avg=3109.06, stdev=75609.95
clat percentiles (usec):
| 1.00th=[ 359], 5.00th=[ 400], 10.00th=[ 490],
| 20.00th=[ 562], 30.00th=[ 627], 40.00th=[ 676],
| 50.00th=[ 734], 60.00th=[ 791], 70.00th=[ 848],
| 80.00th=[ 947], 90.00th=[ 1029], 95.00th=[ 1090],
| 99.00th=[ 1188], 99.50th=[ 2147], 99.90th=[ 526386],
| 99.95th=[ 675283], 99.99th=[3674211]
bw ( KiB/s): min= 1791, max=713984, per=9.77%, avg=418101.46, stdev=163294.27, samples=79
iops : min= 27, max=11156, avg=6532.82, stdev=2551.51, samples=79
lat (usec) : 100=0.01%, 250=0.01%, 500=11.19%, 750=42.47%, 1000=33.19%
lat (msec) : 2=12.57%, 4=0.29%, 20=0.01%, 100=0.01%, 250=0.03%
lat (msec) : 500=0.13%, 750=0.06%, 1000=0.01%, 2000=0.01%, >=2000=0.03%
cpu : usr=1.20%, sys=9.61%, ctx=130850, majf=0, minf=32
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.2%, 16=49.7%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=95.7%, 8=1.6%, 16=2.8%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,258049,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=18679: Sat May 15 20:52:47 2021
write: IOPS=4020, BW=251MiB/s (264MB/s)(14.8GiB/60101msec); 0 zone resets
slat (nsec): min=451, max=354475, avg=1779.34, stdev=1249.23
clat (usec): min=64, max=7947.6k, avg=3128.97, stdev=74256.45
lat (usec): min=76, max=7947.6k, avg=3130.75, stdev=74256.45
clat percentiles (usec):
| 1.00th=[ 347], 5.00th=[ 469], 10.00th=[ 529],
| 20.00th=[ 619], 30.00th=[ 676], 40.00th=[ 717],
| 50.00th=[ 758], 60.00th=[ 807], 70.00th=[ 873],
| 80.00th=[ 955], 90.00th=[ 1029], 95.00th=[ 1090],
| 99.00th=[ 1254], 99.50th=[ 2245], 99.90th=[ 591397],
| 99.95th=[ 666895], 99.99th=[2835350]
bw ( KiB/s): min=49664, max=862976, per=9.77%, avg=417987.46, stdev=166171.81, samples=74
iops : min= 776, max=13484, avg=6531.05, stdev=2596.43, samples=74
lat (usec) : 100=0.01%, 250=0.04%, 500=7.04%, 750=41.16%, 1000=37.84%
lat (msec) : 2=13.33%, 4=0.29%, 20=0.01%, 50=0.01%, 250=0.02%
lat (msec) : 500=0.12%, 750=0.12%, 1000=0.01%, >=2000=0.03%
cpu : usr=1.14%, sys=9.53%, ctx=123456, majf=0, minf=33
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.2%, 16=49.7%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=95.6%, 8=2.7%, 16=1.7%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,241665,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=18680: Sat May 15 20:52:47 2021
write: IOPS=4185, BW=262MiB/s (274MB/s)(15.5GiB/60681msec); 0 zone resets
slat (nsec): min=341, max=293951, avg=1704.35, stdev=1319.12
clat (usec): min=47, max=4817.8k, avg=3113.71, stdev=61482.40
lat (usec): min=70, max=4817.8k, avg=3115.42, stdev=61482.41
clat percentiles (usec):
| 1.00th=[ 359], 5.00th=[ 396], 10.00th=[ 529],
| 20.00th=[ 603], 30.00th=[ 652], 40.00th=[ 701],
| 50.00th=[ 750], 60.00th=[ 799], 70.00th=[ 865],
| 80.00th=[ 947], 90.00th=[ 1029], 95.00th=[ 1090],
| 99.00th=[ 1221], 99.50th=[ 2245], 99.90th=[ 549454],
| 99.95th=[ 658506], 99.99th=[2600469]
bw ( KiB/s): min= 2816, max=697088, per=9.86%, avg=422129.04, stdev=158216.22, samples=77
iops : min= 44, max=10892, avg=6595.77, stdev=2472.13, samples=77
lat (usec) : 50=0.01%, 100=0.01%, 250=0.02%, 500=8.32%, 750=42.05%
lat (usec) : 1000=36.51%
lat (msec) : 2=12.48%, 4=0.28%, 250=0.02%, 500=0.16%, 750=0.10%
lat (msec) : 1000=0.01%, >=2000=0.04%
cpu : usr=1.13%, sys=9.54%, ctx=131181, majf=0, minf=33
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.1%, 16=49.7%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=95.3%, 8=2.4%, 16=2.2%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,253953,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=18681: Sat May 15 20:52:47 2021
write: IOPS=4344, BW=272MiB/s (285MB/s)(16.0GiB/60340msec); 0 zone resets
slat (nsec): min=291, max=249137, avg=1827.58, stdev=1196.20
clat (usec): min=52, max=5665.3k, avg=2794.76, stdev=60057.32
lat (usec): min=62, max=5665.3k, avg=2796.59, stdev=60057.32
clat percentiles (usec):
| 1.00th=[ 420], 5.00th=[ 519], 10.00th=[ 570],
| 20.00th=[ 611], 30.00th=[ 652], 40.00th=[ 701],
| 50.00th=[ 750], 60.00th=[ 807], 70.00th=[ 865],
| 80.00th=[ 947], 90.00th=[ 1020], 95.00th=[ 1074],
| 99.00th=[ 1254], 99.50th=[ 2180], 99.90th=[ 541066],
| 99.95th=[ 616563], 99.99th=[3003122]
bw ( KiB/s): min= 8576, max=924032, per=10.18%, avg=435746.91, stdev=175469.69, samples=77
iops : min= 134, max=14438, avg=6808.55, stdev=2741.71, samples=77
lat (usec) : 100=0.01%, 250=0.01%, 500=3.67%, 750=45.79%, 1000=37.80%
lat (msec) : 2=12.14%, 4=0.28%, 10=0.02%, 250=0.04%, 500=0.12%
lat (msec) : 750=0.09%, 1000=0.01%, 2000=0.01%, >=2000=0.03%
cpu : usr=1.18%, sys=9.88%, ctx=132784, majf=0, minf=32
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.1%, 16=49.7%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=95.6%, 8=2.2%, 16=2.3%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,262145,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=18682: Sat May 15 20:52:47 2021
write: IOPS=4149, BW=259MiB/s (272MB/s)(15.2GiB/60212msec); 0 zone resets
slat (nsec): min=280, max=215755, avg=1774.15, stdev=1156.77
clat (usec): min=51, max=8180.1k, avg=2756.73, stdev=67407.01
lat (usec): min=62, max=8180.1k, avg=2758.51, stdev=67407.01
clat percentiles (usec):
| 1.00th=[ 343], 5.00th=[ 396], 10.00th=[ 465],
| 20.00th=[ 594], 30.00th=[ 660], 40.00th=[ 709],
| 50.00th=[ 750], 60.00th=[ 807], 70.00th=[ 873],
| 80.00th=[ 955], 90.00th=[ 1020], 95.00th=[ 1074],
| 99.00th=[ 1205], 99.50th=[ 2073], 99.90th=[ 549454],
| 99.95th=[ 666895], 99.99th=[2734687]
bw ( KiB/s): min= 1280, max=709760, per=10.23%, avg=438079.12, stdev=147404.22, samples=73
iops : min= 20, max=11090, avg=6844.99, stdev=2303.19, samples=73
lat (usec) : 100=0.01%, 250=0.02%, 500=12.48%, 750=37.27%, 1000=37.32%
lat (msec) : 2=12.37%, 4=0.27%, 10=0.01%, 100=0.01%, 250=0.01%
lat (msec) : 500=0.13%, 750=0.08%, 1000=0.01%, 2000=0.01%, >=2000=0.03%
cpu : usr=1.05%, sys=9.62%, ctx=126117, majf=0, minf=34
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.2%, 16=49.7%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=95.8%, 8=1.7%, 16=2.5%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,249857,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=18683: Sat May 15 20:52:47 2021
write: IOPS=4208, BW=263MiB/s (276MB/s)(15.5GiB/60340msec); 0 zone resets
slat (nsec): min=501, max=725060, avg=1993.07, stdev=1952.97
clat (usec): min=51, max=7931.0k, avg=2921.63, stdev=71299.92
lat (usec): min=58, max=7931.0k, avg=2923.63, stdev=71299.92
clat percentiles (usec):
| 1.00th=[ 363], 5.00th=[ 408], 10.00th=[ 537],
| 20.00th=[ 611], 30.00th=[ 652], 40.00th=[ 701],
| 50.00th=[ 750], 60.00th=[ 807], 70.00th=[ 857],
| 80.00th=[ 938], 90.00th=[ 1029], 95.00th=[ 1090],
| 99.00th=[ 1188], 99.50th=[ 2147], 99.90th=[ 530580],
| 99.95th=[ 616563], 99.99th=[3036677]
bw ( KiB/s): min= 4480, max=952576, per=10.40%, avg=445262.90, stdev=160048.99, samples=73
iops : min= 70, max=14884, avg=6957.23, stdev=2500.77, samples=73
lat (usec) : 100=0.01%, 250=0.02%, 500=6.37%, 750=43.91%, 1000=36.76%
lat (msec) : 2=12.38%, 4=0.27%, 250=0.03%, 500=0.14%, 750=0.08%
lat (msec) : 1000=0.01%, 2000=0.01%, >=2000=0.02%
cpu : usr=1.26%, sys=9.66%, ctx=129860, majf=0, minf=32
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.1%, 16=49.7%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=95.4%, 8=2.1%, 16=2.5%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,253953,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=18684: Sat May 15 20:52:47 2021
write: IOPS=4353, BW=272MiB/s (285MB/s)(16.0GiB/60212msec); 0 zone resets
slat (nsec): min=250, max=154370, avg=1746.22, stdev=903.60
clat (usec): min=57, max=7741.2k, avg=2995.73, stdev=75804.04
lat (usec): min=66, max=7741.2k, avg=2997.48, stdev=75804.04
clat percentiles (usec):
| 1.00th=[ 400], 5.00th=[ 494], 10.00th=[ 537],
| 20.00th=[ 594], 30.00th=[ 644], 40.00th=[ 693],
| 50.00th=[ 742], 60.00th=[ 799], 70.00th=[ 848],
| 80.00th=[ 922], 90.00th=[ 1012], 95.00th=[ 1057],
| 99.00th=[ 1188], 99.50th=[ 2008], 99.90th=[ 522191],
| 99.95th=[ 624952], 99.99th=[3036677]
bw ( KiB/s): min=14336, max=796032, per=10.31%, avg=441480.42, stdev=154796.40, samples=76
iops : min= 224, max=12438, avg=6898.13, stdev=2418.69, samples=76
lat (usec) : 100=0.01%, 250=0.01%, 500=5.51%, 750=45.69%, 1000=37.12%
lat (msec) : 2=11.16%, 4=0.23%, 10=0.01%, 250=0.02%, 500=0.14%
lat (msec) : 750=0.07%, 1000=0.01%, >=2000=0.04%
cpu : usr=1.22%, sys=9.51%, ctx=132934, majf=0, minf=32
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.1%, 16=49.7%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=95.6%, 8=1.8%, 16=2.6%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,262145,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=18685: Sat May 15 20:52:47 2021
write: IOPS=4089, BW=256MiB/s (268MB/s)(15.0GiB/60101msec); 0 zone resets
slat (nsec): min=320, max=566834, avg=1861.58, stdev=1790.88
clat (usec): min=64, max=6615.8k, avg=3239.34, stdev=73118.90
lat (usec): min=73, max=6615.8k, avg=3241.20, stdev=73118.90
clat percentiles (usec):
| 1.00th=[ 388], 5.00th=[ 519], 10.00th=[ 570],
| 20.00th=[ 627], 30.00th=[ 676], 40.00th=[ 717],
| 50.00th=[ 766], 60.00th=[ 824], 70.00th=[ 889],
| 80.00th=[ 971], 90.00th=[ 1045], 95.00th=[ 1090],
| 99.00th=[ 1336], 99.50th=[ 2278], 99.90th=[ 549454],
| 99.95th=[ 666895], 99.99th=[4731175]
bw ( KiB/s): min= 1920, max=840576, per=9.80%, avg=419404.80, stdev=174384.42, samples=75
iops : min= 30, max=13134, avg=6553.20, stdev=2724.76, samples=75
lat (usec) : 100=0.01%, 250=0.02%, 500=3.82%, 750=42.51%, 1000=38.16%
lat (msec) : 2=14.83%, 4=0.33%, 20=0.01%, 50=0.01%, 100=0.01%
lat (msec) : 250=0.03%, 500=0.14%, 750=0.10%, 1000=0.01%, >=2000=0.04%
cpu : usr=1.22%, sys=9.52%, ctx=127596, majf=0, minf=31
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.1%, 16=49.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=95.6%, 8=2.0%, 16=2.4%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,245761,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=18686: Sat May 15 20:52:47 2021
write: IOPS=4246, BW=265MiB/s (278MB/s)(15.7GiB/60679msec); 0 zone resets
slat (nsec): min=241, max=250480, avg=1957.72, stdev=1502.19
clat (usec): min=64, max=6709.6k, avg=2989.49, stdev=61333.95
lat (usec): min=72, max=6709.6k, avg=2991.45, stdev=61333.95
clat percentiles (usec):
| 1.00th=[ 338], 5.00th=[ 379], 10.00th=[ 474],
| 20.00th=[ 578], 30.00th=[ 644], 40.00th=[ 701],
| 50.00th=[ 750], 60.00th=[ 799], 70.00th=[ 865],
| 80.00th=[ 955], 90.00th=[ 1037], 95.00th=[ 1090],
| 99.00th=[ 1287], 99.50th=[ 2245], 99.90th=[ 557843],
| 99.95th=[ 658506], 99.99th=[2734687]
bw ( KiB/s): min= 6144, max=999424, per=9.74%, avg=416730.26, stdev=199136.09, samples=78
iops : min= 96, max=15616, avg=6511.41, stdev=3111.50, samples=78
lat (usec) : 100=0.01%, 250=0.03%, 500=11.32%, 750=38.99%, 1000=35.47%
lat (msec) : 2=13.57%, 4=0.29%, 250=0.04%, 500=0.13%, 750=0.11%
lat (msec) : 1000=0.01%, 2000=0.01%, >=2000=0.02%
cpu : usr=1.26%, sys=9.54%, ctx=133867, majf=0, minf=32
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.2%, 16=49.6%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=95.6%, 8=2.3%, 16=2.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,257664,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=18687: Sat May 15 20:52:47 2021
write: IOPS=4276, BW=267MiB/s (280MB/s)(15.8GiB/60340msec); 0 zone resets
slat (nsec): min=291, max=304621, avg=1776.30, stdev=1315.98
clat (usec): min=42, max=8884.2k, avg=3127.55, stdev=82258.03
lat (usec): min=47, max=8884.2k, avg=3129.33, stdev=82258.03
clat percentiles (usec):
| 1.00th=[ 355], 5.00th=[ 445], 10.00th=[ 529],
| 20.00th=[ 603], 30.00th=[ 644], 40.00th=[ 685],
| 50.00th=[ 734], 60.00th=[ 775], 70.00th=[ 848],
| 80.00th=[ 947], 90.00th=[ 1029], 95.00th=[ 1074],
| 99.00th=[ 1237], 99.50th=[ 2212], 99.90th=[ 530580],
| 99.95th=[ 633340], 99.99th=[2969568]
bw ( KiB/s): min=36864, max=827264, per=10.02%, avg=428944.62, stdev=172763.46, samples=77
iops : min= 576, max=12926, avg=6702.26, stdev=2699.43, samples=77
lat (usec) : 50=0.01%, 100=0.01%, 250=0.02%, 500=6.84%, 750=47.55%
lat (usec) : 1000=32.05%
lat (msec) : 2=12.93%, 4=0.29%, 250=0.03%, 500=0.15%, 750=0.09%
lat (msec) : 1000=0.01%, 2000=0.01%, >=2000=0.03%
cpu : usr=1.11%, sys=9.89%, ctx=130145, majf=0, minf=33
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.2%, 16=49.7%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=95.8%, 8=1.3%, 16=2.9%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,258049,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=18688: Sat May 15 20:52:47 2021
write: IOPS=4180, BW=261MiB/s (274MB/s)(15.5GiB/60680msec); 0 zone resets
slat (nsec): min=320, max=423113, avg=1727.64, stdev=1945.63
clat (usec): min=51, max=8131.3k, avg=3045.09, stdev=76128.69
lat (usec): min=67, max=8131.3k, avg=3046.82, stdev=76128.69
clat percentiles (usec):
| 1.00th=[ 363], 5.00th=[ 502], 10.00th=[ 529],
| 20.00th=[ 586], 30.00th=[ 644], 40.00th=[ 693],
| 50.00th=[ 742], 60.00th=[ 791], 70.00th=[ 857],
| 80.00th=[ 947], 90.00th=[ 1029], 95.00th=[ 1074],
| 99.00th=[ 1254], 99.50th=[ 2180], 99.90th=[ 549454],
| 99.95th=[ 624952], 99.99th=[2969568]
bw ( KiB/s): min= 2048, max=802688, per=9.83%, avg=420794.95, stdev=175673.44, samples=76
iops : min= 32, max=12542, avg=6574.92, stdev=2744.90, samples=76
lat (usec) : 100=0.01%, 250=0.03%, 500=4.97%, 750=47.44%, 1000=34.39%
lat (msec) : 2=12.59%, 4=0.27%, 10=0.01%, 250=0.02%, 500=0.13%
lat (msec) : 750=0.11%, 1000=0.01%, 2000=0.01%, >=2000=0.02%
cpu : usr=1.08%, sys=9.55%, ctx=128651, majf=0, minf=33
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.2%, 16=49.6%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=95.7%, 8=1.6%, 16=2.7%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,253694,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=18689: Sat May 15 20:52:47 2021
write: IOPS=3982, BW=249MiB/s (261MB/s)(14.8GiB/60680msec); 0 zone resets
slat (nsec): min=280, max=387277, avg=1932.22, stdev=1561.47
clat (usec): min=57, max=4858.9k, avg=3422.71, stdev=72444.37
lat (usec): min=75, max=4858.9k, avg=3424.64, stdev=72444.37
clat percentiles (usec):
| 1.00th=[ 351], 5.00th=[ 396], 10.00th=[ 486],
| 20.00th=[ 603], 30.00th=[ 668], 40.00th=[ 717],
| 50.00th=[ 766], 60.00th=[ 824], 70.00th=[ 889],
| 80.00th=[ 979], 90.00th=[ 1045], 95.00th=[ 1074],
| 99.00th=[ 1565], 99.50th=[ 2311], 99.90th=[ 599786],
| 99.95th=[ 901776], 99.99th=[3976201]
bw ( KiB/s): min=10240, max=830080, per=9.90%, avg=423713.32, stdev=162981.80, samples=73
iops : min= 160, max=12970, avg=6620.52, stdev=2546.59, samples=73
lat (usec) : 100=0.01%, 250=0.03%, 500=11.20%, 750=34.52%, 1000=37.47%
lat (msec) : 2=16.13%, 4=0.34%, 10=0.01%, 20=0.01%, 250=0.02%
lat (msec) : 500=0.10%, 750=0.13%, 1000=0.01%, >=2000=0.05%
cpu : usr=1.19%, sys=9.31%, ctx=126103, majf=0, minf=32
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.2%, 16=49.7%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=95.8%, 8=2.5%, 16=1.7%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,241665,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=18690: Sat May 15 20:52:47 2021
write: IOPS=4157, BW=260MiB/s (272MB/s)(15.2GiB/60101msec); 0 zone resets
slat (nsec): min=270, max=489548, avg=1874.40, stdev=2201.22
clat (usec): min=26, max=5668.5k, avg=3186.64, stdev=65014.42
lat (usec): min=28, max=5668.6k, avg=3188.51, stdev=65014.42
clat percentiles (usec):
| 1.00th=[ 351], 5.00th=[ 396], 10.00th=[ 519],
| 20.00th=[ 586], 30.00th=[ 635], 40.00th=[ 701],
| 50.00th=[ 750], 60.00th=[ 807], 70.00th=[ 857],
| 80.00th=[ 947], 90.00th=[ 1045], 95.00th=[ 1090],
| 99.00th=[ 1221], 99.50th=[ 2245], 99.90th=[ 557843],
| 99.95th=[ 624952], 99.99th=[3405775]
bw ( KiB/s): min=22656, max=625920, per=9.70%, avg=415330.08, stdev=157062.66, samples=77
iops : min= 354, max= 9780, avg=6489.53, stdev=2454.10, samples=77
lat (usec) : 50=0.01%, 100=0.01%, 250=0.05%, 500=7.53%, 750=42.06%
lat (usec) : 1000=35.52%
lat (msec) : 2=14.27%, 4=0.24%, 250=0.02%, 500=0.12%, 750=0.14%
lat (msec) : 1000=0.01%, 2000=0.02%, >=2000=0.03%
cpu : usr=1.18%, sys=9.56%, ctx=125911, majf=0, minf=32
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.2%, 16=49.7%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=95.7%, 8=2.2%, 16=2.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,249857,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=4180MiB/s (4383MB/s), 249MiB/s-272MiB/s (261MB/s-285MB/s), io=248GiB (266GB), run=60101-60682msec
Disk stats (read/write):
nvme14n1: ios=0/511242, merge=0/672, ticks=0/10685026, in_queue=10685102, util=98.24%
# 4k random reads (500gb used of 2tb)
...
fio-3.21
Starting 4 processes
Jobs: 4 (f=4): [r(4)][100.0%][r=6350MiB/s][r=1626k IOPS][eta 00m:00s]
onessd: (groupid=0, jobs=4): err= 0: pid=21699: Sat May 15 20:56:53 2021
read: IOPS=1623k, BW=6340MiB/s (6648MB/s)(186GiB/30001msec)
bw ( MiB/s): min= 5815, max= 6409, per=100.00%, avg=6347.80, stdev=18.06, samples=236
iops : min=1488795, max=1640914, avg=1625036.02, stdev=4623.50, samples=236
cpu : usr=14.24%, sys=80.24%, ctx=250, majf=1, minf=21
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=48690616,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=6340MiB/s (6648MB/s), 6340MiB/s-6340MiB/s (6648MB/s-6648MB/s), io=186GiB (199GB), run=30001-30001msec
Disk stats (read/write):
nvme14n1: ios=48480896/0, merge=0/0, ticks=3431562/0, in_queue=3431562, util=99.82%
75% full (1.5tb used of 2tb) (4k random reads)
fio-3.21
Starting 4 processes
Jobs: 4 (f=4): [r(4)][100.0%][r=1791MiB/s][r=459k IOPS][eta 00m:00s]
onessd: (groupid=0, jobs=4): err= 0: pid=82577: Sat May 15 22:43:36 2021
read: IOPS=456k, BW=1782MiB/s (1868MB/s)(52.2GiB/30002msec)
bw ( MiB/s): min= 1560, max= 1808, per=100.00%, avg=1783.92, stdev= 7.71, samples=236
iops : min=399539, max=463076, avg=456683.47, stdev=1974.31, samples=236
cpu : usr=4.84%, sys=91.01%, ctx=159, majf=0, minf=17
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=13683384,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=1782MiB/s (1868MB/s), 1782MiB/s-1782MiB/s (1868MB/s-1868MB/s), io=52.2GiB (56.0GB), run=30002-30002msec
Disk stats (read/write):
nvme14n1: ios=13623945/0, merge=0/0, ticks=3796323/0, in_queue=3796324, util=99.76%
99% full (2tb, outch!)
nessd: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=32
...
fio-3.21
Starting 4 processes
Jobs: 4 (f=4): [r(4)][100.0%][r=320MiB/s][r=81.9k IOPS][eta 00m:00s]
onessd: (groupid=0, jobs=4): err= 0: pid=72911: Sat May 15 22:26:58 2021
read: IOPS=64.7k, BW=253MiB/s (265MB/s)(7581MiB/30003msec)
bw ( KiB/s): min=97968, max=334584, per=99.71%, avg=257979.93, stdev=25538.84, samples=236
iops : min=24492, max=83646, avg=64494.98, stdev=6384.71, samples=236
cpu : usr=0.82%, sys=98.59%, ctx=206, majf=0, minf=19
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=1940728,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=253MiB/s (265MB/s), 253MiB/s-253MiB/s (265MB/s-265MB/s), io=7581MiB (7949MB), run=30003-30003msec
Disk stats (read/write):
nvme14n1: ios=1930020/0, merge=0/0, ticks=3820458/0, in_queue=3820458, util=99.77%
[root@localhost 2]# ../onessd.sh 4 /dev/nvme14n1 4k
onessd: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=32
...
