Octoquadragintuple Actuator Hardware Accelerated Raid!

Hardware Hub Build Logs
1

After many many years of owning no dedicated backup server I’ve finally cobbled together an eclectic mix of parts to make one. This will be a build log to try and make a quiet, fast and cheap backup server.

For the case I ended up coming across a 4U Chenbro tri-fold chassis for a good price. It’s got 12-drive expander backplanes on the front, left and right side for a total of 36 bays:

Untitled
Untitled750×800 223 KB

For drives I’m going with two dozen 14TB mach.2s I picked up for 8 mills a gigabyte that I bought on FOMO impulses.

​​​ ​ ​

​​​ ​ ​

The centerpiece of this build is going to be the raid card.
I decided on Areca due to the amount of control it gives over very low level functions of storage that I haven’t seen anywhere else, as well as the out of band management it has for peace of mind; the last thing I want to encounter one morning is a no boot situation and having to worry about the integrity of my storage.

ARC-1886-16i:

image
image700×700 73.3 KB

The rest of the system is using spare parts I had lying around:
Asus W790 Sage motherboard
Intel w5-3435x CPU
2x 24GB vcolor 5600MT RDIMM
Nvidia a2000 12GB GPU
Silverstone XE02-4677 CPU cooler
Acbel 1200W CRPS PSU

​​​ ​ ​

​​​ ​ ​
I put the system together to get an idea of what I was dealing with:

image
image1920×1280 327 KB

  • The first problem is the CRPS power supply isn’t modular and is making a rat’s nest of cables and blocking airflow.

    • I’ve ordered a FSP modular CRPS PSU to remedy this; I’m likely going to have to make my own cables for it.

  • Another show stopper is the noise this system produces, I want it to be quiet enough for an office environment but the 80x38mm fans are screamers. Even after installation of an octo to prevent the 100% fan ramps during boot and control the fans better during steady state, I can’t get them quiet enough while cooling the backplane expander chips to comfortable levels.

    • I’ve ordered an array of 9RA fans to swap in to get the noise down. In my experience they are usually ~10dB quieter than “industry standard” fans at the same performance level.
      I’ve also ordered some 40mm fans to attach directly to the expander heatsinks (there are 32x32mm mounting provisions on them to do this). Finding 40mms without obnoxiously large motor hubs that restrict CFM is more difficult that I would have imaged.
3 Likes
2

Double column (2x12 drives) hardware raid 60 performance:

FIO high queue depth sequential read of 14.2GB/s 
C:\Users\Administrator>fio --name=seq_read --rw=read --bs=1024k --size=100G --iodepth=64 --direct=1 --numjobs=16 --filename=\\.\PhysicalDrive2
fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread' option to get rid of this warning.
seq_read: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=windowsaio, iodepth=64
...
fio-3.41
Starting 16 threads
Jobs: 16 (f=16): [R(16)][100.0%][r=13.3GiB/s][r=13.6k IOPS][eta 00m:00s]
seq_read: (groupid=0, jobs=1): err= 0: pid=10156: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120859msec)
    slat (usec): min=21, max=3758, avg=65.72, stdev=62.16
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.00
     lat (msec): min=67, max=209, avg=75.49, stdev= 3.04
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  113], 99.95th=[  174],
     | 99.99th=[  199]
   bw (  KiB/s): min=471040, max=878568, per=6.26%, avg=868373.93, stdev=26231.03, samples=238
   iops        : min=  460, max=  857, avg=847.50, stdev=25.58, samples=238
  lat (msec)   : 100=99.89%, 250=0.11%
  cpu          : usr=0.00%, sys=5.79%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=3444: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120857msec)
    slat (usec): min=31, max=2425, avg=62.81, stdev=49.36
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.05
     lat (msec): min=67, max=208, avg=75.49, stdev= 3.09
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  121], 99.95th=[  174],
     | 99.99th=[  199]
   bw (  KiB/s): min=470099, max=882404, per=6.26%, avg=868430.45, stdev=26465.73, samples=238
   iops        : min=  459, max=  861, avg=847.58, stdev=25.81, samples=238
  lat (msec)   : 100=99.88%, 250=0.12%
  cpu          : usr=0.00%, sys=4.96%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=6948: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120851msec)
    slat (usec): min=32, max=2195, avg=62.35, stdev=45.69
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.07
     lat (msec): min=67, max=209, avg=75.49, stdev= 3.11
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  114], 99.95th=[  176],
     | 99.99th=[  201]
   bw (  KiB/s): min=479232, max=880838, per=6.26%, avg=868465.64, stdev=25761.55, samples=238
   iops        : min=  468, max=  860, avg=847.60, stdev=25.12, samples=238
  lat (msec)   : 100=99.88%, 250=0.12%
  cpu          : usr=0.00%, sys=5.79%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=12296: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120880msec)
    slat (usec): min=21, max=7001, avg=212.81, stdev=170.52
    clat (usec): min=256, max=208855, avg=75274.26, stdev=4392.91
     lat (usec): min=383, max=210965, avg=75487.08, stdev=4428.74
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   77], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   78], 99.50th=[   79], 99.90th=[   85], 99.95th=[  176],
     | 99.99th=[  201]
   bw (  KiB/s): min=454656, max=885085, per=6.24%, avg=866190.98, stdev=27916.52, samples=238
   iops        : min=  444, max=  864, avg=845.35, stdev=27.22, samples=238
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.05%, 10=0.11%, 20=0.01%, 50=0.02%
  lat (msec)   : 100=99.69%, 250=0.08%
  cpu          : usr=0.00%, sys=5.79%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=1.1%, >=64=98.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.1%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=13288: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120855msec)
    slat (usec): min=33, max=2332, avg=63.76, stdev=50.00
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.04
     lat (msec): min=67, max=209, avg=75.49, stdev= 3.08
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  114], 99.95th=[  174],
     | 99.99th=[  201]
   bw (  KiB/s): min=471040, max=888561, per=6.26%, avg=868422.66, stdev=26512.55, samples=238
   iops        : min=  460, max=  867, avg=847.58, stdev=25.85, samples=238
  lat (msec)   : 100=99.89%, 250=0.11%
  cpu          : usr=0.00%, sys=5.79%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=2232: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120856msec)
    slat (usec): min=28, max=2371, avg=63.56, stdev=49.64
    clat (msec): min=67, max=206, avg=75.43, stdev= 3.05
     lat (msec): min=67, max=208, avg=75.49, stdev= 3.09
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  117], 99.95th=[  174],
     | 99.99th=[  199]
   bw (  KiB/s): min=470099, max=880352, per=6.26%, avg=868433.45, stdev=26270.63, samples=238
   iops        : min=  459, max=  859, avg=847.55, stdev=25.63, samples=238
  lat (msec)   : 100=99.88%, 250=0.12%
  cpu          : usr=0.00%, sys=4.96%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=12904: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120852msec)
    slat (usec): min=32, max=5940, avg=63.93, stdev=56.29
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.01
     lat (msec): min=67, max=209, avg=75.49, stdev= 3.06
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  112], 99.95th=[  174],
     | 99.99th=[  201]
   bw (  KiB/s): min=473088, max=892665, per=6.26%, avg=868438.32, stdev=26758.69, samples=238
   iops        : min=  462, max=  871, avg=847.55, stdev=26.10, samples=238
  lat (msec)   : 100=99.88%, 250=0.12%
  cpu          : usr=0.00%, sys=4.96%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=5448: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120859msec)
    slat (usec): min=30, max=4664, avg=67.42, stdev=64.54
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.02
     lat (msec): min=67, max=209, avg=75.49, stdev= 3.06
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  114], 99.95th=[  174],
     | 99.99th=[  199]
   bw (  KiB/s): min=473088, max=882688, per=6.26%, avg=868384.60, stdev=26092.65, samples=238
   iops        : min=  462, max=  862, avg=847.52, stdev=25.45, samples=238
  lat (msec)   : 100=99.88%, 250=0.12%
  cpu          : usr=0.00%, sys=5.79%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=6276: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120856msec)
    slat (usec): min=33, max=2532, avg=63.90, stdev=53.03
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.03
     lat (msec): min=67, max=209, avg=75.49, stdev= 3.07
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  120], 99.95th=[  174],
     | 99.99th=[  199]
   bw (  KiB/s): min=468992, max=882404, per=6.26%, avg=868396.92, stdev=26448.50, samples=238
   iops        : min=  458, max=  861, avg=847.53, stdev=25.79, samples=238
  lat (msec)   : 100=99.88%, 250=0.12%
  cpu          : usr=0.00%, sys=5.79%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=5548: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120856msec)
    slat (usec): min=30, max=2475, avg=63.68, stdev=50.96
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.06
     lat (msec): min=67, max=209, avg=75.49, stdev= 3.10
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  115], 99.95th=[  174],
     | 99.99th=[  201]
   bw (  KiB/s): min=470099, max=882404, per=6.26%, avg=868461.95, stdev=26302.18, samples=238
   iops        : min=  459, max=  861, avg=847.60, stdev=25.66, samples=238
  lat (msec)   : 100=99.88%, 250=0.12%
  cpu          : usr=0.00%, sys=4.96%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=8176: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120860msec)
    slat (usec): min=27, max=4632, avg=70.01, stdev=77.26
    clat (msec): min=58, max=207, avg=75.42, stdev= 2.94
     lat (msec): min=59, max=209, avg=75.49, stdev= 2.99
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  101], 99.95th=[  176],
     | 99.99th=[  201]
   bw (  KiB/s): min=467879, max=884456, per=6.26%, avg=868290.44, stdev=26897.19, samples=238
   iops        : min=  456, max=  863, avg=847.40, stdev=26.29, samples=238
  lat (msec)   : 100=99.90%, 250=0.10%
  cpu          : usr=0.00%, sys=5.79%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=5680: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120853msec)
    slat (usec): min=30, max=2442, avg=62.69, stdev=49.87
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.03
     lat (msec): min=67, max=209, avg=75.49, stdev= 3.07
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  114], 99.95th=[  174],
     | 99.99th=[  199]
   bw (  KiB/s): min=469931, max=884544, per=6.26%, avg=868441.67, stdev=26365.51, samples=238
   iops        : min=  458, max=  863, avg=847.56, stdev=25.77, samples=238
  lat (msec)   : 100=99.89%, 250=0.11%
  cpu          : usr=0.00%, sys=4.96%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=10504: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (889MB/s)(100GiB/120843msec)
    slat (usec): min=22, max=2103, avg=62.90, stdev=42.27
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.08
     lat (msec): min=67, max=209, avg=75.49, stdev= 3.12
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  110], 99.95th=[  180],
     | 99.99th=[  203]
   bw (  KiB/s): min=489472, max=891145, per=6.26%, avg=868555.85, stdev=26196.15, samples=238
   iops        : min=  478, max=  870, avg=847.67, stdev=25.54, samples=238
  lat (msec)   : 100=99.88%, 250=0.12%
  cpu          : usr=0.00%, sys=4.97%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=3404: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120859msec)
    slat (usec): min=25, max=3122, avg=66.40, stdev=62.78
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.02
     lat (msec): min=67, max=209, avg=75.49, stdev= 3.06
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  114], 99.95th=[  174],
     | 99.99th=[  201]
   bw (  KiB/s): min=475136, max=884456, per=6.26%, avg=868398.95, stdev=26083.32, samples=238
   iops        : min=  464, max=  863, avg=847.53, stdev=25.43, samples=238
  lat (msec)   : 100=99.88%, 250=0.12%
  cpu          : usr=0.00%, sys=5.79%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=8908: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120857msec)
    slat (usec): min=25, max=2749, avg=63.36, stdev=52.39
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.03
     lat (msec): min=67, max=209, avg=75.49, stdev= 3.07
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  114], 99.95th=[  176],
     | 99.99th=[  199]
   bw (  KiB/s): min=465827, max=886509, per=6.26%, avg=868386.13, stdev=26717.73, samples=238
   iops        : min=  454, max=  865, avg=847.50, stdev=26.11, samples=238
  lat (msec)   : 100=99.89%, 250=0.11%
  cpu          : usr=0.00%, sys=4.96%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=2276: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120854msec)
    slat (usec): min=24, max=2372, avg=62.54, stdev=50.50
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.03
     lat (msec): min=67, max=209, avg=75.49, stdev= 3.08
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  118], 99.95th=[  176],
     | 99.99th=[  199]
   bw (  KiB/s): min=469931, max=884456, per=6.26%, avg=868438.47, stdev=26531.32, samples=238
   iops        : min=  458, max=  863, avg=847.57, stdev=25.93, samples=238
  lat (msec)   : 100=99.89%, 250=0.11%
  cpu          : usr=0.00%, sys=5.79%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=13.2GiB/s (14.2GB/s), 847MiB/s-847MiB/s (888MB/s-889MB/s), io=1600GiB (1718GB), run=120843-120880msec
FIO high queue depth sequential write of 7.7GB/s 
C:\Users\Administrator>fio --name=seq_write --rw=write --bs=1024k --size=100G --iodepth=64 --direct=1 --numjobs=16 --filename=\\.\PhysicalDrive2
fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread' option to get rid of this warning.
seq_write: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=windowsaio, iodepth=64
...
fio-3.41
Starting 16 threads
Jobs: 16 (f=16): [W(16)][100.0%][w=7290MiB/s][w=7289 IOPS][eta 00m:00s]
seq_write: (groupid=0, jobs=1): err= 0: pid=14032: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221783msec); 0 zone resets
    slat (usec): min=99, max=1446, avg=177.51, stdev=40.13
    clat (msec): min=20, max=272, avg=138.43, stdev= 3.08
     lat (msec): min=21, max=272, avg=138.60, stdev= 3.08
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  161],
     | 99.99th=[  255]
   bw (  KiB/s): min=333348, max=513026, per=6.25%, avg=472955.37, stdev=9163.33, samples=438
   iops        : min=  325, max=  501, avg=461.16, stdev= 8.95, samples=438
  lat (msec)   : 50=0.03%, 100=0.03%, 250=99.93%, 500=0.01%
  cpu          : usr=4.51%, sys=3.61%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=10304: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221787msec); 0 zone resets
    slat (usec): min=97, max=1301, avg=178.69, stdev=39.60
    clat (msec): min=21, max=269, avg=138.43, stdev= 3.02
     lat (msec): min=21, max=269, avg=138.61, stdev= 3.01
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  161],
     | 99.99th=[  247]
   bw (  KiB/s): min=333348, max=503968, per=6.25%, avg=472959.15, stdev=8494.24, samples=438
   iops        : min=  325, max=  492, avg=461.16, stdev= 8.31, samples=438
  lat (msec)   : 50=0.03%, 100=0.03%, 250=99.93%, 500=0.01%
  cpu          : usr=4.51%, sys=4.06%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=7828: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221813msec); 0 zone resets
    slat (usec): min=90, max=23174, avg=303.13, stdev=150.53
    clat (msec): min=4, max=161, avg=138.31, stdev= 3.39
     lat (msec): min=4, max=161, avg=138.61, stdev= 3.39
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  140],
     | 99.99th=[  150]
   bw (  KiB/s): min=413615, max=544935, per=6.25%, avg=472951.07, stdev=27665.50, samples=438
   iops        : min=  403, max=  532, avg=461.21, stdev=27.03, samples=438
  lat (msec)   : 10=0.02%, 20=0.01%, 50=0.02%, 100=0.05%, 250=99.89%
  cpu          : usr=3.16%, sys=5.41%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.5%, >=64=99.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.1%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=5508: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221797msec); 0 zone resets
    slat (usec): min=100, max=1440, avg=184.93, stdev=72.21
    clat (msec): min=20, max=267, avg=138.43, stdev= 3.08
     lat (msec): min=21, max=268, avg=138.61, stdev= 3.08
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  159],
     | 99.99th=[  245]
   bw (  KiB/s): min=337341, max=506869, per=6.25%, avg=472953.59, stdev=8014.86, samples=438
   iops        : min=  329, max=  494, avg=461.16, stdev= 7.85, samples=438
  lat (msec)   : 50=0.03%, 100=0.03%, 250=99.93%, 500=0.01%
  cpu          : usr=3.61%, sys=4.51%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=11044: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221799msec); 0 zone resets
    slat (usec): min=101, max=1781, avg=175.46, stdev=38.79
    clat (msec): min=25, max=262, avg=138.44, stdev= 2.90
     lat (msec): min=25, max=262, avg=138.61, stdev= 2.90
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  153],
     | 99.99th=[  249]
   bw (  KiB/s): min=337341, max=503937, per=6.25%, avg=472969.04, stdev=10504.57, samples=438
   iops        : min=  329, max=  492, avg=461.18, stdev=10.27, samples=438
  lat (msec)   : 50=0.02%, 100=0.03%, 250=99.94%, 500=0.01%
  cpu          : usr=4.51%, sys=4.06%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=6388: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221794msec); 0 zone resets
    slat (usec): min=96, max=1671, avg=174.52, stdev=42.44
    clat (msec): min=24, max=264, avg=138.44, stdev= 2.77
     lat (msec): min=24, max=264, avg=138.61, stdev= 2.76
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  148],
     | 99.99th=[  241]
   bw (  KiB/s): min=347321, max=510973, per=6.25%, avg=472964.32, stdev=7839.57, samples=438
   iops        : min=  339, max=  498, avg=461.17, stdev= 7.66, samples=438
  lat (msec)   : 50=0.02%, 100=0.03%, 250=99.94%, 500=0.01%
  cpu          : usr=4.51%, sys=4.06%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=6284: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221794msec); 0 zone resets
    slat (usec): min=98, max=1535, avg=183.91, stdev=52.85
    clat (msec): min=22, max=268, avg=138.43, stdev= 3.08
     lat (msec): min=22, max=268, avg=138.61, stdev= 3.08
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  157],
     | 99.99th=[  247]
   bw (  KiB/s): min=331352, max=503808, per=6.25%, avg=472964.82, stdev=9816.90, samples=438
   iops        : min=  323, max=  492, avg=461.16, stdev= 9.59, samples=438
  lat (msec)   : 50=0.03%, 100=0.03%, 250=99.93%, 500=0.01%
  cpu          : usr=4.06%, sys=4.51%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=7380: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221794msec); 0 zone resets
    slat (usec): min=99, max=1912, avg=176.85, stdev=42.47
    clat (msec): min=40, max=236, avg=138.43, stdev= 2.19
     lat (msec): min=41, max=236, avg=138.61, stdev= 2.18
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  140],
     | 99.99th=[  215]
   bw (  KiB/s): min=375267, max=505856, per=6.25%, avg=472980.18, stdev=9491.25, samples=438
   iops        : min=  366, max=  494, avg=461.19, stdev= 9.25, samples=438
  lat (msec)   : 50=0.01%, 100=0.03%, 250=99.96%
  cpu          : usr=4.06%, sys=4.06%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=12184: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221790msec); 0 zone resets
    slat (usec): min=100, max=1394, avg=177.86, stdev=42.20
    clat (msec): min=21, max=269, avg=138.43, stdev= 3.02
     lat (msec): min=22, max=269, avg=138.61, stdev= 3.01
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  161],
     | 99.99th=[  247]
   bw (  KiB/s): min=333348, max=510973, per=6.25%, avg=472965.50, stdev=8213.60, samples=438
   iops        : min=  325, max=  498, avg=461.17, stdev= 8.04, samples=438
  lat (msec)   : 50=0.03%, 100=0.03%, 250=99.93%, 500=0.01%
  cpu          : usr=4.06%, sys=4.51%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=3772: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221793msec); 0 zone resets
    slat (usec): min=96, max=1541, avg=189.87, stdev=74.73
    clat (msec): min=19, max=270, avg=138.42, stdev= 3.07
     lat (msec): min=20, max=270, avg=138.61, stdev= 3.06
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  157],
     | 99.99th=[  241]
   bw (  KiB/s): min=341333, max=507887, per=6.25%, avg=472956.44, stdev=8538.71, samples=438
   iops        : min=  333, max=  495, avg=461.14, stdev= 8.33, samples=438
  lat (msec)   : 20=0.01%, 50=0.03%, 100=0.03%, 250=99.93%, 500=0.01%
  cpu          : usr=3.61%, sys=4.51%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=8016: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221791msec); 0 zone resets
    slat (usec): min=96, max=1528, avg=175.33, stdev=39.52
    clat (msec): min=22, max=270, avg=138.43, stdev= 3.00
     lat (msec): min=22, max=271, avg=138.61, stdev= 3.00
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  157],
     | 99.99th=[  249]
   bw (  KiB/s): min=337341, max=508921, per=6.25%, avg=472967.45, stdev=8150.29, samples=438
   iops        : min=  329, max=  496, avg=461.16, stdev= 7.97, samples=438
  lat (msec)   : 50=0.03%, 100=0.03%, 250=99.93%, 500=0.01%
  cpu          : usr=4.06%, sys=4.06%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=4952: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221795msec); 0 zone resets
    slat (usec): min=93, max=1873, avg=173.99, stdev=39.40
    clat (msec): min=23, max=264, avg=138.44, stdev= 2.97
     lat (msec): min=24, max=265, avg=138.61, stdev= 2.97
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  159],
     | 99.99th=[  249]
   bw (  KiB/s): min=337341, max=503968, per=6.25%, avg=472971.12, stdev=8466.73, samples=438
   iops        : min=  329, max=  492, avg=461.18, stdev= 8.30, samples=438
  lat (msec)   : 50=0.02%, 100=0.03%, 250=99.93%, 500=0.01%
  cpu          : usr=4.06%, sys=4.06%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=12060: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221798msec); 0 zone resets
    slat (usec): min=100, max=1433, avg=184.05, stdev=54.94
    clat (msec): min=23, max=262, avg=138.43, stdev= 2.87
     lat (msec): min=24, max=262, avg=138.61, stdev= 2.87
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  150],
     | 99.99th=[  245]
   bw (  KiB/s): min=345325, max=502765, per=6.25%, avg=472969.14, stdev=8481.47, samples=438
   iops        : min=  337, max=  490, avg=461.18, stdev= 8.29, samples=438
  lat (msec)   : 50=0.02%, 100=0.03%, 250=99.94%, 500=0.01%
  cpu          : usr=4.06%, sys=4.06%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=8740: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221782msec); 0 zone resets
    slat (usec): min=98, max=1249, avg=175.18, stdev=41.42
    clat (msec): min=20, max=270, avg=138.43, stdev= 3.13
     lat (msec): min=20, max=270, avg=138.60, stdev= 3.12
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  163],
     | 99.99th=[  249]
   bw (  KiB/s): min=327360, max=506869, per=6.25%, avg=472957.63, stdev=8830.83, samples=438
   iops        : min=  319, max=  494, avg=461.15, stdev= 8.65, samples=438
  lat (msec)   : 50=0.03%, 100=0.03%, 250=99.93%, 500=0.01%
  cpu          : usr=4.51%, sys=3.61%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=13744: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221794msec); 0 zone resets
    slat (usec): min=95, max=1541, avg=176.51, stdev=40.54
    clat (msec): min=39, max=237, avg=138.43, stdev= 2.22
     lat (msec): min=40, max=237, avg=138.61, stdev= 2.21
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  142],
     | 99.99th=[  218]
   bw (  KiB/s): min=375267, max=505856, per=6.25%, avg=472974.21, stdev=9013.95, samples=438
   iops        : min=  366, max=  494, avg=461.16, stdev= 8.80, samples=438
  lat (msec)   : 50=0.01%, 100=0.03%, 250=99.96%
  cpu          : usr=4.51%, sys=4.06%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=12840: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221795msec); 0 zone resets
    slat (usec): min=92, max=2811, avg=178.30, stdev=45.55
    clat (msec): min=22, max=270, avg=138.43, stdev= 3.03
     lat (msec): min=23, max=271, avg=138.61, stdev= 3.03
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  159],
     | 99.99th=[  251]
   bw (  KiB/s): min=335345, max=501960, per=6.25%, avg=472963.67, stdev=8683.66, samples=438
   iops        : min=  327, max=  490, avg=461.17, stdev= 8.50, samples=438
  lat (msec)   : 50=0.03%, 100=0.03%, 250=99.93%, 500=0.01%
  cpu          : usr=4.51%, sys=4.06%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=7386MiB/s (7745MB/s), 462MiB/s-462MiB/s (484MB/s-484MB/s), io=1600GiB (1718GB), run=221782-221813msec

These numbers were done on a 1.6TiB test depth to rule out any caching polluting the performance figures.

What this has made me realize is that the old rules-of-thumb formulas for raid speed calculations don’t work anymore.

I was expecting (N-2)(single drive speed) for the raid 6 read speed which should have come out to 10.5GB/s given Seagate’s 524MB/s speed rating for the drive.
and N(single drive speed)/6 for the raid 6 write speed which should have come out to 2.1GB/s.