Octoquadragintuple Actuator Hardware Accelerated Raid!

Hardware Hub Build Logs
1

After many many years of owning no dedicated backup server I’ve finally cobbled together an eclectic mix of parts to make one. This will be a build log to try and make a quiet, fast and cheap backup server.

For the case I ended up coming across a 4U Chenbro tri-fold chassis for a good price. It’s got 12-drive expander backplanes on the front, left and right side for a total of 36 bays:

Untitled
Untitled750×800 223 KB

For drives I’m going with two dozen 14TB mach.2s I picked up for 8 mills a gigabyte that I bought on FOMO impulses.

​​​ ​ ​

​​​ ​ ​

The centerpiece of this build is going to be the raid card.
I decided on Areca due to the amount of control it gives over very low level functions of storage that I haven’t seen anywhere else, as well as the out of band management it has for peace of mind; the last thing I want to encounter one morning is a no boot situation and having to worry about the integrity of my storage.

ARC-1886-16i:

image
image700×700 73.3 KB

The rest of the system is using spare parts I had lying around:
Asus W790 Sage motherboard
Intel w5-3435x CPU
2x 24GB vcolor 5600MT RDIMM
Nvidia a2000 12GB GPU
Silverstone XE02-4677 CPU cooler
Acbel 1200W CRPS PSU

​​​ ​ ​

​​​ ​ ​
I put the system together to get an idea of what I was dealing with:

image
image1920×1280 327 KB

  • The first problem is the CRPS power supply isn’t modular and is making a rat’s nest of cables and blocking airflow.

    • I’ve ordered a FSP modular CRPS PSU to remedy this; I’m likely going to have to make my own cables for it.

  • Another show stopper is the noise this system produces, I want it to be quiet enough for an office environment but the 80x38mm fans are screamers. Even after installation of an octo to prevent the 100% fan ramps during boot and control the fans better during steady state, I can’t get them quiet enough while cooling the backplane expander chips to comfortable levels.

    • I’ve ordered an array of 9RA fans to swap in to get the noise down. In my experience they are usually ~10dB quieter than “industry standard” fans at the same performance level.
      I’ve also ordered some 40mm fans to attach directly to the expander heatsinks (there are 32x32mm mounting provisions on them to do this). Finding 40mms without obnoxiously large motor hubs that restrict CFM is more difficult that I would have imaged.
3 Likes
2

Double column (2x12 drives) hardware raid 60 performance:

FIO high queue depth sequential read of 14.2GB/s 
C:\Users\Administrator>fio --name=seq_read --rw=read --bs=1024k --size=100G --iodepth=64 --direct=1 --numjobs=16 --filename=\\.\PhysicalDrive2
fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread' option to get rid of this warning.
seq_read: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=windowsaio, iodepth=64
...
fio-3.41
Starting 16 threads
Jobs: 16 (f=16): [R(16)][100.0%][r=13.3GiB/s][r=13.6k IOPS][eta 00m:00s]
seq_read: (groupid=0, jobs=1): err= 0: pid=10156: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120859msec)
    slat (usec): min=21, max=3758, avg=65.72, stdev=62.16
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.00
     lat (msec): min=67, max=209, avg=75.49, stdev= 3.04
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  113], 99.95th=[  174],
     | 99.99th=[  199]
   bw (  KiB/s): min=471040, max=878568, per=6.26%, avg=868373.93, stdev=26231.03, samples=238
   iops        : min=  460, max=  857, avg=847.50, stdev=25.58, samples=238
  lat (msec)   : 100=99.89%, 250=0.11%
  cpu          : usr=0.00%, sys=5.79%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=3444: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120857msec)
    slat (usec): min=31, max=2425, avg=62.81, stdev=49.36
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.05
     lat (msec): min=67, max=208, avg=75.49, stdev= 3.09
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  121], 99.95th=[  174],
     | 99.99th=[  199]
   bw (  KiB/s): min=470099, max=882404, per=6.26%, avg=868430.45, stdev=26465.73, samples=238
   iops        : min=  459, max=  861, avg=847.58, stdev=25.81, samples=238
  lat (msec)   : 100=99.88%, 250=0.12%
  cpu          : usr=0.00%, sys=4.96%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=6948: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120851msec)
    slat (usec): min=32, max=2195, avg=62.35, stdev=45.69
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.07
     lat (msec): min=67, max=209, avg=75.49, stdev= 3.11
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  114], 99.95th=[  176],
     | 99.99th=[  201]
   bw (  KiB/s): min=479232, max=880838, per=6.26%, avg=868465.64, stdev=25761.55, samples=238
   iops        : min=  468, max=  860, avg=847.60, stdev=25.12, samples=238
  lat (msec)   : 100=99.88%, 250=0.12%
  cpu          : usr=0.00%, sys=5.79%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=12296: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120880msec)
    slat (usec): min=21, max=7001, avg=212.81, stdev=170.52
    clat (usec): min=256, max=208855, avg=75274.26, stdev=4392.91
     lat (usec): min=383, max=210965, avg=75487.08, stdev=4428.74
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   77], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   78], 99.50th=[   79], 99.90th=[   85], 99.95th=[  176],
     | 99.99th=[  201]
   bw (  KiB/s): min=454656, max=885085, per=6.24%, avg=866190.98, stdev=27916.52, samples=238
   iops        : min=  444, max=  864, avg=845.35, stdev=27.22, samples=238
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.05%, 10=0.11%, 20=0.01%, 50=0.02%
  lat (msec)   : 100=99.69%, 250=0.08%
  cpu          : usr=0.00%, sys=5.79%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=1.1%, >=64=98.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.1%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=13288: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120855msec)
    slat (usec): min=33, max=2332, avg=63.76, stdev=50.00
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.04
     lat (msec): min=67, max=209, avg=75.49, stdev= 3.08
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  114], 99.95th=[  174],
     | 99.99th=[  201]
   bw (  KiB/s): min=471040, max=888561, per=6.26%, avg=868422.66, stdev=26512.55, samples=238
   iops        : min=  460, max=  867, avg=847.58, stdev=25.85, samples=238
  lat (msec)   : 100=99.89%, 250=0.11%
  cpu          : usr=0.00%, sys=5.79%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=2232: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120856msec)
    slat (usec): min=28, max=2371, avg=63.56, stdev=49.64
    clat (msec): min=67, max=206, avg=75.43, stdev= 3.05
     lat (msec): min=67, max=208, avg=75.49, stdev= 3.09
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  117], 99.95th=[  174],
     | 99.99th=[  199]
   bw (  KiB/s): min=470099, max=880352, per=6.26%, avg=868433.45, stdev=26270.63, samples=238
   iops        : min=  459, max=  859, avg=847.55, stdev=25.63, samples=238
  lat (msec)   : 100=99.88%, 250=0.12%
  cpu          : usr=0.00%, sys=4.96%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=12904: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120852msec)
    slat (usec): min=32, max=5940, avg=63.93, stdev=56.29
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.01
     lat (msec): min=67, max=209, avg=75.49, stdev= 3.06
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  112], 99.95th=[  174],
     | 99.99th=[  201]
   bw (  KiB/s): min=473088, max=892665, per=6.26%, avg=868438.32, stdev=26758.69, samples=238
   iops        : min=  462, max=  871, avg=847.55, stdev=26.10, samples=238
  lat (msec)   : 100=99.88%, 250=0.12%
  cpu          : usr=0.00%, sys=4.96%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=5448: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120859msec)
    slat (usec): min=30, max=4664, avg=67.42, stdev=64.54
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.02
     lat (msec): min=67, max=209, avg=75.49, stdev= 3.06
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  114], 99.95th=[  174],
     | 99.99th=[  199]
   bw (  KiB/s): min=473088, max=882688, per=6.26%, avg=868384.60, stdev=26092.65, samples=238
   iops        : min=  462, max=  862, avg=847.52, stdev=25.45, samples=238
  lat (msec)   : 100=99.88%, 250=0.12%
  cpu          : usr=0.00%, sys=5.79%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=6276: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120856msec)
    slat (usec): min=33, max=2532, avg=63.90, stdev=53.03
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.03
     lat (msec): min=67, max=209, avg=75.49, stdev= 3.07
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  120], 99.95th=[  174],
     | 99.99th=[  199]
   bw (  KiB/s): min=468992, max=882404, per=6.26%, avg=868396.92, stdev=26448.50, samples=238
   iops        : min=  458, max=  861, avg=847.53, stdev=25.79, samples=238
  lat (msec)   : 100=99.88%, 250=0.12%
  cpu          : usr=0.00%, sys=5.79%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=5548: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120856msec)
    slat (usec): min=30, max=2475, avg=63.68, stdev=50.96
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.06
     lat (msec): min=67, max=209, avg=75.49, stdev= 3.10
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  115], 99.95th=[  174],
     | 99.99th=[  201]
   bw (  KiB/s): min=470099, max=882404, per=6.26%, avg=868461.95, stdev=26302.18, samples=238
   iops        : min=  459, max=  861, avg=847.60, stdev=25.66, samples=238
  lat (msec)   : 100=99.88%, 250=0.12%
  cpu          : usr=0.00%, sys=4.96%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=8176: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120860msec)
    slat (usec): min=27, max=4632, avg=70.01, stdev=77.26
    clat (msec): min=58, max=207, avg=75.42, stdev= 2.94
     lat (msec): min=59, max=209, avg=75.49, stdev= 2.99
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  101], 99.95th=[  176],
     | 99.99th=[  201]
   bw (  KiB/s): min=467879, max=884456, per=6.26%, avg=868290.44, stdev=26897.19, samples=238
   iops        : min=  456, max=  863, avg=847.40, stdev=26.29, samples=238
  lat (msec)   : 100=99.90%, 250=0.10%
  cpu          : usr=0.00%, sys=5.79%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=5680: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120853msec)
    slat (usec): min=30, max=2442, avg=62.69, stdev=49.87
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.03
     lat (msec): min=67, max=209, avg=75.49, stdev= 3.07
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  114], 99.95th=[  174],
     | 99.99th=[  199]
   bw (  KiB/s): min=469931, max=884544, per=6.26%, avg=868441.67, stdev=26365.51, samples=238
   iops        : min=  458, max=  863, avg=847.56, stdev=25.77, samples=238
  lat (msec)   : 100=99.89%, 250=0.11%
  cpu          : usr=0.00%, sys=4.96%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=10504: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (889MB/s)(100GiB/120843msec)
    slat (usec): min=22, max=2103, avg=62.90, stdev=42.27
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.08
     lat (msec): min=67, max=209, avg=75.49, stdev= 3.12
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  110], 99.95th=[  180],
     | 99.99th=[  203]
   bw (  KiB/s): min=489472, max=891145, per=6.26%, avg=868555.85, stdev=26196.15, samples=238
   iops        : min=  478, max=  870, avg=847.67, stdev=25.54, samples=238
  lat (msec)   : 100=99.88%, 250=0.12%
  cpu          : usr=0.00%, sys=4.97%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=3404: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120859msec)
    slat (usec): min=25, max=3122, avg=66.40, stdev=62.78
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.02
     lat (msec): min=67, max=209, avg=75.49, stdev= 3.06
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  114], 99.95th=[  174],
     | 99.99th=[  201]
   bw (  KiB/s): min=475136, max=884456, per=6.26%, avg=868398.95, stdev=26083.32, samples=238
   iops        : min=  464, max=  863, avg=847.53, stdev=25.43, samples=238
  lat (msec)   : 100=99.88%, 250=0.12%
  cpu          : usr=0.00%, sys=5.79%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=8908: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120857msec)
    slat (usec): min=25, max=2749, avg=63.36, stdev=52.39
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.03
     lat (msec): min=67, max=209, avg=75.49, stdev= 3.07
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  114], 99.95th=[  176],
     | 99.99th=[  199]
   bw (  KiB/s): min=465827, max=886509, per=6.26%, avg=868386.13, stdev=26717.73, samples=238
   iops        : min=  454, max=  865, avg=847.50, stdev=26.11, samples=238
  lat (msec)   : 100=99.89%, 250=0.11%
  cpu          : usr=0.00%, sys=4.96%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_read: (groupid=0, jobs=1): err= 0: pid=2276: Mon Dec 29 10:06:18 2025
  read: IOPS=847, BW=847MiB/s (888MB/s)(100GiB/120854msec)
    slat (usec): min=24, max=2372, avg=62.54, stdev=50.50
    clat (msec): min=67, max=207, avg=75.43, stdev= 3.03
     lat (msec): min=67, max=209, avg=75.49, stdev= 3.08
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   75], 10.00th=[   75], 20.00th=[   75],
     | 30.00th=[   75], 40.00th=[   75], 50.00th=[   75], 60.00th=[   75],
     | 70.00th=[   75], 80.00th=[   75], 90.00th=[   77], 95.00th=[   77],
     | 99.00th=[   77], 99.50th=[   79], 99.90th=[  118], 99.95th=[  176],
     | 99.99th=[  199]
   bw (  KiB/s): min=469931, max=884456, per=6.26%, avg=868438.47, stdev=26531.32, samples=238
   iops        : min=  458, max=  863, avg=847.57, stdev=25.93, samples=238
  lat (msec)   : 100=99.89%, 250=0.11%
  cpu          : usr=0.00%, sys=5.79%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=102400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=13.2GiB/s (14.2GB/s), 847MiB/s-847MiB/s (888MB/s-889MB/s), io=1600GiB (1718GB), run=120843-120880msec
FIO high queue depth sequential write of 7.7GB/s 
C:\Users\Administrator>fio --name=seq_write --rw=write --bs=1024k --size=100G --iodepth=64 --direct=1 --numjobs=16 --filename=\\.\PhysicalDrive2
fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread' option to get rid of this warning.
seq_write: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=windowsaio, iodepth=64
...
fio-3.41
Starting 16 threads
Jobs: 16 (f=16): [W(16)][100.0%][w=7290MiB/s][w=7289 IOPS][eta 00m:00s]
seq_write: (groupid=0, jobs=1): err= 0: pid=14032: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221783msec); 0 zone resets
    slat (usec): min=99, max=1446, avg=177.51, stdev=40.13
    clat (msec): min=20, max=272, avg=138.43, stdev= 3.08
     lat (msec): min=21, max=272, avg=138.60, stdev= 3.08
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  161],
     | 99.99th=[  255]
   bw (  KiB/s): min=333348, max=513026, per=6.25%, avg=472955.37, stdev=9163.33, samples=438
   iops        : min=  325, max=  501, avg=461.16, stdev= 8.95, samples=438
  lat (msec)   : 50=0.03%, 100=0.03%, 250=99.93%, 500=0.01%
  cpu          : usr=4.51%, sys=3.61%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=10304: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221787msec); 0 zone resets
    slat (usec): min=97, max=1301, avg=178.69, stdev=39.60
    clat (msec): min=21, max=269, avg=138.43, stdev= 3.02
     lat (msec): min=21, max=269, avg=138.61, stdev= 3.01
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  161],
     | 99.99th=[  247]
   bw (  KiB/s): min=333348, max=503968, per=6.25%, avg=472959.15, stdev=8494.24, samples=438
   iops        : min=  325, max=  492, avg=461.16, stdev= 8.31, samples=438
  lat (msec)   : 50=0.03%, 100=0.03%, 250=99.93%, 500=0.01%
  cpu          : usr=4.51%, sys=4.06%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=7828: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221813msec); 0 zone resets
    slat (usec): min=90, max=23174, avg=303.13, stdev=150.53
    clat (msec): min=4, max=161, avg=138.31, stdev= 3.39
     lat (msec): min=4, max=161, avg=138.61, stdev= 3.39
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  140],
     | 99.99th=[  150]
   bw (  KiB/s): min=413615, max=544935, per=6.25%, avg=472951.07, stdev=27665.50, samples=438
   iops        : min=  403, max=  532, avg=461.21, stdev=27.03, samples=438
  lat (msec)   : 10=0.02%, 20=0.01%, 50=0.02%, 100=0.05%, 250=99.89%
  cpu          : usr=3.16%, sys=5.41%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.5%, >=64=99.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.1%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=5508: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221797msec); 0 zone resets
    slat (usec): min=100, max=1440, avg=184.93, stdev=72.21
    clat (msec): min=20, max=267, avg=138.43, stdev= 3.08
     lat (msec): min=21, max=268, avg=138.61, stdev= 3.08
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  159],
     | 99.99th=[  245]
   bw (  KiB/s): min=337341, max=506869, per=6.25%, avg=472953.59, stdev=8014.86, samples=438
   iops        : min=  329, max=  494, avg=461.16, stdev= 7.85, samples=438
  lat (msec)   : 50=0.03%, 100=0.03%, 250=99.93%, 500=0.01%
  cpu          : usr=3.61%, sys=4.51%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=11044: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221799msec); 0 zone resets
    slat (usec): min=101, max=1781, avg=175.46, stdev=38.79
    clat (msec): min=25, max=262, avg=138.44, stdev= 2.90
     lat (msec): min=25, max=262, avg=138.61, stdev= 2.90
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  153],
     | 99.99th=[  249]
   bw (  KiB/s): min=337341, max=503937, per=6.25%, avg=472969.04, stdev=10504.57, samples=438
   iops        : min=  329, max=  492, avg=461.18, stdev=10.27, samples=438
  lat (msec)   : 50=0.02%, 100=0.03%, 250=99.94%, 500=0.01%
  cpu          : usr=4.51%, sys=4.06%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=6388: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221794msec); 0 zone resets
    slat (usec): min=96, max=1671, avg=174.52, stdev=42.44
    clat (msec): min=24, max=264, avg=138.44, stdev= 2.77
     lat (msec): min=24, max=264, avg=138.61, stdev= 2.76
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  148],
     | 99.99th=[  241]
   bw (  KiB/s): min=347321, max=510973, per=6.25%, avg=472964.32, stdev=7839.57, samples=438
   iops        : min=  339, max=  498, avg=461.17, stdev= 7.66, samples=438
  lat (msec)   : 50=0.02%, 100=0.03%, 250=99.94%, 500=0.01%
  cpu          : usr=4.51%, sys=4.06%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=6284: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221794msec); 0 zone resets
    slat (usec): min=98, max=1535, avg=183.91, stdev=52.85
    clat (msec): min=22, max=268, avg=138.43, stdev= 3.08
     lat (msec): min=22, max=268, avg=138.61, stdev= 3.08
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  157],
     | 99.99th=[  247]
   bw (  KiB/s): min=331352, max=503808, per=6.25%, avg=472964.82, stdev=9816.90, samples=438
   iops        : min=  323, max=  492, avg=461.16, stdev= 9.59, samples=438
  lat (msec)   : 50=0.03%, 100=0.03%, 250=99.93%, 500=0.01%
  cpu          : usr=4.06%, sys=4.51%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=7380: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221794msec); 0 zone resets
    slat (usec): min=99, max=1912, avg=176.85, stdev=42.47
    clat (msec): min=40, max=236, avg=138.43, stdev= 2.19
     lat (msec): min=41, max=236, avg=138.61, stdev= 2.18
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  140],
     | 99.99th=[  215]
   bw (  KiB/s): min=375267, max=505856, per=6.25%, avg=472980.18, stdev=9491.25, samples=438
   iops        : min=  366, max=  494, avg=461.19, stdev= 9.25, samples=438
  lat (msec)   : 50=0.01%, 100=0.03%, 250=99.96%
  cpu          : usr=4.06%, sys=4.06%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=12184: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221790msec); 0 zone resets
    slat (usec): min=100, max=1394, avg=177.86, stdev=42.20
    clat (msec): min=21, max=269, avg=138.43, stdev= 3.02
     lat (msec): min=22, max=269, avg=138.61, stdev= 3.01
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  161],
     | 99.99th=[  247]
   bw (  KiB/s): min=333348, max=510973, per=6.25%, avg=472965.50, stdev=8213.60, samples=438
   iops        : min=  325, max=  498, avg=461.17, stdev= 8.04, samples=438
  lat (msec)   : 50=0.03%, 100=0.03%, 250=99.93%, 500=0.01%
  cpu          : usr=4.06%, sys=4.51%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=3772: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221793msec); 0 zone resets
    slat (usec): min=96, max=1541, avg=189.87, stdev=74.73
    clat (msec): min=19, max=270, avg=138.42, stdev= 3.07
     lat (msec): min=20, max=270, avg=138.61, stdev= 3.06
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  157],
     | 99.99th=[  241]
   bw (  KiB/s): min=341333, max=507887, per=6.25%, avg=472956.44, stdev=8538.71, samples=438
   iops        : min=  333, max=  495, avg=461.14, stdev= 8.33, samples=438
  lat (msec)   : 20=0.01%, 50=0.03%, 100=0.03%, 250=99.93%, 500=0.01%
  cpu          : usr=3.61%, sys=4.51%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=8016: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221791msec); 0 zone resets
    slat (usec): min=96, max=1528, avg=175.33, stdev=39.52
    clat (msec): min=22, max=270, avg=138.43, stdev= 3.00
     lat (msec): min=22, max=271, avg=138.61, stdev= 3.00
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  157],
     | 99.99th=[  249]
   bw (  KiB/s): min=337341, max=508921, per=6.25%, avg=472967.45, stdev=8150.29, samples=438
   iops        : min=  329, max=  496, avg=461.16, stdev= 7.97, samples=438
  lat (msec)   : 50=0.03%, 100=0.03%, 250=99.93%, 500=0.01%
  cpu          : usr=4.06%, sys=4.06%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=4952: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221795msec); 0 zone resets
    slat (usec): min=93, max=1873, avg=173.99, stdev=39.40
    clat (msec): min=23, max=264, avg=138.44, stdev= 2.97
     lat (msec): min=24, max=265, avg=138.61, stdev= 2.97
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  159],
     | 99.99th=[  249]
   bw (  KiB/s): min=337341, max=503968, per=6.25%, avg=472971.12, stdev=8466.73, samples=438
   iops        : min=  329, max=  492, avg=461.18, stdev= 8.30, samples=438
  lat (msec)   : 50=0.02%, 100=0.03%, 250=99.93%, 500=0.01%
  cpu          : usr=4.06%, sys=4.06%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=12060: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221798msec); 0 zone resets
    slat (usec): min=100, max=1433, avg=184.05, stdev=54.94
    clat (msec): min=23, max=262, avg=138.43, stdev= 2.87
     lat (msec): min=24, max=262, avg=138.61, stdev= 2.87
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  150],
     | 99.99th=[  245]
   bw (  KiB/s): min=345325, max=502765, per=6.25%, avg=472969.14, stdev=8481.47, samples=438
   iops        : min=  337, max=  490, avg=461.18, stdev= 8.29, samples=438
  lat (msec)   : 50=0.02%, 100=0.03%, 250=99.94%, 500=0.01%
  cpu          : usr=4.06%, sys=4.06%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=8740: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221782msec); 0 zone resets
    slat (usec): min=98, max=1249, avg=175.18, stdev=41.42
    clat (msec): min=20, max=270, avg=138.43, stdev= 3.13
     lat (msec): min=20, max=270, avg=138.60, stdev= 3.12
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  163],
     | 99.99th=[  249]
   bw (  KiB/s): min=327360, max=506869, per=6.25%, avg=472957.63, stdev=8830.83, samples=438
   iops        : min=  319, max=  494, avg=461.15, stdev= 8.65, samples=438
  lat (msec)   : 50=0.03%, 100=0.03%, 250=99.93%, 500=0.01%
  cpu          : usr=4.51%, sys=3.61%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=13744: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221794msec); 0 zone resets
    slat (usec): min=95, max=1541, avg=176.51, stdev=40.54
    clat (msec): min=39, max=237, avg=138.43, stdev= 2.22
     lat (msec): min=40, max=237, avg=138.61, stdev= 2.21
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  142],
     | 99.99th=[  218]
   bw (  KiB/s): min=375267, max=505856, per=6.25%, avg=472974.21, stdev=9013.95, samples=438
   iops        : min=  366, max=  494, avg=461.16, stdev= 8.80, samples=438
  lat (msec)   : 50=0.01%, 100=0.03%, 250=99.96%
  cpu          : usr=4.51%, sys=4.06%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
seq_write: (groupid=0, jobs=1): err= 0: pid=12840: Mon Dec 29 10:12:48 2025
  write: IOPS=461, BW=462MiB/s (484MB/s)(100GiB/221795msec); 0 zone resets
    slat (usec): min=92, max=2811, avg=178.30, stdev=45.55
    clat (msec): min=22, max=270, avg=138.43, stdev= 3.03
     lat (msec): min=23, max=271, avg=138.61, stdev= 3.03
    clat percentiles (msec):
     |  1.00th=[  131],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  140], 40.00th=[  140], 50.00th=[  140], 60.00th=[  140],
     | 70.00th=[  140], 80.00th=[  140], 90.00th=[  140], 95.00th=[  140],
     | 99.00th=[  140], 99.50th=[  140], 99.90th=[  140], 99.95th=[  159],
     | 99.99th=[  251]
   bw (  KiB/s): min=335345, max=501960, per=6.25%, avg=472963.67, stdev=8683.66, samples=438
   iops        : min=  327, max=  490, avg=461.17, stdev= 8.50, samples=438
  lat (msec)   : 50=0.03%, 100=0.03%, 250=99.93%, 500=0.01%
  cpu          : usr=4.51%, sys=4.06%, ctx=0, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,102400,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=7386MiB/s (7745MB/s), 462MiB/s-462MiB/s (484MB/s-484MB/s), io=1600GiB (1718GB), run=221782-221813msec

These numbers were done on a 1.6TiB test depth to rule out any caching polluting the performance figures.

What this has made me realize is that the old rules-of-thumb formulas for raid speed calculations don’t work anymore.

I was expecting (N-2)(single drive speed) for the raid 6 read speed which should have come out to 10.5GB/s given Seagate’s 524MB/s speed rating for the drive.
and N(single drive speed)/6 for the raid 6 write speed which should have come out to 2.1GB/s.

3

I got around to “benchmarking” the fans I bought, but the more I think about the method I used the less I like it.

At any rate, here’s a dump of pictures of original 38mm thick fan, the 9RA 38mm thick fan and then the 9RA 25mm thick fan running at either a similar power level for the first two, or the max power level for the 25mm thick fan, with two different mic placements:

noise floor

IMG_20260117_191047
IMG_20260117_1910471920×1440 201 KB

Nidec UltraFlo

IMG_20260117_190858
IMG_20260117_1908581920×1440 187 KB

IMG_20260117_190927
IMG_20260117_1909271920×1440 185 KB

38mm thick 9RA

IMG_20260117_191456
IMG_20260117_1914561920×1440 203 KB

IMG_20260117_191523
IMG_20260117_1915231920×1440 225 KB

25mm thick 9RA

IMG_20260117_191811
IMG_20260117_1918111920×1440 176 KB

IMG_20260117_191718
IMG_20260117_1917181920×1440 210 KB

I didn’t quite get the 10dBA reduction in noise I was looking for, only 5.5dBa reduction from the front mic location measurement and 9.2 dBA from the side mic location on the 38mm thick fans. The 9RA fan does have a more pleasant quality to the noise it produces, it sounds less rattly than the Nidec.

The 25mm thick fan isn’t really comparable in performance to the other two, but it does “throw” the air pretty far with the 25mm>38mm thickness adapter to the point it feels as strong as the 38mm thick fans if you have your hand far enough away.

I’ll be using the 25mm thick fans in the upper chamber of the case since it only needs to pull air through a single backplane of restriction. The bottom chamber of the case is much more restrictive and will need the 38mm fans.

IMG_20260117_184147
IMG_20260117_1841471920×1440 107 KB

1 Like