[ POOR ? ] DISK PERFORMANCE ( 2 SATA SSD ) ZFS RAID1 mirror on ProxMox 8.1

maleko48 · January 20, 2024, 9:32pm

I just completed a clean install of ProxMox 8.1 on a minimal host with the following specs:

 CPU:           Intel i5 10210U    (25W TDP)
                4C/8T -- 1.60GHz - 4.20GHz
                2.10 GHz - 3.80 GHz  nominal

 RAM:           ( 1x )          Crucial 16GB
                                DDR4 2667 MHz

 DISK1:         Transcend 256GB ( TS256GMSA230S )
                      mSATA SSD ( read: 530 MB/s | write: 400 MB/s )
                     03-20-2022 ( 4k random read IOPS: 45000 | write IOPS: 70000 )

 DISK2:         Samsung 870 EVO 250GB ( MZ-77E250B/AM )
                            DRAM cache: 512MB LPDDR4
                             SATA SSD ( read: 560 MB/s | write: 530 MB/s )
                           04-02-2022 ( 4k random read QD1  IOPS: 13000 | write IOPS: 36000 )
                                      ( 4k random read QD32 IOPS: 98000 | write IOPS: 88000 )

 WIFI:           Atheros AR9280 WiFi4 2x2
                   mPCIe   (ATH9K driver)

 NICs:           6x 1Gbit  Intel i211 ver. 3

When attempting to upload disk ISOs via the web GUI the first issue I noticed compared to my old ProxMox 7.4 installation was the web GUI freezing after approximately 1GB of ISO data transferred. Apparently this is somewhat common on the newer ProxMox 8.1, so I switched to a LAN transfer using SCP from my Windows 10 desktop.

While the transfers complete, I noticed the disk performance seems to be poor, averaging anywhere from 40MB/s to 60MB/s after a 5+GB ISO transfer as reported at the end of SCP from PowerShell. There also seems to be some amount of freezing / hitching when it comes to the SCP status updating in the PowerShell terminal, as if the host ProxMox system was getting micro freezes / stuttering. On ProxMox 7.4 the PowerShell SCP transfers at least had consistent updates throughout the transfer if only marginally-better actual throughput.

Peak transfer rates appear to be around 100MB/s to 110MB/s which makes sense when transferring over my 1Gb LAN connection, but then I plugged in an NVMe USB drive (Ventoy, exFAT) I regularly see 300-400+ MB/s transfers out of when copying files on all of my other machines and performed a local copy from it to my ProxMox host after mounting the USB drive and did not see any improvement in disk performance during the transfer.

Next I installed fio on my ProxMox host and ran some benchmarks, all of which seem pretty poor. Here are my benchmark results:

16 PARALLEL 64KiB RANDOM WRITE PROCESSES

root@tallbike:~# fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=64k --size=256m --numjobs=16 --iodepth=16 --runtime=60 --time_based --end_fsync=1
random-write: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=posixaio, iodepth=16
...
fio-3.33
Starting 16 processes
Jobs: 16 (f=16): [F(16)][100.0%][eta 00m:00s]                         
random-write: (groupid=0, jobs=1): err= 0: pid=19529: Sat Jan 20 16:39:40 2024
  write: IOPS=31, BW=2005KiB/s (2053kB/s)(152MiB/77485msec); 0 zone resets
    slat (nsec): min=624, max=392784, avg=4941.50, stdev=10850.05
    clat (usec): min=262, max=11329k, avg=430851.49, stdev=1952550.82
     lat (usec): min=264, max=11329k, avg=430856.43, stdev=1952550.40
    clat percentiles (usec):
     |  1.00th=[     400],  5.00th=[     586], 10.00th=[     693],
     | 20.00th=[     775], 30.00th=[     963], 40.00th=[    1532],
     | 50.00th=[    2212], 60.00th=[    3294], 70.00th=[    4359],
     | 80.00th=[    7701], 90.00th=[   26870], 95.00th=[   76022],
     | 99.00th=[10267657], 99.50th=[11341399], 99.90th=[11341399],
     | 99.95th=[11341399], 99.99th=[11341399]
   bw (  KiB/s): min= 3712, max=109568, per=88.59%, avg=30848.50, stdev=33135.59, samples=10
   iops        : min=   58, max= 1712, avg=482.00, stdev=517.75, samples=10
  lat (usec)   : 500=2.02%, 750=14.71%, 1000=14.79%
  lat (msec)   : 2=15.99%, 4=21.14%, 10=14.42%, 20=5.73%, 50=4.53%
  lat (msec)   : 100=2.06%, >=2000=4.61%
  cpu          : usr=0.02%, sys=0.01%, ctx=1687, majf=0, minf=25
  IO depths    : 1=1.0%, 2=2.2%, 4=4.8%, 8=52.3%, 16=39.7%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.2%, 8=0.6%, 16=4.2%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2427,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=19530: Sat Jan 20 16:39:40 2024
  write: IOPS=30, BW=1961KiB/s (2008kB/s)(148MiB/77485msec); 0 zone resets
    slat (nsec): min=430, max=990824, avg=5115.71, stdev=23488.09
    clat (usec): min=391, max=11330k, avg=440441.59, stdev=1976194.40
     lat (usec): min=396, max=11330k, avg=440446.70, stdev=1976194.33
    clat percentiles (usec):
     |  1.00th=[     486],  5.00th=[     578], 10.00th=[     644],
     | 20.00th=[     898], 30.00th=[    1090], 40.00th=[    1352],
     | 50.00th=[    1893], 60.00th=[    2737], 70.00th=[    4080],
     | 80.00th=[    6390], 90.00th=[   24773], 95.00th=[   69731],
     | 99.00th=[10267657], 99.50th=[11341399], 99.90th=[11341399],
     | 99.95th=[11341399], 99.99th=[11341399]
   bw (  KiB/s): min=  128, max=102656, per=86.60%, avg=30152.00, stdev=34162.09, samples=10
   iops        : min=    2, max= 1604, avg=471.10, stdev=533.79, samples=10
  lat (usec)   : 500=1.77%, 750=11.84%, 1000=11.42%
  lat (msec)   : 2=25.78%, 4=18.24%, 10=14.57%, 20=6.07%, 50=4.68%
  lat (msec)   : 100=0.93%, >=2000=4.72%
  cpu          : usr=0.02%, sys=0.01%, ctx=1738, majf=0, minf=25
  IO depths    : 1=0.8%, 2=1.6%, 4=3.2%, 8=53.5%, 16=40.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.2%, 8=2.2%, 16=2.6%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2374,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=19531: Sat Jan 20 16:39:40 2024
  write: IOPS=35, BW=2249KiB/s (2303kB/s)(170MiB/77485msec); 0 zone resets
    slat (nsec): min=603, max=57011k, avg=25438.19, stdev=1092470.86
    clat (usec): min=337, max=11333k, avg=383805.79, stdev=1845559.40
     lat (usec): min=341, max=11333k, avg=383831.22, stdev=1845555.22
    clat percentiles (usec):
     |  1.00th=[     461],  5.00th=[     515], 10.00th=[     578],
     | 20.00th=[     799], 30.00th=[    1139], 40.00th=[    1663],
     | 50.00th=[    2089], 60.00th=[    2966], 70.00th=[    5080],
     | 80.00th=[    8455], 90.00th=[   21365], 95.00th=[   53216],
     | 99.00th=[10267657], 99.50th=[11341399], 99.90th=[11341399],
     | 99.95th=[11341399], 99.99th=[11341399]
   bw (  KiB/s): min= 1792, max=80640, per=99.39%, avg=34607.70, stdev=27104.22, samples=10
   iops        : min=   28, max= 1260, avg=540.70, stdev=423.50, samples=10
  lat (usec)   : 500=3.89%, 750=14.25%, 1000=8.81%
  lat (msec)   : 2=21.19%, 4=18.95%, 10=15.28%, 20=7.20%, 50=5.25%
  lat (msec)   : 100=1.07%, >=2000=4.11%
  cpu          : usr=0.03%, sys=0.00%, ctx=1858, majf=0, minf=24
  IO depths    : 1=0.9%, 2=1.8%, 4=4.0%, 8=54.4%, 16=38.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.0%, 8=1.6%, 16=3.5%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2723,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=19532: Sat Jan 20 16:39:40 2024
  write: IOPS=26, BW=1693KiB/s (1734kB/s)(128MiB/77485msec); 0 zone resets
    slat (nsec): min=496, max=184596, avg=4439.33, stdev=6057.65
    clat (usec): min=385, max=11268k, avg=510065.35, stdev=2112191.18
     lat (usec): min=386, max=11268k, avg=510069.79, stdev=2112190.97
    clat percentiles (usec):
     |  1.00th=[     529],  5.00th=[     668], 10.00th=[     783],
     | 20.00th=[    1106], 30.00th=[    1418], 40.00th=[    1844],
     | 50.00th=[    2868], 60.00th=[    4752], 70.00th=[    5932],
     | 80.00th=[   16450], 90.00th=[   30802], 95.00th=[ 7482639],
     | 99.00th=[10267657], 99.50th=[11207181], 99.90th=[11207181],
     | 99.95th=[11207181], 99.99th=[11207181]
   bw (  KiB/s): min= 1920, max=71296, per=74.65%, avg=25991.10, stdev=24940.50, samples=10
   iops        : min=   30, max= 1114, avg=406.10, stdev=389.69, samples=10
  lat (usec)   : 500=0.54%, 750=8.10%, 1000=8.63%
  lat (msec)   : 2=23.71%, 4=16.24%, 10=17.27%, 20=8.98%, 50=10.29%
  lat (msec)   : 100=0.78%, >=2000=5.46%
  cpu          : usr=0.02%, sys=0.01%, ctx=1515, majf=0, minf=24
  IO depths    : 1=0.4%, 2=0.8%, 4=2.4%, 8=53.9%, 16=42.5%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.5%, 8=3.5%, 16=1.9%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2050,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=19533: Sat Jan 20 16:39:40 2024
  write: IOPS=29, BW=1879KiB/s (1924kB/s)(142MiB/77485msec); 0 zone resets
    slat (nsec): min=583, max=531406, avg=5011.74, stdev=13503.83
    clat (usec): min=318, max=11268k, avg=459635.72, stdev=2010285.12
     lat (usec): min=322, max=11268k, avg=459640.74, stdev=2010285.85
    clat percentiles (usec):
     |  1.00th=[     498],  5.00th=[     570], 10.00th=[     619],
     | 20.00th=[     766], 30.00th=[    1369], 40.00th=[    1582],
     | 50.00th=[    1795], 60.00th=[    2737], 70.00th=[    3752],
     | 80.00th=[    7373], 90.00th=[   34866], 95.00th=[   88605],
     | 99.00th=[10267657], 99.50th=[11207181], 99.90th=[11207181],
     | 99.95th=[11207181], 99.99th=[11207181]
   bw (  KiB/s): min= 1152, max=71680, per=83.03%, avg=28909.20, stdev=27632.47, samples=10
   iops        : min=   18, max= 1120, avg=451.70, stdev=431.76, samples=10
  lat (usec)   : 500=1.05%, 750=17.45%, 1000=5.93%
  lat (msec)   : 2=28.57%, 4=18.86%, 10=10.07%, 20=4.53%, 50=5.36%
  lat (msec)   : 100=3.25%, >=2000=4.92%
  cpu          : usr=0.03%, sys=0.00%, ctx=1662, majf=0, minf=25
  IO depths    : 1=0.5%, 2=1.1%, 4=2.6%, 8=55.1%, 16=40.7%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.8%, 8=1.9%, 16=3.3%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2275,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=19534: Sat Jan 20 16:39:40 2024
  write: IOPS=51, BW=3289KiB/s (3368kB/s)(249MiB/77485msec); 0 zone resets
    slat (nsec): min=577, max=4472.7k, avg=5006.41, stdev=71019.77
    clat (usec): min=73, max=11324k, avg=262572.27, stdev=1538116.57
     lat (usec): min=83, max=11324k, avg=262577.28, stdev=1538116.37
    clat percentiles (usec):
     |  1.00th=[     433],  5.00th=[     502], 10.00th=[     553],
     | 20.00th=[     660], 30.00th=[     791], 40.00th=[     930],
     | 50.00th=[    1090], 60.00th=[    1680], 70.00th=[    2835],
     | 80.00th=[    4817], 90.00th=[    9896], 95.00th=[   19006],
     | 99.00th=[ 9865004], 99.50th=[10267657], 99.90th=[11341399],
     | 99.95th=[11341399], 99.99th=[11341399]
   bw (  KiB/s): min=  128, max=160768, per=100.00%, avg=56348.22, stdev=50926.41, samples=9
   iops        : min=    2, max= 2512, avg=880.33, stdev=795.76, samples=9
  lat (usec)   : 100=0.03%, 500=4.85%, 750=23.33%, 1000=18.11%
  lat (msec)   : 2=18.21%, 4=12.53%, 10=13.41%, 20=5.07%, 50=0.83%
  lat (msec)   : 100=0.83%, >=2000=2.81%
  cpu          : usr=0.05%, sys=0.00%, ctx=2953, majf=0, minf=25
  IO depths    : 1=0.6%, 2=1.3%, 4=3.0%, 8=53.9%, 16=41.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.1%, 8=1.4%, 16=3.5%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,3982,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=19535: Sat Jan 20 16:39:40 2024
  write: IOPS=65, BW=4180KiB/s (4281kB/s)(316MiB/77483msec); 0 zone resets
    slat (nsec): min=566, max=247754, avg=3578.04, stdev=7011.36
    clat (usec): min=329, max=11273k, avg=206505.01, stdev=1366419.47
     lat (usec): min=331, max=11273k, avg=206508.59, stdev=1366419.46
    clat percentiles (usec):
     |  1.00th=[     363],  5.00th=[     441], 10.00th=[     506],
     | 20.00th=[     619], 30.00th=[     734], 40.00th=[     881],
     | 50.00th=[    1037], 60.00th=[    1205], 70.00th=[    1565],
     | 80.00th=[    1909], 90.00th=[    6128], 95.00th=[   19530],
     | 99.00th=[ 9328133], 99.50th=[10133439], 99.90th=[11207181],
     | 99.95th=[11207181], 99.99th=[11207181]
   bw (  KiB/s): min=  256, max=215680, per=100.00%, avg=64409.10, stdev=73853.44, samples=10
   iops        : min=    4, max= 3370, avg=1006.30, stdev=1153.87, samples=10
  lat (usec)   : 500=9.37%, 750=21.87%, 1000=16.72%
  lat (msec)   : 2=32.92%, 4=6.01%, 10=5.97%, 20=2.49%, 50=1.60%
  lat (msec)   : 100=0.85%, >=2000=2.21%
  cpu          : usr=0.03%, sys=0.03%, ctx=4174, majf=0, minf=22
  IO depths    : 1=0.8%, 2=1.6%, 4=3.4%, 8=50.9%, 16=43.3%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.6%, 8=0.1%, 16=4.3%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,5061,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=19536: Sat Jan 20 16:39:40 2024
  write: IOPS=30, BW=1968KiB/s (2016kB/s)(149MiB/77485msec); 0 zone resets
    slat (nsec): min=502, max=55763k, avg=28444.23, stdev=1142224.25
    clat (usec): min=462, max=11277k, avg=438496.60, stdev=1966217.68
     lat (usec): min=463, max=11277k, avg=438525.04, stdev=1966212.76
    clat percentiles (usec):
     |  1.00th=[     502],  5.00th=[     627], 10.00th=[     881],
     | 20.00th=[    1303], 30.00th=[    1713], 40.00th=[    2507],
     | 50.00th=[    3359], 60.00th=[    4047], 70.00th=[    5276],
     | 80.00th=[   10290], 90.00th=[   21890], 95.00th=[   55837],
     | 99.00th=[10267657], 99.50th=[11207181], 99.90th=[11341399],
     | 99.95th=[11341399], 99.99th=[11341399]
   bw (  KiB/s): min= 1280, max=90880, per=86.86%, avg=30244.90, stdev=28119.99, samples=10
   iops        : min=   20, max= 1420, avg=472.50, stdev=439.33, samples=10
  lat (usec)   : 500=0.92%, 750=6.08%, 1000=5.79%
  lat (msec)   : 2=22.83%, 4=24.05%, 10=19.43%, 20=8.35%, 50=7.01%
  lat (msec)   : 100=0.84%, >=2000=4.70%
  cpu          : usr=0.02%, sys=0.02%, ctx=1668, majf=0, minf=22
  IO depths    : 1=1.0%, 2=2.0%, 4=4.7%, 8=53.7%, 16=38.7%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.1%, 8=1.3%, 16=3.6%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2383,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=19537: Sat Jan 20 16:39:40 2024
  write: IOPS=22, BW=1471KiB/s (1506kB/s)(111MiB/77483msec); 0 zone resets
    slat (nsec): min=598, max=174791, avg=4772.82, stdev=6439.45
    clat (usec): min=380, max=11291k, avg=587091.15, stdev=2259239.93
     lat (usec): min=382, max=11291k, avg=587095.92, stdev=2259239.90
    clat percentiles (usec):
     |  1.00th=[     490],  5.00th=[     619], 10.00th=[     693],
     | 20.00th=[     848], 30.00th=[    1045], 40.00th=[    1844],
     | 50.00th=[    3032], 60.00th=[    5407], 70.00th=[    6652],
     | 80.00th=[   18482], 90.00th=[   45876], 95.00th=[ 7885292],
     | 99.00th=[10267657], 99.50th=[11341399], 99.90th=[11341399],
     | 99.95th=[11341399], 99.99th=[11341399]
   bw (  KiB/s): min= 4352, max=60024, per=64.88%, avg=22591.50, stdev=17794.27, samples=10
   iops        : min=   68, max=  937, avg=352.90, stdev=277.83, samples=10
  lat (usec)   : 500=1.07%, 750=13.92%, 1000=13.36%
  lat (msec)   : 2=13.70%, 4=12.30%, 10=20.55%, 20=7.64%, 50=8.37%
  lat (msec)   : 100=2.81%, >=2000=6.29%
  cpu          : usr=0.02%, sys=0.01%, ctx=1375, majf=0, minf=26
  IO depths    : 1=0.6%, 2=1.2%, 4=2.9%, 8=52.6%, 16=42.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.4%, 8=1.0%, 16=3.6%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1781,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=19538: Sat Jan 20 16:39:40 2024
  write: IOPS=38, BW=2433KiB/s (2491kB/s)(184MiB/77484msec); 0 zone resets
    slat (nsec): min=521, max=171588, avg=4299.19, stdev=6696.82
    clat (usec): min=233, max=11325k, avg=355035.31, stdev=1780157.06
     lat (usec): min=243, max=11325k, avg=355039.61, stdev=1780156.84
    clat percentiles (usec):
     |  1.00th=[     578],  5.00th=[     652], 10.00th=[     693],
     | 20.00th=[     783], 30.00th=[     914], 40.00th=[    1336],
     | 50.00th=[    1844], 60.00th=[    3130], 70.00th=[    5342],
     | 80.00th=[    7308], 90.00th=[   13566], 95.00th=[   32637],
     | 99.00th=[10267657], 99.50th=[11341399], 99.90th=[11341399],
     | 99.95th=[11341399], 99.99th=[11341399]
   bw (  KiB/s): min= 7168, max=105984, per=100.00%, avg=37437.70, stdev=33593.51, samples=10
   iops        : min=  112, max= 1656, avg=584.90, stdev=524.88, samples=10
  lat (usec)   : 250=0.07%, 500=0.51%, 750=15.52%, 1000=16.88%
  lat (msec)   : 2=18.17%, 4=12.50%, 10=22.44%, 20=5.94%, 50=4.18%
  lat (msec)   : >=2000=3.80%
  cpu          : usr=0.01%, sys=0.02%, ctx=2040, majf=0, minf=24
  IO depths    : 1=0.7%, 2=1.3%, 4=3.8%, 8=55.3%, 16=38.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.9%, 8=2.5%, 16=2.6%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2945,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=19539: Sat Jan 20 16:39:40 2024
  write: IOPS=22, BW=1439KiB/s (1473kB/s)(109MiB/77484msec); 0 zone resets
    slat (nsec): min=566, max=87185k, avg=54704.75, stdev=2088795.11
    clat (usec): min=434, max=21715k, avg=599807.14, stdev=2717725.62
     lat (usec): min=440, max=21715k, avg=599861.85, stdev=2717715.09
    clat percentiles (usec):
     |  1.00th=[     594],  5.00th=[     889], 10.00th=[    1090],
     | 20.00th=[    1467], 30.00th=[    1745], 40.00th=[    2212],
     | 50.00th=[    3064], 60.00th=[    4948], 70.00th=[    6521],
     | 80.00th=[   10552], 90.00th=[   29492], 95.00th=[ 7415530],
     | 99.00th=[ 9730786], 99.50th=[17112761], 99.90th=[17112761],
     | 99.95th=[17112761], 99.99th=[17112761]
   bw (  KiB/s): min= 2048, max=70156, per=70.50%, avg=24546.89, stdev=21529.20, samples=9
   iops        : min=   32, max= 1096, avg=383.44, stdev=336.40, samples=9
  lat (usec)   : 500=0.06%, 750=3.04%, 1000=4.82%
  lat (msec)   : 2=28.82%, 4=19.12%, 10=22.73%, 20=10.85%, 50=3.16%
  lat (msec)   : 100=1.89%, >=2000=5.51%
  cpu          : usr=0.02%, sys=0.00%, ctx=1200, majf=0, minf=25
  IO depths    : 1=0.8%, 2=1.8%, 4=3.8%, 8=53.8%, 16=39.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.0%, 8=1.0%, 16=3.9%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1742,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=19540: Sat Jan 20 16:39:40 2024
  write: IOPS=31, BW=2020KiB/s (2069kB/s)(153MiB/77484msec); 0 zone resets
    slat (nsec): min=606, max=3905.3k, avg=5717.06, stdev=79681.42
    clat (usec): min=61, max=11330k, avg=427501.48, stdev=1947347.61
     lat (usec): min=68, max=11330k, avg=427507.20, stdev=1947347.01
    clat percentiles (usec):
     |  1.00th=[     330],  5.00th=[     469], 10.00th=[     537],
     | 20.00th=[     644], 30.00th=[     750], 40.00th=[     947],
     | 50.00th=[    1385], 60.00th=[    2343], 70.00th=[    4621],
     | 80.00th=[    6652], 90.00th=[   22152], 95.00th=[   70779],
     | 99.00th=[10267657], 99.50th=[11341399], 99.90th=[11341399],
     | 99.95th=[11341399], 99.99th=[11341399]
   bw (  KiB/s): min=  384, max=89984, per=89.25%, avg=31076.70, stdev=32190.88, samples=10
   iops        : min=    6, max= 1406, avg=485.50, stdev=502.88, samples=10
  lat (usec)   : 100=0.04%, 250=0.37%, 500=7.52%, 750=21.83%, 1000=13.04%
  lat (msec)   : 2=12.51%, 4=11.61%, 10=17.70%, 20=4.37%, 50=4.42%
  lat (msec)   : 100=2.00%, >=2000=4.58%
  cpu          : usr=0.03%, sys=0.00%, ctx=1878, majf=0, minf=27
  IO depths    : 1=0.9%, 2=1.6%, 4=3.4%, 8=51.2%, 16=42.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.5%, 8=0.9%, 16=3.7%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2446,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=19541: Sat Jan 20 16:39:40 2024
  write: IOPS=23, BW=1518KiB/s (1555kB/s)(115MiB/77485msec); 0 zone resets
    slat (nsec): min=612, max=214570, avg=4616.52, stdev=7431.44
    clat (usec): min=395, max=11313k, avg=568907.55, stdev=2230274.32
     lat (usec): min=395, max=11313k, avg=568912.16, stdev=2230273.99
    clat percentiles (usec):
     |  1.00th=[     412],  5.00th=[     486], 10.00th=[     523],
     | 20.00th=[     783], 30.00th=[    1139], 40.00th=[    2057],
     | 50.00th=[    3392], 60.00th=[    4228], 70.00th=[    5669],
     | 80.00th=[    7504], 90.00th=[   34866], 95.00th=[ 7952401],
     | 99.00th=[10267657], 99.50th=[11341399], 99.90th=[11341399],
     | 99.95th=[11341399], 99.99th=[11341399]
   bw (  KiB/s): min=  128, max=56432, per=66.93%, avg=23306.90, stdev=18569.01, samples=10
   iops        : min=    2, max=  881, avg=364.00, stdev=290.00, samples=10
  lat (usec)   : 500=6.69%, 750=12.30%, 1000=4.57%
  lat (msec)   : 2=15.72%, 4=17.95%, 10=25.63%, 20=4.03%, 50=5.55%
  lat (msec)   : 100=1.47%, >=2000=6.09%
  cpu          : usr=0.00%, sys=0.02%, ctx=1450, majf=0, minf=24
  IO depths    : 1=0.6%, 2=1.2%, 4=3.3%, 8=52.0%, 16=43.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.4%, 8=1.1%, 16=3.5%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1838,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=19542: Sat Jan 20 16:39:40 2024
  write: IOPS=26, BW=1706KiB/s (1747kB/s)(129MiB/77483msec); 0 zone resets
    slat (nsec): min=588, max=167818, avg=4713.36, stdev=6958.02
    clat (usec): min=214, max=11328k, avg=506359.97, stdev=2107537.00
     lat (usec): min=219, max=11328k, avg=506364.69, stdev=2107536.70
    clat percentiles (usec):
     |  1.00th=[     363],  5.00th=[     545], 10.00th=[     693],
     | 20.00th=[    1369], 30.00th=[    1926], 40.00th=[    2409],
     | 50.00th=[    3523], 60.00th=[    4359], 70.00th=[    6456],
     | 80.00th=[   11731], 90.00th=[   26084], 95.00th=[ 7415530],
     | 99.00th=[10267657], 99.50th=[11341399], 99.90th=[11341399],
     | 99.95th=[11341399], 99.99th=[11341399]
   bw (  KiB/s): min= 1152, max=70284, per=75.29%, avg=26217.80, stdev=21818.23, samples=10
   iops        : min=   18, max= 1098, avg=409.60, stdev=340.88, samples=10
  lat (usec)   : 250=0.10%, 500=4.31%, 750=7.51%, 1000=3.83%
  lat (msec)   : 2=14.92%, 4=24.50%, 10=22.91%, 20=9.10%, 50=5.96%
  lat (msec)   : 100=1.45%, >=2000=5.42%
  cpu          : usr=0.03%, sys=0.00%, ctx=1234, majf=0, minf=24
  IO depths    : 1=1.0%, 2=2.1%, 4=5.2%, 8=56.9%, 16=34.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=94.6%, 8=2.6%, 16=2.9%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2065,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=19543: Sat Jan 20 16:39:40 2024
  write: IOPS=42, BW=2694KiB/s (2759kB/s)(204MiB/77484msec); 0 zone resets
    slat (nsec): min=493, max=120497, avg=3806.14, stdev=4390.69
    clat (usec): min=455, max=11323k, avg=320570.20, stdev=1692826.76
     lat (usec): min=459, max=11323k, avg=320574.00, stdev=1692826.63
    clat percentiles (usec):
     |  1.00th=[     510],  5.00th=[     594], 10.00th=[     644],
     | 20.00th=[     775], 30.00th=[     963], 40.00th=[    1434],
     | 50.00th=[    1663], 60.00th=[    2147], 70.00th=[    3326],
     | 80.00th=[    5080], 90.00th=[   17171], 95.00th=[   32637],
     | 99.00th=[ 9865004], 99.50th=[10267657], 99.90th=[11341399],
     | 99.95th=[11341399], 99.99th=[11341399]
   bw (  KiB/s): min= 1024, max=108633, per=100.00%, avg=41544.80, stdev=41036.58, samples=10
   iops        : min=   16, max= 1697, avg=649.00, stdev=641.18, samples=10
  lat (usec)   : 500=0.31%, 750=17.32%, 1000=13.76%
  lat (msec)   : 2=26.09%, 4=16.62%, 10=12.84%, 20=4.11%, 50=4.11%
  lat (msec)   : 100=1.41%, >=2000=3.43%
  cpu          : usr=0.03%, sys=0.01%, ctx=2681, majf=0, minf=22
  IO depths    : 1=0.4%, 2=0.9%, 4=2.2%, 8=52.3%, 16=44.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.5%, 8=1.7%, 16=1.8%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,3262,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
random-write: (groupid=0, jobs=1): err= 0: pid=19544: Sat Jan 20 16:39:40 2024
  write: IOPS=36, BW=2314KiB/s (2370kB/s)(175MiB/77483msec); 0 zone resets
    slat (nsec): min=593, max=206726, avg=4376.85, stdev=7532.13
    clat (usec): min=66, max=11268k, avg=373183.75, stdev=1821180.30
     lat (usec): min=76, max=11268k, avg=373188.12, stdev=1821180.14
    clat percentiles (usec):
     |  1.00th=[     437],  5.00th=[     498], 10.00th=[     586],
     | 20.00th=[     816], 30.00th=[    1221], 40.00th=[    1532],
     | 50.00th=[    1647], 60.00th=[    2311], 70.00th=[    3556],
     | 80.00th=[    7308], 90.00th=[   19530], 95.00th=[   50594],
     | 99.00th=[10267657], 99.50th=[11207181], 99.90th=[11207181],
     | 99.95th=[11207181], 99.99th=[11207181]
   bw (  KiB/s): min= 2048, max=120704, per=100.00%, avg=39558.78, stdev=37580.95, samples=9
   iops        : min=   32, max= 1886, avg=618.00, stdev=587.10, samples=9
  lat (usec)   : 100=0.07%, 500=5.35%, 750=11.71%, 1000=10.49%
  lat (msec)   : 2=29.09%, 4=15.77%, 10=9.64%, 20=8.17%, 50=4.57%
  lat (msec)   : 100=1.14%, >=2000=4.00%
  cpu          : usr=0.03%, sys=0.00%, ctx=2064, majf=0, minf=23
  IO depths    : 1=0.7%, 2=1.7%, 4=3.7%, 8=52.5%, 16=41.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.3%, 8=0.4%, 16=4.3%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2802,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=34.0MiB/s (35.7MB/s), 1439KiB/s-4180KiB/s (1473kB/s-4281kB/s), io=2635MiB (2763MB), run=77483-77485msec

SINGLE 1MiB RANDOM WRITE PROCESS

root@tallbike:~# fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=1m --size=16g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1
random-write: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=posixaio, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [F(1)][100.0%][eta 00m:00s]                        
random-write: (groupid=0, jobs=1): err= 0: pid=20261: Sat Jan 20 16:44:44 2024
  write: IOPS=42, BW=42.8MiB/s (44.9MB/s)(3253MiB/75932msec); 0 zone resets
    slat (usec): min=7, max=278, avg=34.79, stdev=18.02
    clat (usec): min=113, max=10481k, avg=20565.17, stdev=399885.78
     lat (usec): min=124, max=10481k, avg=20599.96, stdev=399886.05
    clat percentiles (usec):
     |  1.00th=[     125],  5.00th=[     182], 10.00th=[     198],
     | 20.00th=[     231], 30.00th=[     265], 40.00th=[     293],
     | 50.00th=[     326], 60.00th=[     355], 70.00th=[     396],
     | 80.00th=[     453], 90.00th=[     545], 95.00th=[     660],
     | 99.00th=[     979], 99.50th=[    1221], 99.90th=[ 7885292],
     | 99.95th=[ 9462350], 99.99th=[10536092]
   bw (  KiB/s): min=14336, max=1314816, per=100.00%, avg=554991.67, stdev=339826.25, samples=12
   iops        : min=   14, max= 1284, avg=541.92, stdev=331.97, samples=12
  lat (usec)   : 250=25.02%, 500=61.02%, 750=10.67%, 1000=2.43%
  lat (msec)   : 2=0.49%, 4=0.06%, 10=0.03%, >=2000=0.28%
  cpu          : usr=0.18%, sys=0.02%, ctx=4703, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,3253,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=42.8MiB/s (44.9MB/s), 42.8MiB/s-42.8MiB/s (44.9MB/s-44.9MB/s), io=3253MiB (3411MB), run=75932-75932msec

On ProxMox 7.4 I never had the web GUI freezing issue and my throughput was a little better than in 8.1. On PxMx 7.4 roughly the first 1.50GB - 2.00GB of ISO transfer would move quick, then slow down a bit. My initial thoughts with that behavior were my SSDs’ onboard write caches were just filling up and the behavior was normal, but I’m not sure anymore.

I am also wondering if there could be some kind of BIOS misconfiguration not alotting sufficient PCIe bandwidth to my storage drives? I do have a fully-unlocked BIOS and I have tried resetting it back to stock “Optimized Defaults” but with no improvements unfortunately.

Am I missing something here?

I just want to know if this is normal / expected behavior, or if I have a misconfiguration somewhere that is holding me back. I know my hardware is trash-tier in the grand scheme of things, but I thought SATA SSDs would yield a little better throughput than this.

Thanks!

jode · January 22, 2024, 5:03am

You could try a fstrim -a from the proxmox terminal.
In case the raid1 config is a zfs mirror, check out the zpool trim <pool> command, also from the terminal.

risk · January 22, 2024, 6:20am

What record size are you using?

What’s atop or iostat telling you about the physical device while you’re running the test?

You’re getting a crazy low number of iops for the 1M test, it doesn’t look right at all

MetalizeYourBrain · January 22, 2024, 2:11pm

Could it be the mirror pool is just being slowed down by the slowest drive?

maleko48 · January 22, 2024, 2:15pm

I went ahead and nuked that particular ProxMox 8.1 install in order to do some BIOS tuning and testing. Also because I wanted to install Windows 10 on my unit temporarily for some disk tests and to run the Transcend OEM monitor tool once to see if it had any firmware updates available for the SSD.

Here are some of those test results run from a clean install of Windows 10 (I forgot to run Crystal Diskmark):

SEQUENTIAL READ/WRITE (MB/s):

SEQUENTIAL READ/WRITE (IOPs):

FIO (Win10) – 4k RAND WRITES QD32 4G 60sec

FIO (Win10) – 4k RAND WRITES QD32 4G 60sec + FSYNC

FIO (Win10) – 4k RAND WRITES QD1 4G 60sec

I noticed while doing the ATTO disk tests that the Transcend 230S SSD temps were getting pretty warm, though the disk tests’ results seem un-changed despite the high temperatures. Regardless, I re-pasted everything in the unit and installed an internal fan to circulate and distribute the warm air inside the unit. Subsequent tests yield much lower, stable disk temps, but I really don’t think those were any part of the issue here based on the results from the tests performed inside Windows while the SSD was quite hot. (FWIW, the unit can run at 3.9 - 4.2 GHz indefinitely, so I don’t think clock speeds are the issue either.)

So tonight when I get home from work I plan to re-install ProxMox 8.1 again and am open to all suggestions and input regarding configurations to make. I plan to keep at this until I am happy with a base ProxMox ZFS install that performs its best before I actually start using it for anything.

I can see, but the clean install with nothing on it was mostly stock in that regard aside from some block/record size changes, and a couple HA services disabled. Would this be something that needs to be tweaked at initial ProxMox installation or set thereafter?

As far as my previous install, I had set the block / record sizes to 64k afaik, iostat was just showing the 40MB/s bursts, but nothing was going consistent. I did not have atop installed, but can install it this next go-around. I don’t quite know how to use these tools to the level you probably want me to, I will need some guidance. I only know how to run them and look at the default screen they present mostly.

I will peform another suite of disk tests from Windows tonight on both disks before I blow Windows away and re-install ProxMox again. I know these are only budget SATA disks and don’t expect more than a few hundred MB/s individually on Windows and am aware of the penalty regarding random reads/writes, etc… not sure how all of that translates to a mirror on ZFS though.

I am just getting to know all of this better as my last go-around with this unit was a stock ProxMox 7.4 install with pfSense running for over a year or more on these same disks. Maybe I just didnt realize how poor the throughput was then because I wasn’t looking for anything more once I got pfSense going.

jode · January 22, 2024, 3:17pm

No, it shouldn’t. Trim operations, if necessary, are typically performed by the OS. I suggested this as a troubleshooting step.

Have you benchmarked the Samsung drive as well?
While the performance of the Transcend drive has been documented below current industry standard, it does not explain the (even lower) performance you observed under Proxmox.
Maybe an issue with the Samsung? Or an issue running both in a raid config?

MetalizeYourBrain · January 22, 2024, 5:09pm

Exactly! Those tests you posted absolutely show the Transcend drive to be slower not by marging of error so that must have to do something with it. About ZFS performance I don’t know the FS too well too but it could be some quirk of the filesystem that makes the pool extremely slow.