Hi,
I made a new server build initially for ESXi but it’s driving me crazy with slow rand4k read performance. The hardware is
AMD Ryzen 5900X
ASRock Rack X570D4U-2L2T
Kingston 32gbx2 ECC (KSM32ED8/32ME) on Memory QVL list
Broadcom MegaRAID 9560-16i PCIe 4.0 RAID controller
ICY DOCK ToughArmor MB720MK-B V2
Samsung PM9A3 (2 disks in RAID1 mode)
All traffic through the RAID card is SLOW in Ubuntu and ESXI, but fast in Windows 2022.
All drivers are updated for mobo and raid card and OS. Traffic to NVME bootdrives are fine in both Linux and Ubuntu / ESXi.
I’m mostly interested in READ speed because of database performance.
fio under Ubuntu
sudo fio --filename=/dev/sda --direct=1 --rw=randread --bs=4k --size=4g --ioengine=posixaio --iodepth=1 --runtime=30 --time_based --group_reporting --name=iops-test-job --readonly
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=posixaio, iodepth=1
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=22.1MiB/s][r=5651 IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=1): err= 0: pid=1675: Tue Feb 15 13:41:27 2022
read: IOPS=5646, BW=22.1MiB/s (23.1MB/s)(662MiB/30001msec)
slat (nsec): min=410, max=90470, avg=1348.97, stdev=288.25
clat (usec): min=103, max=721, avg=175.30, stdev=22.66
lat (usec): min=105, max=722, avg=176.64, stdev=22.66
clat percentiles (usec):
| 1.00th=[ 165], 5.00th=[ 167], 10.00th=[ 169], 20.00th=[ 169],
| 30.00th=[ 172], 40.00th=[ 172], 50.00th=[ 174], 60.00th=[ 174],
| 70.00th=[ 174], 80.00th=[ 176], 90.00th=[ 178], 95.00th=[ 180],
| 99.00th=[ 343], 99.50th=[ 351], 99.90th=[ 359], 99.95th=[ 359],
| 99.99th=[ 363]
bw ( KiB/s): min=22488, max=22728, per=99.99%, avg=22585.29, stdev=60.58, samples=59
iops : min= 5622, max= 5682, avg=5646.27, stdev=15.16, samples=59
lat (usec) : 250=98.31%, 500=1.69%, 750=0.01%
cpu : usr=1.68%, sys=1.98%, ctx=169433, majf=0, minf=43
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=169413,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=22.1MiB/s (23.1MB/s), 22.1MiB/s-22.1MiB/s (23.1MB/s-23.1MB/s), io=662MiB (694MB), run=30001-30001msec
Disk stats (read/write):
sda: ios=168826/0, merge=0/0, ticks=27860/0, in_queue=0, util=99.73%
fio under Windows
C:\Users\Administrator>fio.exe --name=baseline --rw=randread --direct=1 --size=4g --iodepth=1 --bs=4k --ioengine=windowsaio --filename=\\.\PhysicalDrive1 --time_based --runtime=30
fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread' option to get rid of this warning.
baseline: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=windowsaio, iodepth=1
fio-3.27
Starting 1 thread
Jobs: 1 (f=1): [r(1)][100.0%][r=65.1MiB/s][r=16.7k IOPS][eta 00m:00s]
baseline: (groupid=0, jobs=1): err= 0: pid=4672: Tue Feb 15 03:43:32 2022
read: IOPS=16.7k, BW=65.4MiB/s (68.6MB/s)(1962MiB/30001msec)
slat (nsec): min=2234, max=278125, avg=2507.26, stdev=832.63
clat (usec): min=4, max=2076, avg=56.73, stdev=18.54
lat (usec): min=19, max=2082, avg=59.24, stdev=18.59
clat percentiles (usec):
| 1.00th=[ 19], 5.00th=[ 43], 10.00th=[ 44], 20.00th=[ 44],
| 30.00th=[ 46], 40.00th=[ 47], 50.00th=[ 64], 60.00th=[ 65],
| 70.00th=[ 65], 80.00th=[ 67], 90.00th=[ 78], 95.00th=[ 78],
| 99.00th=[ 79], 99.50th=[ 80], 99.90th=[ 85], 99.95th=[ 103],
| 99.99th=[ 215]
bw ( KiB/s): min=65808, max=67547, per=100.00%, avg=67032.86, stdev=384.55, samples=59
iops : min=16452, max=16886, avg=16757.83, stdev=96.14, samples=59
lat (usec) : 10=0.01%, 20=2.14%, 50=46.73%, 100=51.07%, 250=0.04%
lat (usec) : 500=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%
cpu : usr=0.00%, sys=3.33%, ctx=0, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=502397,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=65.4MiB/s (68.6MB/s), 65.4MiB/s-65.4MiB/s (68.6MB/s-68.6MB/s), io=1962MiB (2058MB), run=30001-30001msec
In order to bugfix the server is in dual boot with Ubuntu and Windows server. Both fresh vanilla installations. If i boot in Windows the raid is fast, Ubuntu its slow.
I tried changing a bunch of stuff in the bios.
I tried installing VM SQL and a search in 50secs in Linux and 5 secs in windows, so the fio results must be accurate. Crystaldiskmark and ATTO shows similar results.
I’m going crazy here, any help will be greatly appreciated.