[SOLVED] TrueNAS with LSI9211-8i HBA and Encrypted Striped Mirror pool extremely slow performance!

drizzlezap · January 31, 2022, 10:32pm

Hey There

It’s me again, following situation:

My server has TrueNAS installed on a mirror of 2 Samsung 870 EVOs.

It also has a striped Pool of 3 x 2(Mirror) vdevs. I chose this because I was able to spare the drives for the capacity loss and because I aimed for maximum performance with spinning rust. I am using this as a VM storage

I recently bought an Optane P4801x to use as SLOG and activate sync writes. Before installing it I wanted to see how much performance I was going to loose with the SLOG installed and sync writes active in comparsion to sync write off.

I actually never tested my speed before as i was under the impression that I should have plenty so this was the first time I did.

I have a dataset on the pool that is shared through NFS to a Proxmox install. I created an ubuntu server VM and ran the Phoronix Test Suite on it. I used the 'pts/iozone` benchmark to determine my disk performance.

The result are honestly quite funny.

Write Performance:

21MB/s (sync off)
7Mb/s (sync on and SLOG installed)

Now, as slow as spinning rust may be, I assume there must be something wrong here, as other users are getting way more than me.

I think around 400MB/s should be realistic?

I read somewhere that I may have misaligned vdev because the installation on SSDs makes the system use an ashift of 9 instead of 12 needed for 4k drives? Or is it maybe the other way around?

However, I was not able to confirm that with the commands provided in those threads, mainly:

zdb | grep ashift
zdb -c <pool name> | grep ashift

My drives are 512E drives, meaning they are actually 4k but reporting as 512, does that maybe cause a problem?

Running fdisk /dev/<drive> reports:

Media secor size is 512

Unfortunately, I do not know enough about ZFS to sort this issue out myself and my own research hasn’t helped me out so far. Does anybody know, what might have gone wrong here?

Server Specs

TrueNAS 12.0-U7 on Samsung 870 EVO (Mirror)
Supermicro X8DTE-F Motherboard
2 x Xeon X5690
168GB DDR3 ECC RAM
3 x Fujitsu D2607 crossflashed to LSI 9211-8i (IT mode confirmed) (PCIe 2.0 x8)
8 x IronWolf Pro 8TB, 512E
Optane P4801x (PCIe 2.0 x8)
Intel X520 (PCIe 2.0 x4)

Appreciate time and effort in advance

Edit:
Changed title for the people from the future searching for this.

Solution:

Enable AES-NI Option in BIOS if you are using those HBAs and encrypted pools!

thro · January 31, 2022, 11:04pm

Yeah ASHIFT of 9 for 512 byte sectors will definitely fuck this up.

TLDR: you need to re-create the pool with ASHIFT=12 manually.

Long PC industry BS-practices-caused explanation follows…

Basically if that’s what ZFS is reporting, it is trying to write blocks in 512 byte sizes at the drive level

Which means that if the drive really has 4k sectors, every single 512 byte write will turn into something inside the drive’s controller like… drum roll…

1x 4k (real) sector read
modify 4k sector to incorporate the modified 512b part of it in drive’s controller
1x 4k write to the medium

So writing 4k would be that x8 - because even though you’re writing 4k the drive is doing it 512b at a time using the above method due to it actually having 4k sectors which is the smallest read/write possible internally. Which also means 8x the writes to the drive!

You’re screwing the performance by what… 8x at least, before even considering things like increased wear and tear causing wear levelling, caches being invalidated/less efficiently used, etc.

You’ll need to destroy/recreate pool for this I think, but set your ASHIFT to 12 for 4096 byte sectors would be my general advice, but also check with ZFS gurus to see if it may even be beneficial to set something larger on SSD media (I’m not an expert on ZFS on SSD).

This is really something to watch out for with ZFS, and I’m surprised there are SSDs on the market reporting 512b sectors. The reason for drives to do that is for compatibility (albeit with the above penalties - i.e., it “works” rather than fail to init/read/write or whatever) with legacy platforms that assume 512b (e.g… DOS? Windows 95?). Which hasn’t been a thing for… many years now.

ZFS can only (automatically) go by what the drive reports - unless you tell it otherwise. There are hard drives that do this too, and performance tanks on them just as bad/worse.

Now, if you use ASHIFT=12 in a 512b drive? You’re just writing larger chunks at once(?). Given windows filesystems default to 4k cluster size now IIRC anyway - I’m surprised ZFS doesn’t default to 4k now to be honest, unless there’s some other issue I’m not aware of with doing that.

Sure, a 512b write on a 512 sector drive would be 4k of writes, but who’s writing 512b at a time in the 21st century??

Log · February 1, 2022, 12:05am

From what I hear, there is quite a bit of enterprise hardware out there that still can’t handle 4K. But I agree, it really needs to just stop already.

Part of the issue is that the ZFS have decided that trusting the sector size the drive reports is the correct behavior and use a whitelist to set it differently on known drives. What has screwed over so many people it’s not even funny, which goes counter to ZFS’s goal of being fairly easy and just working decently without getting to into details.

If I had any say I’d force it to throw a warning before it automatically uses 512, explaining it’s probably false with a severe performance impact. That or just demand explicit setting of ashift so people have to ask the question. I’ve given passing thought more than once about cobbling together a “frontend” script for ZFS that warns about these kind of common fuckups, and explains context and alternatives.

Also, good synopsis.

thro · February 1, 2022, 1:59am

For what it’s worth, re: drive detection I agree with the ZFS team. Broken hardware be broken. If you need legacy 512b drives, they should be on the market for whatever ridiculous price the OEM demands. Don’t report 512b sectors to the OS when its bullshit and you’re really 4k internally.

99.99% of the market will require (due to internally really having) 4k sectors these days so that should be what drives report.

That would be really neat. I agree there should definitely be a warning presented; something like:

"Your drive is reporting it uses 512b sectors possibly for legacy compatibility reasons. 4k sector size is strongly recommended as using 512b sectors will ‘work’ but cause severe performance penalties and additional wear on your media. We strongly recommend you use 4k sectors, test and only rebuild with 512b sectors if you experience problems.

Changing the sector size requires a pool rebuild which in turn requires backup and restore of your media so do not choose 512b sectors unless you really know you need it"

drizzlezap · February 1, 2022, 7:50am

Thanks for the in-depth explanation. I read somehwere that TreuNAS should actually use ashift 12 by default, but because I installed the system on SSD it went for ashift 9 instead. However, I don’t know if that is true.

Also I would like to point out that I was not able to confirm ashift 9 on my pools, as the commands mentioned do not work on my system for some reason. But I guess it must be that as ‘fdisk’ reports 512?

It is very understandable now why performance will crash when ashift is set wrong from your explanation. The sad thing is that I even researched this topic when I was buying the drives, but somehow I didn’t make the connection with ZFS and ashift when I was setting everything up.

Learned something new I guess. I have a second TrueNAS system that I use for backup, can I just Backup the VM there and restore them once the pool is setup correctly or will there might be some incompatibility with the new pool? As it should be just data, I guess not?

So even with HDDs that have 512E sectors it is reccomended to use ashift 12, right? I guess I will have to recreate all my Pools on both systems then.

That would have indeed been very handy in my case.

Thanks for the help and informations! Guess I know, what I will be doing this week.

Exard3k · February 1, 2022, 10:30am

FreeBSD is sometimes different. In the case of zdb, the correct TrueNAS syntax is:

zdb -U /data/zfs/zpool.cache | grep ashift

default directory is /etc or /boot, but FreeBSD likes to store the cachefile in /data of all things. Not your fault, caused headache to me as well

Trooper_ish · February 1, 2022, 12:38pm

512e is 4K physical, representing as 512 bytes.
Kinda less good for write amplification and random reading.
Might want to use ashift=12, and ensure partition aligns to sector boundaries

https://i.dell.com/sites/doccontent/shared-content/data-sheets/Documents/512e_4Kn_Disk_Formats_120413.pdf

drizzlezap · February 1, 2022, 1:19pm

Exard3k:

FreeBSD is sometimes different. In the case of zdb, the correct TrueNAS syntax is:
zdb -U /data/zfs/zpool.cache | grep ashift
default directory is /etc or /boot, but FreeBSD likes to store the cachefile in /data of all things. Not your fault, caused headache to me as well

Thanks a lot, that worked! However, I am not sure I like the information that came from it. Seems like ashift on my pool is already set to 12, which is bad news for me as I really wouldn’t know where to look next.

What else could cause my performance issues?

That is exactly the document i read when researching that topic, it’s pretty good.

Is there a way I can check for misalignment on an existing pool and maybe even fix it without destroying it first?

Exard3k · February 1, 2022, 1:40pm

fio. To check if drive performance is also bad locally. Write your fio-file to a (new) dataset that has compression, primarycache and secondarycache disabled. post results. I’m at work so don’t have time to post a full fio copy&paste command.

drizzlezap · February 1, 2022, 2:36pm

I used following command and ran it inside the /mnt directory of the Pool:

fio --rw=readwrite --name=test --size=50000M --direct=1 --bs=1024k

fio 5GB

root@freenas:/mnt/Zpool-24TB-Mirror/Proxmox-Cluster-VM-Storage # fio --rw=readwrite --name=test --size=5000M --direct=1 --bs=1024k
test: (g=0): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.27
Starting 1 process
test: Laying out IO file (1 file / 5000MiB)
Jobs: 1 (f=1): [M(1)][5.8%][r=59.9MiB/s,w=48.0MiB/s][r=59,w=47 IOPS][eta 00m:49sJobs: 1 (f=1): [M(1)][7.4%][r=34.1MiB/s,w=48.1MiB/s][r=34,w=48 IOPS][eta 00m:50sJobs: 1 (f=1): [M(1)][9.1%][r=33.8MiB/s,w=47.7MiB/s][r=33,w=47 IOPS][eta 00m:50sJobs: 1 (f=1): [M(1)][10.9%][r=45.1MiB/s,w=48.1MiB/s][r=45,w=48 IOPS][eta 00m:49Jobs: 1 (f=1): [M(1)][12.7%][r=40.9MiB/s,w=46.9MiB/s][r=40,w=46 IOPS][eta 00m:48Jobs: 1 (f=1): [M(1)][14.5%][r=47.2MiB/s,w=47.2MiB/s][r=47,w=47 IOPS][eta 00m:47Jobs: 1 (f=1): [M(1)][16.7%][r=56.1MiB/s,w=48.1MiB/s][r=56,w=48 IOPS][eta 00m:45Jobs: 1 (f=1): [M(1)][18.5%][r=32.0MiB/s,w=48.0MiB/s][r=32,w=48 IOPS][eta 00m:44Jobs: 1 (f=1): [M(1)][20.4%][r=52.9MiB/s,w=48.0MiB/s][r=52,w=47 IOPS][eta 00m:43Jobs: 1 (f=1): [M(1)][23.6%][r=41.7MiB/s,w=48.7MiB/s][r=41,w=48 IOPS][eta 00m:42Jobs: 1 (f=1): [M(1)][25.5%][r=46.0MiB/s,w=48.0MiB/s][r=46,w=48 IOPS][eta 00m:41Jobs: 1 (f=1): [M(1)][27.3%][r=49.0MiB/s,w=47.0MiB/s][r=49,w=47 IOPS][eta 00m:40Jobs: 1 (f=1): [M(1)][29.1%][r=38.0MiB/s,w=48.0MiB/s][r=37,w=47 IOPS][eta 00m:39Jobs: 1 (f=1): [M(1)][31.5%][r=62.1MiB/s,w=47.0MiB/s][r=62,w=47 IOPS][eta 00m:37Jobs: 1 (f=1): [M(1)][33.3%][r=52.1MiB/s,w=48.0MiB/s][r=52,w=48 IOPS][eta 00m:36Jobs: 1 (f=1): [M(1)][35.2%][r=32.0MiB/s,w=48.0MiB/s][r=31,w=47 IOPS][eta 00m:35Jobs: 1 (f=1): [M(1)][37.0%][r=47.4MiB/s,w=48.4MiB/s][r=47,w=48 IOPS][eta 00m:34Jobs: 1 (f=1): [M(1)][38.9%][r=46.9MiB/s,w=47.9MiB/s][r=46,w=47 IOPS][eta 00m:33Jobs: 1 (f=1): [M(1)][40.7%][r=46.2MiB/s,w=48.2MiB/s][r=46,w=48 IOPS][eta 00m:32Jobs: 1 (f=1): [M(1)][42.6%][r=38.9MiB/s,w=46.9MiB/s][r=38,w=46 IOPS][eta 00m:31Jobs: 1 (f=1): [M(1)][44.4%][r=51.1MiB/s,w=48.1MiB/s][r=51,w=48 IOPS][eta 00m:30Jobs: 1 (f=1): [M(1)][46.3%][r=52.1MiB/s,w=47.0MiB/s][r=52,w=47 IOPS][eta 00m:29Jobs: 1 (f=1): [M(1)][48.1%][r=55.1MiB/s,w=48.1MiB/s][r=55,w=48 IOPS][eta 00m:28Jobs: 1 (f=1): [M(1)][50.0%][r=54.6MiB/s,w=47.7MiB/s][r=54,w=47 IOPS][eta 00m:27Jobs: 1 (f=1): [M(1)][51.9%][r=45.2MiB/s,w=47.2MiB/s][r=45,w=47 IOPS][eta 00m:26Jobs: 1 (f=1): [M(1)][53.7%][r=35.7MiB/s,w=47.6MiB/s][r=35,w=47 IOPS][eta 00m:25Jobs: 1 (f=1): [M(1)][55.6%][r=33.0MiB/s,w=48.0MiB/s][r=33,w=48 IOPS][eta 00m:24Jobs: 1 (f=1): [M(1)][57.4%][r=37.0MiB/s,w=48.0MiB/s][r=37,w=48 IOPS][eta 00m:23Jobs: 1 (f=1): [M(1)][59.3%][r=51.0MiB/s,w=48.0MiB/s][r=51,w=48 IOPS][eta 00m:22Jobs: 1 (f=1): [M(1)][61.1%][r=47.0MiB/s,w=47.0MiB/s][r=47,w=47 IOPS][eta 00m:21Jobs: 1 (f=1): [M(1)][63.0%][r=45.0MiB/s,w=45.0MiB/s][r=45,w=45 IOPS][eta 00m:20Jobs: 1 (f=1): [M(1)][64.8%][r=41.9MiB/s,w=44.9MiB/s][r=41,w=44 IOPS][eta 00m:19Jobs: 1 (f=1): [M(1)][66.7%][r=59.2MiB/s,w=47.1MiB/s][r=59,w=47 IOPS][eta 00m:18Jobs: 1 (f=1): [M(1)][68.5%][r=50.9MiB/s,w=47.0MiB/s][r=50,w=46 IOPS][eta 00m:17Jobs: 1 (f=1): [M(1)][70.4%][r=44.7MiB/s,w=46.7MiB/s][r=44,w=46 IOPS][eta 00m:16Jobs: 1 (f=1): [M(1)][72.2%][r=44.7MiB/s,w=46.7MiB/s][r=44,w=46 IOPS][eta 00m:15Jobs: 1 (f=1): [M(1)][74.1%][r=33.9MiB/s,w=46.9MiB/s][r=33,w=46 IOPS][eta 00m:14Jobs: 1 (f=1): [M(1)][75.9%][r=34.0MiB/s,w=47.0MiB/s][r=34,w=47 IOPS][eta 00m:13Jobs: 1 (f=1): [M(1)][77.8%][r=53.3MiB/s,w=46.4MiB/s][r=53,w=46 IOPS][eta 00m:12Jobs: 1 (f=1): [M(1)][79.6%][r=42.0MiB/s,w=47.0MiB/s][r=42,w=47 IOPS][eta 00m:11Jobs: 1 (f=1): [M(1)][80.0%][r=35.0MiB/s,w=48.0MiB/s][r=35,w=48 IOPS][eta 00m:11Jobs: 1 (f=1): [M(1)][81.8%][r=43.0MiB/s,w=48.0MiB/s][r=43,w=48 IOPS][eta 00m:10Jobs: 1 (f=1): [M(1)][83.6%][r=35.0MiB/s,w=47.0MiB/s][r=35,w=47 IOPS][eta 00m:09Jobs: 1 (f=1): [M(1)][85.5%][r=50.0MiB/s,w=47.0MiB/s][r=50,w=47 IOPS][eta 00m:08Jobs: 1 (f=1): [M(1)][87.3%][r=48.7MiB/s,w=47.7MiB/s][r=48,w=47 IOPS][eta 00m:07Jobs: 1 (f=1): [M(1)][89.1%][r=45.5MiB/s,w=47.5MiB/s][r=45,w=47 IOPS][eta 00m:06Jobs: 1 (f=1): [M(1)][92.6%][r=54.1MiB/s,w=48.0MiB/s][r=54,w=48 IOPS][eta 00m:04Jobs: 1 (f=1): [M(1)][94.4%][r=50.6MiB/s,w=47.6MiB/s][r=50,w=47 IOPS][eta 00m:03Jobs: 1 (f=1): [M(1)][96.3%][r=37.0MiB/s,w=48.0MiB/s][r=37,w=48 IOPS][eta 00m:02Jobs: 1 (f=1): [M(1)][98.1%][r=48.0MiB/s,w=48.0MiB/s][r=47,w=47 IOPS][eta 00m:01Jobs: 1 (f=1): [M(1)][100.0%][r=62.1MiB/s,w=47.1MiB/s][r=62,w=47 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=71750: Tue Feb  1 06:32:49 2022
  read: IOPS=45, BW=45.7MiB/s (48.0MB/s)(2452MiB/53612msec)
    clat (usec): min=85, max=401, avg=214.21, stdev=27.37
     lat (usec): min=85, max=402, avg=214.45, stdev=27.41
    clat percentiles (usec):
     |  1.00th=[  133],  5.00th=[  174], 10.00th=[  180], 20.00th=[  190],
     | 30.00th=[  202], 40.00th=[  212], 50.00th=[  219], 60.00th=[  225],
     | 70.00th=[  229], 80.00th=[  235], 90.00th=[  243], 95.00th=[  251],
     | 99.00th=[  277], 99.50th=[  289], 99.90th=[  314], 99.95th=[  326],
     | 99.99th=[  404]
   bw (  KiB/s): min=22483, max=77513, per=99.88%, avg=46776.42, stdev=13101.18, samples=106
   iops        : min=   21, max=   75, avg=44.94, stdev=12.81, samples=106
  write: IOPS=47, BW=47.5MiB/s (49.8MB/s)(2548MiB/53612msec); 0 zone resets
    clat (usec): min=19751, max=75819, avg=20812.53, stdev=1438.82
     lat (usec): min=19760, max=75837, avg=20829.84, stdev=1438.84
    clat percentiles (usec):
     |  1.00th=[20055],  5.00th=[20317], 10.00th=[20579], 20.00th=[20579],
     | 30.00th=[20579], 40.00th=[20579], 50.00th=[20579], 60.00th=[20841],
     | 70.00th=[20841], 80.00th=[20841], 90.00th=[21103], 95.00th=[21103],
     | 99.00th=[23462], 99.50th=[25297], 99.90th=[39584], 99.95th=[47449],
     | 99.99th=[76022]
   bw (  KiB/s): min=42922, max=51097, per=100.00%, avg=48702.92, stdev=1209.27, samples=106
   iops        : min=   41, max=   49, avg=46.84, stdev= 1.23, samples=106
  lat (usec)   : 100=0.18%, 250=46.40%, 500=2.46%
  lat (msec)   : 20=0.46%, 50=50.48%, 100=0.02%
  cpu          : usr=0.08%, sys=2.23%, ctx=20395, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2452,2548,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0,  percentile=100.00%, depth=1

Run status group 0 (all jobs):
    READ: bw=45.7MiB/s (48.0MB/s), 45.7MiB/s-45.7MiB/s (48.0MiB/s-48.0MiB/s), io=2452MiB (2571MB), run=53612-53612msec
    WRITE: bw=47.5MiB/s (49.8MB/s), 47.5MiB/s-47.5MiB/s (49.8MiB/s-49.8MiB/s), io=2548MiB (2672MB), run=53612-53612msec

fio 50GB

root@freenas:/mnt/Zpool-24TB-Mirror # fio --rw=readwrite --name=test --size=50000M --direct=1 --bs=1024k
test: (g=0): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.27
Starting 1 process
test: Laying out IO file (1 file / 50000MiB)
Jobs: 1 (f=1): [M(1)][0.0%][r=2031KiB/s,w=1015KiB/s][r=1,w=0 IOPS][eta 10h:24m:5Jobs: 1 (f=1): [M(1)][0.0%][r=3078KiB/s,w=4104KiB/s][r=3,w=4 IOPS][eta 05h:02m:5Jobs: 1 (f=1): [M(1)][0.1%][r=7175KiB/s,w=8200KiB/s][r=7,w=8 IOPS][eta 02h:40m:1Jobs: 1 (f=1): [M(1)][0.2%][r=40.8MiB/s,w=44.8MiB/s][r=40,w=44 IOPS][eta 44m:32sJobs: 1 (f=1): [M(1)][0.3%][r=20.9MiB/s,w=15.9MiB/s][r=20,w=15 IOPS][eta 39m:01sJobs: 1 (f=1): [M(1)][0.3%][r=9079KiB/s,w=9.85MiB/s][r=8,w=9 IOPS][eta 39m:32s]Jobs: 1 (f=1): [M(1)][0.5%][r=31.2MiB/s,w=30.2MiB/s][r=31,w=30 IOPS][eta 32m:36sJobs: 1 (f=1): [M(1)][0.5%][r=1022KiB/s][r=0 IOPS][eta 36m:03s]Jobs: 1 (f=1): [M(1)][0.5%][r=2029KiB/s,w=3044KiB/s][r=1,w=2 IOPS][eta 01h:02m:0Jobs: 1 (f=1): [M(1)][0.5%][r=2041KiB/s,w=5104KiB/s][r=1,w=4 IOPS][eta 01h:03m:3Jobs: 1 (f=1): [M(1)][0.6%][r=2056KiB/s,w=3084KiB/s][r=2,w=3 IOPS][eta 01h:05m:2Jobs: 1 (f=1): [M(1)][0.6%][r=2045KiB/s,w=1022KiB/s][r=1,w=0 IOPS][eta 01h:07m:3Jobs: 1 (f=1): [M(1)][0.6%][r=2029KiB/s,w=2029KiB/s][r=1,w=1 IOPS][eta 01h:09m:3Jobs: 1 (f=1): [M(1)][0.6%][r=3078KiB/s,w=4104KiB/s][r=3,w=4 IOPS][eta 01h:10m:4Jobs: 1 (f=1): [M(1)][0.6%][r=2045KiB/s,w=3068KiB/s][r=1,w=2 IOPS][eta 01h:12m:1Jobs: 1 (f=1): [M(1)][0.6%][r=2023KiB/s,w=3035KiB/s][r=1,w=2 IOPS][eta 01h:13m:4Jobs: 1 (f=1): [M(1)][1.7%][r=252MiB/s,w=278MiB/s][r=252,w=278 IOPS][eta 27m:28sJobs: 1 (f=1): [M(1)][2.6%][r=235MiB/s,w=249MiB/s][r=234,w=248 IOPS][eta 17m:53sJobs: 1 (f=1): [M(1)][3.3%][r=174MiB/s,w=184MiB/s][r=174,w=184 IOPS][eta 14m:26sJobs: 1 (f=1): [M(1)][3.9%][r=147MiB/s,w=147MiB/s][r=146,w=146 IOPS][eta 12m:37sJobs: 1 (f=1): [M(1)][4.4%][r=122MiB/s,w=121MiB/s][r=122,w=121 IOPS][eta 11m:31sJobs: 1 (f=1): [M(1)][4.9%][r=114MiB/s,w=105MiB/s][r=113,w=104 IOPS][eta 10m:46sJobs: 1 (f=1): [M(1)][5.2%][r=82.4MiB/s,w=92.4MiB/s][r=82,w=92 IOPS][eta 10m:18sJobs: 1 (f=1): [M(1)][5.5%][r=61.9MiB/s,w=83.8MiB/s][r=61,w=83 IOPS][eta 10m:01sJobs: 1 (f=1): [M(1)][5.8%][r=80.5MiB/s,w=76.5MiB/s][r=80,w=76 IOPS][eta 09m:42sJobs: 1 (f=1): [M(1)][6.1%][r=63.0MiB/s,w=71.0MiB/s][r=63,w=71 IOPS][eta 09m:30sJobs: 1 (f=1): [M(1)][6.4%][r=72.6MiB/s,w=65.7MiB/s][r=72,w=65 IOPS][eta 09m:19sJobs: 1 (f=1): [M(1)][6.6%][r=71.8MiB/s,w=62.8MiB/s][r=71,w=62 IOPS][eta 09m:08sJobs: 1 (f=1): [M(1)][6.9%][r=53.8MiB/s,w=59.8MiB/s][r=53,w=59 IOPS][eta 09m:02sJobs: 1 (f=1): [M(1)][7.1%][r=56.9MiB/s,w=57.9MiB/s][r=56,w=57 IOPS][eta 08m:57sJobs: 1 (f=1): [M(1)][7.3%][r=46.1MiB/s,w=55.1MiB/s][r=46,w=55 IOPS][eta 08m:53sJobs: 1 (f=1): [M(1)][7.5%][r=44.9MiB/s,w=54.9MiB/s][r=44,w=54 IOPS][eta 08m:50sJobs: 1 (f=1): [M(1)][7.7%][r=48.2MiB/s,w=52.2MiB/s][r=48,w=52 IOPS][eta 08m:47sJobs: 1 (f=1): [M(1)][7.9%][r=39.0MiB/s,w=52.1MiB/s][r=39,w=52 IOPS][eta 08m:46sJobs: 1 (f=1): [M(1)][8.1%][r=51.9MiB/s,w=50.9MiB/s][r=51,w=50 IOPS][eta 08m:43sJobs: 1 (f=1): [M(1)][8.3%][r=39.0MiB/s,w=49.0MiB/s][r=39,w=49 IOPS][eta 08m:42sJobs: 1 (f=1): [M(1)][8.5%][r=49.0MiB/s,w=49.0MiB/s][r=49,w=49 IOPS][eta 08m:39sJobs: 1 (f=1): [M(1)][8.7%][r=47.0MiB/s,w=49.0MiB/s][r=47,w=49 IOPS][eta 08m:37sJobs: 1 (f=1): [M(1)][8.8%][r=46.9MiB/s,w=48.9MiB/s][r=46,w=48 IOPS][eta 08m:35sJobs: 1 (f=1): [M(1)][9.1%][r=56.2MiB/s,w=49.1MiB/s][r=56,w=49 IOPS][eta 08m:32sJobs: 1 (f=1): [M(1)][9.2%][r=44.0MiB/s,w=49.0MiB/s][r=43,w=48 IOPS][eta 08m:31sJobs: 1 (f=1): [M(1)][9.4%][r=46.0MiB/s,w=48.0MiB/s][r=45,w=47 IOPS][eta 08m:29sJobs: 1 (f=1): [M(1)][9.6%][r=40.0MiB/s,w=49.0MiB/s][r=40,w=49 IOPS][eta 08m:28sJobs: 1 (f=1): [M(1)][9.8%][r=69.0MiB/s,w=48.0MiB/s][r=69,w=48 IOPS][eta 08m:24sJobs: 1 (f=1): [M(1)][10.1%][r=58.5MiB/s,w=48.6MiB/s][r=58,w=48 IOPS][eta 08m:21Jobs: 1 (f=1): [M(1)][10.2%][r=39.6MiB/s,w=48.6MiB/s][r=39,w=48 IOPS][eta 08m:20Jobs: 1 (f=1): [M(1)][10.4%][r=34.1MiB/s,w=48.1MiB/s][r=34,w=48 IOPS][eta 08m:20Jobs: 1 (f=1): [M(1)][10.6%][r=51.0MiB/s,w=48.0MiB/s][r=51,w=48 IOPS][eta 08m:18Jobs: 1 (f=1): [M(1)][10.8%][r=44.0MiB/s,w=48.0MiB/s][r=44,w=48 IOPS][eta 08m:16Jobs: 1 (f=1): [M(1)][11.0%][r=50.0MiB/s,w=47.0MiB/s][r=50,w=47 IOPS][eta 08m:15Jobs: 1 (f=1): [M(1)][11.2%][r=44.6MiB/s,w=47.5MiB/s][r=44,w=47 IOPS][eta 08m:13Jobs: 1 (f=1): [M(1)][11.4%][r=61.0MiB/s,w=48.0MiB/s][r=61,w=48 IOPS][eta 08m:10Jobs: 1 (f=1): [M(1)][11.6%][r=51.9MiB/s,w=46.9MiB/s][r=51,w=46 IOPS][eta 08m:09Jobs: 1 (f=1): [M(1)][11.8%][r=40.1MiB/s,w=47.1MiB/s][r=40,w=47 IOPS][eta 08m:08Jobs: 1 (f=1): [M(1)][11.9%][r=27.8MiB/s,w=47.6MiB/s][r=27,w=47 IOPS][eta 08m:08Jobs: 1 (f=1): [M(1)][12.1%][r=55.9MiB/s,w=46.9MiB/s][r=55,w=46 IOPS][eta 08m:06Jobs: 1 (f=1): [M(1)][12.3%][r=43.7MiB/s,w=47.7MiB/s][r=43,w=47 IOPS][eta 08m:05Jobs: 1 (f=1): [M(1)][12.5%][r=39.8MiB/s,w=46.7MiB/s][r=39,w=46 IOPS][eta 08m:04Jobs: 1 (f=1): [M(1)][12.6%][r=32.9MiB/s,w=47.8MiB/s][r=32,w=47 IOPS][eta 08m:04Jobs: 1 (f=1): [M(1)][12.8%][r=51.2MiB/s,w=46.1MiB/s][r=51,w=46 IOPS][eta 08m:02Jobs: 1 (f=1): [M(1)][13.0%][r=63.4MiB/s,w=46.6MiB/s][r=63,w=46 IOPS][eta 08m:00Jobs: 1 (f=1): [M(1)][13.2%][r=49.0MiB/s,w=47.0MiB/s][r=48,w=46 IOPS][eta 07m:58Jobs: 1 (f=1): [M(1)][13.4%][r=34.1MiB/s,w=47.1MiB/s][r=34,w=47 IOPS][eta 07m:58Jobs: 1 (f=1): [M(1)][13.6%][r=39.0MiB/s,w=47.0MiB/s][r=39,w=47 IOPS][eta 07m:57Jobs: 1 (f=1): [M(1)][13.8%][r=67.8MiB/s,w=46.9MiB/s][r=67,w=46 IOPS][eta 07m:54Jobs: 1 (f=1): [M(1)][14.0%][r=53.2MiB/s,w=46.1MiB/s][r=53,w=46 IOPS][eta 07m:53Jobs: 1 (f=1): [M(1)][14.2%][r=47.0MiB/s,w=48.0MiB/s][r=47,w=48 IOPS][eta 07m:51Jobs: 1 (f=1): [M(1)][14.4%][r=44.0MiB/s,w=46.0MiB/s][r=44,w=46 IOPS][eta 07m:51Jobs: 1 (f=1): [M(1)][14.6%][r=48.6MiB/s,w=44.6MiB/s][r=48,w=44 IOPS][eta 07m:49Jobs: 1 (f=1): [M(1)][14.7%][r=38.6MiB/s,w=46.5MiB/s][r=38,w=46 IOPS][eta 07m:49Jobs: 1 (f=1): [M(1)][14.9%][r=33.1MiB/s,w=48.1MiB/s][r=33,w=48 IOPS][eta 07m:48Jobs: 1 (f=1): [M(1)][15.1%][r=42.0MiB/s,w=48.0MiB/s][r=41,w=47 IOPS][eta 07m:47Jobs: 1 (f=1): [M(1)][15.3%][r=66.7MiB/s,w=46.8MiB/s][r=66,w=46 IOPS][eta 07m:45Jobs: 1 (f=1): [M(1)][15.5%][r=50.0MiB/s,w=47.0MiB/s][r=50,w=47 IOPS][eta 07m:43Jobs: 1 (f=1): [M(1)][15.7%][r=58.1MiB/s,w=47.0MiB/s][r=58,w=47 IOPS][eta 07m:41Jobs: 1 (f=1): [M(1)][15.9%][r=60.5MiB/s,w=47.6MiB/s][r=60,w=47 IOPS][eta 07m:39Jobs: 1 (f=1): [M(1)][16.1%][r=48.9MiB/s,w=47.9MiB/s][r=48,w=47 IOPS][eta 07m:38Jobs: 1 (f=1): [M(1)][16.3%][r=46.6MiB/s,w=47.6MiB/s][r=46,w=47 IOPS][eta 07m:37Jobs: 1 (f=1): [M(1)][16.5%][r=48.0MiB/s,w=48.0MiB/s][r=48,w=48 IOPS][eta 07m:35Jobs: 1 (f=1): [M(1)][16.7%][r=30.1MiB/s,w=47.1MiB/s][r=30,w=47 IOPS][eta 07m:35Jobs: 1 (f=1): [M(1)][16.8%][r=40.0MiB/s,w=48.0MiB/s][r=40,w=48 IOPS][eta 07m:34Jobs: 1 (f=1): [M(1)][17.0%][r=56.0MiB/s,w=47.0MiB/s][r=56,w=47 IOPS][eta 07m:33Jobs: 1 (f=1): [M(1)][17.2%][r=51.1MiB/s,w=47.0MiB/s][r=51,w=47 IOPS][eta 07m:31Jobs: 1 (f=1): [M(1)][17.5%][r=62.9MiB/s,w=46.9MiB/s][r=62,w=46 IOPS][eta 07m:29Jobs: 1 (f=1): [M(1)][17.6%][r=35.1MiB/s,w=47.1MiB/s][r=35,w=47 IOPS][eta 07m:29Jobs: 1 (f=1): [M(1)][17.8%][r=66.0MiB/s,w=46.0MiB/s][r=66,w=46 IOPS][eta 07m:27Jobs: 1 (f=1): [M(1)][18.0%][r=43.7MiB/s,w=46.6MiB/s][r=43,w=46 IOPS][eta 07m:26Jobs: 1 (f=1): [M(1)][18.2%][r=33.0MiB/s,w=47.0MiB/s][r=33,w=47 IOPS][eta 07m:25Jobs: 1 (f=1): [M(1)][18.4%][r=44.0MiB/s,w=47.0MiB/s][r=43,w=46 IOPS][eta 07m:24Jobs: 1 (f=1): [M(1)][18.6%][r=43.3MiB/s,w=47.2MiB/s][r=43,w=47 IOPS][eta 07m:23Jobs: 1 (f=1): [M(1)][18.8%][r=61.9MiB/s,w=46.9MiB/s][r=61,w=46 IOPS][eta 07m:21Jobs: 1 (f=1): [M(1)][19.0%][r=70.9MiB/s,w=47.0MiB/s][r=70,w=46 IOPS][eta 07m:19Jobs: 1 (f=1): [M(1)][19.2%][r=58.2MiB/s,w=47.1MiB/s][r=58,w=47 IOPS][eta 07m:17Jobs: 1 (f=1): [M(1)][19.4%][r=43.9MiB/s,w=47.9MiB/s][r=43,w=47 IOPS][eta 07m:16Jobs: 1 (f=1): [M(1)][19.6%][r=55.3MiB/s,w=47.2MiB/s][r=55,w=47 IOPS][eta 07m:15Jobs: 1 (f=1): [M(1)][19.8%][r=46.0MiB/s,w=47.0MiB/s][r=46,w=47 IOPS][eta 07m:14Jobs: 1 (f=1): [M(1)][20.0%][r=40.0MiB/s,w=46.0MiB/s][r=39,w=45 IOPS][eta 07m:13Jobs: 1 (f=1): [M(1)][20.1%][r=39.9MiB/s,w=44.9MiB/s][r=39,w=44 IOPS][eta 07m:12Jobs: 1 (f=1): [M(1)][20.3%][r=47.2MiB/s,w=47.2MiB/s][r=47,w=47 IOPS][eta 07m:11Jobs: 1 (f=1): [M(1)][20.5%][r=50.5MiB/s,w=45.6MiB/s][r=50,w=45 IOPS][eta 07m:10Jobs: 1 (f=1): [M(1)][20.7%][r=48.0MiB/s,w=47.0MiB/s][r=47,w=46 IOPS][eta 07m:09Jobs: 1 (f=1): [M(1)][20.9%][r=51.1MiB/s,w=48.0MiB/s][r=51,w=48 IOPS][eta 07m:07Jobs: 1 (f=1): [M(1)][21.1%][r=48.0MiB/s,w=47.0MiB/s][r=48,w=47 IOPS][eta 07m:06Jobs: 1 (f=1): [M(1)][21.3%][r=56.2MiB/s,w=47.3MiB/s][r=56,w=47 IOPS][eta 07m:05Jobs: 1 (f=1): [M(1)][21.5%][r=61.0MiB/s,w=47.0MiB/s][r=61,w=47 IOPS][eta 07m:03Jobs: 1 (f=1): [M(1)][21.7%][r=42.1MiB/s,w=47.1MiB/s][r=42,w=47 IOPS][eta 07m:02Jobs: 1 (f=1): [M(1)][21.9%][r=52.1MiB/s,w=46.1MiB/s][r=52,w=46 IOPS][eta 07m:01Jobs: 1 (f=1): [M(1)][22.1%][r=45.9MiB/s,w=46.9MiB/s][r=45,w=46 IOPS][eta 07m:00Jobs: 1 (f=1): [M(1)][22.3%][r=47.1MiB/s,w=48.1MiB/s][r=47,w=48 IOPS][eta 06m:59Jobs: 1 (f=1): [M(1)][22.4%][r=41.9MiB/s,w=46.9MiB/s][r=41,w=46 IOPS][eta 06m:58Jobs: 1 (f=1): [M(1)][22.6%][r=49.1MiB/s,w=46.1MiB/s][r=49,w=46 IOPS][eta 06m:57Jobs: 1 (f=1): [M(1)][22.8%][r=49.0MiB/s,w=47.0MiB/s][r=49,w=47 IOPS][eta 06m:56Jobs: 1 (f=1): [M(1)][23.0%][r=33.0MiB/s,w=47.0MiB/s][r=32,w=46 IOPS][eta 06m:55Jobs: 1 (f=1): [M(1)][23.1%][r=34.0MiB/s,w=47.0MiB/s][r=34,w=47 IOPS][eta 06m:55Jobs: 1 (f=1): [M(1)][23.3%][r=47.0MiB/s,w=47.0MiB/s][r=47,w=47 IOPS][eta 06m:54Jobs: 1 (f=1): [M(1)][23.5%][r=43.0MiB/s,w=46.0MiB/s][r=42,w=45 IOPS][eta 06m:53Jobs: 1 (f=1): [M(1)][23.7%][r=41.9MiB/s,w=46.9MiB/s][r=41,w=46 IOPS][eta 06m:52Jobs: 1 (f=1): [M(1)][23.8%][r=36.1MiB/s,w=46.1MiB/s][r=36,w=46 IOPS][eta 06m:52Jobs: 1 (f=1): [M(1)][24.1%][r=56.7MiB/s,w=47.7MiB/s][r=56,w=47 IOPS][eta 06m:50Jobs: 1 (f=1): [M(1)][24.3%][r=40.1MiB/s,w=47.1MiB/s][r=40,w=47 IOPS][eta 06m:49Jobs: 1 (f=1): [M(1)][24.4%][r=51.0MiB/s,w=47.0MiB/s][r=51,w=47 IOPS][eta 06m:48Jobs: 1 (f=1): [M(1)][24.7%][r=58.2MiB/s,w=47.4MiB/s][r=58,w=47 IOPS][eta 06m:46Jobs: 1 (f=1): [M(1)][24.9%][r=56.1MiB/s,w=47.1MiB/s][r=56,w=47 IOPS][eta 06m:45Jobs: 1 (f=1): [M(1)][25.1%][r=56.1MiB/s,w=48.2MiB/s][r=56,w=48 IOPS][eta 06m:43Jobs: 1 (f=1): [M(1)][25.3%][r=39.9MiB/s,w=47.9MiB/s][r=39,w=47 IOPS][eta 06m:42Jobs: 1 (f=1): [M(1)][25.5%][r=56.1MiB/s,w=47.0MiB/s][r=56,w=47 IOPS][eta 06m:41Jobs: 1 (f=1): [M(1)][25.7%][r=53.2MiB/s,w=48.1MiB/s][r=53,w=48 IOPS][eta 06m:40Jobs: 1 (f=1): [M(1)][25.9%][r=57.9MiB/s,w=47.0MiB/s][r=57,w=46 IOPS][eta 06m:38Jobs: 1 (f=1): [M(1)][26.0%][r=36.0MiB/s,w=48.0MiB/s][r=36,w=48 IOPS][eta 06m:38Jobs: 1 (f=1): [M(1)][26.3%][r=49.1MiB/s,w=48.1MiB/s][r=49,w=48 IOPS][eta 06m:36Jobs: 1 (f=1): [M(1)][26.4%][r=51.0MiB/s,w=47.0MiB/s][r=51,w=47 IOPS][eta 06m:35Jobs: 1 (f=1): [M(1)][26.6%][r=37.0MiB/s,w=48.0MiB/s][r=37,w=48 IOPS][eta 06m:35Jobs: 1 (f=1): [M(1)][26.8%][r=42.6MiB/s,w=47.6MiB/s][r=42,w=47 IOPS][eta 06m:34Jobs: 1 (f=1): [M(1)][27.0%][r=64.3MiB/s,w=46.5MiB/s][r=64,w=46 IOPS][eta 06m:32Jobs: 1 (f=1): [M(1)][27.2%][r=40.0MiB/s,w=48.0MiB/s][r=39,w=47 IOPS][eta 06m:31Jobs: 1 (f=1): [M(1)][27.4%][r=52.1MiB/s,w=48.1MiB/s][r=52,w=48 IOPS][eta 06m:30Jobs: 1 (f=1): [M(1)][27.6%][r=33.0MiB/s,w=48.0MiB/s][r=33,w=48 IOPS][eta 06m:29Jobs: 1 (f=1): [M(1)][27.7%][r=55.9MiB/s,w=48.0MiB/s][r=55,w=47 IOPS][eta 06m:28Jobs: 1 (f=1): [M(1)][27.9%][r=50.9MiB/s,w=48.0MiB/s][r=50,w=47 IOPS][eta 06m:27Jobs: 1 (f=1): [M(1)][28.2%][r=51.9MiB/s,w=47.9MiB/s][r=51,w=47 IOPS][eta 06m:25Jobs: 1 (f=1): [M(1)][28.4%][r=43.2MiB/s,w=47.2MiB/s][r=43,w=47 IOPS][eta 06m:24Jobs: 1 (f=1): [M(1)][28.5%][r=38.0MiB/s,w=48.0MiB/s][r=38,w=48 IOPS][eta 06m:24Jobs: 1 (f=1): [M(1)][28.7%][r=52.2MiB/s,w=47.1MiB/s][r=52,w=47 IOPS][eta 06m:22Jobs: 1 (f=1): [M(1)][28.9%][r=50.9MiB/s,w=47.0MiB/s][r=50,w=46 IOPS][eta 06m:21Jobs: 1 (f=1): [M(1)][29.1%][r=50.9MiB/s,w=46.9MiB/s][r=50,w=46 IOPS][eta 06m:20Jobs: 1 (f=1): [M(1)][29.3%][r=41.1MiB/s,w=48.1MiB/s][r=41,w=48 IOPS][eta 06m:19Jobs: 1 (f=1): [M(1)][29.5%][r=52.1MiB/s,w=47.0MiB/s][r=52,w=47 IOPS][eta 06m:18Jobs: 1 (f=1): [M(1)][29.7%][r=37.1MiB/s,w=47.1MiB/s][r=37,w=47 IOPS][eta 06m:17Jobs: 1 (f=1): [M(1)][29.9%][r=54.9MiB/s,w=46.9MiB/s][r=54,w=46 IOPS][eta 06m:16Jobs: 1 (f=1): [M(1)][30.0%][r=47.0MiB/s,w=47.0MiB/s][r=47,w=47 IOPS][eta 06m:15Jobs: 1 (f=1): [M(1)][30.2%][r=31.0MiB/s,w=47.0MiB/s][r=31,w=47 IOPS][eta 06m:15Jobs: 1 (f=1): [M(1)][30.4%][r=53.8MiB/s,w=47.8MiB/s][r=53,w=47 IOPS][eta 06m:13Jobs: 1 (f=1): [M(1)][30.6%][r=53.6MiB/s,w=47.6MiB/s][r=53,w=47 IOPS][eta 06m:12Jobs: 1 (f=1): [M(1)][30.8%][r=39.9MiB/s,w=47.9MiB/s][r=39,w=47 IOPS][eta 06m:11Jobs: 1 (f=1): [M(1)][31.0%][r=51.2MiB/s,w=48.1MiB/s][r=51,w=48 IOPS][eta 06m:10Jobs: 1 (f=1): [M(1)][31.2%][r=36.0MiB/s,w=49.0MiB/s][r=36,w=49 IOPS][eta 06m:09Jobs: 1 (f=1): [M(1)][31.3%][r=58.0MiB/s,w=47.0MiB/s][r=58,w=47 IOPS][eta 06m:08Jobs: 1 (f=1): [M(1)][31.6%][r=57.1MiB/s,w=47.0MiB/s][r=57,w=47 IOPS][eta 06m:06Jobs: 1 (f=1): [M(1)][31.8%][r=52.0MiB/s,w=48.0MiB/s][r=52,w=48 IOPS][eta 06m:05Jobs: 1 (f=1): [M(1)][32.0%][r=47.9MiB/s,w=47.9MiB/s][r=47,w=47 IOPS][eta 06m:04Jobs: 1 (f=1): [M(1)][32.1%][r=46.7MiB/s,w=46.7MiB/s][r=46,w=46 IOPS][eta 06m:03Jobs: 1 (f=1): [M(1)][32.3%][r=25.0MiB/s,w=47.0MiB/s][r=25,w=47 IOPS][eta 06m:03Jobs: 1 (f=1): [M(1)][32.5%][r=53.6MiB/s,w=46.6MiB/s][r=53,w=46 IOPS][eta 06m:02Jobs: 1 (f=1): [M(1)][32.6%][r=43.9MiB/s,w=46.9MiB/s][r=43,w=46 IOPS][eta 06m:01Jobs: 1 (f=1): [M(1)][32.8%][r=40.1MiB/s,w=46.1MiB/s][r=40,w=46 IOPS][eta 06m:00Jobs: 1 (f=1): [M(1)][33.0%][r=53.1MiB/s,w=47.0MiB/s][r=53,w=47 IOPS][eta 05m:59Jobs: 1 (f=1): [M(1)][33.3%][r=73.1MiB/s,w=46.0MiB/s][r=73,w=46 IOPS][eta 05m:57Jobs: 1 (f=1): [M(1)][33.5%][r=47.9MiB/s,w=45.9MiB/s][r=47,w=45 IOPS][eta 05m:56Jobs: 1 (f=1): [M(1)][33.6%][r=48.0MiB/s,w=47.0MiB/s][r=48,w=47 IOPS][eta 05m:55Jobs: 1 (f=1): [M(1)][33.8%][r=48.0MiB/s,w=47.0MiB/s][r=48,w=47 IOPS][eta 05m:54Jobs: 1 (f=1): [M(1)][34.0%][r=47.0MiB/s,w=47.0MiB/s][r=47,w=47 IOPS][eta 05m:53Jobs: 1 (f=1): [M(1)][34.2%][r=48.3MiB/s,w=47.3MiB/s][r=48,w=47 IOPS][eta 05m:52Jobs: 1 (f=1): [M(1)][34.4%][r=53.7MiB/s,w=47.8MiB/s][r=53,w=47 IOPS][eta 05m:51Jobs: 1 (f=1): [M(1)][34.5%][r=29.8MiB/s,w=13.9MiB/s][r=29,w=13 IOPS][eta 05m:51Jobs: 1 (f=1): [M(1)][34.5%][r=3056KiB/s][r=2 IOPS][eta 05m:53s]Jobs: 1 (f=1): [M(1)][35.3%][r=17.0MiB/s,w=19.0MiB/s][r=17,w=19 IOPS][eta 08m:01Jobs: 1 (f=1): [M(1)][35.3%][r=1024KiB/s][r=1 IOPS][eta 08m:03s]Jobs: 1 (f=1): [M(1)][35.7%][r=86.9MiB/s,w=87.9MiB/s][r=86,w=87 IOPS][eta 08m:02Jobs: 1 (f=1): [M(1)][35.7%][r=4067KiB/s,w=3050KiB/s][r=3,w=2 IOPS][eta 08m:04s]Jobs: 1 (f=1): [M(1)][36.0%][r=33.0MiB/s,w=37.8MiB/s][r=32,w=37 IOPS][eta 08m:15Jobs: 1 (f=1): [M(1)][36.1%][r=30.9MiB/s,w=36.9MiB/s][r=30,w=36 IOPS][eta 08m:14Jobs: 1 (f=1): [M(1)][36.1%][r=2056KiB/s,w=3084KiB/s][r=2,w=3 IOPS][eta 08m:15s]Jobs: 1 (f=1): [M(1)][36.8%][r=161MiB/s,w=164MiB/s][r=161,w=164 IOPS][eta 08m:03Jobs: 1 (f=1): [M(1)][37.4%][r=140MiB/s,w=137MiB/s][r=139,w=136 IOPS][eta 07m:53Jobs: 1 (f=1): [M(1)][37.6%][r=62.0MiB/s,w=55.1MiB/s][r=62,w=55 IOPS][eta 07m:50Jobs: 1 (f=1): [M(1)][37.7%][r=27.8MiB/s,w=23.8MiB/s][r=27,w=23 IOPS][eta 07m:49Jobs: 1 (f=1): [M(1)][37.7%][r=2043KiB/s,w=1021KiB/s][r=1,w=0 IOPS][eta 07m:51s]Jobs: 1 (f=1): [M(1)][37.8%][r=11.0MiB/s,w=5104KiB/s][r=10,w=4 IOPS][eta 08m:02sJobs: 1 (f=1): [M(1)][38.0%][r=52.4MiB/s,w=62.1MiB/s][r=52,w=62 IOPS][eta 07m:59Jobs: 1 (f=1): [M(1)][38.3%][r=62.1MiB/s,w=41.7MiB/s][r=62,w=41 IOPS][eta 07m:56Jobs: 1 (f=1): [M(1)][38.6%][r=62.5MiB/s,w=67.4MiB/s][r=62,w=67 IOPS][eta 07m:52Jobs: 1 (f=1): [M(1)][38.9%][r=61.9MiB/s,w=58.0MiB/s][r=61,w=58 IOPS][eta 07m:49Jobs: 1 (f=1): [M(1)][38.9%][r=22.0MiB/s,w=24.0MiB/s][r=21,w=23 IOPS][eta 07m:49Jobs: 1 (f=1): [M(1)][39.0%][r=3075KiB/s,w=3075KiB/s][r=3,w=3 IOPS][eta 07m:50s]Jobs: 1 (f=1): [M(1)][39.0%][r=17.6MiB/s,w=14.6MiB/s][r=17,w=14 IOPS][eta 07m:57Jobs: 1 (f=1): [M(1)][39.2%][r=61.8MiB/s,w=63.8MiB/s][r=61,w=63 IOPS][eta 07m:54Jobs: 1 (f=1): [M(1)][39.5%][r=61.4MiB/s,w=59.5MiB/s][r=61,w=59 IOPS][eta 07m:50Jobs: 1 (f=1): [M(1)][39.8%][r=62.1MiB/s,w=72.8MiB/s][r=62,w=72 IOPS][eta 07m:46Jobs: 1 (f=1): [M(1)][40.0%][r=63.0MiB/s,w=63.0MiB/s][r=62,w=62 IOPS][eta 07m:43Jobs: 1 (f=1): [M(1)][40.1%][r=16.1MiB/s,w=12.0MiB/s][r=16,w=12 IOPS][eta 07m:43Jobs: 1 (f=1): [M(1)][40.1%][r=2052KiB/s,w=1026KiB/s][r=2,w=1 IOPS][eta 07m:45s]Jobs: 1 (f=1): [M(1)][40.4%][r=50.4MiB/s,w=58.2MiB/s][r=50,w=58 IOPS][eta 07m:48Jobs: 1 (f=1): [M(1)][40.7%][r=61.1MiB/s,w=60.1MiB/s][r=61,w=60 IOPS][eta 07m:44Jobs: 1 (f=1): [M(1)][40.8%][r=63.0MiB/s,w=33.9MiB/s][r=63,w=33 IOPS][eta 07m:42Jobs: 1 (f=1): [M(1)][41.1%][r=62.2MiB/s,w=77.7MiB/s][r=62,w=77 IOPS][eta 07m:38Jobs: 1 (f=1): [M(1)][41.4%][r=41.3MiB/s,w=61.0MiB/s][r=41,w=60 IOPS][eta 07m:35Jobs: 1 (f=1): [M(1)][41.4%][r=27.1MiB/s,w=29.1MiB/s][r=27,w=29 IOPS][eta 07m:35Jobs: 1 (f=1): [M(1)][41.5%][r=2045KiB/s][r=1 IOPS][eta 07m:36s]Jobs: 1 (f=1): [M(1)][41.5%][r=20.7MiB/s,w=22.7MiB/s][r=20,w=22 IOPS][eta 07m:36Jobs: 1 (f=1): [M(1)][41.6%][r=3056KiB/s,w=3056KiB/s][r=2,w=2 IOPS][eta 07m:37s]Jobs: 1 (f=1): [M(1)][41.6%][r=23.0MiB/s,w=22.0MiB/s][r=22,w=21 IOPS][eta 07m:37Jobs: 1 (f=1): [M(1)][41.7%][r=1017KiB/s][r=0 IOPS][eta 07m:38s]Jobs: 1 (f=1): [M(1)][41.7%][r=14.8MiB/s,w=16.8MiB/s][r=14,w=16 IOPS][eta 07m:38Jobs: 1 (f=1): [M(1)][41.8%][r=4100KiB/s,w=4100KiB/s][r=4,w=4 IOPS][eta 07m:39s]Jobs: 1 (f=1): [M(1)][41.8%][r=16.8MiB/s,w=11.8MiB/s][r=16,w=11 IOPS][eta 07m:40Jobs: 1 (f=1): [M(1)][42.1%][r=83.5MiB/s,w=76.7MiB/s][r=83,w=76 IOPS][eta 07m:35Jobs: 1 (f=1): [M(1)][42.3%][r=62.0MiB/s,w=53.3MiB/s][r=62,w=53 IOPS][eta 07m:32Jobs: 1 (f=1): [M(1)][42.5%][r=33.3MiB/s,w=32.3MiB/s][r=33,w=32 IOPS][eta 07m:31Jobs: 1 (f=1): [M(1)][42.5%][r=4104KiB/s,w=6156KiB/s][r=4,w=6 IOPS][eta 07m:32s]Jobs: 1 (f=1): [M(1)][42.6%][r=16.6MiB/s,w=9000KiB/s][r=16,w=8 IOPS][eta 07m:34sJobs: 1 (f=1): [M(1)][42.7%][r=30.8MiB/s,w=22.9MiB/s][r=30,w=22 IOPS][eta 07m:34Jobs: 1 (f=1): [M(1)][42.9%][r=34.2MiB/s,w=39.1MiB/s][r=34,w=39 IOPS][eta 07m:32Jobs: 1 (f=1): [M(1)][43.1%][r=62.1MiB/s,w=42.7MiB/s][r=62,w=42 IOPS][eta 07m:29Jobs: 1 (f=1): [M(1)][43.3%][r=63.0MiB/s,w=72.7MiB/s][r=63,w=72 IOPS][eta 07m:26Jobs: 1 (f=1): [M(1)][43.6%][r=61.9MiB/s,w=57.0MiB/s][r=61,w=57 IOPS][eta 07m:23Jobs: 1 (f=1): [M(1)][43.7%][r=25.0MiB/s,w=26.0MiB/s][r=25,w=26 IOPS][eta 07m:22Jobs: 1 (f=1): [M(1)][43.7%][r=3056KiB/s][r=2 IOPS][eta 07m:23s]Jobs: 1 (f=1): [M(1)][43.9%][r=46.7MiB/s,w=31.8MiB/s][r=46,w=31 IOPS][eta 07m:26Jobs: 1 (f=1): [M(1)][44.1%][r=62.6MiB/s,w=53.9MiB/s][r=62,w=53 IOPS][eta 07m:23Jobs: 1 (f=1): [M(1)][44.4%][r=62.1MiB/s,w=65.0MiB/s][r=62,w=64 IOPS][eta 07m:19Jobs: 1 (f=1): [M(1)][44.7%][r=63.2MiB/s,w=70.0MiB/s][r=63,w=69 IOPS][eta 07m:16Jobs: 1 (f=1): [M(1)][44.9%][r=54.1MiB/s,w=41.3MiB/s][r=54,w=41 IOPS][eta 07m:13Jobs: 1 (f=1): [M(1)][44.9%][r=2043KiB/s,w=1021KiB/s][r=1,w=0 IOPS][eta 07m:15s]Jobs: 1 (f=1): [M(1)][45.1%][r=29.4MiB/s,w=33.3MiB/s][r=29,w=33 IOPS][eta 07m:18Jobs: 1 (f=1): [M(1)][45.4%][r=62.1MiB/s,w=63.1MiB/s][r=62,w=63 IOPS][eta 07m:15Jobs: 1 (f=1): [M(1)][45.6%][r=61.8MiB/s,w=88.2MiB/s][r=61,w=88 IOPS][eta 07m:11Jobs: 1 (f=1): [M(1)][45.9%][r=62.4MiB/s,w=82.8MiB/s][r=62,w=82 IOPS][eta 07m:07Jobs: 1 (f=1): [M(1)][46.2%][r=62.3MiB/s,w=58.4MiB/s][r=62,w=58 IOPS][eta 07m:04Jobs: 1 (f=1): [M(1)][46.2%][r=3072KiB/s,w=3072KiB/s][r=3,w=3 IOPS][eta 07m:05s]Jobs: 1 (f=1): [M(1)][46.3%][r=12.0MiB/s,w=8208KiB/s][r=12,w=8 IOPS][eta 07m:11sJobs: 1 (f=1): [M(1)][46.5%][r=45.9MiB/s,w=38.1MiB/s][r=45,w=38 IOPS][eta 07m:09Jobs: 1 (f=1): [M(1)][46.8%][r=61.6MiB/s,w=70.4MiB/s][r=61,w=70 IOPS][eta 07m:06Jobs: 1 (f=1): [M(1)][47.1%][r=62.1MiB/s,w=72.9MiB/s][r=62,w=72 IOPS][eta 07m:02Jobs: 1 (f=1): [M(1)][47.4%][r=62.7MiB/s,w=63.7MiB/s][r=62,w=63 IOPS][eta 06m:59Jobs: 1 (f=1): [M(1)][47.5%][r=44.8MiB/s,w=59.4MiB/s][r=44,w=59 IOPS][eta 06m:57Jobs: 1 (f=1): [M(1)][47.6%][r=3078KiB/s,w=4104KiB/s][r=3,w=4 IOPS][eta 06m:58s]Jobs: 1 (f=1): [M(1)][47.8%][r=45.6MiB/s,w=42.7MiB/s][r=45,w=42 IOPS][eta 07m:01Jobs: 1 (f=1): [M(1)][48.1%][r=62.0MiB/s,w=44.6MiB/s][r=62,w=44 IOPS][eta 06m:58Jobs: 1 (f=1): [M(1)][48.3%][r=61.4MiB/s,w=64.3MiB/s][r=61,w=64 IOPS][eta 06m:56Jobs: 1 (f=1): [M(1)][48.5%][r=62.2MiB/s,w=52.5MiB/s][r=62,w=52 IOPS][eta 06m:53Jobs: 1 (f=1): [M(1)][48.7%][r=50.2MiB/s,w=45.3MiB/s][r=50,w=45 IOPS][eta 06m:51Jobs: 1 (f=1): [M(1)][48.7%][r=4096KiB/s,w=7168KiB/s][r=4,w=7 IOPS][eta 06m:52s]Jobs: 1 (f=1): [M(1)][49.0%][r=47.2MiB/s,w=44.3MiB/s][r=47,w=44 IOPS][eta 06m:54Jobs: 1 (f=1): [M(1)][49.2%][r=60.1MiB/s,w=72.9MiB/s][r=60,w=72 IOPS][eta 06m:51Jobs: 1 (f=1): [M(1)][49.5%][r=62.5MiB/s,w=69.3MiB/s][r=62,w=69 IOPS][eta 06m:47Jobs: 1 (f=1): [M(1)][49.8%][r=62.6MiB/s,w=70.3MiB/s][r=62,w=70 IOPS][eta 06m:44Jobs: 1 (f=1): [M(1)][49.9%][r=42.3MiB/s,w=48.2MiB/s][r=42,w=48 IOPS][eta 06m:42Jobs: 1 (f=1): [M(1)][49.9%][r=3068KiB/s][r=2 IOPS][eta 06m:43s]Jobs: 1 (f=1): [M(1)][50.1%][r=13.0MiB/s,w=15.0MiB/s][r=13,w=15 IOPS][eta 06m:47Jobs: 1 (f=1): [M(1)][50.2%][r=31.5MiB/s,w=30.5MiB/s][r=31,w=30 IOPS][eta 06m:46Jobs: 1 (f=1): [M(1)][50.4%][r=62.5MiB/s,w=51.8MiB/s][r=62,w=51 IOPS][eta 06m:43Jobs: 1 (f=1): [M(1)][50.7%][r=62.1MiB/s,w=80.6MiB/s][r=62,w=80 IOPS][eta 06m:39Jobs: 1 (f=1): [M(1)][51.0%][r=66.0MiB/s,w=70.0MiB/s][r=66,w=69 IOPS][eta 06m:36Jobs: 1 (f=1): [M(1)][51.5%][r=128MiB/s,w=131MiB/s][r=128,w=131 IOPS][eta 06m:29Jobs: 1 (f=1): [M(1)][51.6%][r=3078KiB/s,w=2052KiB/s][r=3,w=2 IOPS][eta 06m:29s]Jobs: 1 (f=1): [M(1)][53.8%][r=577MiB/s,w=579MiB/s][r=577,w=579 IOPS][eta 05m:56Jobs: 1 (f=1): [M(1)][54.2%][r=88.6MiB/s,w=87.6MiB/s][r=88,w=87 IOPS][eta 05m:51Jobs: 1 (f=1): [M(1)][54.2%][r=2048KiB/s][r=2 IOPS][eta 05m:52s]Jobs: 1 (f=1): [M(1)][54.5%][r=12.0MiB/s,w=9206KiB/s][r=11,w=8 IOPS][eta 06m:06sJobs: 1 (f=1): [M(1)][56.6%][r=495MiB/s,w=504MiB/s][r=494,w=504 IOPS][eta 05m:37Jobs: 1 (f=1): [M(1)][56.9%][r=81.1MiB/s,w=77.2MiB/s][r=81,w=77 IOPS][eta 05m:33Jobs: 1 (f=1): [M(1)][57.1%][r=62.5MiB/s,w=58.7MiB/s][r=62,w=58 IOPS][eta 05m:31Jobs: 1 (f=1): [M(1)][57.4%][r=62.1MiB/s,w=41.7MiB/s][r=62,w=41 IOPS][eta 05m:28Jobs: 1 (f=1): [M(1)][57.4%][r=5120KiB/s,w=5120KiB/s][r=5,w=5 IOPS][eta 05m:29s]Jobs: 1 (f=1): [M(1)][57.5%][r=10.0MiB/s,w=10.0MiB/s][r=10,w=10 IOPS][eta 05m:39Jobs: 1 (f=1): [M(1)][57.8%][r=61.2MiB/s,w=51.6MiB/s][r=61,w=51 IOPS][eta 05m:36Jobs: 1 (f=1): [M(1)][58.0%][r=62.0MiB/s,w=46.5MiB/s][r=62,w=46 IOPS][eta 05m:34Jobs: 1 (f=1): [M(1)][58.3%][r=62.1MiB/s,w=63.0MiB/s][r=62,w=63 IOPS][eta 05m:31Jobs: 1 (f=1): [M(1)][58.5%][r=61.7MiB/s,w=69.5MiB/s][r=61,w=69 IOPS][eta 05m:28Jobs: 1 (f=1): [M(1)][58.6%][r=25.2MiB/s,w=18.1MiB/s][r=25,w=18 IOPS][eta 05m:28Jobs: 1 (f=1): [M(1)][58.6%][r=4091KiB/s,w=3068KiB/s][r=3,w=2 IOPS][eta 05m:28s]Jobs: 1 (f=1): [M(1)][58.8%][r=30.8MiB/s,w=26.0MiB/s][r=30,w=26 IOPS][eta 05m:30Jobs: 1 (f=1): [M(1)][59.1%][r=62.6MiB/s,w=84.1MiB/s][r=62,w=84 IOPS][eta 05m:26Jobs: 1 (f=1): [M(1)][59.3%][r=61.4MiB/s,w=54.6MiB/s][r=61,w=54 IOPS][eta 05m:24Jobs: 1 (f=1): [M(1)][59.6%][r=62.4MiB/s,w=78.7MiB/s][r=62,w=78 IOPS][eta 05m:21Jobs: 1 (f=1): [M(1)][59.9%][r=61.9MiB/s,w=97.2MiB/s][r=61,w=97 IOPS][eta 05m:17Jobs: 1 (f=1): [M(1)][60.1%][r=28.5MiB/s,w=27.6MiB/s][r=28,w=27 IOPS][eta 05m:16Jobs: 1 (f=1): [M(1)][60.0%][r=2045KiB/s][r=1 IOPS][eta 05m:17s]Jobs: 1 (f=1): [M(1)][60.3%][r=30.3MiB/s,w=24.4MiB/s][r=30,w=24 IOPS][eta 05m:19Jobs: 1 (f=1): [M(1)][60.5%][r=62.6MiB/s,w=50.9MiB/s][r=62,w=50 IOPS][eta 05m:17Jobs: 1 (f=1): [M(1)][60.8%][r=61.9MiB/s,w=74.9MiB/s][r=61,w=74 IOPS][eta 05m:14Jobs: 1 (f=1): [M(1)][61.1%][r=61.3MiB/s,w=61.3MiB/s][r=61,w=61 IOPS][eta 05m:11Jobs: 1 (f=1): [M(1)][61.3%][r=62.6MiB/s,w=49.9MiB/s][r=62,w=49 IOPS][eta 05m:09Jobs: 1 (f=1): [M(1)][61.3%][r=3093KiB/s,w=1031KiB/s][r=3,w=1 IOPS][eta 05m:10s]Jobs: 1 (f=1): [M(1)][61.4%][r=30.1MiB/s,w=25.2MiB/s][r=30,w=25 IOPS][eta 05m:12Jobs: 1 (f=1): [M(1)][61.7%][r=62.0MiB/s,w=81.4MiB/s][r=62,w=81 IOPS][eta 05m:09Jobs: 1 (f=1): [M(1)][61.9%][r=61.1MiB/s,w=65.0MiB/s][r=61,w=64 IOPS][eta 05m:06Jobs: 1 (f=1): [M(1)][62.2%][r=62.9MiB/s,w=50.1MiB/s][r=62,w=50 IOPS][eta 05m:03Jobs: 1 (f=1): [M(1)][62.4%][r=62.1MiB/s,w=61.1MiB/s][r=62,w=61 IOPS][eta 05m:01Jobs: 1 (f=1): [M(1)][62.5%][r=5109KiB/s,w=2043KiB/s][r=4,w=1 IOPS][eta 05m:01s]Jobs: 1 (f=1): [M(1)][62.5%][r=12.8MiB/s,w=8055KiB/s][r=12,w=7 IOPS][eta 05m:04sJobs: 1 (f=1): [M(1)][62.8%][r=62.5MiB/s,w=43.9MiB/s][r=62,w=43 IOPS][eta 05m:01Jobs: 1 (f=1): [M(1)][63.0%][r=62.2MiB/s,w=67.1MiB/s][r=62,w=67 IOPS][eta 04m:59Jobs: 1 (f=1): [M(1)][63.3%][r=62.1MiB/s,w=60.2MiB/s][r=62,w=60 IOPS][eta 04m:56Jobs: 1 (f=1): [M(1)][63.6%][r=63.0MiB/s,w=58.1MiB/s][r=62,w=58 IOPS][eta 04m:53Jobs: 1 (f=1): [M(1)][63.8%][r=37.2MiB/s,w=57.8MiB/s][r=37,w=57 IOPS][eta 04m:51Jobs: 1 (f=1): [M(1)][63.7%][r=3093KiB/s,w=1031KiB/s][r=3,w=1 IOPS][eta 04m:52s]Jobs: 1 (f=1): [M(1)][63.8%][r=22.7MiB/s,w=5044KiB/s][r=22,w=4 IOPS][eta 04m:54sJobs: 1 (f=1): [M(1)][64.1%][r=62.3MiB/s,w=66.1MiB/s][r=62,w=66 IOPS][eta 04m:51Jobs: 1 (f=1): [M(1)][64.3%][r=63.0MiB/s,w=59.1MiB/s][r=62,w=59 IOPS][eta 04m:49Jobs: 1 (f=1): [M(1)][64.7%][r=62.0MiB/s,w=82.4MiB/s][r=62,w=82 IOPS][eta 04m:45Jobs: 1 (f=1): [M(1)][64.9%][r=53.1MiB/s,w=80.7MiB/s][r=53,w=80 IOPS][eta 04m:43Jobs: 1 (f=1): [M(1)][65.1%][r=33.7MiB/s,w=36.6MiB/s][r=33,w=36 IOPS][eta 04m:41Jobs: 1 (f=1): [M(1)][65.1%][r=2050KiB/s,w=2050KiB/s][r=2,w=2 IOPS][eta 04m:42s]Jobs: 1 (f=1): [M(1)][65.1%][r=9216KiB/s,w=11.0MiB/s][r=9,w=11 IOPS][eta 04m:44sJobs: 1 (f=1): [M(1)][65.2%][r=5074KiB/s,w=3044KiB/s][r=4,w=2 IOPS][eta 04m:44s]Jobs: 1 (f=1): [M(1)][65.4%][r=39.4MiB/s,w=44.3MiB/s][r=39,w=44 IOPS][eta 04m:42Jobs: 1 (f=1): [M(1)][65.6%][r=66.3MiB/s,w=51.7MiB/s][r=66,w=51 IOPS][eta 04m:40Jobs: 1 (f=1): [M(1)][66.6%][r=250MiB/s,w=256MiB/s][r=249,w=256 IOPS][eta 04m:28Jobs: 1 (f=1): [M(1)][67.2%][r=157MiB/s,w=151MiB/s][r=156,w=151 IOPS][eta 04m:21Jobs: 1 (f=1): [M(1)][67.5%][r=58.1MiB/s,w=56.1MiB/s][r=58,w=56 IOPS][eta 04m:18Jobs: 1 (f=1): [M(1)][67.5%][r=2050KiB/s,w=1025KiB/s][r=2,w=1 IOPS][eta 04m:19s]Jobs: 1 (f=1): [M(1)][67.7%][r=22.6MiB/s,w=16.7MiB/s][r=22,w=16 IOPS][eta 04m:22Jobs: 1 (f=1): [M(1)][68.0%][r=66.0MiB/s,w=73.7MiB/s][r=66,w=73 IOPS][eta 04m:19Jobs: 1 (f=1): [M(1)][68.3%][r=82.8MiB/s,w=80.9MiB/s][r=82,w=80 IOPS][eta 04m:16Jobs: 1 (f=1): [M(1)][68.8%][r=118MiB/s,w=114MiB/s][r=118,w=114 IOPS][eta 04m:11Jobs: 1 (f=1): [M(1)][69.1%][r=50.1MiB/s,w=59.0MiB/s][r=50,w=58 IOPS][eta 04m:08Jobs: 1 (f=1): [M(1)][69.0%][r=3035KiB/s,w=3035KiB/s][r=2,w=2 IOPS][eta 04m:09s]Jobs: 1 (f=1): [M(1)][69.1%][r=13.8MiB/s,w=14.8MiB/s][r=13,w=14 IOPS][eta 04m:11Jobs: 1 (f=1): [M(1)][69.2%][r=28.2MiB/s,w=21.4MiB/s][r=28,w=21 IOPS][eta 04m:10Jobs: 1 (f=1): [M(1)][69.5%][r=62.0MiB/s,w=52.3MiB/s][r=62,w=52 IOPS][eta 04m:07Jobs: 1 (f=1): [M(1)][69.8%][r=61.5MiB/s,w=67.3MiB/s][r=61,w=67 IOPS][eta 04m:05Jobs: 1 (f=1): [M(1)][70.1%][r=62.1MiB/s,w=71.8MiB/s][r=62,w=71 IOPS][eta 04m:02Jobs: 1 (f=1): [M(1)][70.3%][r=54.9MiB/s,w=58.9MiB/s][r=54,w=58 IOPS][eta 04m:00Jobs: 1 (f=1): [M(1)][70.3%][r=4100KiB/s,w=3075KiB/s][r=4,w=3 IOPS][eta 04m:00s]Jobs: 1 (f=1): [M(1)][74.7%][r=1099MiB/s,w=1107MiB/s][r=1098,w=1106 IOPS][eta 03Jobs: 1 (f=1): [M(1)][75.5%][r=178MiB/s,w=169MiB/s][r=177,w=168 IOPS][eta 03m:06Jobs: 1 (f=1): [M(1)][75.5%][r=3068KiB/s,w=3068KiB/s][r=2,w=2 IOPS][eta 03m:06s]Jobs: 1 (f=1): [M(1)][75.6%][r=20.8MiB/s,w=21.8MiB/s][r=20,w=21 IOPS][eta 03m:06Jobs: 1 (f=1): [M(1)][75.6%][r=3065KiB/s,w=1021KiB/s][r=2,w=0 IOPS][eta 03m:06s]Jobs: 1 (f=1): [M(1)][75.8%][r=28.2MiB/s,w=28.2MiB/s][r=28,w=28 IOPS][eta 03m:07Jobs: 1 (f=1): [M(1)][75.7%][r=2041KiB/s,w=1020KiB/s][r=1,w=0 IOPS][eta 03m:08s]Jobs: 1 (f=1): [M(1)][75.9%][r=17.0MiB/s,w=18.0MiB/s][r=17,w=18 IOPS][eta 03m:10Jobs: 1 (f=1): [M(1)][75.9%][r=4059KiB/s,w=3044KiB/s][r=3,w=2 IOPS][eta 03m:10s]Jobs: 1 (f=1): [M(1)][77.0%][r=240MiB/s,w=257MiB/s][r=240,w=257 IOPS][eta 03m:00Jobs: 1 (f=1): [M(1)][77.2%][r=78.5MiB/s,w=61.6MiB/s][r=78,w=61 IOPS][eta 02m:58Jobs: 1 (f=1): [M(1)][77.2%][r=2048KiB/s,w=1024KiB/s][r=2,w=1 IOPS][eta 02m:58s]Jobs: 1 (f=1): [M(1)][77.2%][r=9216KiB/s,w=11.0MiB/s][r=9,w=11 IOPS][eta 02m:58sJobs: 1 (f=1): [M(1)][83.9%][r=1669MiB/s,w=1669MiB/s][r=1668,w=1668 IOPS][eta 01Jobs: 1 (f=1): [M(1)][83.9%][r=2041KiB/s,w=1020KiB/s][r=1,w=0 IOPS][eta 01m:56s]Jobs: 1 (f=1): [M(1)][84.0%][r=10.0MiB/s,w=9216KiB/s][r=10,w=9 IOPS][eta 01m:57sJobs: 1 (f=1): [M(1)][84.1%][r=21.0MiB/s,w=14.0MiB/s][r=21,w=14 IOPS][eta 01m:56Jobs: 1 (f=1): [M(1)][84.1%][r=2045KiB/s,w=2045KiB/s][r=1,w=1 IOPS][eta 01m:56s]Jobs: 1 (f=1): [M(1)][84.2%][r=15.0MiB/s,w=15.0MiB/s][r=15,w=15 IOPS][eta 01m:57Jobs: 1 (f=1): [M(1)][84.2%][r=1022KiB/s][r=0 IOPS][eta 01m:57s]Jobs: 1 (f=1): [M(1)][84.6%][r=26.9MiB/s,w=30.9MiB/s][r=26,w=30 IOPS][eta 01m:59Jobs: 1 (f=1): [M(1)][84.9%][r=62.2MiB/s,w=66.1MiB/s][r=62,w=66 IOPS][eta 01m:57Jobs: 1 (f=1): [M(1)][85.2%][r=62.7MiB/s,w=67.6MiB/s][r=62,w=67 IOPS][eta 01m:54Jobs: 1 (f=1): [M(1)][85.2%][r=21.0MiB/s,w=21.0MiB/s][r=21,w=21 IOPS][eta 01m:54Jobs: 1 (f=1): [M(1)][85.3%][r=2048KiB/s,w=2048KiB/s][r=2,w=2 IOPS][eta 01m:54s]Jobs: 1 (f=1): [M(1)][85.3%][r=31.3MiB/s,w=15.7MiB/s][r=31,w=15 IOPS][eta 01m:54Jobs: 1 (f=1): [M(1)][85.4%][r=21.6MiB/s,w=15.7MiB/s][r=21,w=15 IOPS][eta 01m:53Jobs: 1 (f=1): [M(1)][85.7%][r=55.7MiB/s,w=45.9MiB/s][r=55,w=45 IOPS][eta 01m:51Jobs: 1 (f=1): [M(1)][85.9%][r=62.4MiB/s,w=56.7MiB/s][r=62,w=56 IOPS][eta 01m:49Jobs: 1 (f=1): [M(1)][86.2%][r=62.4MiB/s,w=68.1MiB/s][r=62,w=68 IOPS][eta 01m:47Jobs: 1 (f=1): [M(1)][86.4%][r=62.4MiB/s,w=56.7MiB/s][r=62,w=56 IOPS][eta 01m:45Jobs: 1 (f=1): [M(1)][86.4%][r=17.0MiB/s,w=12.0MiB/s][r=17,w=12 IOPS][eta 01m:45Jobs: 1 (f=1): [M(1)][86.4%][r=2019KiB/s,w=2019KiB/s][r=1,w=1 IOPS][eta 01m:45s]Jobs: 1 (f=1): [M(1)][86.5%][r=9197KiB/s,w=14.0MiB/s][r=8,w=13 IOPS][eta 01m:45sJobs: 1 (f=1): [M(1)][86.7%][r=40.5MiB/s,w=30.9MiB/s][r=40,w=30 IOPS][eta 01m:43Jobs: 1 (f=1): [M(1)][87.0%][r=62.5MiB/s,w=74.2MiB/s][r=62,w=74 IOPS][eta 01m:41Jobs: 1 (f=1): [M(1)][87.2%][r=62.1MiB/s,w=50.5MiB/s][r=62,w=50 IOPS][eta 01m:39Jobs: 1 (f=1): [M(1)][87.5%][r=62.3MiB/s,w=75.9MiB/s][r=62,w=75 IOPS][eta 01m:37Jobs: 1 (f=1): [M(1)][87.7%][r=62.0MiB/s,w=50.4MiB/s][r=62,w=50 IOPS][eta 01m:35Jobs: 1 (f=1): [M(1)][87.7%][r=4096KiB/s][r=4 IOPS][eta 01m:35s]Jobs: 1 (f=1): [M(1)][88.1%][r=62.1MiB/s,w=57.1MiB/s][r=62,w=57 IOPS][eta 01m:33Jobs: 1 (f=1): [M(1)][88.3%][r=61.9MiB/s,w=55.0MiB/s][r=61,w=55 IOPS][eta 01m:31Jobs: 1 (f=1): [M(1)][88.5%][r=62.0MiB/s,w=73.8MiB/s][r=62,w=73 IOPS][eta 01m:29Jobs: 1 (f=1): [M(1)][88.8%][r=61.8MiB/s,w=58.0MiB/s][r=61,w=57 IOPS][eta 01m:27Jobs: 1 (f=1): [M(1)][88.9%][r=35.5MiB/s,w=21.7MiB/s][r=35,w=21 IOPS][eta 01m:26Jobs: 1 (f=1): [M(1)][88.9%][r=3068KiB/s,w=2045KiB/s][r=2,w=1 IOPS][eta 01m:26s]Jobs: 1 (f=1): [M(1)][89.2%][r=52.6MiB/s,w=40.2MiB/s][r=52,w=40 IOPS][eta 01m:24Jobs: 1 (f=1): [M(1)][89.5%][r=61.2MiB/s,w=70.9MiB/s][r=61,w=70 IOPS][eta 01m:22Jobs: 1 (f=1): [M(1)][89.7%][r=62.9MiB/s,w=40.3MiB/s][r=62,w=40 IOPS][eta 01m:20Jobs: 1 (f=1): [M(1)][89.9%][r=62.3MiB/s,w=47.7MiB/s][r=62,w=47 IOPS][eta 01m:19Jobs: 1 (f=1): [M(1)][90.1%][r=45.9MiB/s,w=50.8MiB/s][r=45,w=50 IOPS][eta 01m:17Jobs: 1 (f=1): [M(1)][90.1%][r=2060KiB/s,w=4120KiB/s][r=2,w=4 IOPS][eta 01m:17s]Jobs: 1 (f=1): [M(1)][90.2%][r=10.0MiB/s,w=4096KiB/s][r=10,w=4 IOPS][eta 01m:17sJobs: 1 (f=1): [M(1)][90.4%][r=60.9MiB/s,w=46.4MiB/s][r=60,w=46 IOPS][eta 01m:15Jobs: 1 (f=1): [M(1)][90.7%][r=63.4MiB/s,w=37.5MiB/s][r=63,w=37 IOPS][eta 01m:13Jobs: 1 (f=1): [M(1)][90.9%][r=62.2MiB/s,w=83.6MiB/s][r=62,w=83 IOPS][eta 01m:11Jobs: 1 (f=1): [M(1)][91.2%][r=62.0MiB/s,w=49.4MiB/s][r=62,w=49 IOPS][eta 01m:09Jobs: 1 (f=1): [M(1)][91.3%][r=43.3MiB/s,w=31.5MiB/s][r=43,w=31 IOPS][eta 01m:08Jobs: 1 (f=1): [M(1)][91.3%][r=3050KiB/s,w=3050KiB/s][r=2,w=2 IOPS][eta 01m:08s]Jobs: 1 (f=1): [M(1)][91.5%][r=28.5MiB/s,w=26.5MiB/s][r=28,w=26 IOPS][eta 01m:07Jobs: 1 (f=1): [M(1)][91.7%][r=63.1MiB/s,w=66.0MiB/s][r=63,w=66 IOPS][eta 01m:05Jobs: 1 (f=1): [M(1)][92.0%][r=62.1MiB/s,w=58.1MiB/s][r=62,w=58 IOPS][eta 01m:03Jobs: 1 (f=1): [M(1)][92.2%][r=62.1MiB/s,w=73.7MiB/s][r=62,w=73 IOPS][eta 01m:01Jobs: 1 (f=1): [M(1)][92.5%][r=62.1MiB/s,w=67.9MiB/s][r=62,w=67 IOPS][eta 00m:59Jobs: 1 (f=1): [M(1)][92.6%][r=29.0MiB/s,w=26.0MiB/s][r=29,w=26 IOPS][eta 00m:58Jobs: 1 (f=1): [M(1)][92.6%][r=2041KiB/s][r=1 IOPS][eta 00m:58s]Jobs: 1 (f=1): [M(1)][92.9%][r=60.2MiB/s,w=59.3MiB/s][r=60,w=59 IOPS][eta 00m:56Jobs: 1 (f=1): [M(1)][93.1%][r=62.1MiB/s,w=57.2MiB/s][r=62,w=57 IOPS][eta 00m:54Jobs: 1 (f=1): [M(1)][93.4%][r=62.0MiB/s,w=59.1MiB/s][r=62,w=59 IOPS][eta 00m:52Jobs: 1 (f=1): [M(1)][93.8%][r=62.1MiB/s,w=82.4MiB/s][r=62,w=82 IOPS][eta 00m:49Jobs: 1 (f=1): [M(1)][93.9%][r=42.3MiB/s,w=39.4MiB/s][r=42,w=39 IOPS][eta 00m:48Jobs: 1 (f=1): [M(1)][93.9%][r=2045KiB/s,w=3068KiB/s][r=1,w=2 IOPS][eta 00m:48s]Jobs: 1 (f=1): [M(1)][93.9%][r=2058KiB/s,w=10.0MiB/s][r=2,w=10 IOPS][eta 00m:48sJobs: 1 (f=1): [M(1)][93.9%][r=3035KiB/s][r=2 IOPS][eta 00m:48s]Jobs: 1 (f=1): [M(1)][94.1%][r=10.0MiB/s,w=23.1MiB/s][r=10,w=23 IOPS][eta 00m:47Jobs: 1 (f=1): [M(1)][94.2%][r=24.8MiB/s,w=31.7MiB/s][r=24,w=31 IOPS][eta 00m:46Jobs: 1 (f=1): [M(1)][94.4%][r=62.0MiB/s,w=64.0MiB/s][r=62,w=63 IOPS][eta 00m:44Jobs: 1 (f=1): [M(1)][94.7%][r=62.4MiB/s,w=78.0MiB/s][r=62,w=78 IOPS][eta 00m:42Jobs: 1 (f=1): [M(1)][94.9%][r=62.0MiB/s,w=66.9MiB/s][r=62,w=66 IOPS][eta 00m:40Jobs: 1 (f=1): [M(1)][95.2%][r=43.7MiB/s,w=39.7MiB/s][r=43,w=39 IOPS][eta 00m:38Jobs: 1 (f=1): [M(1)][95.2%][r=2064KiB/s,w=3096KiB/s][r=2,w=3 IOPS][eta 00m:38s]Jobs: 1 (f=1): [M(1)][95.2%][r=13.9MiB/s,w=12.9MiB/s][r=13,w=12 IOPS][eta 00m:38Jobs: 1 (f=1): [M(1)][95.5%][r=62.4MiB/s,w=68.1MiB/s][r=62,w=68 IOPS][eta 00m:36Jobs: 1 (f=1): [M(1)][95.7%][r=63.1MiB/s,w=52.4MiB/s][r=63,w=52 IOPS][eta 00m:34Jobs: 1 (f=1): [M(1)][96.1%][r=61.1MiB/s,w=80.8MiB/s][r=61,w=80 IOPS][eta 00m:31Jobs: 1 (f=1): [M(1)][96.3%][r=62.0MiB/s,w=55.2MiB/s][r=62,w=55 IOPS][eta 00m:29Jobs: 1 (f=1): [M(1)][96.3%][r=25.0MiB/s,w=26.0MiB/s][r=25,w=26 IOPS][eta 00m:29Jobs: 1 (f=1): [M(1)][96.3%][r=3068KiB/s,w=3068KiB/s][r=2,w=2 IOPS][eta 00m:29s]Jobs: 1 (f=1): [M(1)][96.5%][r=21.0MiB/s,w=17.0MiB/s][r=21,w=17 IOPS][eta 00m:28Jobs: 1 (f=1): [M(1)][96.8%][r=61.4MiB/s,w=64.3MiB/s][r=61,w=64 IOPS][eta 00m:25Jobs: 1 (f=1): [M(1)][97.1%][r=62.3MiB/s,w=64.2MiB/s][r=62,w=64 IOPS][eta 00m:23Jobs: 1 (f=1): [M(1)][97.3%][r=62.4MiB/s,w=63.4MiB/s][r=62,w=63 IOPS][eta 00m:21Jobs: 1 (f=1): [M(1)][97.6%][r=62.0MiB/s,w=76.8MiB/s][r=62,w=76 IOPS][eta 00m:19Jobs: 1 (f=1): [M(1)][97.6%][r=2041KiB/s,w=1020KiB/s][r=1,w=0 IOPS][eta 00m:19s]Jobs: 1 (f=1): [M(1)][97.9%][r=59.8MiB/s,w=60.8MiB/s][r=59,w=60 IOPS][eta 00m:17Jobs: 1 (f=1): [M(1)][98.2%][r=62.2MiB/s,w=85.5MiB/s][r=62,w=85 IOPS][eta 00m:14Jobs: 1 (f=1): [M(1)][98.5%][r=62.2MiB/s,w=55.4MiB/s][r=62,w=55 IOPS][eta 00m:12Jobs: 1 (f=1): [M(1)][98.7%][r=63.1MiB/s,w=67.0MiB/s][r=63,w=66 IOPS][eta 00m:10Jobs: 1 (f=1): [M(1)][99.0%][r=40.0MiB/s,w=55.9MiB/s][r=39,w=55 IOPS][eta 00m:08Jobs: 1 (f=1): [M(1)][99.0%][r=2054KiB/s,w=9243KiB/s][r=2,w=9 IOPS][eta 00m:08s]Jobs: 1 (f=1): [M(1)][99.0%][r=9206KiB/s,w=14.0MiB/s][r=8,w=13 IOPS][eta 00m:08sJobs: 1 (f=1): [M(1)][99.4%][r=54.4MiB/s,w=65.9MiB/s][r=54,w=65 IOPS][eta 00m:05Jobs: 1 (f=1): [M(1)][99.5%][r=62.1MiB/s,w=55.3MiB/s][r=62,w=55 IOPS][eta 00m:04Jobs: 1 (f=1): [M(1)][99.9%][r=62.1MiB/s,w=69.9MiB/s][r=62,w=69 IOPS][eta 00m:01s]
test: (groupid=0, jobs=1): err= 0: pid=71154: Tue Feb  1 06:25:26 2022
  read: IOPS=31, BW=31.5MiB/s (33.0MB/s)(24.4GiB/794895msec)
    clat (usec): min=71, max=1134.4k, avg=25356.23, stdev=91859.96
     lat (usec): min=71, max=1134.4k, avg=25356.44, stdev=91860.02
    clat percentiles (usec):
     |  1.00th=[    85],  5.00th=[   161], 10.00th=[   180], 20.00th=[   202],
     | 30.00th=[   217], 40.00th=[   229], 50.00th=[   245], 60.00th=[ 15795],
     | 70.00th=[ 15795], 80.00th=[ 15795], 90.00th=[ 16057], 95.00th=[ 16712],
     | 99.00th=[530580], 99.50th=[616563], 99.90th=[792724], 99.95th=[851444],
     | 99.99th=[994051]
   bw (  KiB/s): min= 1528, max=2211512, per=100.00%, avg=33657.76, stdev=94944.55, samples=1504
   iops        : min=    1, max= 2159, avg=32.25, stdev=92.77, samples=1504
  write: IOPS=31, BW=31.4MiB/s (33.0MB/s)(24.4GiB/794895msec); 0 zone resets
    clat (usec): min=98, max=87298, avg=6409.12, stdev=9168.51
     lat (usec): min=108, max=87316, avg=6424.56, stdev=9171.04
    clat percentiles (usec):
     |  1.00th=[  133],  5.00th=[  180], 10.00th=[  184], 20.00th=[  186],
     | 30.00th=[  190], 40.00th=[  200], 50.00th=[  215], 60.00th=[  227],
     | 70.00th=[ 9634], 80.00th=[20579], 90.00th=[20841], 95.00th=[21103],
     | 99.00th=[21365], 99.50th=[21365], 99.90th=[25297], 99.95th=[27395],
     | 99.99th=[49021]
   bw (  KiB/s): min= 1528, max=2176766, per=100.00%, avg=43163.14, stdev=105481.71, samples=1172
   iops        : min=    1, max= 2125, avg=41.53, stdev=103.03, samples=1172
  lat (usec)   : 100=0.97%, 250=57.05%, 500=2.48%, 750=0.02%, 1000=0.02%
  lat (msec)   : 2=0.09%, 4=0.95%, 10=1.44%, 20=21.67%, 50=13.01%
  lat (msec)   : 100=0.03%, 250=0.31%, 500=1.35%, 750=0.54%, 1000=0.07%
  lat (msec)   : 2000=0.01%
  cpu          : usr=0.06%, sys=80.44%, ctx=84381, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=25009,24991,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0,  percentile=100.00%, depth=1

Run status group 0 (all jobs):
    READ: bw=31.5MiB/s (33.0MB/s), 31.5MiB/s-31.5MiB/s (33.0MiB/s-33.0MiB/s), io=24.3GiB (26.2GB), run=794895-794895msec
    WRITE: bw=31.4MiB/s (33.0MB/s), 31.4MiB/s-31.4MiB/s (33.0MiB/s-33.0MiB/s), io=24.3GiB (26.2GB), run=794895-794895msec

Exard3k · February 1, 2022, 5:23pm

Sounds about right if the dataset has sync=always

drizzlezap · February 1, 2022, 5:27pm

This is with sync disabled

anon34403656 · February 1, 2022, 7:42pm

I am far from being an expert on this, but the bandwidth to the disks seems to be extremely low. If I am interpreting this correctly fio reported like 50MB bandwidth in read as well as in write.

drizzlezap · February 2, 2022, 4:45pm

Yes indeed, it is horribly slow and I am confident there is something wrong on either the hardware or software side. I have another TrueNAS server for backup and it reaches 400+MB/s on a RaidZ2 Pool. Drives are the non-pro version, but I don’t think that is relevant.

I am starting to think there is something wrong with the HBAs, but they are in IT mode and on the latest p20 firmware version, so I wouldn’t know what I could change there. Maybe there is something wrong with my motherboard settings, but then again I wouldn’t know any setting that could cause this issue.

On a side note, I uninstalled (and disabled in the BIOS) the BootROM from the HBAs as the server wouldn’t start with them installed, but besides making it unable to boot the system from the HBA I am not aware it could cause my problem. Also, I had to install the last version of FreeNAS and upgrade to TrueNAS from the WebGUI as I wasn’t able to install TrueNAS directly on my server, which was also a weird experience.

Log · February 2, 2022, 6:14pm

Have you tried testing the raw disks with FIO, without ZFS?

Exard3k · February 2, 2022, 6:24pm

If you have some on-board SATA, you could check if the problem persists without the HBA. With all other things being equal, might be worth checking. My TrueNAS never had problems with drive detection between different SATA-controllers, but I don’t have an HBA for HDDs, so can’t share any experience with that. The last HBA I had was an Adaptec 2940 if you still remember them

drizzlezap · February 2, 2022, 8:13pm

I did in fact not test raw performance on the disks themself, but I have the feeling that the issue is somewhere else, as I was able to partially solve the issue.

So, I found a thread about a guy complaining about the same performance I was experiencing (30-40MB/s) and he even had the same HBAs. (genuine LSI 9211-8i doe) So appareantly encryption causes some weird issues that I haven’t fully understood so far.

However, after he removed veracrypt from the drives the speeds went up to what he was expecting.

That made me think I might have the same issues, as I have all my Pools encrypted. So I checked my BIOS and found that the AES-NI option was disabled. After enabling it, these are the speeds now:

TruenasHost

Read Write Test:

Command

fio --rw=readwrite --name=test --size=50000M --direct=1 --bs=1024k

Result

READ: bw=204MiB/s (214MB/s), 204MiB/s-204MiB/s (214MB/s-214MB/s), io=24.4GiB (26.2GB), run=122574-122574msec
WRITE: bw=204MiB/s (214MB/s), 204MiB/s-204MiB/s (214MB/s-214MB/s), io=24.4GiB (26.2GB), run=122574-122574msec

Random Write Test:

Command

fio--name=random-write --ioengine=posixaio --rw=randwrite --bs=1m --size=32g --numjobs=1 --iodepth=1 --runtime=120 --time_based --end_fsync=1

Random Write Result

WRITE: bw=632MiB/s (663MB/s), 632MiB/s-632MiB/s (663MB/s-663MB/s), io=74.1GiB(79.5GB), run=120001-120001msec

VM on Proxmox Cluster (set to Host CPU)

Read Write Test:

Command

fio --rw=readwrite --name=test --size=50000M --direct=1 --bs=1024k

Result

READ: bw=553MiB/s (580MB/s), 553MiB/s-553MiB/s (580MB/s-580MB/s), io=24.4GiB(26.2GB), run=45246-45246msec
WRITE: bw=552MiB/s (579MB/s), 552MiB/s-552MiB/s (579MB/s-579MB/s), io=24.4GiB(26.2GB), run=45246-45246msec

Random Write Test:

Command

fio--name=random-write --ioengine=posixaio --rw=randwrite --bs=1m --size=32g --numjobs=1 --iodepth=1 --runtime=120 --time_based --end_fsync=1

Random Write Result

WRITE: bw=480MiB/s (503MB/s), 480MiB/s-480MiB/s (503MB/s-503MB/s), io=56.6GiB (60.8GB), run=120926-120926msec

The results of the VM were even better with kvm64 with 590 random write.

So one would think my problem is solved right, the results are awesome! Well, I am not sure these result are actually real, or if there is some ZFS magic going on behind the scenes.

When running iozone or FlexibleIO benchmark from the Phoronix Test Suite inside the VM my results are still terrible. I am not sure how fio tests are made and if I can take the results serious, but from watching my TrueNAS traffic, the speed surely went up to 700+ MiB/s while the test was running, so I guess the results are valid?

Also does anybody know, why the Phoronix Suite is producing such horrible results?

Are there any other tests I can do to verify that my problems have been solved?

Is there another reason why AES-NI appareantly solved the problem besides my assumption encryption is causing issues with the HBAs?

Exard3k · February 2, 2022, 8:38pm

now we’re talking.

They use atime=on last time I checked, because it’s the default setting. Guess why BTRFS and ZFS suck in Phoronix Suite results And because ZFS is a bitch to benchmark as you might have noticed.

Too slow for ARC (unless your network is capped at that 700 for whatever reason). I’m getting full 10G on cache hit = ~950MiB/s. Writes should be slower than reads on these kinds of data. If you get 400-500 on writes, we’re good.

zfs send/receive, resilver or scrub utilize the drives very efficiently. That’s where I see the biggest numbers on my server. send and scrub are obviously reads, but always feels good to see those 250MB/s in the reporting menu graphs

drizzlezap · February 2, 2022, 9:51pm

Oh okay, and that let’s the performance crash to 25MB/s? Seems extremely low.

Also, thanks for mentioning atime ! I somehow had activated it on the VM Pool in my desperation.

Well actually I think that is just the speed cap or the pool, as I have a stripe of 3 mirror and the drives are rated at 200MB/s. So I guess 660MB/s (random write) on the host i close enough. Also, like mentioned before these test were made with atime on. Setting it off gave it like ~10% speed increase!

Maybe I’ll add another Mirror to the pool soon!

I just activated sync writes to test my speed with the Optane installed and I am getting 350MB/s (random write) from within the VM which I am happy with overall. For less critical VMs I will probably create another pool with sync off.

So I guess we can say the Problem is solved? I sure hope so, this was awful.

Which graphs you mean? Those that don’t work on chromium based browsers?

Here’s the Jira if anybody wants to read into it:
https://jira.ixsystems.com/browse/NAS-113730

Thanks for all the help to everybody!

PS: How can I mark a thread as solved? In the sticky Wendell said I can choose the Option from the 3dot menu on the posts, but I don’t see any option.

Exard3k · February 2, 2022, 10:01pm

You’re just late for the party. Worked fine until last update.

You’ve come a long way from 7MB/s to this. Looks like a solid server. Oh and pls setup a periodic task for scrub+backup, that Linus Video got me triggered so hard