ZFS - slow reads / high wait

Pool details:

  • 5x 4tb ironwolfs in raidz
  • 200G SSD as l2arc
  • 50G nvme SSD as ZIL
  • xattr=sa
  • atime=off
  • ashift=9

Copy source
(notice the wait, but reading 5x rate as transfer is moving)


Could the ashift not being set to 12 cause this slowness??

This is the manual for your drive, correct? (I assumed non-pro)

Assuming it is, they’ve got 4k sectors. Setting ashift to 12 would definitely help, but it’s probably not the cause of an 80% speed reduction.

These aren’t on SATA port multipliers, right?

actually I am using a 36port sas expander with my HBA card…

And yes. Non pro drives


Alright, are all 36 ports populated? If so, you might be running into issues there, but that’s only likely to happen if all disks are active.

My recommendation is to try direct-attachment and ashift=12.

No. only about half the ports… with two ports going to HBA.
I started last night copying files off the zvol as a backup so I can blow away the pool and create it again with ashift=12…

random thought…
I wish I had a motherboard with more PCIe slots so I could install my other sas hba and use multipathd with my sas expander…

shouldn’t be a problem then. If it’s a somewhat reputable expander, you should be fine.

I don’t think ashift=9 is causing that much performance loss, but it could be.

How about those SLOG and Cache devices? Are they old? I’ve seen pool performance suffer because of an SSD that’s choking itself to death.

intel 36 port sas expander
slog is 280G optane u.2
caches are SATA3 ssds ( about 1 months old)

1 Like

Yeah, probably isn’t the solid state in that case.

Neither the expander.

Let’s see what ashift=12 does before I make any more suggestions.

1 Like

Can I change ashift on the pool without blowing it away?

ashift is a readonly pool property. You need to recreate it, unfortunately.

restoring the files now… but going super slow from the backup system…but I also turned on dedup… just to see what happens. lol

1 Like

That’s what dedup does, it slows down your array.

after destroying the zvol and creating a new one without dedup… i confirmed that the slowness is on the backup array side. The restore is moving at the same speed regardless of dedup on or off.

cpu %(green) and wait % (blue) - dedup on - left
cpu %(green) and wait % (blue) - dedup off - right
…same scale…


Still doing some testing… but I may have figured out the performance issue…

More testing is still on order.

1 Like

Hmm, interesting.

What are you thinking it is?

I used whole disks instead of creating a partition on each of them first.

I can’t believe I didn’t mention that.

ZFS requires you to use the entire disk so it can align itself properly. Otherwise, you get your exact situation (worst case).

im thinking about using gdisk to align the partitions on 4096 sector boundaries…???

So are you saying… sda or sda1

You must use the whole disk if you want full performance.


You’ll notice, later on, that ZFS creates two partitions. One is data, one is pool metadata, I think. They’re aligned properly by the util. It’s a bit complicated and I don’t know all the details, but essentially if you just use the raw disk, you don’t have to worry.