Pool details:
Copy source
(notice the wait, but reading 5x rate as transfer is moving)
Destination
Could the ashift not being set to 12 cause this slowness??
This is the manual for your drive, correct? (I assumed non-pro)
Assuming it is, they’ve got 4k sectors. Setting ashift to 12 would definitely help, but it’s probably not the cause of an 80% speed reduction.
These aren’t on SATA port multipliers, right?
actually I am using a 36port sas expander with my HBA card…
And yes. Non pro drives
Alright, are all 36 ports populated? If so, you might be running into issues there, but that’s only likely to happen if all disks are active.
My recommendation is to try direct-attachment and ashift=12.
No. only about half the ports… with two ports going to HBA.
I started last night copying files off the zvol as a backup so I can blow away the pool and create it again with ashift=12…
random thought…
I wish I had a motherboard with more PCIe slots so I could install my other sas hba and use multipathd with my sas expander…
shouldn’t be a problem then. If it’s a somewhat reputable expander, you should be fine.
I don’t think ashift=9 is causing that much performance loss, but it could be.
How about those SLOG and Cache devices? Are they old? I’ve seen pool performance suffer because of an SSD that’s choking itself to death.
intel 36 port sas expander
slog is 280G optane u.2
caches are SATA3 ssds ( about 1 months old)
Yeah, probably isn’t the solid state in that case.
Neither the expander.
Let’s see what ashift=12 does before I make any more suggestions.
Can I change ashift on the pool without blowing it away?
ashift is a readonly pool property. You need to recreate it, unfortunately.
restoring the files now… but going super slow from the backup system…but I also turned on dedup… just to see what happens. lol
That’s what dedup does, it slows down your array.
after destroying the zvol and creating a new one without dedup… i confirmed that the slowness is on the backup array side. The restore is moving at the same speed regardless of dedup on or off.
cpu %(green) and wait % (blue) - dedup on - left
cpu %(green) and wait % (blue) - dedup off - right
…same scale…
Still doing some testing… but I may have figured out the performance issue…
More testing is still on order.
Hmm, interesting.
What are you thinking it is?
I used whole disks instead of creating a partition on each of them first.
I can’t believe I didn’t mention that.
ZFS requires you to use the entire disk so it can align itself properly. Otherwise, you get your exact situation (worst case).
im thinking about using gdisk to align the partitions on 4096 sector boundaries…???
So are you saying… sda or sda1
You must use the whole disk if you want full performance.
/dev/sda
You’ll notice, later on, that ZFS creates two partitions. One is data, one is pool metadata, I think. They’re aligned properly by the util. It’s a bit complicated and I don’t know all the details, but essentially if you just use the raw disk, you don’t have to worry.
