Improving ZFS scrub performance? - Hot Prod Replace & Resilver, hot/cold spare

Moin moin!

I’m not sure what actually happened, but i seem to have observed an increase in zpool scrub performance, after the following scenario

See also, HDD burn-in testing, or how i learned to fail fast and love the RMA - #4 by Platypussy

I’m running a 4-wide Z1 pool of 6TB HGST (WD) Ultrastars, and 1 cold spare. I just put 'em into prod, without prior validation, trusting HGSTs reputation. Alas, one of my hot HDDs failed. No drama, i’ve just zpool replaceed it with the cold spare, and the RMA yielded a “refurbished to HGST specification” replacement unit.

That got me thinking - i didn’t burn-in those HDDs.

So, that’s what i decided to do, see thread above.

I went ahead, zpool replaceing/resilvering, and badblocksing every single member of my Z1 pool, until that refurb unit became the cold spare again.

So, that’s my “hot-prod-burn-in” procedure.

BUT, here’s the kicker!

This whole ordeal seems to have improved the pools zpool scrub performance, significantly! I’m not sure why, and i’m left failing to provide a scientific changelog of all the properties/settings - i’ve changed a lot, especially caching (L1/L2/metadata/sync/read/write) - i’ve even added a chonkers crypto-dataset.

Prior to the above procedure, this pool has been well-used by lots of applications, DSs, ZVOLs for VMs, ZVOLs for VMs with LUKS crypto, encrypted DSs.

Hypothesis: replacing/resilvering all those HDDs caused internal restructuring / optimization. (i didn’t look at the source code)

Now, i’m scrubbing at ~660MB/s on a 4-wide Z1 pool of fancy 6TB Ultrastars. (HPE MicroServer Gen10+, with the i3-9100F upgrade, <50% CPU load avg)

EDIT:
Of course, that’s insufficient - let me fill-in some specs: Ubuntu 20.04.3 LTS, zfs-0.8.3-1ubuntu12.12 …yeah, none of that fancy 2.x, yet. :wink:

(edit 2: fixed typos)