I have set special_small_blocks to 64K which in my instance combined w/ metadata should be ~75G (8.67G + 66.3G) but my sVDEV is only filled to 23G.
Blocks LSIZE PSIZE ASIZE avg comp %Total Type
332K 41.5G 4.33G 8.67G 26.7K 9.58 0.09 L1 Total
block psize lsize asize
size Count Size Cum. Count Size Cum. Count Size Cum.
512: 32.7K 16.4M 16.4M 32.7K 16.4M 16.4M 0 0 0
1K: 49.6K 59.4M 75.7M 49.6K 59.4M 75.7M 0 0 0
2K: 15.5K 41.1M 117M 15.5K 41.1M 117M 0 0 0
4K: 483K 1.89G 2.01G 15.1K 82.4M 199M 113K 452M 452M
8K: 63.8K 642M 2.64G 18.0K 208M 407M 513K 4.11G 4.55G
16K: 83.0K 1.82G 4.46G 208K 3.38G 3.78G 93.4K 1.98G 6.53G
32K: 293K 13.4G 17.8G 58.7K 2.86G 6.64G 218K 10.1G 16.6G
64K: 486K 43.0G 60.9G 67.5K 5.73G 12.4G 566K 49.7G 66.3G
128K: 79.1M 9.88T 9.94T 80.1M 10.0T 10.0T 79.1M 9.88T 9.95T
$ zpool list -v
NAME SIZE ALLOC FREE
special - - - - - - - - -
mirror-2 199G 23.3G 176G - - 11% 11.7% - ONLINE
wwn-0x5002538c402f3ace-part4 200G - - - - - - - ONLINE
wwn-0x5002538c402f3afc-part4 200G - - - - - - - ONLINE
wwn-0x5002538c402f3823-part4 200G - - - - - - - ONLINE
I do want to say my setup is a little complex as it doubles as the boot drive and a special vdev in a 3-way Mirror. I have partitioned 200G just for the special vdev itself. I am running on a fresh install of Proxmox 9.1 with the pool running on the host itself. Any ideas on why there is a size discrepancy? I have made sure to turn off SMB and NFS shares while zfs rewrite -rv /tank was running.
Did you add the special VDEV to an existing pool?
With a few exceptions ZFS generally only uses newly added VDEV’s for new data going forward, it generally does not rebalance or move existing things to new VDEV’s.
Example: When you add an additional data VDEV to an existing pool, ZFS will use it for new writes, but it will not expand existing data and balance it to the newly added VDEV.
For special devices I think (but am not 100% sure) that metadata is generally moved to the new devices, but small files are only written to the new special VDEV for new writes. Existing small files stay where they were prior to adding the special VDEV. (at least until rewritten)
There are ways to force the issue (like, reading and re-writing all of your data) but in general it is best to just leave it alone and let it do it’s thing. Either that or just create a backup, destroy the pool and create it from scratch when you reconfigure it like this.
Edit:
I noticed you used the new “zfs rewrite” command. I probably wouldn’t have. At least if you rely on snapshots.
Snapshots are by their design read only, so when you run “zfs rewrite” on a dataset with snapshots, these snapshots become duplicated and now take even more space.
Maybe that is where your allocation is going, and why you aren’t seeing the expected results.
Yes I am aware of the limitations of ZFS rewrite with snapshots but I do not have any snapshots. Subsequent zfs rewrite -rv /tank did not increase the size allocated to the sVDEV from its current 23G. I want to reiterate that the special VDEV is on the same device as the boot drive (rpool) just on separate partitions if that matters.
Also it seems like metadata isn’t even on the sVDEV? Running find /tank -type f -print0 | xargs .... shows it taxing the HDD pool with zero activity on the sVDEV. iostat shows 80% util on the HDDs but my understanding is that this workload should be accelerated?
$ zpool status -vvv
pool: rpool
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-SAMSUNG_MZ7KM480HAHP-00005_S2HSNX0H508033-part3 ONLINE 0 0 0
ata-SAMSUNG_MZ7KM480HAHP-00005_S2HSNX0H508401-part3 ONLINE 0 0 0
ata-SAMSUNG_MZ7KM480HAHP-00005_S2HSNX0H508422-part3 ONLINE 0 0 0
errors: No known data errors
pool: tank
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
wwn-0x5000cca253c8e637-part1 ONLINE 0 0 0
wwn-0x5000cca253c744ae-part1 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
ata-WDC_WUH721816ALE6L4_2CGRLEZP ONLINE 0 0 0
ata-WUH721816ALE6L4_2BJMBDBN ONLINE 0 0 0
special
mirror-2 ONLINE 0 0 0
wwn-0x5002538c402f3ace-part4 ONLINE 0 0 0
wwn-0x5002538c402f3afc-part4 ONLINE 0 0 0
wwn-0x5002538c402f3823-part4 ONLINE 0 0 0
errors: No known data errors
$ zfs get special_small_blocks tank
NAME PROPERTY VALUE SOURCE
tank special_small_blocks 64K local
$ zfs get special_small_blocks rpool
NAME PROPERTY VALUE SOURCE
rpool special_small_blocks 0 default