Hey kind of new to zfs and proxmox.
Set up a ZFS raid 10 for a client and im noticing some weird utilization.
can someone explain to me like im 5 why my 8TB virtual disc consume 17.7 TB of space on my zpool>?
thanks
Could you post output of
zpool status
and
zpool get all | grep ashift ?
I assume you are using the default 16k block size?
1 Like
root@KZIA-PVE-2018:~# zpool status
pool: data-pool
state: ONLINE
scan: scrub repaired 0B in 04:41:10 with 0 errors on Sun Mar 9 06:05:11 2025
config:
NAME STATE READ WRITE CKSUM
data-pool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-WDC_WD101EFBX-68B0AN0_VH20XALM ONLINE 0 0 0
ata-WDC_WD101EFBX-68B0AN0_VH213LJM ONLINE 0 0 0
ata-WDC_WD101EFBX-68B0AN0_VH236R4M ONLINE 0 0 0
ata-WDC_WD101EFBX-68B0AN0_VH238BUM ONLINE 0 0 0
logs
wwn-0x5002538f55107514-part1 ONLINE 0 0 0
cache
wwn-0x5002538f55107514-part2 ONLINE 0 0 0
errors: No known data errors
pool: ssd-pool
state: ONLINE
scan: scrub repaired 0B in 00:15:28 with 0 errors on Sun Mar 9 00:39:30 2025
config:
NAME STATE READ WRITE CKSUM
ssd-pool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-Samsung_SSD_870_EVO_2TB_S753NL0Y104542V ONLINE 0 0 0
ata-Samsung_SSD_870_EVO_2TB_S753NL0Y104545D ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
ata-Samsung_SSD_870_EVO_2TB_S753NL0Y104534H ONLINE 0 0 0
ata-Samsung_SSD_870_EVO_2TB_S753NL0Y113272V ONLINE 0 0 0
errors: No known data errors
root@KZIA-PVE-2018:~#
root@KZIA-PVE-2018:~# zpool get all | grep ashift
data-pool ashift 12 local
ssd-pool ashift 12 local
root@KZIA-PVE-2018:~#
it looks like my block size is the set to 128k
is that my problem?
8.8TB disk * 2 for mirror = 17.6TB. The extra .1TB is probably overhead of configuration stuff.
1 Like
Are there snapshots on the dataset or ZVOL?
Try zfs get usedbysnapshots pool/path/dataset
1 Like
so not only do i lose half my data pool to raid 10 but the data gets counted twice against my total space?
root@KZIA-PVE-2018:~# zpool status
pool: data-pool
state: ONLINE
scan: scrub repaired 0B in 04:41:10 with 0 errors on Sun Mar 9 06:05:11 2025
config:
NAME STATE READ WRITE CKSUM
data-pool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-WDC_WD101EFBX-68B0AN0_VH20XALM ONLINE 0 0 0
ata-WDC_WD101EFBX-68B0AN0_VH213LJM ONLINE 0 0 0
ata-WDC_WD101EFBX-68B0AN0_VH236R4M ONLINE 0 0 0
ata-WDC_WD101EFBX-68B0AN0_VH238BUM ONLINE 0 0 0
logs
wwn-0x5002538f55107514-part1 ONLINE 0 0 0
cache
wwn-0x5002538f55107514-part2 ONLINE 0 0 0
errors: No known data errors
pool: ssd-pool
state: ONLINE
scan: scrub repaired 0B in 00:15:28 with 0 errors on Sun Mar 9 00:39:30 2025
config:
NAME STATE READ WRITE CKSUM
ssd-pool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-Samsung_SSD_870_EVO_2TB_S753NL0Y104542V ONLINE 0 0 0
ata-Samsung_SSD_870_EVO_2TB_S753NL0Y104545D ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
ata-Samsung_SSD_870_EVO_2TB_S753NL0Y104534H ONLINE 0 0 0
ata-Samsung_SSD_870_EVO_2TB_S753NL0Y113272V ONLINE 0 0 0
errors: No known data errors
root@KZIA-PVE-2018:~# zpool get all | grep ashift
data-pool ashift 12 local
ssd-pool ashift 12 local
root@KZIA-PVE-2018:~#
root@KZIA-PVE-2018:~# zfs get recordsize data-pool
NAME PROPERTY VALUE SOURCE
data-pool recordsize 128K default
root@KZIA-PVE-2018:~# zfs get volblocksize data-pool
NAME PROPERTY VALUE SOURCE
data-pool volblocksize - -
root@KZIA-PVE-2018:~# zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
data-pool/vm-102-disk-0@replicate_102-0_1741611605 7.98G - 7.26T -
ssd-pool/vm-100-disk-0@replicate_100-0_1741612985 407M - 344G -
ssd-pool/vm-102-disk-1@replicate_102-0_1741611605 257M - 91.9G -
ssd-pool/vm-102-disk-2@replicate_102-0_1741611605 16.5M - 124G -
ssd-pool/vm-102-disk-3@replicate_102-0_1741611605 10.5G - 186G -
root@KZIA-PVE-2018:~# zpool list -v
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
data-pool 36.4T 10.0T 26.4T - - 0% 27% 1.00x ONLINE -
raidz1-0 36.4T 10.0T 26.4T - - 0% 27.5% - ONLINE
ata-WDC_WD101EFBX-68B0AN0_VH20XALM 9.10T - - - - - - - ONLINE
ata-WDC_WD101EFBX-68B0AN0_VH213LJM 9.10T - - - - - - - ONLINE
ata-WDC_WD101EFBX-68B0AN0_VH236R4M 9.10T - - - - - - - ONLINE
ata-WDC_WD101EFBX-68B0AN0_VH238BUM 9.10T - - - - - - - ONLINE
logs - - - - - - - - -
wwn-0x5002538f55107514-part1 50.0G 16K 49.5G - - 0% 0.00% - ONLINE
cache - - - - - - - - -
wwn-0x5002538f55107514-part2 1.36T 201G 1.17T - - 0% 14.4% - ONLINE
ssd-pool 3.62T 783G 2.86T - - 7% 21% 1.00x ONLINE -
mirror-0 1.81T 397G 1.42T - - 7% 21.4% - ONLINE
ata-Samsung_SSD_870_EVO_2TB_S753NL0Y104542V 1.82T - - - - - - - ONLINE
ata-Samsung_SSD_870_EVO_2TB_S753NL0Y104545D 1.82T - - - - - - - ONLINE
mirror-1 1.81T 386G 1.44T - - 7% 20.8% - ONLINE
ata-Samsung_SSD_870_EVO_2TB_S753NL0Y104534H 1.82T - - - - - - - ONLINE
ata-Samsung_SSD_870_EVO_2TB_S753NL0Y113272V 1.82T - - - - - - - ONLINE
root@KZIA-PVE-2018:~#
root@KZIA-PVE-2018:~# zfs get compressratio data-pool
NAME PROPERTY VALUE SOURCE
data-pool compressratio 1.06x -
root@KZIA-PVE-2018:~# zpool status
pool: data-pool
state: ONLINE
scan: scrub repaired 0B in 04:41:10 with 0 errors on Sun Mar 9 06:05:11 2025
config:
NAME STATE READ WRITE CKSUM
data-pool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-WDC_WD101EFBX-68B0AN0_VH20XALM ONLINE 0 0 0
ata-WDC_WD101EFBX-68B0AN0_VH213LJM ONLINE 0 0 0
ata-WDC_WD101EFBX-68B0AN0_VH236R4M ONLINE 0 0 0
ata-WDC_WD101EFBX-68B0AN0_VH238BUM ONLINE 0 0 0
logs
wwn-0x5002538f55107514-part1 ONLINE 0 0 0
cache
wwn-0x5002538f55107514-part2 ONLINE 0 0 0
errors: No known data errors
pool: ssd-pool
state: ONLINE
scan: scrub repaired 0B in 00:15:28 with 0 errors on Sun Mar 9 00:39:30 2025
config:
NAME STATE READ WRITE CKSUM
ssd-pool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-Samsung_SSD_870_EVO_2TB_S753NL0Y104542V ONLINE 0 0 0
ata-Samsung_SSD_870_EVO_2TB_S753NL0Y104545D ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
ata-Samsung_SSD_870_EVO_2TB_S753NL0Y104534H ONLINE 0 0 0
ata-Samsung_SSD_870_EVO_2TB_S753NL0Y113272V ONLINE 0 0 0
errors: No known data errors
root@KZIA-PVE-2018:~#
root@KZIA-PVE-2018:~# zfs get reservation,refreservation data-pool
NAME PROPERTY VALUE SOURCE
data-pool reservation none default
data-pool refreservation none default
root@KZIA-PVE-2018:~# qm list
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
100 ZST4LOG running 4096 500.00 3532902
102 ZST2FS2 running 32768 175.00 677616
root@KZIA-PVE-2018:~#
root@KZIA-PVE-2018:~# zfs get usedbysnapshots data-pool
NAME PROPERTY VALUE SOURCE
data-pool usedbysnapshots 0B -
root@KZIA-PVE-2018:~#
Snapshotted data is stored with the same level of pool redundancy, yes.
So this shows you have data in use by snapshots.
This isn’t recursive, so might be misleading. For a summary of space used by snapshots over the entire pool, try:
zfs get -r -t filesystem,volume usedbysnapshots data-pool
1 Like
A handy shortcut is zfs -o space.
1 Like
Somewhere i read that this is do to the refreservation setting.
What happens if i set refreservation to none
LOL
root@kzia-pve5-2020:~# zfs get -r -t filesystem,volume usedbysnapshots data-pool
NAME PROPERTY VALUE SOURCE
data-pool usedbysnapshots 0B -
data-pool/vm-101-disk-0 usedbysnapshots 0B -
data-pool/vm-101-disk-1 usedbysnapshots 0B -
data-pool/vm-102-disk-0 usedbysnapshots 0B -
root@kzia-pve5-2020:~#
Thanks everyone for your help.
thank you
AME USED AVAIL REFRESERV
data-pool 22.2T 4.92T none
data-pool/vm-101-disk-0 6.60T 5.46T 6.60T
data-pool/vm-101-disk-1 203G 4.93T 203G
data-pool/vm-102-disk-0 15.4T 13.8T 8.85T
ssd-pool 2.39T 1.12T none
ssd-pool/vm-100-disk-0 852G 1.61T 508G
ssd-pool/vm-101-disk-0 122G 1.15T 122G
ssd-pool/vm-102-disk-1 270G 1.29T 178G
ssd-pool/vm-102-disk-2 429G 1.42T 305G
ssd-pool/vm-102-disk-3 440G 1.37T 254G
ssd-pool/vm-103-disk-0 3M 1.12T 3M
ssd-pool/vm-103-disk-1 60.9G 1.15T 60.9G
ssd-pool/vm-104-disk-0 279G 1.18T 279G
root@kzia-pve5-2020:~#
im still pretty confused. Why does one vmdisk have 13TB of available space? What does this refreserv do>? what happens when i set it to none?
You have RAIDZ1 for blockstorage.
That RAW disk from your VM is basically in a zvol (blockstorage).
You fell into the ZFS blockstorage trap.
That is why you should never ever use RAIDZ for VMs, always mirrors.
Unless you really know about pool geometry and stuff.
Here is what happened.
You have 4 disks with the default volblocksize of 16k. Disks are ashift 12 = 4k.
Since you use RAIDZ1 with four disks, you would expect the storage of 3 disks. You expect the you get 28.92TB.
But there is a catch.
ZFS is a sneaky bastard, since you don’t directly see this, other than that a 1TB VM disk needs more than 1TB of storage.
BTW I don’t know what these logs and caches drives are, but I am 99% sure that you are better off without them and you probably misunderstood how these work.
TLDR: Rebuild that setup. Use mirrors for VM data, not RAIDZ.
If you need lots of storage, move data away from VMs inside datasets that are based on RAIDZ.
I highly doubt that, unless you manually changed it. I am not talking about the recordsize but the volblocksize.
If you really set this to 128k, then yeah, you have a huge problem. Fragmentation and rw amplification.
2 Likes
Thank you for your response. I am sure there is much about ZFS i do not understand, but am trying to learn.
No worries.
Guide for ZFS newcomers:
- Don’t use blockstorage for files, but datasets
- Separate files from VM data
- Don’t use RAIDZ for blockstorage
- Use RAIDZ2 for big files that are sequentially read and written
- Max out RAM
- If you need special vdev, you have to add it before you move data to your pool
- Don’t use L2ARC or SLOG from the start. There is a high chance that your workload won’t need it. If needed you can still add it later on.
1 Like