ZFS dedup exploration

So I have been testing a small machine with dedup for use with a few VMs on centos7 using kvm/libvirtd/qemu.

system info:

CPU E5-1620 v3 @ 3.50GHz
125GB RAM
zpool = 2x 500GB HDDs

pool: oddly1
state: ONLINE
scan: scrub repaired 0B in 1h32m with 0 errors on Tue Dec  3 05:38:34 2019

config:
        NAME                        STATE     READ WRITE CKSUM
        oddly1                      ONLINE       0     0     0
          mirror-0                  ONLINE       0     0     0
            wwn-0x5000c500501c03e6  ONLINE       0     0     0
            wwn-0x50014ee2b313479c  ONLINE       0     0     0

errors: No known data errors
dedup: DDT entries 1445627, size 826B on disk, 266B in core
bucket              allocated                       referenced
______   ______________________________   ______________________________
refcnt   blocks   LSIZE   PSIZE   DSIZE   blocks   LSIZE   PSIZE   DSIZE
------   ------   -----   -----   -----   ------   -----   -----   -----
1    1.37M    175G    175G    175G    1.37M    175G    175G    175G
2    8.56K   1.07G   1.07G   1.07G    18.2K   2.28G   2.28G   2.28G
4      136     17M     17M     17M      556   69.5M   69.5M   69.5M
8       19   2.38M   2.38M   2.38M      228   28.5M   28.5M   28.5M
16       10   1.25M   1.25M   1.25M      235   29.4M   29.4M   29.4M
32        4    512K    512K    512K      168     21M     21M     21M
64        2    256K    256K    256K      136     17M     17M     17M
128        2    256K    256K    256K      382   47.8M   47.8M   47.8M
256K        1    128K    128K    128K     433K   54.2G   54.2G   54.2G
Total    1.38M    176G    176G    176G    1.81M    232G    232G    232G
-
[root@workstationfed01 ~]# zpool list
NAME     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
oddly1   464G   178G   286G         -    56%    38%  1.31x  ONLINE  -  

How do you calculate how much is cached data vs how much is metadata … the arc_summary command?

I do have another machine with less RAM and more space and a higher dedup ratio…

zpool list
NAME       SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
proxpool  1.90T   219G  1.69T         -    37%    11%  3.77x  ONLINE  -
[root@lsrvfedora27 ~]# free -g
              total        used        free      shared  buff/cache   available
Mem:             62          52           7           0           2           8
 pool: proxpool
 state: ONLINE
  scan: scrub repaired 0B in 0h24m with 0 errors on Tue Dec  3 04:09:29 2019
config:

	NAME                                                        STATE     READ WRITE CKSUM
	proxpool                                                    ONLINE       0     0     0
	  mirror-0                                                  ONLINE       0     0     0
	    pci-0000:04:00.0-sas-exp0x500065b36789abff-phy1-lun-0   ONLINE       0     0     0
	    pci-0000:04:00.0-sas-exp0x500065b36789abff-phy9-lun-0   ONLINE       0     0     0
	  mirror-1                                                  ONLINE       0     0     0
	    pci-0000:04:00.0-sas-exp0x500065b36789abff-phy2-lun-0   ONLINE       0     0     0
	    pci-0000:04:00.0-sas-exp0x500065b36789abff-phy10-lun-0  ONLINE       0     0     0
	  mirror-2                                                  ONLINE       0     0     0
	    pci-0000:04:00.0-sas-exp0x500065b36789abff-phy3-lun-0   ONLINE       0     0     0
	    pci-0000:04:00.0-sas-exp0x500065b36789abff-phy11-lun-0  ONLINE       0     0     0
	  mirror-3                                                  ONLINE       0     0     0
	    pci-0000:04:00.0-sas-exp0x500065b36789abff-phy4-lun-0   ONLINE       0     0     0
	    pci-0000:04:00.0-sas-exp0x500065b36789abff-phy12-lun-0  ONLINE       0     0     0
	  mirror-4                                                  ONLINE       0     0     0
	    pci-0000:04:00.0-sas-exp0x500065b36789abff-phy5-lun-0   ONLINE       0     0     0
	    pci-0000:04:00.0-sas-exp0x500065b36789abff-phy13-lun-0  ONLINE       0     0     0
	  mirror-5                                                  ONLINE       0     0     0
	    pci-0000:04:00.0-sas-exp0x500065b36789abff-phy6-lun-0   ONLINE       0     0     0
	    pci-0000:04:00.0-sas-exp0x500065b36789abff-phy14-lun-0  ONLINE       0     0     0
	  mirror-6                                                  ONLINE       0     0     0
	    pci-0000:04:00.0-sas-exp0x500065b36789abff-phy7-lun-0   ONLINE       0     0     0
	    pci-0000:04:00.0-sas-exp0x500065b36789abff-phy15-lun-0  ONLINE       0     0     0

errors: No known data errors

This machine has many VMs