ZFS replication source and destination sizes don't match

Operating Systems & Open Source Linux
, ,
1

I’d like to start this off on a positive note :3

I said I’d never touch ZFS again but TrueNAS made quite for formidable offer (thanks ixSystems!) but that doesn’t mean it’s been a bumpy-less road but the TrueNAS and OpenZFS communities and their vast documentation have been absolutely helpful!

Now, onto what’s going on.

I needed to convert my RAIDZ2 4x16TB vdev to a mirrored 2x16TB vdev because I wanted to free up two drives for other projects and I wasn’t realistically going to occupy 16TB at breakneck speed.

So, I replicated the contents of the spinning disc vdev (creatively named tank) to my SSD-based vdev’s dataset for storing this data, moby/tank.

I ran replication on it and while tank was 2.26TB, moby/tank was 2.24TB and I cannot for the life of me figured out why.

I have no recollection what changed between my first and second attempts (they were both full filesystem replications) but somehow during my second attempt, the source and destination both matched 2.26TB so I called that a success and destroyed tank and recreated it.

Now comes the restoration and where the problems are starting to arise, I restore moby/tank/tank (the psudeo-root of the tank vdev is the tank dataset) to tank/tank and… now the sizes is mismatched.

2.26TB on moby/tank but 2.24TB on tank. I’ve heard of size increases due to mismatched compression parameters but this makes no sense to me, how does something shrink when it’s supposed to be a raw (because, encryption with post-replication decryption) full filesystem replication.

Here’s some stuff that makes it kind of confusing to me

> $ zfs list moby/tank moby/tank/tank moby/tank/proxmox -o name,logicalreferenced,logicalused,referenced,used
NAME               LREFER  LUSED     REFER   USED
moby/tank             86K  2.93T      314K  2.26T
moby/tank/tank      97.5K  2.88T      326K  2.21T
moby/tank/proxmox   84.5K  46.2G      279K  54.5G

> $ zfs list tank tank/tank tank/proxmox                -o name,logicalreferenced,logicalused,referenced,used
NAME               LREFER  LUSED     REFER   USED
tank                  86K  2.93T      240K  2.24T
tank/tank             99K  2.88T      272K  2.20T
tank/proxmox        84.5K  46.2G      216K  42.7G

The logical used is the same, the refer is almost the same (a slight increment, even) but refer and used have gone down… after a raw transfer?

I’ve looked at the snapshot directories and don’t see a reduced number of snapshots so, what gives?

The only property difference I can think of that comes to mind is that tank (and its descendents) have read-only disabled and moby/tank is still read-only. I have attempted replications four or five times with the exact same results.

What’s going on?

2

Compare dataset and pool settings.

zpool get all <poolname>
zfs get all <poolname>/<dataset>

  • Are both using the same block size on the devices (ashift=9/12)?
  • Are both using special devices (to store metadata), or not?
  • Are both using the same recordsize on the dataset?
  • Are both using compression (or not)?

These are settings that affect the physical size on device despite the data being identical between pools. There are possible more settings, but these are the obvious ones.

3 Likes
3

Are both using the same block size on the devices (ashift=9/12)?

Both are setup through TrueNAS so probably…

> $ zdb -U /data/zfs/zpool.cache | grep ashift
            ashift: 12
            ashift: 12
            ashift: 12
            ashift: 12
            ashift: 12
            ashift: 12

Seems like this is in line with TrueNAS’s sane defaults

Are both using special devices (to store metadata), or not?

Nope, neither of them touch special devices at all

Are both using the same recordsize on the dataset?

NAME            PROPERTY                VALUE                   SOURCE
tank/tank       recordsize              128K                    default
tank/proxmox    recordsize              128K                    default
moby/tank/tank  recordsize              128K                    default

Are both using compression (or not)?

In theory, for the purposes of raw filesystem replication, this shouldn’t matter at all as the datasets are encrypted and tank/tank has no clue about the decryption keys needed for moby/tank/tank and so can only deal with raw streams (I had to export the keys beforehand and then edit the JSON file and re-import it for decryption to work)

Sidenote, not really related to this but two things about ZFS really tick me off, replication screws up encryption roots and there’s no way to change it (there are some posts that indicate using zfs change-key might work but that changes the key itself, not the key root) and two, there is no de-fragmentation mechanism, that’s probably one thing I miss from BTRFS but moaning aside, there’s a reason that despite it all, I still use ZFS :3