I need help send | recv an encrypted recursive zfs dataset

Operating Systems & Open Source Linux
, ,
1

Hello everyone,

this is my zpool and I have encrypted datasets on it. So far so good!

  pool: tank
 state: ONLINE
config:

	NAME                                     STATE     READ WRITE CKSUM
	tank                                     ONLINE       0     0     0
	  mirror-0                               ONLINE       0     0     0
	    ata-WDC_WD4002FYYZ-01B7CB1_K3GXMK3L  ONLINE       0     0     0
	    ata-WDC_WD4002FYYZ-01B7CB1_K3GXMW2L  ONLINE       0     0     0
	  mirror-1                               ONLINE       0     0     0
	    ata-WDC_WD4002FYYZ-01B7CB1_K3GXR4YL  ONLINE       0     0     0
	    ata-WDC_WD4002FYYZ-01B7CB1_K3GY56HL  ONLINE       0     0     0

errors: No known data errors

And this is my encrypted dataset with children.

NAME                    USED  AVAIL     REFER  MOUNTPOINT
tank                    168G  6.96T       96K  none
tank/data               115G  6.96T      192K  none
tank/data/a            10.1G  6.96T     10.1G  /mnt/a
tank/data/b            23.4G  6.96T     23.4G  /mnt/b
tank/data/c            81.8G  6.96T     81.8G  /mnt/c

I used to have this structure on a “normal” dataset and I could make offline backups easily with zfs send -R | zfs recv on an external HDD. I want to do this in case the house is burning, lightning is damaging something or whatever could happen. I have done that since FreeBSD 10. Now that OpenZFS 2.0 is available on GNU/Linux I gave Ubuntu Server 22.04 a try and I chose to encrypt the dataset. But no matter what I do, I can not receive the dataset on the HDD. I currently have it formatted as EXT4 and I do zfs send -Rw tank/data@snap > /mnt/usb/backup.zfs in order to have at least something outside the house in case I need to recover.
Here is some additional information.
This is the first few lines of a zpool history so you can see what I do to create those datasets. Maybe I do something wrong here already

History for 'tank':
2022-06-30.09:21:52 zpool create -m none -o ashift=12 tank mirror /dev/disk/by-id/ata-WDC_WD4002FYYZ-01B7CB1_K3GXMK3L /dev/disk/by-id/ata-WDC_WD4002FYYZ-01B7CB1_K3GXMW2L mirror /dev/disk/by-id/ata-WDC_WD4002FYYZ-01B7CB1_K3GXR4YL /dev/disk/by-id/ata-WDC_WD4002FYYZ-01B7CB1_K3GY56HL
2022-06-30.09:28:54 zfs create tank/vm -o compression=lz4 -o xattr=sa -o recordsize=64K -o atime=off -o mountpoint=none -o encryption=on -o keyformat=raw -o keylocation=file:///mnt/zfskey/zfs.key
2022-06-30.09:29:35 zfs create tank/vm/test -o mountpoint=none
2022-06-30.09:30:50 zfs destroy tank/vm/test
2022-06-30.09:31:17 zfs create tank/data -o compression=lz4 -o xattr=sa -o recordsize=64K -o atime=off -o mountpoint=none -o encryption=on -o keyformat=raw -o keylocation=file:///mnt/zfskey/zfs.key
2022-06-30.09:32:42 zfs create tank/vm/win2k16 -o mountpoint=/mnt/win2k16
2022-06-30.09:32:49 zfs create tank/vm/win2k19 -o mountpoint=/mnt/win2k19
2022-06-30.09:32:58 zfs create tank/vm/unifi -o mountpoint=/mnt/unifi
2022-06-30.09:33:14 zfs create tank/vm/wireguard -o mountpoint=/mnt/wireguard

And this is how it fails. I am clearly missing something here.

fox@owl:~$ sudo zfs send -vRw tank/data@dienstag | sudo zfs recv -F zbackup/data
full send of tank/data@dienstag estimated size is 69.6K
full send of tank/data/a@dienstag estimated size is 23.3G
full send of tank/data/b@dienstag estimated size is 9.95G
full send of tank/data/c@dienstag estimated size is 81.6G
total estimated size is 115G
cannot receive new filesystem stream: zfs receive -F cannot be used to destroy an encrypted filesystem or overwrite an unencrypted one with an encrypted one
fox@owl:~$ sudo zfs send -vRw tank/data@dienstag | sudo zfs recv zbackup/data
full send of tank/data@dienstag estimated size is 69.6K
full send of tank/data/a@dienstag estimated size is 23.3G
full send of tank/data/b@dienstag estimated size is 9.95G
full send of tank/data/c@dienstag estimated size is 81.6G
total estimated size is 115G
cannot receive new filesystem stream: destination 'zbackup/data' exists
must specify -F to overwrite it
fox@owl:~$ sudo zfs send -vR tank/data@dienstag | sudo zfs recv zbackup/data
cannot send tank/data@dienstag: encrypted dataset tank/data may not be sent with properties without the raw flag
warning: cannot send 'tank/data@dienstag': backup failed
cannot receive: failed to read from stream
fox@owl:~$ sudo zfs send -vR tank/data@dienstag | sudo zfs recv -F zbackup/data
cannot send tank/data@dienstag: encrypted dataset tank/data may not be sent with properties without the raw flag
warning: cannot send 'tank/data@dienstag': backup failed
cannot receive: failed to read from stream

As you can see here I created the backup pool and dataset the same way as the main pool. I think I need to do that because otherwise -R and -w can not be used with zfs send, right? Here is the first few lines of the backup pool history.

History for 'zbackup':
2022-07-05.15:43:19 zpool create -f -m none -o ashift=12 zbackup /dev/disk/by-id/scsi-SSeagate_M3_Portable_NM16JP5J
2022-07-05.15:46:03 zfs create zbackup/data -o mountpoint=none -o compression=lz4 -o xattr=sa -o atime=off -o encryption=on -o keyformat=raw -o keylocation=file:///mnt/zfskey/zfs.key

I am not active here a lot so if I screwed up how I should have made this look, please let me know and I will do my best to fix it.
If you have any idea how I could do proper backups again, please let me know.

I wish every reader a nice day :slight_smile:

2

Did the target set always have encryption?

Is this the first and only snapshot to send (parent and children)?

If there are older snapshots, perhaps try an incremental snapshot?

If there is only one set of recursive snapshots to send, perhaps send to a different target, and after successful transfer, remove the old data target?

I can’t remember the correct tags, but incramental is like
zfs send -IRvw tank@montag%dienstag | zfs receive zbackup/data

Something like that? Man page will have the flag for incramental.

3

Both the main and backup pool hold a dataset the is created equally. You can see that in the information above.

For the second question I am not sure what you mean. I do zfs snap -r tank/data@day and then I try to send it with it’s children to the backup pool with zfs send -R.

There is nothing old interfering because I have created everything fresh from new HDDs. You can see the time stamps in the information above.

There are two parent datasets which I try to backup but for the sake of simplicity I have just shown one. The issue should be the same. I don’t really have other targets to send them to. I have three backup HDDs in total. But they’re all equal.

4

Thanks,

so if fresh and new, you should not need to create /data on the zbackup set, the receive function should create this from the sending dataset?
Perhaps that is why it is saying the destination exists, and cannot be destroyed

If you wish to preserve the settings in zbackup/data, then add an extra / at the end, and tank/data (plus /a and b and c) will be copied across to zbackup/data/data and also /a /b /c datasets.

5

Like, the last errors were read errors, as it was unable to decrypt the stream (no -w means you need to provide the key to decrypt as part of the send)

The rest of the errors about dataset already existing, (and the system unable to destroy) is because it should be creating the /data dataset itself, as well as children.

Later on you can incramentally stream changes between sets of snapshots of one dataset(and children) to another with matching old snapshots.

But for a full send, the target needs to have a trailing forward slash / at the end