What's the common setup for ZFS encrypted home server

lets say we have the following setup:

  • Home server of a ATX case and micro ATX mobo
  • ZFS pool on 3 disks and boot system on SSD
  • ZFS pool is encrypted. the decryption passphrase is not in a file but the user should be prompted for it.

Here is the problem:

  • is this server is headless (without monitor, keyboard, mouse) how would it boot without passphrase
  • adding the decryption key in a file to the system defies the whole idea of encrypting the disks I would like the user to be the one able to decrypt the data
  • So if the system is headless how would it get decrypted?

The question is:

  • what is the common setup in this case? do you always have the decryption file on disk?
  • isn’t having the decryption key in the same system that is encrypted introduce a threat. as if the system is stolen or acquired by any third-party then the encryption wouldn’t be beneficial?
  • is there other ways in this case to give the system the passphrase from the user?

Disk encryption is only useful here when the data is at rest, or if you have to RMA or dispose of a disk without worrying about the data being recovered.

The chances of a casual opportunist criminal stealing your server/disks, and being interested in your data or even knows what ZFS is, is very unlikely. Best guess is that all they care about is converting stolen items into cash. If you have professionals after your data, you very likely have bigger things to worry about! :laughing:

My personal experience with FDE is only ever applied because of a policy requiring it, something like PCI Access Controls (interacting with cardholder data).

Threat level changes for everyone and every situation but you can look into:

  • a remote console so you can insert passphrase on boot
  • dracut-crypt-ssh or similar which brings up ssh during initram so you can remotely unlock it (requires unencrypted /boot). There are similar tools but they all boil down to copy the ssh keys binaries to initram.
  • If FreeBSD, maybe use a NanoBSD read only image as root and then unlock the FS after boot. Run all programs from jails inside the encrypted volume.

If someone steals your server they shouldn’t be able to break into your data. If your threat level is evil maid tier then I guess none of those work.

I disagree. Every computer storing personal data should have its disks encrypted.

3 Likes

I’m using luks and unlocking over dropbear ssh embedded into initramfs.

Perhaps there’s a similar script/plugin for ZFS?

1 Like

I have my Proxmox install on a EXT formatted SSD. It boots all the way to running.
The data array is encrypted, with prompt
I have to remote / SSH in, run “zfs load-key” enter the key at promt, then “zfs mount -a”

I can’t remember if I needed to set “canmount=noauto” but the system will boot all the way without my array mounted, and just run without that data available.

  • I didn’t know about loading ssh into initramfs method.
  • and also dropbear is a good starting point to follow.
  • I also found this gist which should lead me to more details.
  • I also found PiKVM but it felt a bit much

thanks for the pointers

There’s zfsbootmenu that can do remote SSH unlock via dracut-crypt-ssh.

3 Likes

Seconded, I highly recommend zfsbootmenu. Solves many of the potential issues with booting into a zfs on root install (so long as you set up taking automatic snapshots before and after an update). As it’s a preboot environment that doesn’t rely on the main OS working.

Kernel version too new and can’t boot? Revert to previous snapshot, remotely if you like (assuming you set up the mentioned SSH functionality). And it works with zfs’s native encryption.

2 Likes

Hmm it’s headless but do you have access to a usb port? Physically or through some ipkvm?

Allow the machine to post and enter the boot phase, and calculate the time it takes from power on to the password entry.

Now always keep this time in mind for restart/off-on. When the machine is waiting for the password, you connect a usb device that must imitate the HID/keyboard correctly and automatically inject the password and finish with enter and then the machine boots itself and the device can be disconnected immediately.

The question is what usb device would be able to do this… you will probably find a few of them.

If disk encryption supports, for example, hardware keys for decryption, you can also use yubikey in a similar way.
And since this is a home server, you probably have physical access to it to some extent. Perhaps you have the ability to connect a long usb cable and then away from the server you can decrypt the machine.

You can think of some cheap sbc, for example, pi-zero, and make a remote keyboard out of it, i.e. kvm in total simplification.

This way you have easy access as a keyboard on the server and you log in to sbc via ssh as needed. You obviously don’t keep any passwords/keys on the sbc itself.
Although I would rather choose something like zeropi(or NanoPi NEO-LTS if $$$) if it must have eth.

Personally, I prefer simplified external solutions and the server is supposed to be itself without any strange procedures within it. :wink:

I’ve been thinking about this for a while.
1 Bluetooth beakon with some sort of crypto handshake and exchange.
2 same but over ssh to something like I pizero in a separate wifi network.
Device in a ceiling cavity of otherwise hidden. Stolen computer is out of range, so no decrypt.
Could use another luks key slot to use a USB flash drive or yubi key when travelling.
And there is always the fallback of 9f entering a password.

Since we’re on the topic if encryption, there’s an elephant in the room I’d like to bring up that some don’t know may be there.

Currently, the implementation of encryption in ZFS has multiple unresolved edge case bugs, which can cause kernel panics, pool corruption or even pool destruction not just in the home pool, but in ones that receive a send as well apparently. Usually ZFS bugs are remarkably safe in regards to data, but not this time.

I don’t personally use native encryption yet, and had kinda forgotten the issues (and the very disappointing dev response) but this thread thankfully reminded me.

Here is a doc created and maintained by Rincebrain that gives a good summary of the reported issues and their status. Apparently it needs to be updated again: OpenZFS open encryption bugs (public RO) - Google Drive

And here is how to easily see all the open encryption related issues on github: Issues · openzfs/zfs · GitHub

Basically I highly recommend looking through the bugs to if what you want to do looks like one of those.

The data hazard is mitigated by not using send/recv of encrypted pools, and instead using rsync or something similar to replicate data for the time being. Personally, I’d stick with LUKs on a server if I wanted encryption for now. Not ideal, but time tested and stable.

I’ll repeat this post in my ZFS update thread later so more people are exposed to it hopefully.

2 Likes

But the bottom line is still theoretically the same, let’s use encryption, right? :wink: It’s definitely a good reminder for those who rarely make backups.

If not LUKs then what else for zfs?

On some machines I use a different approach to encryption. I don’t encrypt the entire drive, I create encrypted containers with the data that must always be kept secret. Of course, you can have a fully encrypted disk as well as containers, which I also do.

The concept of full disk encryption needs to be considered in terms of threat vectors. Some people forget that this protects our data only in certain situations.
It will not protect our data in the event of an OS hack when the drives are decrypted. And it won’t protect our data if the attacker physically takes over the server with power backup, but this is no longer a home user scenario and more of an FBI open up.

At work, we use encrypted veracrpt containers that are never decrypted on the server. They are made available to the end device where decryption takes place. In this way, the data on the server, regardless of the attack vector, is always encrypted, although of course it is quite a specific model and in many cases it cannot be used.

1 Like

Interesting. I haven’t made enough research on issues around ZFS encryption. I sort of took it for granted that it works.

I guess I’ll keep clear of send/recv then. I went with ZFS to avoid stacking LVM/LUKS/btrfs and I would like to continue with ZFS for the sake of simplicity.

Thanks for the heads up