Hey all, I didn’t find a recent thread on this topic. I wonder if anyone yet installed Ubuntu 22.04 (or any of its flavours) with root on ZFS? Are you happy with it? Do you use it in “production”?
I’m about to update my main rig to 22.04, with a new install. It will likely be based on desktop Xubuntu. I hoped root on ZFS via the installer would be “official” by now, however instead it looks like they almost pulled it out before release. Seems a bit discouraging, but nevertheless I want to hear your thoughts.
I got ZFS via Ubuntu 22.04 on my Laptop. Implementation is a bit different than your usual ZFS-on-root approach, but works flawlessly. 20.04 also had ZFS support and 22.04 didn’t change much. It’s still very basic when it comes to options and you only can do a single-disk stripe vdev during install, but 22.04 now also offers native disk encryption.
Thanks, good to know that it is in a similar state as before, at least. It implies that few things would have had an opportunity to break recently, and that we can trust the (collective) experience accumulated with previous releases. I didn’t try it seriously before, but I will now.
I guess you can add a second drive for redundancy after install? Not that I would be in a hurry with that, nightly snapshots sent elsewhere would suffice for a decent integrity of the root fs (at least compared to what I ever had with ext4).
Like you can with every ZFS. I’m not sure how it works for e.g. mirroring the bpool (boot pool) as I only use striped vdevs and bpool just sits on one disk. But I can confirm that zpool add and zpool attach work in my 2x NVMe configuration for the rpool. Once it boots and runs, it’s just ZFS and works with all features. And version is 2.0, so a rather recent release by Ubuntu standards.
If you want ZFS-on-Root, Ubuntu Installer is the easiest way to get it on Linux. And I’ve never seen mismatch of kernel versions and ZFS versions that fuck up the system even with 20.04. Ubuntu conservative release schedule, especially on an LTS, pretty much guarantees ZFS to work.
I also edited my grub so I can boot into snapshots. Really nice for people like me , being clumsy and doing stupid things at times.
Ubuntu 22.04 is really ZFS for the people. I’d like to see more configuration options like mirror or RAIDZ with multiple disks during install and maybe some GUI tool, but other than that there is no alternative other than doing it manually, which is quite a bit of work.
Take what I’m about to say with a grain of salt, as I’m far from being an authority on this but…
I’ve been running root on ZFS on my Debian install from around the release time of Debian 10. The Debian installer did not and still doesn’t have a built in option for root on ZFS, so I had to install it by following the instructions provided by the OpenZFS docs.
In the time I’ve been daily driving my Debian install, I’ve had no issues but again… grain of salt.
I have been thinking about reinstalling and maybe distro hopping though. So I’ve looked into Ubuntu and decided no, for the same reasons I did last time, but I did go through the OpenZFS instructions for installing root on ZFS - Ubuntu — OpenZFS documentation
The instructions for 22.04 currently uses Canonical’s zsys, which has some pretty significant issues and has been put on the back burner by the devs. So I ran through the instructions for the most recent version of Ubuntu that didn’t ship with zsys. And aside from a few changes to the release name in the sources.list section, and a small issue with connecting to the internet which may have been my own doing… the install went just fine… in a VM.
Make of that what you will.
Copied from the TrueNAS forum to save time.
I’ll give you a quote from Matthew Ahrens, one of the cofounders of ZFS at Sun Microsystems and current ZFS developer at Delphix, and let you decide.
"There’s nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem. If you use UFS, EXT, NTFS, btrfs, etc without ECC RAM, you are just as much at risk as if you used ZFS without ECC RAM. Actually, ZFS can mitigate this risk to some degree if you enable the unsupported ZFS_DEBUG_MODIFY flag (zfs_flags=0x10). This will checksum the data while at rest in memory, and verify it before writing to disk, thus reducing the window of vulnerability from a memory error.
I would simply say: if you love your data, use ECC RAM. Additionally, use a filesystem that checksums your data, such as ZFS."
[EDIT] For any other intrepid root on ZFS new installers… probably specifically non-US English users. I did run into another minor issue with Gnome and opening the gnome-terminal. Basically, it would not open, but KDE’s konsole would. Turns out that if I swapped my main language to US English and then back to my preferred (ahem… far superior British English), it would fix the issue with opening the gnome-terminal.
Thanks, I understand from the bug report that the unreadiness of zsys is the reason for ZFS on root still being experimental. I would not want to follow a checklist though, part of the attractiveness for me is that the root fs is set up by the installer - I usually organize my systems so that reinstalling is fairly easy if something breaks. That would be defeated by even a Gentoo-style checklist
I’m not entirely up to date yet about what zsys brings to the table, from my reading of Didier Roche’s slightly old blog posts it seems to be mostly about tracking parallel branches of the system. I’d like to be able to rollback a failed system update, but I’m perfectly fine with that rollback being irreversible.
I’m somewhat familiar with BEadm on OpenSolaris/Illumos, from the pre-Oracle era.
Nice. Are you doing this by a method unrelated to zsys?
The Ubuntu installer still has ZFS support, but it was almost removed for 22.04 and it no longer installs zsys. At the moment, this HOWTO still uses zsys, but that will be probably be removed in the near future.
Not your usual flavour of debian, but I am running root on zfs on my proxmox workstation, didn’t use the installer on this machine as I originally installed without zfs and migrated to it afterwards, but I have deployed mutiple proxmox VMs and boxes now and it’s as simple as choosing zfs as an option
For added bonus, I also send daily to my truenas server using znapzend
root@pve:~# pveversion
pve-manager/7.1-12/b3c09de3 (running kernel: 5.13.19-6-pve)
root@pve:~# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 11 (bullseye)
Release: 11
Codename: bullseye
root@pve:~# zpool status rpool
pool: rpool
state: ONLINE
scan: scrub repaired 0B in 00:01:17 with 0 errors on Sun May 8 00:25:19 2022
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
nvme1n1p2 ONLINE 0 0 0
errors: No known data errors
root@pve:~# znapzendztatz -r
USED LAST SNAPSHOT DATASET
112K znapzend-2022-05-17-000000 osxwork
288G znapzend-2022-05-17-000000 osxwork/vm-103-disk-0
0B znapzend-2022-05-17-000000 rpool
0B znapzend-2022-05-17-000000 [email protected]:hd01/ZFS-Snapshots/PVE
0B znapzend-2022-05-17-000000 rpool/ROOT
0B znapzend-2022-05-17-000000 [email protected]:hd01/ZFS-Snapshots/PVE/ROOT
5.95G znapzend-2022-05-17-000000 rpool/ROOT/PBS1
1.39G znapzend-2022-05-17-000000 [email protected]:hd01/ZFS-Snapshots/PVE/ROOT/PBS1
5.25M znapzend-2022-05-17-000000 rpool/ROOT/PBS1/etc
6.75M znapzend-2022-05-17-000000 [email protected]:hd01/ZFS-Snapshots/PVE/ROOT/PBS1/etc
0B znapzend-2022-05-17-000000 rpool/ROOT/PBS1/home
0B znapzend-2022-05-17-000000 [email protected]:hd01/ZFS-Snapshots/PVE/ROOT/PBS1/home
2.92G znapzend-2022-05-17-000000 rpool/ROOT/PBS1/usr
3.24G znapzend-2022-05-17-000000 [email protected]:hd01/ZFS-Snapshots/PVE/ROOT/PBS1/usr
12.0G znapzend-2022-05-17-000000 rpool/ROOT/PBS1/var
13.1G znapzend-2022-05-17-000000 [email protected]:hd01/ZFS-Snapshots/PVE/ROOT/PBS1/var
Personally I’d steer clear of root on ZFS with any Linux given the hostility between kernel and ZFS development over licensing. Kernel changes have broken ZFS in the past and it’s clear a number including Linus himself don’t care about that.
Small boot/system partition/drive, data on ZFS is what I’d do.
I mean you can make it work. Maybe you’ll be safe if you stick only to official Ubuntu kernels. But I’d avoid the potential headache myself.
That’s interesting, I didn’t know about it. I’ve looked at Jim Salter’s Sanoid/Syncoid that has been around for a while, and also Zrep, but not yet in enough depth to decide which to use. Znapzend is apparently another alternative. They all seem to have slightly different focus. Did you consider other softwares too? If so, what made you decide on znapsend?
I see your point, and I agree about the license being an issue. Though it is an issue for ZFS generally too. I would be ok with taking the risk, “political” changes are different from software breakages in that they happen with some warning, so that I can plan ahead. Also, I foresee sticking to the official kernels, at least the hwe branch.
(I haven’t had the need to compile a kernel since 2012, when experimenting with PCIe passthrough and XEN, and I don’t expect to need it any time soon.)
The ZnapZend configuration is stored as properties in the ZFS filesystem itself.
That means that wherever I move a zfs pool, it’s backup/snapshot config is automatically there as long as I start the znapzend daemon.
All other solutions still require a daemon or a cron log, plus local config files
For me, it keeps things simple and I like simple, and I often forgot custom configs months after I have done them, and have to re-learn them to understand what I did
I agree, I had in mind “ZFS in Linux contexts generally” but failed to specify that. The license mismatch is a real thing (in contrast to with BSD/Illumos), and I consider Linux devs’ attitude mostly a downstream effect of that. It’s less than ideal either way.
