Hi folks! I’m hoping some of you might help me with a new build.
I want to set up an Ubuntu Server w/Docker and ZFS for file-/media services (like Plex, NextCloud, Calibre +++). All the hardware is purchased and I have an idea about how I want to set up my drives. In concept I am trying to segregate the data from the OS / system as much as possible and have all data on ZFS pools for redundancy and easy backup.
I have some familiarization with Ubuntu, Docker and ZFS, but I fall short when it comes to how I can separate the Docker data volumes out to a disk preferably on individual datasets pr. container so that each one can be configured with ZFS properties fit for the content (e.g. setting recordsize=16K for database data).
I don’t know if this is even possible without installing Docker itself on a ZFS filesystem (i.e. Root on ZFS). However, I have read that his Root on ZFS together with Docker might be a bit troublesome so I was thinking to just use a single drive for my root wo. ZFS and rather store the Docker data itself on a separate ZFS pool.
The drawing shows what I would like to achieve in theory. The red color illustrating the parts I could need some advice and guidance on and how to actually implement in practice. I am quite new at all this so it might be that I need some detail feed in with a teaspoon 
I’m working towards something, but I currently have ZFS on root (just toggled option to use ZFS during Ubuntu installation). I don’t have any advice to offer, but please let me know how this configuration worked for you & if you have any tips! Excellent diagram btw
As you already noticed, there is no official support for zfs-on-root on Ubuntu Server. Ubuntu desktop however has ZFS-on-root as an install option. Repurposing Desktop with additional server-related packages may be an acceptable compromise to you.
The official ZFS documentation on github has a step-by-step guide how to make zfs-on-root yourself for various distributions, including Ubuntu.
There are always options, some are easier than others.
I’m not sure if you can change Docker volumes or bind mounts away from /var/lib, but if Docker can be configured to run of e.g. off /mnt/docker or whatever, you can use NVMe as boot drive and create pool with all remaining disks and mount datasets where you want them via zfs set mountpoint=/mnt pool/dataset
But I really have to pass on the Docker issue otherwise.
Having everything in one single pool comes with several advantages. You can use the two small, but very nice enterprise-grade, SSDs for caching or special vdev and the NVMe as a L2ARC. That will make a nice tiered pool that can cover small recordsizes in reads and writes and still having 16TB of space. Not only will general performance improve, but administration, maintenance and backups are so much easier with a single pool. I like your striped mirror approach, as it not only is more performant, but also allows you quick and easy expansion or removal of an entire mirror of your pool in the future.
Regarding your nice drawing, you don’t want docker-data/volumes and pool-name/movies. In a single pool, with a dataset geared to small recordsize and other docker-optimizing properties, you end up with pool-name/docker-data/volumes, and also give datasets for media huge recordsizes (1M or higher) ,change caching and compression for these kinds of data.
The flexibility of ZFS to tailor and create filesystems at will (datasets,directories, mounts however you want to call it), to fit into any use case, is something I learned to appreciate a lot myself.
