Building Custom Kernel with ZFS Built In - Updated 0.8 or higher

[timtierney]

By popular demand. I’m posting an update to my Linux Kernel + ZFS custom build guide. As always, feedback/suggestions appreciated. I’m definitely ‘not’ a kernel building/git expert.

Things To Note / Assumptions

This procedure is written on Ubuntu 19.04. Reason being, it’s my daily-driver distro. That’s what I’m most comfortable with. However, I assume a lot of this procedure can be applied to other Distros with some adjustments.

Starting with ZFS 0.8 the SPL module will be included in the main ZFS project code. So building SPL is no longer necessary. However, if you are building 0.7.*, you’ll need to include SPL. See my previous post linked above for details.

Kernel and ZFS Git repo will be located in ~/build. For this guide, I’ll be building Kernel 5.0.9 and ZFS 0.8.0-RC4

Install Prerequisites

Install required dependencies/tools to build the Linux kernel.

sudo apt install git build-essential kernel-package fakeroot libncurses5-dev libssl-dev ccache flex bison  libelf-dev

Install required dpendencies/tools to build ZFS

sudo apt install build-essential autoconf libtool gawk alien fakeroot linux-headers-$(uname -r) zlib1g-dev uuid-dev libattr1-dev libblkid-dev libselinux-dev libudev-dev libdevmapper-dev

Download Linux kernel and ZFS Git repos

mkdir ~/build && cd ~/build

git clone git://git.launchpad.net/~ubuntu-kernel-test/ubuntu/+source/linux/+git/mainline-crack ubuntu_kernel

cd ./ubuntu_kernel
git checkout tags/v5.0.9
cd ../

git clone https://github.com/zfsonlinux/zfs.git

cd ./zfs
git checkout tags/zfs-0.8.0-rc4
cd ../

Note: To determine the correct tags. I usually use git tag or navigate to the gitub page and look at Releases.

Download/apply Ubuntu specific patches

Ubuntu has some Distro specific patches they apply to the kernel prior to building. Download and apply them.

Navigate your favorite web browser to the Ubuntu Kernel Mainline site. Find the kernel version you’re looking to build (in our example v5.0.9).

Download all the patches listed on the page into the ‘./build’ directory. I usually use wget. Ex:

wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.0.9/0001-base-packaging.patch
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.0.9/0002-UBUNTU-SAUCE-add-vmlinux.strip-to-BOOT_TARGETS1-on-p.patch

…etc…etc…

The final result will look like this:

timtierney@darter-pop:~/build$ ls
0001-base-packaging.patch
0002-UBUNTU-SAUCE-add-vmlinux.strip-to-BOOT_TARGETS1-on-p.patch
0003-UBUNTU-SAUCE-tools-hv-lsvmbus-add-manual-page.patch
0004-debian-changelog.patch
0005-configs-based-on-Ubuntu-5.0.0-14.15.patch
ubuntu_kernel
zfs

Now patch the kernel. Make sure you apply them in numerical order.

cd ./ubuntu_kernel
patch -p1 < ../0001-base-packaging.patch
patch -p1 < ../0002-UBUNTU-SAUCE-add-vmlinux.strip-to-BOOT_TARGETS1-on-p.patch

…etc…etc…

Copy Working Kernel Config and Prepare

Make sure you’re in the ‘ubuntu_kernel’ folder and execute the following

cp /boot/config-"$(uname -r)" .config
yes '' | make oldconfig
make prepare scripts

Configure, Build, and Install ZFS

cd ../zfs
sh autogen.sh
./configure --prefix=/ --libdir=/lib --includedir=/usr/include --datarootdir=/usr/share --enable-linux-builtin=yes --with-linux=$HOME/build/ubuntu_kernel --with-linux-obj=$HOME/build/ubuntu_kernel
./copy-builtin $HOME/build/ubuntu_kernel
make -j <number of CPU cores>
sudo make install

Enable ZFS and Build The Kernel

cd ../ubuntu_kernel
make menuconfig

When the menu displays, move the cursor down to Filesystems and press Enter.

Highlight ZFS and press ‘Y’ to include it in your .config file.

‘Tab’ over to Save and press enter. Accept the default filename of ‘.config’ and press Enter. Then ‘Tab’ over to Exit and press Enter

make clean
make -j <number of CPU cores> deb-pkg LOCALVERSION=-custom

Install Kernel DEB Package

Inside the ‘build’ directory, there wil now be several ‘.deb’ packages. Install them!

cd ../
sudo dpkg -i *.deb

Reboot and you should be on the latest LInux kernel with ZFS support.

[fsm]

Thanks for the guide! Spent a while playing around with Kernel building/Root on ZFS over the weekend.

When booting the new kernel, dmesg seemed to report zfs 0.8 loading, but no pool upgrades were available, and the new flags weren’t available for my pools/datasets. Haven’t had any time to look into this yet, but I’m wondering if the 0.7 packages installed during the root on ZFS guide are interfering. Uninstalling them results in an unbootable system even with the custom kernel though, so they must be required, unless I’ve done something wrong Any advice here would be welcome - I’ll be taking another look later in the week!

Few things to note from my experimentation/research.

1. Until SIMD is supported again/the limitations of not having SIMD support have been worked around, I’d strongly suggest also building in this patch to revert the export change. Just don’t go sharing your compiled kernel, as you’ll be breaking GPL.
   https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specific/linux/kernel/export_kernel_fpu_functions.patch

Without SIMD support ZFS performance on encrypted datasets drops immensely - so you really need this patch if you want to use native encryption currently.

Open issue here: https://github.com/zfsonlinux/zfs/issues/8793

2. Friendly pointer - You can use nproc to return the number of Threads in the system you’re building on. So you can simplify make -j <number of CPU cores> to make -j $(nproc)

3. I ran through the Root on ZFS guide for 18.04, and can report it works fine on 19.04 with no changes (Apart from using disco instead of bionic for debootstrap and your sources.list)

4. if your kernel config doesn’t get generated at /boot/config you can also find it at /lib/modules/$(uname -r)/build/.config

[fsm]

Following on from this - my issue was indeed with the 0.7 packages. After a bit of experimentation, I’ve now got a fully functioning natively encrypted root on zfs install, running ZFS 0.8, linux kernel 5.1.4 with ZFS built in/ export change reversed on Ubuntu 19.04. For installation, I combined this guide with this + my own experimentation and built my own zfs 0.8 binaries (Anyone planning on doing this, the build scripts for the initramfs are slightly broken… If you don’t copy zfs-functions into /etc/zfs/ - you’ll end up with a kernel panic on boot. See here

From what I understand (Someone please correct me if I’m wrong!) ZFS is split into the user packages and the kernel packages. What we’re doing by building ZFS into the kernel, only means we don’t need to bother with the dkms module. We still need to install the user packages along with the initramfs image - as these are not integrated into the kernel.

Happy to write up a guide for the whole process, though I’m sure the official zfsonlinux guides will be updated before long

[Bytales]

Hi there, i tried this procedure on a linux Mint 19.1 Tara. Installing the deb packages worked except for a minor Error:

Error: (dkmas apport): kernel package Linux-headers-5.0.9-custom is not supported
Error! Bad return Status for module build on kernel: 5.0.9-custom (x86_64)
Consult /var/lib/dkmas/amdgpu/19.10-812274/build/make.log for more Information

I dont know what that means, but after restart, running “uname -r” Displays 5.0.9 custom, and it seems ZFS 0.8.0rc4 is onboard.

Any ideea what that error is or if ist bad ?
Also, would like to know, what would i Need to do to make a build with the latest Version ? As now there are more kernel Versions available. 5.1.0 is the latest finished, and 5.2 is in the release candidate stage.

[fsm]

Not sure about the error, seems like the amdgpu module failed to build. Check the logs - can you update it?

FYI - don’t use rc4 -it’s a pre release build, and stable is now out… The official 0.8.1 release came out yesterday as well… Use git tag to confirm tags.

When using git, to build another kernel, just checkout the kernel tag you want, or download the source elsewhere

[Bytales]

I built 5.1.9 with ZFS 0.8.1, which is what im running now on top of mint 19.1, and its working. Still meddling with the ZFS, its amazing to have it working here in linux. So in the future, i will build the combo, install it, and then install AMD gpu drivers to make sure everything runs okay.
Tell me, the step:
cp /boot/config-"$(uname -r)" .config
Does it copy something from the original kernel ? i dont get it ? it copies the “config-@kernelversion” into the new to be built kerne ?
What is it ?

[Novastark]

I was just trying to use one of the ubuntu mainline pre-comiled 5.25 kernels @ http://kernel.ubuntu.com/~kernel-ppa/mainline/?C=N;O=D

and I didn’t even notice the absence of ZFS when my modules were being rebuilt. I’m not doing zfs on root, only /home atm, so I made it as far as the login screen before I realized I was doing it wrong

In hindsight I realized I hadn’t read about dkms support for ZFS and even now there’s not much info about the zfs-dkms package. Anyone know if it’s safe/maintained or should I buckle down and do a custom kernel like this guide outlines? I was trying to test if kernel 5.25 would fix a VFIO error I was getting so I was trying to take a shortcut.

I guess in the end I’m wondering if there any con’s to using DKMS other than it being less performant vs a custom kernel? It’d be nice if I could just swap between kernel versions as I test without having to rebuild everything everytime. But if I have to I have to.

Edit Well, I threw caution to the wind and tried installing the zfs-dkms package. It rebuilt the zpl module for the standard 5.0 kernel that comes with pop but failed to do so for 5.25 (complained that the headers weren’t supported). Looking at the ZoL webpage it looks like ZoL might not be ready for 5.2 and I’m just wasting my time? I found a phoronix news article saying 0.8.1 was supposed to have 5.2 support but I’m guessing the timing was off because the official site only says up to 5.1.

I guess I’ll try 5.1 but I’m not to hopeful. We’ll see.