So I taught UEFI how to boot from ZFS... but

Operating Systems & Open Source Linux
1

What is all this about?

I was looking for a way to boot directly from a ZFS-Root pool in order to include kernel and initramfs into snapshots for easy rollback. I know there is GRUB2 which has ZFS read-only capabilities but its “bloated” and thus pretty slow on boot and ain’t nobody got time for that…
Another solution would have been rEFInd which allows loading drivers from the ESP partition. There is a lot of available drivers provided by Pete Batard’s EfiFs. Unfortunately trying this always featured a bunch of driver-related errors claiming the location it’s reading is corrupted, followed by loading the vmlinuz image with an empty string handed over to the ZFS target dropping me into emergency shell. So dead end here too.

I eventually got curious about EFISTUB booting so I fired up an EFI shell and tried loading the driver directly (this is also found on the driver website):

\> fs0:
fs0:\> load \path\to\driver\zfs_x64.efi

Success! Now we need to load the kernel and add a few parameters (found in ArchWiki):

fs0:\> \vmlinuz-linux zfs=<poolname> rw initrd=/initramfs-linux.img

And a few seconds later it successfully loaded Arch Linux on ZFS, allowing me to login.

Okay, it works but this is super clunky and not automated in the slightest. I want this to be a seamless experience.

Attempt #1 - Using a script:

I put the above commands into a little script archzfs.nsh and saved it to the ESP. Then I generated a boot entry using efibootmgr:

# efibootmgr --create --disk /dev/sdc --part 1 --gpt --label "Arch Linux ZFS" --loader \archzfs.nsh

Which again worked in EFI shell however any attempt to boot it automatically ended in skipping the entry and booting the next OS on another drive. Actively choosing it by pressinf ‘F2’ and navigating through the BIOS didn’t work either (it flickered briefly but nothing happened).

Attempt #2 - BIOS modding:

Disclaimer: This is dangerous with a tendency to brick your motherboard. Do NOT do this unless you know what you are doing!

The necessary tools and instructions are available on Pete Batard’s Github Wiki and the Win-Raid Forum.

What I am doing here is the following:

  1. Turning the .efi driver into a BIOS compatible .ffs file using FFSTool
  2. Downloading your BIOS firmware from your motherboard manufacturer’s website
  3. Opening the firmware using UEFITool and search for “DXE” (I recommend looking very closely at the Win-Raid guide)
  4. Find the undermost DXE driver entry and insert your .ffs driver after it
  5. Save this and flash it to your motherboard

As I am using an Asus board which has Asus’ Flashback it was of pretty much no danger to me as I could replace the firmware as long as the board is powered. Your mileage may vary so again, do NOT try this unless you know what you are doing.

The result:

The motherboard posted successfully and entering drivers in EFI shell shows the ZFS driver is loaded - this allows the command

fs0:\> \vmlinuz-linux zfs=tank rw initrd=/initramfs-linux.img

to work without needing to specifically load the driver as it’s now part of the firmware.

However, booting by EFISTUB still does not work and results in brief flickering and nothing else as described before. I used this command and a bunch of variations of it to generate the entry:

# efibootmgr --create --disk /dev/sdc --part 1 --gpt --label "Arch Linux Native-ZFS" --loader \vmlinuz-linux --unicode "zfs=tank rw initrd=/initramfs.img"

I couldn’t get it to work but still I have a feeling of being very close to the solution. Since I ran out of ideas I welcome your input on this topic. Maybe someone can figure it out or points to the simple error I am doing. :smiley:

Thanks for reading!

6 Likes
2

No ideas as of yet, but I’ll definitely have to have a crack at this when I get a couple free days.

3 Likes
3

This probably isn’t your problem, but is it possible to use UUID for efibootmgr --disk just in case your devices get mixed up in /dev?

Hope you’re able to figure it out. This is something I’ve always wanted to do.

2 Likes
4

Do you have non-zfs disks in your system? If they’re all ZFS, it shouldn’t matter. It will see the disk, load the pool and you’re off to the races.

I would try unplugging any non-zfs disks just to see if it works. (moonshot though)

1 Like
5

The trick is, as I understand it, that efibootmgr takes the UUID to the corresponding device it finds at /dev/sdX. If you issue efibootmgr -v it looks like this:

$ efibootmgr -v
BootCurrent: 0001
Timeout: 1 seconds
BootOrder: 0001
Boot0001* UEFI OS	HD(1,GPT,fa1065b9-60e1-499d-b642-44583340f223,0x800,0x80000)/File(\EFI\BOOT\BOOTX64.EFI)..BO

This is the ESP residing on my SSD. This one houses my kernel and initramfs though as it’s my workhorse. The test drive currently is detached. :smiley:

There is a total of 3 drives and they all hold their own ZFS pool in some form (in brackets I explain the partitioning layout I manually set):

  • 1x NVMe SSD - “nvme” (4 partitions: ESP (kernel location), ZFS-Root, SLOG, L2ARC)
  • 1x WD Red 3TB - “hdd”
  • 1x 60GB HDD with USB3 enclosure - “tank” (2 partitions: ESP, ZFS-Root (kernel location)) <-- This is the interesting one

I wonder if the driver is actually loaded at the point of calling EFISTUB or if it’s just automatically loaded whenever an EFI shell is booting. That’s one of the many things I don’t know about UEFI.

6

yeah, I’m not a pro when it comes to UEFI. Might have to work with OVMF to try and figure it out.

7

The UUID will always identify the same drive. Depending on what other storage devices you have plugged in, a drive might be on /dev/sda sdb sdc etc. For instance, I have a server with 3 drive bays. If I have a flash drive in the usb port, it mounts at /dev/sdc and pushes one of the hard drives to /dev/sdd. The UUID always follows the same drive/partition/whatever which is why you usually see the root mount point in fstab point at a UUID and not a/dev/sdX.

Presumably, you’re not connecting/disconnecting disks while working on this project, so it shouldn’t be causing you any problems, but if the final solution should account for this.

Also, I have no experience messing with the UEFI, so I’m not sure how much of that carries over or what caveats the UEFI introduces…

Wish I could be of more help.

8

Are there any benefits over rEFInd + ZBM (zfsbootmgr)?