The Ultimate Home Server - Herd of Netboot Raspberry Pi? Sure

Part 2 of this is coming featuring Ansible tips from @geerlingguy if I can talk him into it haha

O84HvSe

(He is really good!! also welcome!!)

13 Likes

I actually did… and just got to chapter 6 in the ansible book.

1 Like

One thing to note

Docker will allow containers to use NFS shares, however Docker itself will not run from an NFS share. Dockers default storage driver is overlay2 , which uses overlayfs , which as far as I know does not support NFS. An alternative would be to use an iSCSI share instead.

It’s a fairly important point considering the popularity of both Raspberry Pi’s & Docker are in the homelab.

The question is, could you use the zfs filesystem driver with Docker on an NFS share?

2 Likes

Heh, thanks! I’ve been meaning to cover netboot in a video someday (and to convert all my racked up Pis to it, so I can ditch their hard-to-access microSD cards), but until then, I’ll be pointing people at this forum post. Thanks for posting all the details!

14 Likes

@geerlingguy covers the new beta of Network Install:

5 Likes

Yeah, the inclusion of pi 4 images is nice, I still have my original self-compiled image running and it’s still going strong. And Void has a lot of packages in the repo. But, for the Pi, especially if you run one with <2GB of RAM, I would highly recommend Alpine if it fits your needs.

Heck, even with maintaining a bunch of SD cards, Alpine Linux on the Pi is running by default in Diskless Mode (“frugal mode”), which reads the image from an SD card (or storage of choice) and then loads the whole system onto RAM. If you want, you can unplug the SD card after the boot is over and it will be fine, so long as you don’t have any partition from the SD still mounted (only for safety of the SD, the system will still be fine without it, even forcefully unplugged).

I’m running Alpine on my pi 3 from an ancient 2 GB microSD card. Whenever I do updates, I just do a lbu commit and the directories I have set to get backed up are stored on the SD card. I can then install whatever crap I want, but after a reboot, it will revert to the previous state from the last commit and not have anything else installed, as those were only installed in RAM and not committed to disk. That way, I get basically 0 read or writes on the SD card besides when doing an update and a lbu commit.

What lbu commit does is creating a tar archive on whatever mount point you give it. It doesn’t have to be on the SD card, you can boot a completely generic Alpine image via PXE and have Alpine get its own tar.gz archive to extract on its RAM FS. This way, you even cut the network cable (so long as you don’t use PoE obviously) and you can eliminate network read/writes, you don’t need to have your / partition mounted on a NFS server.

Jeff is a L1T Forum member? Noice.

@geerlingguy @rustyb0y welcome to the forum, both of you. A bit late on my part, but nonetheless, welcome! Hope you’ll feel like home here, we have a lot of interesting topics that you can follow, from Wendell’s posts, to our forum members blogs featuring anything from server stuff, audio stuff, or pop culture things. Here’s another post in which I mention the most prominent blogs on the forum:

1 Like

then perhaps the thing to do is back the iscsi lun on the fileserver as a zfs block device for the actual docker containers. to appear to the rpi as a natively formatted partition (onto which that can be overlayfs). so a really good question is what is the most effecient and performant filesystem to put onto that zfs backed iscsi lun, once it also has a working overlayfs sitting ontop. for your docker build times etc. thats an interesting thing to benchmark.

moving on. if you bind mount specific folders into the containers. for actual data, those can be zfs folders or datasets. which can then be doubly shared over nfs

you also want to be able to re-create the docker containers should the monolithic partition become bad or messed up. so if it includes docker volumes which are not bind mounted then those need to be either recreatable from an immutable template. or regularly backed up elsewhere. then restored… hence the suggestion of bind mounted folders

perhaps a more elaborate and interesting solution is kubernetes. but that would be much more complex to configure and setup. also its hard to get extra rpis right now for a cluster. unless they are some other different alternative device

Agreed, I generally advice new admins, who is looking into learning - to consider recycling a old laptop and/or desktop that they can ask around for “free” (especially netbooks)

Unless they are space (ie. living in a dorm), or energy constraint - most “do X on PI” tutorials, can be replaced by a general computer - and is much cheaper, and less intimidating to experiment on a “PC” for newcomers.

PI, to me should primarily be used in IOT use case. For “server” workload use case, its more of for rule of cool, and youtube keyword optimisation. (nothing against jeff, its cool)

Nice write up, picked up a bunch of dell Wyse 3040’s and netbooted them using parts of this guide as a reference. I used the MAC instead of getting the output of vcgencmd.
Is there a simple way to automatically update the tftp server to point to a new kernel after upgrading it? I know apt has a post upgrade routine I can add a bit into, or just have it in my bashrc to check every restart and update the pxelinux file with the new kernel.
Just wondering if there’s a quicker a neater solution that anyone has used?

1 Like

The documentation on the Raspberry Pi web site (Raspberry Pi Documentation - Remote Access) suggests:

nfsroot=1.2.3.4:/nfsroot/rpi-pxe/rpi1,vers=4.1,proto=tcp

When starting up a recent Raspbian image the kernel (5.15.30) messages seem to indicate NFS 4.1 support for root. My NFS server (Debian 11) communicates using NFS 4.2 if not specified so setting vers=4.1 is required in my case. I assume if you have an older NFS server, the version negotiated would be old enough to not be required in the cmdline.txt file but network booting might mysteriously stop working if the NFS server is updated.

2 Likes

This might be muddying the waters but the Raspberry Pi documentation - Network booting (Raspberry Pi Documentation - Raspberry Pi Hardware) seems to indicate that network booting is possible on Raspberry Pi 400, 4, 3B, 3B+ and 2B v1.2.

  • For Raspberry Pi 4 and 400 you can update the EEPROM if needed
  • For Raspberry Pi 3 you might be able to enable network book using the built-in firmware
  • For Raspberry Pi 2B v1.2 and 3 you might need an SD card with an updated bootcode.bin - Special bootcode.bin-only boot mode (Raspberry Pi Documentation - Raspberry Pi Hardware)

As @MadMatt says, BerryBoot might help out with older hardware and WiFi booting with iSCSI but their web site does not link to the images for Raspberry Pi 0,1,2 last updated June 2019 so maybe not a good long term option.

nfsroot=1.2.3.4:/nfsroot/rpi-pxe/rpi1,vers=3

^^^ this!! after 2 days of fiddling! thank you (pi4 and vers=4 here)

THANK YOU!

5 Likes

So instead of thinking about important stuff, I looked at the example fstab and wondered what the “pass” number should actually be in this case. “0”? I’m going to go with “0”. Is the pass number actually used by anything these days?

topic for another video. Since you’ve done navidrome, some sort of steam/game cache. It looks like it required port 53 for DNS… BUT I am already running pihole.

Not sure if helpful, but I’ve been meaning to do this for a long time and keep getting hung up on things not working as simply as the guide or videos make it appear.

In any case, the nfs mounting part of getting a pi to mount directories wasn’t clear and at this point I’ve managed to learn and forget how to do it twice.

I use a Synology NAS, so I’m working within the bounds of what it has for dials and switches in terms of TFTP and NFS shares and permissions. For my fellow Synology users, pay close attention to the “Mount path” notice on the bottom left of the NFS Permissions tab of the shared folder edit dialog. All NFS Shares have the Synology Volume name prepended to the share name.

So instead of running sudo mount 10.0.0.5:rpi-pxe/rpi3 /nfs/root you gotta run sudo mount 10.0.0.5:/volume1/rpi-pxe/rpi3 /nfs/root
*assuming that your share is on a volume named “volume1” and your synology is at IP address 10.0.0.5.

Permissions and read/write issues are still a little beyond my understanding. For the most part it seems like to edit most things I have to be running as root on the pi.

Hope that helps!

I’ve gone through this tutorial as well as the one linked on mikejmcfarlane. I’ve gotten pretty far along but I’m a bit stuck and have a lot of questions.

First question, what version of NFS is everyone using here? It seems like Wendel is using NFS version 4. I was having trouble setting up the exports properly with NFS version 4. What seemed to work for me was creating the pi user and group on my NFS/TFP server with the same uid and guid as the pi users on the pis. Here is what my exports look like:

/nfs/pi-pxe x.x.x.x/x(rw,async,all_squash,no_subtree_check,anonuid=1000,anongid=1000)

/nfs/pi-tftpboot x.x.x.x/x(rw,async,all_squash,no_subtree_check,anonuid=1000,anongid=1000)

I finally got my NFS server and mounts working as well as TFTP and added option 66 on my DHCP server.

I was able to get the raspberry pi to boot but I ran into a kernel panic. I got the kernel panic with both the 64bit and 32bit images.

I attached a screenshot of my kernel panic.

So just to summarize.

  1. What version of NFS is everyone here using or is Wendel using?
  2. Can I get some examples of how the NFS exports are set up?
  3. Did everyone else have to create a raspberry pi user on the NFS/TFP server so that the uid and guid match?
  4. Any ideas on why I’m getting the kernel panic?

Quick update I went through the whole process again using an old raspberry pi image from 2020 (buster). I got the same kernel panic issue. I double-checked everything. Not sure what I’m doing wrong.

Hi @wendell if you get a chance to look at this, any ideas?

Thanks!

I’m not netbooting RPi, but Odroid HC4, N2+ and RockPro64 (well, the last one ain’t netbooted anymore, because it’s a router now, but did during some testing).

Using NFS v4 on my server, but some devices will just downgrade themselves to nfs v3. Which is fine.

The kernel says something about init=. It almost looks as if you are not booting your init system, which on Raspberry Pi OS is systemd.

Do a few things. Mount your SD card like normal, do an apt update and apt install nfs-common. It should already be installed. Then try mounting the NFS server locally on the pi using mount.nfs ip-of-nfs-server:/path/to/rootfs /mnt/. If it works, good, edit your /mnt/etc/fstab and replace the / with the nfs server’s mount point, kinda like so:

# <device>                        <dir> <type> <options>         <dump> <fsck>
ip-of-nfs-server:/path/to/rootfs    /    nfs    defaults,noatime    0    0

Note that normally you would have fsck = 1 for root, but since root is on NFS, no need to do that, the NFS server will take care of file system checks (if you’re using ZFS, you have your scheduled zfs scrubs). And comment any lines about a swap partition if you have any of these.

Then you poweroff, unplug the SD and try booting from the network again. If that doesn’t work, let us know.

No.

Here’s what my export looks like (my TFTP and NFS server are the same, using the rootfs/boot folder for tftp, makes life easier):

/mnt/netboot/rootfs/hc4         bikyhc4.local(rw,sync,no_root_squash,no_subtree_check)

You want your NFS to use synchronous writes, not async. It is slower, but you want to make sure changes are written to the nfs server, or you’re going to be in big trouble. You likely don’t want to set the user id and gid to 1000, because on the RPi, it is going to be id 0 (root) that is mounting the FS. So it’s probably your rootfs that is failing to mount during boot.

1 Like

I was going to post about how I was upset this wouldn’t work for me, as I use CUCM on my home network (don’t ask) and DHCP option 150 is set, but it looks like if I use option 66 with a hostname specifically it should not interfere with Callmanager. So adds this to list of projects.

Actually, now that this is mentioned… I use options 209 and 210, next-server and just pxelinux.cfg file (kernel, initrd, dtb and append options). I didn’t really follow Wendell’s guide though, since I’m not netboting RPis, but it should be similar. Although I am using u-boot and petitboot, no idea what the pi bootloader is (probably some weird broadcom blob, the RPi boots using its GPU :face_vomiting: ).

Buy a managed switch and separate your VOIP network. You should not have VOIP in the same subnet as your other stuff anyway. Have you DHCP option 66, 150 and whatelse split per subnet.

Hi Biky, appreciate your response.

I went through this again. Iterated 4 of 5 times again change my config on the exports and also trying different /etc/fstab configs. I always end up with a kernel panic.

This is my current export config now:

/nfs/pi-pxe x.x.x.x/x(rw,sync,no_root_squash,no_subtree_check)
/nfs/pi-tftpboot x.x.x.x/x(rw,sync,no_root_squash,no_subtree_check)

I also noticed an error with the RCP service:Sep 14 13:26:45 nfs-server rpcbind[56535]: cannot open file = /run/rpcbind/rpcbind.xdr for writing
Sep 14 13:26:45 nfs-server rpcbind[56535]: cannot save any registration
Sep 14 13:26:45 nfs-server rpcbind[56535]: cannot open file = /run/rpcbind/portmap.xdr for writing
Sep 14 13:26:45 nfs-server rpcbind[56535]: cannot save any registration

I was able to fix these by creating the files it was complaining about, and then also changing the permissions of /run/rpcbind recursively with this command:

chown -R rpc:rpc /run/rpcbind

I also made sure selinux is not blocking anything but setting things up in permissive mode.

These are my TFTP logs at boot when the PI boots up:
Sep 14 13:33:06 nfs-server in.tftpd[56626]: tftp: client does not accept options
Sep 14 13:33:06 nfs-server in.tftpd[56627]: Client ::ffff:192.168.31.40 finished f024980d/config.txt
Sep 14 13:33:07 nfs-server in.tftpd[56631]: Client ::ffff:192.168.31.40 finished f024980d/start4.elf
Sep 14 13:33:07 nfs-server in.tftpd[56632]: Client ::ffff:192.168.31.40 finished f024980d/fixup4.dat
Sep 14 13:33:08 nfs-server in.tftpd[56635]: Client ::ffff:192.168.31.40 finished f024980d/config.txt
Sep 14 13:33:08 nfs-server in.tftpd[56639]: Client ::ffff:192.168.31.40 finished f024980d/config.txt
Sep 14 13:33:09 nfs-server in.tftpd[56642]: Client ::ffff:192.168.31.40 finished f024980d/bcm2711-rpi-4-b.dtb
Sep 14 13:33:09 nfs-server in.tftpd[56644]: Client ::ffff:192.168.31.40 finished f024980d/overlays/overlay_map.dtb
Sep 14 13:33:09 nfs-server in.tftpd[56646]: Client ::ffff:192.168.31.40 finished f024980d/config.txt
Sep 14 13:33:09 nfs-server in.tftpd[56648]: Client ::ffff:192.168.31.40 finished f024980d/overlays/vc4-fkms-v3d-pi4.dtbo
Sep 14 13:33:10 nfs-server in.tftpd[56650]: Client ::ffff:192.168.31.40 finished f024980d/cmdline.txt
Sep 14 13:33:12 nfs-server in.tftpd[56663]: Client ::ffff:192.168.31.40 finished f024980d/kernel7l.img

This is my cmdline.txt from my NFS/TFTP share:

console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=192.168.31.68:/nfs/pi-pxe/rpi001 rw ip=dhcp elevator=deadline rootwait

This is my fstab on the NFS server, full path /nfs/pi-pxe/rpi001/etc/fstab. You will notice that in my latest update I commented out the /boot mount, but I’ve tried it both ways. I’ve also played around with the fsck option in both the /boot and / mount:

proc            /proc           proc    defaults          0       0
#PARTUUID=7daedbb9-01  /boot           vfat    defaults          0       2
#PARTUUID=7daedbb9-02  /               ext4    defaults,noatime  0       1
#192.168.31.68:/nfs/rpi-tftpboot/f024980d  /boot           nfs    defaults          0       0
192.168.31.68:/nfs/rpi-pxe/rpi001  /               nfs    defaults,noatime  0       0
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that

These are my nfs and rcp packages:
libtirpc-1.3.2-1.0.1.el9.x86_64
libnfsidmap-2.5.4-10.el9.x86_64
rpcbind-1.2.6-2.el9.x86_64
nfs-utils-2.5.4-10.el9.x86_64
sssd-nfs-idmap-2.6.2-4.0.2.el9_0.1.x86_64

This is my kernel version on my nfs/tftp server:

Linux  5.15.0-1.43.4.2.el9uek.x86_64 #2 SMP Wed Aug 17 14:50:30 PDT 2022 x86_64 x86_64 x86_64 GNU/Linux

And this is my distro version:

Red Hat Enterprise Linux release 9.0 (Plow)
Oracle Linux Server release 9.0

The reason I’m trying oracle linux is because I also wanted to test kernel updates without reboots, with ksplice (Zero downtime updates for Oracle Linux). But that’s just a side note in case anyone is wondering why I’m using oracle linux.

I’m still getting the kernel panics, willing to try more things or provide more logs and information if anyone else has ideas. Thanks again!