The Ultimate Home Server - Herd of Netboot Raspberry Pi? Sure

aaaskew · April 3, 2022, 2:31pm

The documentation on the Raspberry Pi web site (Raspberry Pi Documentation - Remote Access) suggests:

nfsroot=1.2.3.4:/nfsroot/rpi-pxe/rpi1,vers=4.1,proto=tcp

When starting up a recent Raspbian image the kernel (5.15.30) messages seem to indicate NFS 4.1 support for root. My NFS server (Debian 11) communicates using NFS 4.2 if not specified so setting vers=4.1 is required in my case. I assume if you have an older NFS server, the version negotiated would be old enough to not be required in the cmdline.txt file but network booting might mysteriously stop working if the NFS server is updated.

aaaskew · April 3, 2022, 2:56pm

This might be muddying the waters but the Raspberry Pi documentation - Network booting (Raspberry Pi Documentation - Raspberry Pi Hardware) seems to indicate that network booting is possible on Raspberry Pi 400, 4, 3B, 3B+ and 2B v1.2.

For Raspberry Pi 4 and 400 you can update the EEPROM if needed
For Raspberry Pi 3 you might be able to enable network book using the built-in firmware
For Raspberry Pi 2B v1.2 and 3 you might need an SD card with an updated bootcode.bin - Special bootcode.bin-only boot mode (Raspberry Pi Documentation - Raspberry Pi Hardware)

As @MadMatt says, BerryBoot might help out with older hardware and WiFi booting with iSCSI but their web site does not link to the images for Raspberry Pi 0,1,2 last updated June 2019 so maybe not a good long term option.

JohnnyDanger · April 11, 2022, 12:44am

nfsroot=1.2.3.4:/nfsroot/rpi-pxe/rpi1,vers=3

^^^ this!! after 2 days of fiddling! thank you (pi4 and vers=4 here)

THANK YOU!

swa · April 13, 2022, 12:48am

So instead of thinking about important stuff, I looked at the example fstab and wondered what the “pass” number should actually be in this case. “0”? I’m going to go with “0”. Is the pass number actually used by anything these days?

JohnnyDanger · April 13, 2022, 2:06am

topic for another video. Since you’ve done navidrome, some sort of steam/game cache. It looks like it required port 53 for DNS… BUT I am already running pihole.

shadowimmage · June 18, 2022, 6:01am

Not sure if helpful, but I’ve been meaning to do this for a long time and keep getting hung up on things not working as simply as the guide or videos make it appear.

In any case, the nfs mounting part of getting a pi to mount directories wasn’t clear and at this point I’ve managed to learn and forget how to do it twice.

I use a Synology NAS, so I’m working within the bounds of what it has for dials and switches in terms of TFTP and NFS shares and permissions. For my fellow Synology users, pay close attention to the “Mount path” notice on the bottom left of the NFS Permissions tab of the shared folder edit dialog. All NFS Shares have the Synology Volume name prepended to the share name.

So instead of running sudo mount 10.0.0.5:rpi-pxe/rpi3 /nfs/root you gotta run sudo mount 10.0.0.5:/volume1/rpi-pxe/rpi3 /nfs/root
*assuming that your share is on a volume named “volume1” and your synology is at IP address 10.0.0.5.

Permissions and read/write issues are still a little beyond my understanding. For the most part it seems like to edit most things I have to be running as root on the pi.

Hope that helps!

bmzt · September 13, 2022, 2:41am

I’ve gone through this tutorial as well as the one linked on mikejmcfarlane. I’ve gotten pretty far along but I’m a bit stuck and have a lot of questions.

First question, what version of NFS is everyone using here? It seems like Wendel is using NFS version 4. I was having trouble setting up the exports properly with NFS version 4. What seemed to work for me was creating the pi user and group on my NFS/TFP server with the same uid and guid as the pi users on the pis. Here is what my exports look like:

/nfs/pi-pxe x.x.x.x/x(rw,async,all_squash,no_subtree_check,anonuid=1000,anongid=1000)

/nfs/pi-tftpboot x.x.x.x/x(rw,async,all_squash,no_subtree_check,anonuid=1000,anongid=1000)

I finally got my NFS server and mounts working as well as TFTP and added option 66 on my DHCP server.

I was able to get the raspberry pi to boot but I ran into a kernel panic. I got the kernel panic with both the 64bit and 32bit images.

I attached a screenshot of my kernel panic.

So just to summarize.

What version of NFS is everyone here using or is Wendel using?
Can I get some examples of how the NFS exports are set up?
Did everyone else have to create a raspberry pi user on the NFS/TFP server so that the uid and guid match?
Any ideas on why I’m getting the kernel panic?

Quick update I went through the whole process again using an old raspberry pi image from 2020 (buster). I got the same kernel panic issue. I double-checked everything. Not sure what I’m doing wrong.

Hi @wendell if you get a chance to look at this, any ideas?

Thanks!

ThatGuyB · September 13, 2022, 11:49pm

I’m not netbooting RPi, but Odroid HC4, N2+ and RockPro64 (well, the last one ain’t netbooted anymore, because it’s a router now, but did during some testing).

Using NFS v4 on my server, but some devices will just downgrade themselves to nfs v3. Which is fine.

The kernel says something about init=. It almost looks as if you are not booting your init system, which on Raspberry Pi OS is systemd.

Do a few things. Mount your SD card like normal, do an apt update and apt install nfs-common. It should already be installed. Then try mounting the NFS server locally on the pi using mount.nfs ip-of-nfs-server:/path/to/rootfs /mnt/. If it works, good, edit your /mnt/etc/fstab and replace the / with the nfs server’s mount point, kinda like so:

# <device>                        <dir> <type> <options>         <dump> <fsck>
ip-of-nfs-server:/path/to/rootfs    /    nfs    defaults,noatime    0    0

Note that normally you would have fsck = 1 for root, but since root is on NFS, no need to do that, the NFS server will take care of file system checks (if you’re using ZFS, you have your scheduled zfs scrubs). And comment any lines about a swap partition if you have any of these.

Then you poweroff, unplug the SD and try booting from the network again. If that doesn’t work, let us know.

No.

Here’s what my export looks like (my TFTP and NFS server are the same, using the rootfs/boot folder for tftp, makes life easier):

/mnt/netboot/rootfs/hc4         bikyhc4.local(rw,sync,no_root_squash,no_subtree_check)

You want your NFS to use synchronous writes, not async. It is slower, but you want to make sure changes are written to the nfs server, or you’re going to be in big trouble. You likely don’t want to set the user id and gid to 1000, because on the RPi, it is going to be id 0 (root) that is mounting the FS. So it’s probably your rootfs that is failing to mount during boot.

NephtonLabs · September 14, 2022, 1:18am

I was going to post about how I was upset this wouldn’t work for me, as I use CUCM on my home network (don’t ask) and DHCP option 150 is set, but it looks like if I use option 66 with a hostname specifically it should not interfere with Callmanager. So adds this to list of projects.

ThatGuyB · September 14, 2022, 2:28am

Actually, now that this is mentioned… I use options 209 and 210, next-server and just pxelinux.cfg file (kernel, initrd, dtb and append options). I didn’t really follow Wendell’s guide though, since I’m not netboting RPis, but it should be similar. Although I am using u-boot and petitboot, no idea what the pi bootloader is (probably some weird broadcom blob, the RPi boots using its GPU ).

Buy a managed switch and separate your VOIP network. You should not have VOIP in the same subnet as your other stuff anyway. Have you DHCP option 66, 150 and whatelse split per subnet.

bmzt · September 14, 2022, 5:50pm

Hi Biky, appreciate your response.

I went through this again. Iterated 4 of 5 times again change my config on the exports and also trying different /etc/fstab configs. I always end up with a kernel panic.

This is my current export config now:

/nfs/pi-pxe x.x.x.x/x(rw,sync,no_root_squash,no_subtree_check)
/nfs/pi-tftpboot x.x.x.x/x(rw,sync,no_root_squash,no_subtree_check)

I also noticed an error with the RCP service:Sep 14 13:26:45 nfs-server rpcbind[56535]: cannot open file = /run/rpcbind/rpcbind.xdr for writing
Sep 14 13:26:45 nfs-server rpcbind[56535]: cannot save any registration
Sep 14 13:26:45 nfs-server rpcbind[56535]: cannot open file = /run/rpcbind/portmap.xdr for writing
Sep 14 13:26:45 nfs-server rpcbind[56535]: cannot save any registration

I was able to fix these by creating the files it was complaining about, and then also changing the permissions of /run/rpcbind recursively with this command:

chown -R rpc:rpc /run/rpcbind

I also made sure selinux is not blocking anything but setting things up in permissive mode.

These are my TFTP logs at boot when the PI boots up:
Sep 14 13:33:06 nfs-server in.tftpd[56626]: tftp: client does not accept options
Sep 14 13:33:06 nfs-server in.tftpd[56627]: Client ::ffff:192.168.31.40 finished f024980d/config.txt
Sep 14 13:33:07 nfs-server in.tftpd[56631]: Client ::ffff:192.168.31.40 finished f024980d/start4.elf
Sep 14 13:33:07 nfs-server in.tftpd[56632]: Client ::ffff:192.168.31.40 finished f024980d/fixup4.dat
Sep 14 13:33:08 nfs-server in.tftpd[56635]: Client ::ffff:192.168.31.40 finished f024980d/config.txt
Sep 14 13:33:08 nfs-server in.tftpd[56639]: Client ::ffff:192.168.31.40 finished f024980d/config.txt
Sep 14 13:33:09 nfs-server in.tftpd[56642]: Client ::ffff:192.168.31.40 finished f024980d/bcm2711-rpi-4-b.dtb
Sep 14 13:33:09 nfs-server in.tftpd[56644]: Client ::ffff:192.168.31.40 finished f024980d/overlays/overlay_map.dtb
Sep 14 13:33:09 nfs-server in.tftpd[56646]: Client ::ffff:192.168.31.40 finished f024980d/config.txt
Sep 14 13:33:09 nfs-server in.tftpd[56648]: Client ::ffff:192.168.31.40 finished f024980d/overlays/vc4-fkms-v3d-pi4.dtbo
Sep 14 13:33:10 nfs-server in.tftpd[56650]: Client ::ffff:192.168.31.40 finished f024980d/cmdline.txt
Sep 14 13:33:12 nfs-server in.tftpd[56663]: Client ::ffff:192.168.31.40 finished f024980d/kernel7l.img

This is my cmdline.txt from my NFS/TFTP share:

console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=192.168.31.68:/nfs/pi-pxe/rpi001 rw ip=dhcp elevator=deadline rootwait

This is my fstab on the NFS server, full path /nfs/pi-pxe/rpi001/etc/fstab. You will notice that in my latest update I commented out the /boot mount, but I’ve tried it both ways. I’ve also played around with the fsck option in both the /boot and / mount:

proc            /proc           proc    defaults          0       0
#PARTUUID=7daedbb9-01  /boot           vfat    defaults          0       2
#PARTUUID=7daedbb9-02  /               ext4    defaults,noatime  0       1
#192.168.31.68:/nfs/rpi-tftpboot/f024980d  /boot           nfs    defaults          0       0
192.168.31.68:/nfs/rpi-pxe/rpi001  /               nfs    defaults,noatime  0       0
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that

These are my nfs and rcp packages:
libtirpc-1.3.2-1.0.1.el9.x86_64
libnfsidmap-2.5.4-10.el9.x86_64
rpcbind-1.2.6-2.el9.x86_64
nfs-utils-2.5.4-10.el9.x86_64
sssd-nfs-idmap-2.6.2-4.0.2.el9_0.1.x86_64

This is my kernel version on my nfs/tftp server:

Linux  5.15.0-1.43.4.2.el9uek.x86_64 #2 SMP Wed Aug 17 14:50:30 PDT 2022 x86_64 x86_64 x86_64 GNU/Linux

And this is my distro version:

Red Hat Enterprise Linux release 9.0 (Plow)
Oracle Linux Server release 9.0

The reason I’m trying oracle linux is because I also wanted to test kernel updates without reboots, with ksplice (Zero downtime updates for Oracle Linux). But that’s just a side note in case anyone is wondering why I’m using oracle linux.

I’m still getting the kernel panics, willing to try more things or provide more logs and information if anyone else has ideas. Thanks again!

bmzt · September 14, 2022, 5:58pm

These is another way to have your PIs boot to the TFTP server without option 66 set. You have to update your eeprom on your raspberry Pi. Wendel mentions this slightly in the video he made about this process but he doesn’t go into detail, but one of the links in the original post has a section about this.

Here is the link

And look for the section “Configure the eeprom to include PXE boot options”. I actually ran through this process and it worked on one of my PIs.

NephtonLabs · September 14, 2022, 6:12pm

Fair point and good practice, but eeeeugh subnetting and routing on my home network.
I used to have some subnets, but the routing between them always seemed unreliable, so I eventually flattened the whole thing.
Maybe now it would be better, but I also don’t like how the traffic would have to route or the reengineering on everything else I’d have to do to avoid bouncing traffic off the inside interface of the firewall.
This goes firmly in the “one day” pile, but thank you for the reminder.

Dutch_Master · September 14, 2022, 7:30pm

Just a thought: did you set the permissions correctly on the NFS share? Is it owned by root or another user?

ThatGuyB · September 14, 2022, 7:41pm

TFTP is fine, if you get to the point in the RPi where you get the kernel panic, then you are booting, at least getting the initramfs up.

I’m more curious on your proc /proc proc line, usually that gets mounted automatically.

Anyway, the kernel panic is still likely in the NFS mounting phase.

Good point. chown -R root:root /nfs/rpi-pxe-pi001. Might have some trouble with the user, but it should be easily solvable if you manage to boot.

Dutch_Master · September 14, 2022, 8:42pm

Perhaps transferring data is too slow, add some (more?) wait time?

ThatGuyB · September 14, 2022, 9:06pm

It has the rootwait option and I believe bmzt has a local network.

Dutch_Master · September 14, 2022, 10:17pm

So, to recap, a summary so far:

RPi boot into PXE mode works
TFTP server successfully transfers required files
Kernel is started successfully
kernel can’t find the NFS share and panics

However, as can be seen above, the kernel line includes the rootwait option and w/o a specified timeout (as is the case here) it’ll wait indefinitely, Or at least, it should. But, something still times out and causes the kernel panic. And that is looking like an interesting search

bmzt · September 17, 2022, 5:01pm

Thanks who everyone who chipped in. Finally got this working!

I’m not sure what the issue was. I made a bunch of changes and kept iterating until I got it working. Here are a few things I did:

Started from scratch using a different raspberry pi image. This time I downloaded the raspberry pi image from January 2022. The 32 bit old stable image. I chose this image because Wendel made his video in February, and I figured he used the latest image available at that time, which was the January image.
Switched to using NFSv3. I hard coded NFSv3 on the server and client side. I tried to disable NFSv4 completely
I added both no_root_squash,no_all_squash to my exports
I added sec=sys. I did this because I was trying to disable the idmap thing in NFSv4 but I kept it even though I’m using NFSv3
I recreated my folders on my NFS/TFP and copied everything over from the new raspberry flashed SD card

Currently this is now what my exports look like, if anyone else is having issue I thought I would provide my current config that is working:

/nfs/pi-pxe x.x.x.x(rw,sync,no_root_squash,no_all_squash,no_subtree_check,sec=sys)
/nfs/pi-tftpboot x.x.x.x(rw,sync,no_root_squash,no_all_squash,no_subtree_check,sec=sys)

My client side etc/fstab:

proc            /proc           proc    defaults          0       0
x.x.x.x:/nfs/pi-tftpboot/RPISERIAL  /boot           nfs    defaults,vers=3,proto=tcp,noatime          0       0
x.x.x.x:/nfs/pi-pxe/rpi001  /               nfs    defaults,vers=3,proto=tcp,noatime  0       0

I have one follow up question, but will probably have more after I start playing around with this a bit more.

Does anyone know if there are any big drawbacks to using NFSv3 vs NFSv4?

wayland · September 28, 2022, 11:23pm

Anything with an operating system or memory management will crash. Microcontrollers only run the one program you put in them so only crash if you made a mistake. Oversimplification because you might be using libraries someone else wrote.