Journey into SBCs (ThatGuyB in ARMland)

I wasn’t fond of layer 4, I’ll look into that. But TBH I really don’t feel like breaking what’s working.

I could increase the mtu size a little, to adjust just barely for wireguard, but I’m ok that things work. I’d also have to look at the TCP window size, I can only assume it would be a kernel parameter somewhere (I know I had to deal with something among those lines at work at some point, but I wasn’t the one who fixed it).

Pretty sure he’d running a BSD OS based firewall so…
pf.conf(5) - OpenBSD manual pages (same on FreeBSD)
There’s also a similar setting for ipfw(2) and npf (npf.conf(5) - NetBSD Manual Pages)

Still running Linux, haven’t gotten around to switching to OpenBSD yet. I have everything ready, just need to dd an SD card, boot it and install it on my router’s emmc.

The CM3588 is available at the store with the case and 16GB ram versions. It seems the chip also has AV1 decoding capabilities. Great investment for media server.

I’ll try to get one when GitHub - edk2-porting/edk2-rk3588: EDK2 UEFI firmware for Rockchip RK3588 platforms supports it so I can try generic purpose arm ISOs. I’m not using hacked SBC OS images.

Hi @diizzy, could you describe in more detail the random breakage you experienced in the configuration of OpenWrt when upgrading? I am considering replacing my router with a device I created.

Either UCI or underlying syntax changes and breaks causing your device not to boot or at least to not make network interface come online. I’ve “degraded” all my OpenWRT devices as dumb APs and run FreeBSD on something to act as a router/firewall/gateway instead.

2 Likes

Thanks, @diizzy, for answering my question. Since the device I will be installing OpenWRT on will be a mini-computer, I am first going to create a virtual machine and install OpenWRT on it; this will be done for testing to see if everything is working as expected. Then, after a couple of updates to OpenWRT, I will install OpenWRT on my Mini-Computer.

Be aware that pretty much any instruction not supported by baseline x86-64/amd64 are disabled and will need patching to be enabled.

1 Like

Today, while I was messing with the odroid n2+, it suddenly froze and stopped working. Powering it back on was not working on the same port. I moved it on another port, but wouldn’t boot. I tried all kinds of things, like going back a kernel (because I also updated today, so I don’t remember if I booted the new kernel or not), returning to runit and a bunch more. The emmc is still readable. The only thing that made it work, was to boot from NFS (thank god for petitboot and pxe booting support).

At this point it smells to me like a broken OS, but I can’t point my fingers at anything. I wish I had done my testings in a VM, but I didn’t really want to power on my hypervisor (the AC can barely keep up as it is).

I just now managed to boot with an ancient 5.10 kernel I had laying around (extracted from armbian ages ago) and the thing finally showed something on the screen and it was corrupted disk. Which is weird, because the partition did get mounted on 2 other devices.

fsck from util-linux 2.38.1
e2fsck 1.46.5 (30-Dec-2021)
ext2fs_open2: Bad magic number in super-block
fsck.ext2: Superblock invalid, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/mmcblk1

The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>

Time for a backup of the data and a fsck of the emmc (maybe even a clean wipe). It’s been a while since I had any kind of file system corruption (more than a year I think) and I’m tired of dealing with that (that was the whole reason I bought emmc drives). I think I’ll either go full-retard and run zfs on emmc, or return to root-on-nfs-on-zfs (which is on a ssd mirrored zpool).

The original emmc setup for the n2+ was based on armbian’s installation, so it contained its own u-boot on emmc in the first blocks. I then removed the rootfs, kept the partition and just slapped my own rootfs, along with a generic kernel from my distro’s repos and a uInitrd generated from the system’s initramfs. Everything worked for a (long) while. but the board was booting with u-boot.

The n2+ has a switch to change between petitboot (SPI) and sd / emmc (well, technically just bypass spi flash). I’ve always kept it on u-boot (bypass SPI), but I’m pretty sure the n2+ should work with petitboot just fine (my hc4 does, so why not).

If I break the partition, I’m going to clean install and switch to petitboot. From my very old testing (which can be found in this very thread), petitboot with pxe was working great, so I see no reason it shouldn’t work with a local disk.


The moral of the story today: careful with some boards, they won’t display stuff on the screen when booting, until after stuff already happened (like after they mount the root partition), so you’re kinda screwed if you don’t have a way to connect to them via a serial connection. I have some uart connections for these, but I was just lazy and found other means to discover the problem. In some cases, I might not afford to be lazy and would need to connect to serial to troubleshoot what’s going on.

The bright side is that some boards, particularly the odroids, have firmware that will display stuff on the screen (petitboot in this case), so you know it’s not your board that’s defective, but a broken OS or sd / emmc (although symptoms won’t always be obvious). Still, unlike x86, some boards won’t even display u-boot (like my odroid doesn’t - my rockpro64 does).

The price you pay for SBCs is offset by your time and frustration if you don’t know what you’re doing and particularly if you don’t use a supported OS and try to force your way with it, where you could break things.

3 Likes

I smacked myself into a brick wall when I realized that I can’t use root-on-nfs if I’m using vlans, because I don’t have another NIC. I forgot that I take an IP via DHCP, mount root on nfs, then if I want to switch to vlans, I need to remove the IP from the (only) eth interface, set a bridge, set the vlans and then somehow magically “resume activity.”

That ain’t gonna happen. By the time eth0 just gets its IP unset, the system crashes (as you literally don’t have access to stuff like /usr to run the IP commands that you actually need). It could be doable if I figure out how to do an alpine-style diskless install (i.e. instead of root-on-nfs to do root-on-tmpfs, but copy the sysutils from nfs).

Weirdly enough, even after formatting the disk and copying my data back, I can’t boot anymore. I’ve not clue what I did wrong, or what I broke, but it’s almost certainly something to do with either the kernel image or initramfs (uInitrd).

I’ll need to check tomorrow, but I’m kinda glad I’ve got to attempt to move to petitboot now on my n2+ too. I just need to figure out which part of the system is broken. Worse case, now that the system is broken, I could try nixos on the n2+, but I’m not too fond of having my s6 plans on this board go to waste.

1 Like

You should be able to do your IP setup before mounting root, with an initrd or something like that.

2 Likes

I’m not entirely sure about that. You can certainly assign an IP in the initramfs (that’s why there’s even a bootloader command line argument and is how all root-on-nfs systems work). But the VLAN part is questionable.

The parameters (for any bootloader) I found were these.

 ip=<client-ip>:<server-ip>:<gw-ip>:<netmask>:<hostname>:<device>:<autoconf>:<dns0-ip>:<dns1-ip>:<ntp0-ip> 

So, if I boot via DHCP, or even if I use a static IP, as long as I don’t break that NFS connection, I’m ok.

If, after the system’s passed the native vlan part and I want to configure a VLAN (assuming I got the vlan set to both native and trunk on my switch, which I do only for now, while I’m running root-on-nfs to recover my main emmc volume), then I’d be breaking that connection and lose access to the rootfs.

At best, I can use a different native vlan on the interface itself (native traffic on eth0), but add a bridge with its own set of tagged interfaces and completely ignore the native one. That way, I’d keep the NFS connection, but that’s not exactly what I want to do. I’d rather have native VLAN spit all non-tagged traffic to a junk vlan and keep all my good traffic to a dedicated vlan for that.

From what I found online, there might be a way to add a VLAN bridge to an initramfs. I found this “initramfs-tools” package, but it only mentions bonding and vlans, but not bridges (I use a bridge and then VLANs on top of that).

And idk if an initramfs even comes with the 8021q module on it, I’d probably have to include it in.

I found this, which gets me closer, but it’s using busybox. I’m not sure if it’s going to bite me in the @$$ during the boot process (I can only assume only certain tools from busybox will be used, but then call my own init after rootfs gets mounted from a nfs export).

I’d need to mod that shell script to use ip commands instead of ifconfig and brctl (easy). I also didn’t know switch_root existed, which is brilliant. I think I could make use of that to run a tmpfs rootfs (I’d have a baller setup if I’d have enough RAM).

I was being driven insane by being unable to boot from local fs even after I formatted. I even attached a uart connector to my n2+! And yet I couldn’t see anything past the petitboot getting loaded up (funnily enough, petitboot uses u-boot to boot up, didn’t know that, I assume it uses the 1st stage, but I’d have to read into it).

When booting my latest kernel (and the one I knew worked before the system crashed), I was getting nothing on the screen, no LED blinking and nothing on the serial console. When booting an ancient armbian 5.10 kernel (that always worked to some degree), the thing loaded up right away. Now, the thing is, I got errors that the rootfs was corrupted again.

In one mistake, I managed to also back up lost+found and I slapped that back onto the freshly formatted rootfs. That was pretty much my own undoing and the reason why the system failed to boot with fs errors. But just the fact that nothing even showed up on serial with the 6.6 kernel, but things worked on 5.10, is weird. I feel something else might be going on.

To be fair, previously I was booting with u-boot straight from emmc (bypassing petitboot). But I do recall switching to petitboot and it was still working (or at least I think so).

So I gave it another go. Formatted everything fresh, deleted the backed up lost+found and rsync’ed the data back to emmc. If I get another error about file system corruption, it’s clearly the emmc is dying (which idk how that would happen, I had no swap and I barely used it, the board was powered off most of its life).

Aaand… with 6.6 still no dice. Works with 5.10. However, fsck runs automatically in 5.10 and shows, quote:

Begin: Will now check root file system ... fsck from util-linux 2.36.1
[/sbin/fsck.ext4 (1) -- /dev/mmcblk1p1] fsck.ext4 -a -C0 /dev/mmcblk1p1
e2fsck: Get a newer version of e2fsck!

/dev/mmcblk1p1 ********** WARNING: Filesystem still has errors **********

fsck exited with status code 12
done.
Failure: File system check of the root filesystems failed
The root filesystem on /dev/mmcblk1p1 requires a manual fsck

This doesn’t make any sense. It’s a freshly formatted ext4 fs. My last hope (besides doing a uSD card) is dd’ing my old fs image from 2022 (my original dd of the working system). That one has u-boot on it, so I’m kinda hesitant, but… I want this to work and I don’t care if I have to wipe petitboot if I must (although I prefer the bypass switch).

The uart connector proved itself useless today, only because petitboot doesn’t output to serial after it’s loaded, but also because when trying to kexec into the newer kernel, nothing was happening, while with 5.10, at least I was getting output. But I’d be getting the same output like on the monitor (the init log), so why use serial at all then… well, it would be useful if you’d be installing BSDs on your boards, or other OS that have no gpu drivers to display to hdmi out.

Well, TBH, it’s not all that bad with uart. I knew the cr2032 battery was dead, because the date on the system was always reverted to 1970, but actually the serial output reported low voltage on the rtc battery and that the date is not accurate / reliable, which was a nice touch (there were other stuff thrown in from u-boot, which you don’t get to see on the display, but they were mostly unimportant to what the issue I was dealing with was.

No dice. I’m going mad. I even flashed a new version of armbian (with probably a new u-boot).

Found U-Boot script /boot/boot.scr
8133 bytes read in 2 ms (3.9 MiB/s)
## Executing script at 08000000
U-boot default fdtfile: amlogic/meson-g12b-odroid-n2-plus.dtb
Current variant: n2-plus
For variant n2-plus (dash version, 2021.07 or up), set default fdtfile: amlogic/meson-g12b-odroid-n2-plus.dtb
161 bytes read in 1 ms (157.2 KiB/s)
Current fdtfile after armbianEnv: amlogic/meson-g12b-odroid-n2-plus.dtb
Mainline bootargs: root=UUID=528518a1-8367-456d-86c1-8479e59704b9 rootwait rootfstype=ext4 splash=verbose console=ttyAML0,115200 console=tty1 consoleblank=0 coherent_pool=2M loglevel=1 ubooty
63157836 bytes read in 2867 ms (21 MiB/s)
42177024 bytes read in 1915 ms (21 MiB/s)
53504 bytes read in 4 ms (12.8 MiB/s)
Failed to load '/boot/dtb/amlogic/overlay/meson-fixup.scr'
## Loading init Ramdisk from Legacy Image at 13000000 ...
   Image Name:   uInitrd
   Image Type:   AArch64 Linux RAMDisk Image (gzip compressed)
   Data Size:    63157772 Bytes = 60.2 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 04080000
   Booting using the fdt blob at 0x4080000
   Loading Ramdisk to 3c3c4000, end 3ffff60c ... OK
   Loading Device Tree to 000000003c34e000, end 000000003c3c3fff ... OK

Starting kernel ...

And this time, there’s the NIC LED blinking, on both the board and on the switch (probably a new u-boot).

But I realized the fs errors on linux 5.10 were my own bad. I somehow ended up with the root-on-nfs fstab in the emmc. Dunno how I did that, but I booted 5.10 successfully on the board with petitboot.


Holly crap, during use, I got a EXT4-fs error on the monitor:

[  278.750545] EXT4-fs error (device mmcblk1p1): ext4_mb_generate_buddy:805: group 33, block bitmap and bg descriptor inconsistent: 32 vs 9 free clusters
[  278.774800] Aborting journal on device mmcblk1p1-8.
[  278.783422] EXT4-fs (mmcblk1p1): Remounting filesystem read-only

MOTHE–

grep mmc /proc/mounts 
/dev/mmcblk1p1 / ext4 ro,noatime,errors=remount-ro,commit=600 0 0

Unstinking real. That’s the whole reason I got emmc instead of using sd cards and yet, here we are. So many frustrations because of this. I’m immediately working on root-on-nfs with proper vlan setup and I don’t care if I have to make a new vlan just for the native stuff. AAAAAAAAAAAAAAAAAAAHHHHHH.

I’ve been using MicroSD cards for years but either way, corruption sucks…

1 Like

I’ve left the zfs-dkms installation overnight and it’s still running. It seems to be stuck in a loop.

 HOSTCC  scripts/kconfig/util.o
 HOSTLD  scripts/kconfig/conf
 SYNC    include/config/auto.conf
 HOSTCC  scripts/basic/fixdep
 HOSTCC  scripts/kconfig/conf.o
 HOSTCC  scripts/kconfig/confdata.o
 HOSTCC  scripts/kconfig/expr.o
 HOSTCC  scripts/kconfig/lexer.lex.o
 HOSTCC  scripts/kconfig/menu.o
 HOSTCC  scripts/kconfig/parser.tab.o
 HOSTCC  scripts/kconfig/preprocess.o
 HOSTCC  scripts/kconfig/symbol.o
 HOSTCC  scripts/kconfig/util.o
 HOSTLD  scripts/kconfig/conf
...
 HOSTCC  scripts/basic/fixdep
 HOSTCC  scripts/kconfig/conf.o
 HOSTCC  scripts/kconfig/confdata.o
 HOSTCC  scripts/kconfig/expr.o
 HOSTCC  scripts/kconfig/lexer.lex.o
 HOSTCC  scripts/kconfig/menu.o
 HOSTCC  scripts/kconfig/parser.tab.o
 HOSTCC  scripts/kconfig/preprocess.o
 HOSTCC  scripts/kconfig/symbol.o
 HOSTCC  scripts/kconfig/util.o
 HOSTLD  scripts/kconfig/conf
 SYNC    include/config/auto.conf
 HOSTCC  scripts/basic/fixdep
 HOSTCC  scripts/kconfig/conf.o
 HOSTCC  scripts/kconfig/confdata.o
 HOSTCC  scripts/kconfig/expr.o
 HOSTCC  scripts/kconfig/lexer.lex.o
 HOSTCC  scripts/kconfig/menu.o
 HOSTCC  scripts/kconfig/parser.tab.o
 HOSTCC  scripts/kconfig/preprocess.o
 HOSTCC  scripts/kconfig/symbol.o
 HOSTCC  scripts/kconfig/util.o

Ok, apparently, this wasn’t zfs, but dkms itself getting in a loop. Additionally, no matter what I’m doing now, I can’t the n2+ to boot my distro’s kernels anymore. It’s just that nothing’s happening when running petitboot.

With u-boot, at least there’s some minimal feedback from the LEDs and it’s probably checking the emmc.

Going to flash TowBoot on the n2+ SPI.

1 Like

After countless hours of struggle, I managed to get my board at least booting. TowBoot doesn’t have HDMI output (which I find insane), but at least I managed to get uart to work using screen (idk what I was doing wrong, but minicom wouldn’t capture input to send through the uart connector).

Both PXE and local emmc work. Somehow my distro kernel doesn’t support netbooting, but the armbian kernel does. I haven’t figured out the magic (I’m thinking the armbian kernel has nfs built-in, while on my distro it’s a kernel module that needs to be loaded after the FS gets mounted… so root will never be mounted). The default is local disk.

Through PXE, I can make it boot either with root-on-nfs or with root on local disk (giving the root= cmdline argument). What I find a bit crazy is that towboot (and probably implicitly u-boot) requires its own PXE config and doesn’t even respect option 209, despite it respecting option 210. I had to make a config 01-ma-ca-dd-re-ss, instead of having the default ma-ca-dd-re-ss file that any self-respecting bootloader supports. And to add insult to injury, instead of getting stuff from “tftp://ip:/path/file,” it instead wanted the format “path/file” and then it grabbed it from the tftp server.

I’m already kinda hating towboot (the project itself is fine, I can’t blame them too much, because the n2+ is only a best-effort support). Since kexec is out of the question, now I can finally return to things I should be doing. I never managed to get petitboot to work, which is a shame, because I really liked the ability to net-boot on demand by flipping a switch (I could still technically do that, but I’d probably need to slap it on an sd card, since now towboot is in the SPI flash).

Now I’m hoping my hc4 will never just suddenly stop working with nixos (nixos at least officially supports it, but not with petitboot, but with u-boot, meaning you have to wipe the SPI flash on it).

2 Likes

As far as serial goes, GitHub - tio/tio: A serial device I/O tool or possibly GitHub - wtarreau/bootterm: The terminal written for its users by its users might be of interest.

1 Like

By the way @diizzy, how are you dealing with zfs datasets shared on multiple IPs with the same option in FreeBSD? In zfs set sharenfs, I can only set -maproot=root,-network=subnet/mask, but I cannot use the network multiple times.

For now, I got around by adding an additional line in /etc/exports (well, the only line, since zfs does it in /etc/zfs/exports) with the same share as the one from the sharenfs property, but a different subnet.

Can you do that in a single line? By the way, are you even using the sharenfs property?

1 Like