Journey into SBCs (ThatGuyB in ARMland)

Hardware Hub
261

Took me a while to come back to this project. I migrated from my Pi 2 fully over to the RockPro64 NAS. ZFS and NFS works great (FreeBSD only allows a single file system to be shared via NFS, so I’ve just been making dedicated zfs paths for all my needs). I also had to migrate the last bit from my pi 2, the tftpd server, to rkpr64. Easy enough, the FreeBSD docs are great.

I’m working on getting NixOS to work the hc4. Ideally I would make it boot from NFS, but last I tried, nix was trying to find its partition UUID and failing (or so I remember). But this time I’ll be writing it to an SD card and try to actually boot it locally (would also help in case my infrastructure goes down, if the hc4 boots by itself, which would be the point of a backup server, which I’m planning to use it as, but testing is easier with the rootfs on nfs).

I’m not sure I want to deal with unsupported OS on this thing anymore. I have data copies between the ssd pool and the hdd pool on the rkpr64 NAS, but I’d rather have another pool dedicated to backups. Then, this’ll give me an opportunity to work with nixos, which I plan on learning anyway. Doing it on aarch64 is probably not ideal as my first experience with it, but given that the thing is only a config file and a nix-store path, it should in theory be a similar experience on any architecture (just the bootloader differs).

Given that I’m not planning on doing much with the hc4 other than a backup server, I might eventually switch to a nixos VM (maybe even a firecracker, if I can make the docker hub to work on opennebula - more details on my ranting thread or what are you working on today thread).

It seems that, as of now, my setup is pretty much stable. It’s still not ideal, like my rockpro64 router being unable run any kernel greater than 5.18, because the realtek 88x2bu is hot garbage (and this is the only kernel I managed to make the module work, despite the driver apparently being available in 6+). But for the most part I’m happy with my setup.

In all fairness, I’d like to get rid of the threadripper and get something smaller and more power efficient. Initially I was planning to make this a NAS and hypervisor, but compared to my n2+ and rkpr64, to run 24/7 this thing requires probably around 10 times the power of these 2 combined (which is still not much given the compute horsepower, but it’d be mostly idle anyway, making arm sbcs that sip power a no-brainer for mostly idle systems).

My original goal was to make an affordable, easy to replicate infrastructure with arm sbcs and make the services either highly available through the hypervisors or through other failover means. The only problem was that I was hoping to make this with 2 switches, which certainly isn’t enough given most failover scenarios (with split brain problem - unless I can make an external system serve as a check, like a ping to my router at least, if not an outright separate arbiter / tie-breaker). I need 3 switches and an SBC and potentially a NAS on all 3 of them (maybe running ceph on all 3). This makes it raise in price really fast.

While I like the datacenter-in-a-box potential for this, I think I should restrict myself from big purchases and stick to what I’ve got. I can do a theoretical build and explain the procedures step by step, but investing in the hardware seems a bit excessive at this point for me, especially because I don’t expect the necessity of 99.99% uptime (heck, I want my infrastructure to be rebooted often and have the services just switch over during the failover while the other half of the infrastructure is rebooting).

Over the past year, I’ve been interested in a lot of hot stuff coming out, like the Khadas vim4, Radxa Rock 5 and other stuff, but I think the only thing I might add to my infrastructure would be a QuartzPro64, or whatever the name will end up being for a Pine64 board to replace the RockPro64, with a rk3588 and 16GB of RAM, only because from experience, the pine64 stuff seems to be the most supported SBCs around by other distributions. If people want to stick with Debian / Armbian, then most boards would do fine anyway (although most official distros don’t ever get an upgraded kernel outside the original one, which typically ends up being deprecated by the time of the launch anyway).

2 Likes
262

Just plop in a PCIe in your RockPro64 running as router and you’re done? :slight_smile:
QuartzPro64 is probably going to be interesiting in a year or so once drivers and hopefully have been ironed out. If there’s an UEFI project that might accelerate things a bit at last like with the Quartz64 board. GitHub - jaredmcneill/quartz64_uefi: A port of Tianocore EDK II to Rockchip RK356x based SBCs.

2 Likes
263

Problem is that I want the PCI-E for a 2.5 Gbps Ethernet (preferably 2 ports), so I’m keeping it open for when that time comes, just in case. I might be looking for another USB 3.0 WiFi AC or AX that is supported by mainline Linux (not sure if I can find any Intel ones, the reason I’m stuck with linux on my router is precisely the wifi drivers).

QuartzPro64 will likely be the next board I buy, unless I start deploying too much stuff and need an additional N2+ or something.

1 Like
264

I see, that’s going to be a bit hard with no bifurcation support and PCIe 2.0 x2?
Just grab a cheapo router and use it as a wireless bridge either with vendor firmware or something like OpenWrt, it’ll save you a ton issues and time.

1 Like
265

By x2, I meant 2 Ethernet ports, not 2 PCI-E devices, like a QNAP QXG-2G2T-I225.

1 Like
266

Which are 2 separate controllers on the same card, so no go…

1 Like
267

Why wouldn’t it work? My HP i340-t4 (4 port 1Gbps NIC) worked on a Celeron J3455, which was just a bare basic 4 lane PCI-E with no bifurcation.

And are you sure it’s 2 controllers? Because in most product specifications, I only see i225-lm being mentioned.

1 Like
268

Because i340 is a single controller, i225 and 226 series are single port controllers

2 Likes
269

Thank you for pointing that out. I might want to reconsider, but as of right now, I wasn’t planning to purchase the NIC anyway. I’ll need something that can work on the rockpro64, be multi-gigabit (support 2.5Gbps, not just 10G) and which has 2 ports. Might be a big ask to find something like this. But maybe if I can get proper 10G that doesn’t suck a lot of power and change to a switch that has at least 4 SFP+ ports, then maybe (then I’ll also want the switch to be silent, another hard find).

1 Like
270

No worries, you have the 10G ones by Intel but I don’t think they do 2.5G compatible ones at least no yet. Broadcom have PHYs but no controllers it seems…

1 Like
271

Just managed to boot NixOS on the HC4 via PXE through petitboot. The problem is that NixOS (and Armbian IIRC) don’t have keyboard support when they are KExec’ed. Void with Armbian kernel worked somehow.

Just booted back into void, added my ssh key in the nixos user and I logged in via SSH. This is exciting, but I need to get it to work locally via u-boot. For one, the HC4 needs to have its flash wiped to remove petitboot and IIRC, u-boot only works from SD on the HC4.

The NixOS wiki say to use nix-build to create a u-boot image, but I didn’t have a nix system to do that (until now). The other problem now is that the nix-build is for cross-compiling aarch64, but I’m on aarch64 already, I need to find how to build it natively.

I might end up doing another hackery and copy the nix rootfs, dd armbian (which has u-boot on the sd already), change the partition UUIDs to what nixos expects, point the symbolic links of what u-boot expects to what nixos needs to boot and pray it works.

For my rockpro64, that’s what I’m doing (and whenever there’s an update, I just create the uInitrd and then change the symlinks in /boot to point to the new uinitrd, kernel image and dtb and u-boot just boots without a fuss).

On FreeBSD I don’t have to do anything. I’d prefer that, because the HC4 is a supported platform (allegedly), to have the OS take care of all the boot changes. The last thing I want from my backup server is to power it on only to find out that there was an update which I forgot to test with a reboot, I removed the old kernel and stuff and now I need to mount the sd somewhere and change the symlinks).

1 Like
272

I made NixOS boot via u-boot on the HC4. After I tested it by pushing the SPI flash skip button and made sure it worked, I powered the board off, removed the microSD card and wiped the SPI flash. Plugged the uSD back in and now the keyboard is also working. But nixos still automatically logs me in, for whatever reason. I need to rebuild it and it has no example config file in /etc/nixos (in fact, the folder is empty, idk how it even boots).

Before I forget, here are the steps I took on the hc4. So, like mentioned before, I wrote the nix img to the SD card and booted it via petitboot by PXE boot. There, I followed this wiki, but instead of cross compiling, I did it straight and then I dd’ed to the SD (still mounted, lol) and it worked ok.

https://nixos.wiki/wiki/NixOS_on_ARM/ODROID-HC4

export NIXPKGS_ALLOW_UNFREE=1
nix-build -I "nixpkgs=$PWD" -A ubootOdroidC4
sudo dd status=progress if=result/u-boot.bin of=/dev/mmcblk0 conv=fsync,notrunc bs=512 seek=1

This did the trick, then I proceeded with the spi skip, wipe and finished the setup.

I’m glad I can now focus on playing with nixos and the rest of my infrastructure. And I’m sure glad the HC4 finally got a stable, supported system on it. If only I could go back to tinkering more… The Pi 4 could use an android on it (for me to remote into it from the internet). I don’t like that I still have some SBCs that see no use (I tested my alpine pi 3 router as a on-the-go router and it worked wonderful, but my pi 2 and 4 now are unused).

1 Like
273

I’ve learned the configuration.nix and implemented it on my HC4. Now my HC4 is running with all the stuff configured from that, including ZFS. It’s still booting from a ext4 formatted sd card, but that’s fine.

What’s not fine is that I’ve hit the same bug with the board with u-boot as I used to have with armbian. The weird part is that when I had petitboot on the spi, armbian would reboot fine. I’m not even entirely sure if I needed to remove petitboot here, because nixos seems like a well built system.

The configuration.nix has an entry for the bootloader to update /boot/extlinux/extlinux.conf, which petitboot could’ve read. I probably could’ve formatted the sd however I wanted anyway (like, with root on zfs, but running zfs on a sd card… I don’t think I’m brave enough for that). Still, I might revert to petitboot and reformat the SD just to see if reboot will function normally with petitboot.

I never thought booting on aarch64 could be this easy. When doing pxeboot on petitboot, it was easy enough, I just modified the symlinks to the kernel, uinitrd and dtb. Didn’t need to update the pxe boot conf. But instead of doing symlinks, I could’ve just edited the pxe conf. The extliinux.conf is similar to the pxe boot conf, just that it’s local (and I know petitboot can read that file automatically). And nixos is doing that for me.

There’s a small partition /dev/mmcblkp1 that seems to only contain RPi type boot files. I bet I don’t need it, as u-boot just loads from the space before partition 1. Once the sd is wiped and petitboot flashed again, I should be able to just load the extlinux.conf file automatically. I’m hoping reboot will start working, because having to unplug and replug the system whenever I want to reboot (as if it was a poweroff) is horrible. But I’d be more than happy to keep working with petitboot, I was sad to see it go from my hc4. I hope it works.

One thing that didn’t quite work with petitboot, from what I remember was loading the zfs kernel module. Not sure if that’s a limitation from KExec or if it was just hardware not supported. But I’ll obviously make a backup of the ubooted sd before I do anything stupid.

1 Like
274

It’s an absolute miracle I managed to get petitboot back when I did. Now I figured how to do it in linux and it was easy. I first needed to flash an image called spiupdate, then copy the spiboot.img in the root of that partition. Doable with dd, I was just being an idiot (in my defense, I found no instructions on how to do that before).

https://minskio.co.uk/restoring-petitboot-on-an-odroid-hc4/

1 Like
275

Currently trying to install Debian on another SD card for the HC4. Not entirely sure what is wrong with the NixOS boot, but petitboot will not find the extlinux.conf or the linux image, kernel and dtb. It will just not show the Disk boot entry, which is bananas.

I’m not ditching NixOS, but I’m testing Debian (using the lovely petitboot pxeboot from hardkernel’s http servers, which makes it so easy to install - basically a live netboot install without flashing a USB, try doing that with x86). I want to test if it has anything to do with the uboot image that’s flashed on the nixos sd card. I won’t mind wiping the install (and it’d be a breeze restoring it) if that makes nixos bootable from petitboot.

One thing I noticed is that, if I boot nixos by holding the SPI bypass button and then reboot, the SPI is getting read/loaded anyway. Meaning that, in absence of anything in the SPI on the board (using u-boot on the SD), then a reboot will try to load a nothingburger (I know I’m using that wrong). In a reboot with petitboot on the spi flash, it does load petitboot properly.

Would installing u-boot on the spi make more sense? Probably. Is it worth it? I don’t think so for me, I’m a self-proclaimed petitboot fan. If I can get my boot configs to get discovered by the bootloader, then I’d prefer that. Again, the reason I’m doing all this is because I want my board to be rebootable, I don’t want to have to unplug and replug power for the board to boot using u-boot on the SD (cold boot), I want it to be rebootable on warm boot.

And if I go with a fresh install of nixos for giggle, in theory, just adding my backed up nixos conf file will make the system config just like before (except I don’t want to deal with downloading a bunch of crap again, even if it’s only 2 or 3GB, if I already have them on another SD that I can just rsync between).

1 Like
276

The most annoying thing I find with “supported” ARM SoCs on Linux is the number of hoops you need to do to get something usable because of the non consistent nature of distros.

Given the relatively sparse support of the HC4 I’d just throw LibreELEC on it and call it a day even if you’re just going to use it as a NAS device… :wink:

1 Like
277

NixOS supports the HC4, but reboots don’t work, most likely because of a quirk in the warm boot process (first going to the spi, which you need to wipe to use the built-in u-boot on the SD card). Everything worked surprisingly well with NixOS.

My odroid n2+ gets around that by having an spi bypass toggle, but the hc4 only has a momentary switch. I haven’t tested if permanently pressing it does anything for the reboots, but that’s a last resort (I could solder a switch on the board if needed if that ends up working).

Copying rootfs’es around ain’t fun (especially when using sd cards). I left it overnight, I need to boot and see if petitboot still recognizes the boot partition. It seems u-boot was using an extlinux.conf to boot on nixos (which is smart), but debian is using a boot.scr script (I think it’s a u-boot thing). It seems the hc4 might be finding that boot.scr and reading it (at least that’s what may have been happening armbian - on the n2+ I have to edit the symlinks to the new kernel, initramfs and dtb, but the scr stays the same).

1 Like
278

I’m having quite the struggle with the hc4. It wouldn’t be as bad if I wasn’t such a brainlet and make nixos bootable via nfs temporarily. Editing the fstab doesn’t work, I think the initrd might not have support for nfsroot built-in and need to add that to a running system in the config. Problem is, the system will not work properly in chroot, because systemd commands refuse to run in chroot (RIP). Another reason to not use it. That leads me to only trying it on an already working system (chicken and egg problem - almost, I can get nixos to boot fine as a one off).

Now, the problem I’m facing is that nixos has its own funky config dependent on the partition uuids. The boot.scr and extlinux are not what makes petitboot see the sd card as a bootable device (I checked the partition table and the root partition of nixos and the boot of the other debian sd card were both set properly with bootable flag).

As long as I copy the initrd, kernel image and dtb on my tftp server and give the proper boot parameter (the “init” part of nixos), I can boot via tftp and load the local FS of the original nixos sd card. But the same fails on the debian sd card, even if I change the root uuid to the old sd card’s uuid. I believe nixos also makes use of partuuids and labels, so I need to switch the that too.

I wiped the nixos sd card, installing debian on it (this time debian 11, because in my previous netinstall, I forgot to switch the mirror codename and suite to bullseye, despite the fact that I booted the 11 kernel and initrd). I partitioned it manually similarly to how nixos had it before. I have a backup of the rootfs of nixos which I slapped on there and modified the uuids, part and labels to be the exact same as the original nixos image’s.

And bummer, petitboot doesn’t auto-detect the boot stuff. But just discovered that there is a kboot.conf file that petitboot is looking for on pb-discover. I need to look into that.

1 Like
279

kboot.conf gets detected, but doesn’t get selected as an auto boot option. Getting closer. I’ll come back to this later.

1 Like
280

Ok, kboot.conf managed to load the system as configured in petitboot… but petitboot doesn’t auto boot it.

Found that on debian, the thing that made the boot option appear in petitboot (and get auto-booted) was the u-boot boot.ini file. NixOS image didn’t have one, so it wasn’t getting detected obviously. It’s interesting that petitboot is able to boot u-boot boot.ini files, but these are absolute f***fests…

...just look at this! 
ODROIDC4-UBOOT-CONFIG

setenv bootlabel "NixOS"

setenv board "odroidc4"
setenv display_autodetect "true"
setenv hdmimode "1080p60hz"
#setenv monitor_onoff "false" # true or false
setenv monitor_onoff "true" # true or false
setenv overscan "100"
setenv sdrmode "auto" 
setenv voutmode "hdmi"
setenv disablehpd "false"
setenv cec "true"
setenv disable_vu7 "true"
setenv max_freq_a55 "1908"    # 1.908 GHz, default value
setenv maxcpus "4"
setenv enable_wol "0"

# Set load addresses
setenv dtb_loadaddr "0x10000000"
setenv dtbo_addr_r "0x11000000"
setenv k_addr "0x1100000"
setenv loadaddr "0x1B00000"
setenv initrd_loadaddr "0x3700000"

if test "${variant}" = "hc4"; then
       setenv max_freq_a55 "1800"
fi

load mmc ${devno}:1 ${loadaddr} config.ini \
    && ini generic ${loadaddr}
if test "x${overlay_profile}" != "x"; then
    ini overlay_${overlay_profile} ${loadaddr}
fi

setenv condev "console=ttyS0,115200n8"   # on both
setenv condev "console=ttyS0,115200n8"   # on both


### Normal HDMI Monitors
if test "${display_autodetect}" = "true"; then hdmitx edid; fi
if test "${hdmimode}" = "custombuilt"; then setenv cmode "modeline=${modeline}"; fi
if test "${cec}" = "true"; then setenv cec_enable "hdmitx=cec3f"; fi
if test "${disable_vu7}" = "false"; then setenv hid_quirks "usbhid.quirks=0x0eef:0x0005:0x0004"; fi

# Boot Args
setenv bootargs "root=UUID=44444444-4444-4444-8888-888888888888 init=/nix/store/5fbhlfm4y8m9200xi802c083jg9bcdvb-nixos-system-hc4-23.05.1748.e11142026e2/init zfs.zfs_arc_max=2147483648 nohibernate loglevel=4 rootwait rw ${condev} ${amlogic} no_console_suspend fsck.repair=yes net.ifnames=0 elevator=noop hdmimode=${hdmimode} cvbsmode=576cvbs max_freq_a55=${max_freq_a55} maxcpus=${maxcpus} voutmode=${voutmode} ${cmode} disablehpd=${disablehpd} cvbscable=${cvbscable} overscan=${overscan} ${hid_quirks} monitor_onoff=${monitor_onoff} logo=osd0,loaded ${cec_enable} sdrmode=${sdrmode} enable_wol=${enable_wol}"

# Load kernel, dtb and initrd
#load mmc ${devno}:1 ${k_addr} Image.gz
load mmc ${devno}:1 ${k_addr} vmlinuz

#load mmc ${devno}:1 ${k_addr} Image
#load mmc ${devno}:1 ${dtb_loadaddr} amlogic/meson64_odroid${variant}.dtb
load mmc ${devno}:1 ${dtb_loadaddr} dtb

#load mmc ${devno}:1 ${initrd_loadaddr} uInitrd
load mmc ${devno}:1 ${initrd_loadaddr} initrd.img
fdt addr ${dtb_loadaddr}

if test "x{overlays}" != "x"; then
    fdt resize ${overlay_resize}
    for overlay in ${overlays}; do
        load mmc ${devno}:1 ${dtbo_addr_r} amlogic/overlays/${board}/${overlay}.dtbo \
            && fdt apply ${dtbo_addr_r}
    done
fi

# unzip the kernel
unzip ${k_addr} ${loadaddr}

# boot
booti ${loadaddr} ${initrd_loadaddr} ${dtb_loadaddr}

I can’t figure out how these things are loaded, but somehow, the kernel and initrd addresses manage to point to the actual long path name that nixos creates, but the dtb is not detected (the address points to some /tmp location). I’d prefer an easy config like kboot.conf, but for the love of KISS, I can’t figure out how to make kboot auto-boot. If I get one or the other, then I’ll have a bootable and rebootable nixos system on the hc4.

The journey continues…

2 Likes