Journey into SBCs (ThatGuyB in ARMland)

Unfortunately PI’s are crap in general, you’re usually much better off with Rockchip and/or Allwinner depending on application.

2 Likes

Looked at ODROID and Libre too… neither looks too bad from the right source that isn’t gouging you… For me it would be to tinker with HA of services for the home for giggles. :slight_smile:

I also had a crazy idea of creating my own PC cooling system controller like Aqua Computer :slight_smile:

2 Likes

The Odroid N2+ is probably the one I recommend the most. I prefer Amlogic. Besides, The N2+ is $75 and while it has half the RAM of a supposed RPi 4, at least the price is consistent and there is supply for it.

1 Like

…and poor mainline support at best which makes Rockchip and Allwinner are much better in that regard. I would highly advice against getting hardware that has no documentation available simply because you’re at the mercy of some vendor fork which will become obsolete quickly and not to mention near impossible to fix issues. While Raspberry Pi does have pretty good mainline support more or less is by the Rasperry Pi Foundation solely and no documentation available for others so once they stop maintaining it it’s a dead end not to mention all other quirky hardware choices and issues. Not saying that Rockchip and Allwinner are the greatest thing on earth but they’re much better choices in that regard, you also have NXP but they’re usually quite a bit more expensive and more network oriented but nice SoCs. There are also other’s such as Ti but they’re not really targeting “generic” usage, more using their SoCs as controllers etc.

2 Likes

In an exercise of frustration, tonight the same steps that worked previously to migrate from the old armbian kernel to a generic release in the repo failed. I wanted to test booting into Linux 5.18, but I failed. It’s late, so I won’t try 5.15 or a generic 5.10 right now, maybe tomorrow.

This one was a test on my Odroid N2+. I installed dracut, as always, generated an initrd (because that’s what u-boot seems to be using), generated a u-boot Image, pointed the symlinks to the new entries, regenerated the boot.scr… doesn’t seem like it’s working.

Maybe I’m misremembering and I never got a non-armbian or non-manjaro kernel to actually boot on u-boot on other OS. I’ll have to give my own thread a skim. Oh, it’s going to be such a pain, why am I writing walls-of-text?

1 Like

You probably can’t boot a generic kernel due to mismatching dtb/dts files

1 Like

The dtbs are coming on their own, they get pushed to a folder /boot/dtb/kernel-version, so I don’t think it’s a mismatch. I also made sure that the dtb file was there.

1 Like

…and the kernel knows which one to use?

1 Like

I assume it should know, but I’m far too unknowledgeable on the subject.

1 Like

So, long story short, I bricked my Void install on my N2+.

Long story long, I was trying to get any kernel to work on the N2+. Kernel 5.15 worked, but I had to be a smartass and retry 5.18 again, just to be sure. Of course, it didn’t work. No worries, I say, just have to plug it in to my RPi and move some symlinks back around and everything will be fine, I did this more than 20 times in the past 2 days. Said and done, go back to 5.15.

Plug the n2+ in, nothing happens. Usually the blue LED lights up, meaning it is booting / reading data, kind of like a storage activity LED. I got worried, so I wanted to move back to the armbian binaries that I know worked, although the previous test, 5.15 was doing great. Plug the SD card to my Pi and to my surprise, my partition was gone. Supposedly magic superblock mismatch or missing or shenanigans. So I reflashed armbian, plugged it in, waited for it to resize the partition, update… ok, great.

By this time, I remembered why I didn’t run Armbian on neither the HC4, nor the N2+. apt is broken. It works, it updates fine, but I had to wait one and a half hours for a damn fugging apt search zfs after a relatively decent apt update. This was happening both with both the default debian repos, and ones close to me, with apt update after each repo change. Just apt search is going ape sh*t.

I finally try installing zfs… no headers installed. I can’t use apt search to look for kernel headers, because… duh. So, I try random names until I hit linux-headers, then I get prompted by multiple providers and find the meson64 headers that I was running. Installed it, then removed and readded zfs-dkms. Still couldn’t build zfs, because no makefile was found.

So I just poweroff, plug the SD in my Pi, mount the partition, remove most of the debian junk out, place the void rootfs, copy the backed up lib modules, firmware, fstab and stuff… unmount, plug the SD… no post. After it was working just a few minutes ago. Plug the SD in my Pi… magic block errors. Un-fugging-real!

So, now I attached my USB HDD to my Pi. And the first thing that happened was that my / partition was unmounted / not detected, with no commands working. Which happens when that thing draws a lot of power from the USB port. The problem is that the HDD is attached through a powered USB hub and this thing still draws enough power to mess with the m.2 SSD enclosure. Rebooted with the HDD unplugged, booted fine, plugged the HDD. Ok, I’m not in my OS and writing this update.

After all the headaches I had with microSD cards, I’m sick and tired of it all. Tomorrow or sometimes soon I am plugging in my Pi 2 and setting up a tftp server and a NFS server on it. Will probably use the USB HDD on it with the powered USB hub. I don’t know if this is going to be a permanent solution, I don’t really want it to. At the very least I want a device with eMMC to run some temporary FS on for the other SBCs.

I shall see if the netboot tutorial for Raspberry Pis translates well to other ARM SBCs. I kinda doubt it, given that they need to make use of dtbs. I have a tutorial on how to make a net boot.ini config, but I don’t know how to include the dtbs path.

2 Likes

Just a word of advice, many SoCs are still very much in WIP state and Amlogic (linux-meson) is one of them so I don’t see why you’d try to shoehorn mainline onto it. As far as Linux and ARM goes LibreELEC seems like a pretty good “measuring stick” when it comes to mainline support. For now I’d only use Amlogic to consume media (CoreELEC) as support isn’t there yet. SD controllers can be hard and qurky, just look at how many issues RPi has and that is by the community deemed as “well supported” hardware. You might very well be bitten doing intensive I/O tasks (not recommended on SD-cards) on hardware with “beta-ish” software at best.

2 Likes

I know some stuff aren’t supported. But most of what I need works. As for corrupting the SD, it didn’t happen with just the n2, I had this happen to me with microSDs that I plugged into the RockPro64 as well. So I’m pretty sure it’s the RPi’s quirky USB controller.

1 Like

Drivers have been a bit troublesome on both Linux including various forks and at least FreeBSD regarding the RK3399 but they should be stable now with some minor quirks. I have yet to corrupt a memory card on my RockPro64 boards and some are quite abused to be honest :wink:
I have experienced myself quirky results with Amlogic and their vendor fork (via CoreELEC and Lakka) and given that linux-meson is still a relatively young project I wouldn’t be surprised if you run into corruption issues. CoreELEC does however seem to be quite reliable in that regard now though…

1 Like

Could be, but again, the corruption seems to always happen either when I plug my SD card into my RPi, which uses a broadcom chip, or when I plug it out. I saw myself, the partition was good when running lsblk, then after mounting it, I get the magic number error, then another lsblk results in the partition being gone. A dmesg shows the same error about magic number afterwards. So I can say with a high amount of certainty that the odroids and rockpro64 aren’t at fault here.

2 Likes

The RPi family are known for sd card corruption but they can be due to a number of issues

Can’t help you much more than that as I only have Allwinner, Amlogic and Rockchip devices

2 Likes

I set up a tftp server and I was ready to start migrating. Tried compressing / and realized I forgot to exclude /dev, /proc, /sys and other stuff. So I ctrl+c out of tar. On next compression, I hit a bug on meson-mmc (at least, dmesg was showing error messages coming from meson-mmc) and had lots of read issues (tar was verbose and showing the read errors on the files that failed to get archived). Only got fixed after a reboot. Could also be because I was basically archiving /dev files to a tar archive and when I saw the output, that’s when I realized how dumb that was.

Normally I would just rsync, but I was too lazy to set up root authentication on my target, in order to preserve all the files and folders ownership. Taring and extracting makes some sense, except that you need additional space somewhere for the archive itself. Not a big deal for a rootfs.

Unfortunately, the tutorial Wendell made for the RPi does not translate well with other SBCs. I need to figure out how to point each SBC to their supposed boot partition and rootfs. Maybe based on the IP they get. But now comes the questionable part. Petitboot has a different IP than the one the HC4 gets when the OS boots up. So that’s probably going to be a problem, but at least it seems to get a consistent IP address. The even weirder part? They use the same MAC address, I looked into the DHCP leases. Maybe that won’t happen once I configure IP binding in DHCP.

I know that I need a pxeboot.cfg file for the SBCs to read, but unlike RPis, my SBCs have different bootloaders and different dtbs, so I would need one for each. I need to get to read about that.

At least, once I get this going, it will get the SD card problem out of the way and allow me to test more readily, without needing to constantly plug the SD back and forth between my main PC (the pi 4) and the SBC I’m working on. But another wall is about to be hit, as I want to have VLANs on my switch and my SBCs, so I might need to make use of the native / untagged VLAN in order to boot, then once the OS loads up, it will set trunk. At least it shouldn’t impact security that much, as the bridge to LXD containers will be tagged, but in access mode. But still mildly annoying that I have to use the native VLAN. This is why having multiple ethernet ports, with one (or two for redundancy) for management is a nice to have. But this is a budget build, made to be replicable. Technically it doesn’t even need a managed switch, but I chose to do that because I’m insane.

2 Likes

I am getting closer. Managed to configure the dhcpd just with the option 209 and 210, but it seems like petitboot is grabbing the config file as a kernel image file instead. So I need to find a way to make petitboot read it as a config file.

On the plus side, if I power up the hc4 without an SD card and set it to manually grab the config file from the tftp server, it does grab it properly and it loads it. But it doesn’t boot. Bummer. I can see a NFS conection from the hc4’s IP address to the NFS server, I can ping the hc4, but it doesn’t seem to boot up properly, as sshd doesn’t start up.

Petitboot does know how to read pxelinux.cfg files, like mentioned before. But a cool thing about it is that you can also give it the dtb parameter and it will grab it just like all the other options, like kernel, initrd and boot arguments.

I’m really undecided if I should also slap petitboot on the n2+. It’s more of an alpha release, but I should work pretty similar. I don’t know if u-boot has any advantage over petitboot. I mean, it’s likely that Petitboot can’t boot OpenBSD, I read that it could technically boot FreeBSD, but still, I’d rule it out for the RockPro64. But any other advantages u-boot might have over petitboot? I don’t know, I’d like to hear about it.

Back to the hc4. Technically, the boot.ini file that came with the original official Ubuntu release had a few more lines that mentioned a kernel load address and other stuff. But I don’t think that should be necessary for a NFS rootfs. I could be wrong though. But if the opposite is the case, then I’m pretty screwed, at least in terms of using the Ubuntu binary blobs.

I found some instructions online on how to boot the hc4 and install debian netboot using the ppa linuxfactory repos, which have similar settings to what I got and it doesn’t seem like they use the loadaddr options either. It’s been probably 4h since I started working on this today, I should probably get some sleep.

2 Likes

Just tell the kernel to mount filesystem using NFS or iSCSI?
https://www.kernel.org/doc/Documentation/filesystems/nfs/nfsroot.txt
https://wiki.archlinux.org/title/ISCSI/Boot

While not being 100% diskless it’ll most likely save you from most issues you’re seeing and due to how ARM and MIPS it’ll most likely be less painful than trying to shoehorn in PXE boot or do custom u-boot builds.

1 Like

Worth the trouble looking at that tutorial with netboot. I looked up the linuxfactory ppa and found a pxelinux.cfg file there. Grabbed it, looked at the config and found a few differences. Managed to fix the output on the hc4, as previously it was not working. This was really helpful, because now I figured out the issue.

I forgot to edit fstab. REEEEEEEEEE. And now I can boot into Void.

I still need to fix the config / PXE autoconfiguration.


And I was fixing while writing this. Was a wrong entry in the DHCP server. I’ll probably post what worked in a future comment. But got the configuration to auto-load. Now I need to make it boot from it automatically.

2 Likes

And it didn’t work at first, but then I looked into the config and saw that I didn’t have the option to boot from a network device. Added that and now I get auto-boot. Sweet!

Yes, that’s what I did.

I have a diskless boot now. Well, technically, rootfs is still on nfs, but no disks attached. So I guess that counts. It’s no frugal install, as in, booting rootfs into RAM, but this does the trick for now.

Not sure about the custom u-boot build, but it may be an option for the n2+ if I can’t make u-boot automatically boot from tftp. The plan was always for the n2+ and other workers in my infrastructure to boot from netboot and have their rootfs mounted on a NFS on a zpool on the HC4.

Petitboot seems to work wonders and I am very tempted to put it on the n2+. After I get done with the testing on the HC4 and manage to get ZFS to install properly, I’ll probably move back to an SD card, as this device needs to be standalone-ish. The RockPro64, I’ll get an eMMC module for it and install either OpenBSD or FreeBSD on it, once I figure out how. The netboot approach is a good test ground for now and might use it on the rkpr64 to see how I can make it work, but just like the hc4, it will also need to be standalone, as it will be the first device to start on my network: my router.

3 Likes