Journey into SBCs (ThatGuyB in ARMland)

Hardware
1

This thread is dedicated to my woes into the SBC world outside the Raspberry Pi ecosystem. I will be talking hardware and software, challenges and my usual rants.

As some of the forum members already know, I have recently purchased a RockPro64, an Odroid N2+ and an Odroid HC4 (the later which hasn’t arrived yet). I’ve been fighting with both the RkPr64 and the N2+.

My final goal is to make an easy to replicate infrastructure for people to run services at home on the cheap with some optional form of hardware high-availability if your budget affords it, all on single board computers that aren’t too power hungry. These can serve as a DNS, web server, mail, monitoring, file server, nextcloud and whatever else you can run on linux and / or OCI containers (docker, podman, k8s etc.).

As mentioned in the intro, this thread will be mostly serve as my journal to what I’m trying to do, what I’ve done so far, how things progress and allow others to keep up with whatever I’m doing (yay social media), but anyone who wants to post their own SBC adventures and notes can feel free to join me in this thread, I don’t mind converting it into a General thread about SBC journeys and / or Q&A and / or tips and tricks and / or helpdesk thread strictly for SBCs.

6 Likes
2

I have been fighting with both getting OpenBSD to boot on the RockPro64 and with Void and Alpine to boot on the Odroid N2+

On the N2+, I recently discovered that, to get alpine working, I may need to use the Alpine forked u-boot which was converted to work with tar.gz files. Alpine by default for SBCs / aarch64 installations, doesn’t install itself in disk mode, but in diskless mode, what is normally known on other distros as “frugal install” with boot option “toram.”

Although Alpine has the Odroid N2+ in its aarch64 image, there are no documentations on how to install it. From looking over the Arch ARM wiki, it seems like all that would be needed would be to just make a vfat partition, copy the files over, dd the u-boot.bin onto the SD card itself and should be good to go. But that doesn’t work.

I have managed to get Void to boot using the Manjaro bootloader and Manjaro Linux Kernel 5.16, but I somehow borked my install while trying to remove the manjaro splash screen from the boot screen. Actually, I borked my entire SD card, partition 2 was unreadable, with partition 1 being corrupted, but still readable, but most folders and files in it had weird characters.

I’m currently taking a short break, but I’ll go back to try to get Void to run, as it was the closest I could get to a workable system.

3 Likes
3

There is some weird glichiness going on with my setup.

So, as I mentioned, I managed to get Void running by just copying the manjaro install /boot over to my Void install’s /boot dir (formatted as vfat of course) and dd’ing u-boot.bin onto the SD card where I put Void.

I edited /etc/fstab and added the PARTUUID for my SD card, then edited /boot/boot.ini and replaced Manjaro’s SD PARTUUID with my partition table’s. This enable Void to boot.

However, the glitchiness comes when trying to insert my password. I did reset the root password, but that’s not it. After I type in “root” and hit enter, I can only type one character on the keyboard before the TTY decides to press the enter key again. Then if I try to type root again, the TTY just inserts enter again. And this goes on for a bit, until I am able to type again, but then the cycle repeats.

I tell to myself: “fine, I don’t need a TTY, I’ll just enable SSH and dhcpd-eth0.” Chroot into the SD card, make some symlinks as per how runit works, “bam, I’m done.” Oh no, I plug the SD card in the N2+, insert the ethernet cable, connect the power, the board comes up, but I get no IP address. I can see some activity on the n2+ and the switch network ports, likely DHCP requests, but for some reason, my pi 3 doesn’t serve it any IP.

dhcpd.leases doesn’t show anything new. I did remove the previous entry before, I thought maybe that may somehow interfere, but it didn’t matter, the board doesn’t get any IP. I did boot into manjaro again and made sure the interface name was called eth0, and so it was. I’m out of options for now, other than running armbian on this, because I refuse to run manjaro, even if it has a shiny new kernel that has mainline support for most of the things on the odroid n2+.

2 Likes
4

I haven’t had the energy to play around with trying to get alpine or void to run on the Odroid N2+, for now I’ve been playing with Manjaro, as I had the SD prepared anyway.

I’m glad to see that both the Odroid N2+ and the Raspberry Pi 3 have vlan support. It was as easy as adding links using the ip command. For alpine on the pi 3, I had to also install vlan and modprobe 8021q (vlan support is IEEE 802.1Q for those unfamiliar)

I didn’t do anything crazy yet, just set the switch ports on my new Zyxel XGS1210-12 in trunk mode and left native vlan enabled, set static ips on the virtual interfaces on the SBCs and tried pinging and ssh-ing via the new vlan. After that I disabled the native vlan and only left the port in trunk mode, to make sure the things actually use the vlan tagging and not the native vlan and ssh still worked, but the ping on the native vlan port’s IP address didn’t work, so all good.

I’m still salty that I can’t access the SSH interface on this switch, even though it is running a SSH server.

I’ll probably start preparing my switch to fully utilize vlans now, in preparation for my future infrastructure. I’ll need to get the RockPro64 going soon. Unfortunately, while testing to get alpine or void working on the Odroid n2+ on the SD card I wanted to use with the RockPro64, I discovered that my 2GB SD card is throwing out I/O errors in dmesg, so it’s probably on its way out.

I only have 4x 2GB microSD cards, one is the one above about to die, the other is running Alpine on the Pi 3 splendidly, one serves as my /boot device on my Pi 4 main desktop, to get me into the 256GB M.2 SSD / partition, and the last one has a backup copy of an old version of Raspbian (debian 8 I think) that I kept just in case I wanted to use it to update the firmware on my Pi 4 and other rpis.

Then I have 2x 64GB and 1x 32GB microSD cards that I used to store photos and videos on my phones and tablets along the years. And I don’t like being excessive with the storage if I know I am not going to use it, but I don’t have other uses for those uSD cards. I have 2x 16GB SD cards left, one in my Pi 2 running Void, one that was originally running Void on the Pi 3, but that now has Manjaro on the Odroid N2+ for testing.

I will probably repurpose the Raspbian SD card for the RkPr64 if it doesn’t throw the same errors that the other 2GB SD card is showing. Those things have to be probably 20 years old by now. I know for certain that one of the 64GB SD cards I have is about 8 years old and I had the 2GB SD cards back in the days when I had a Samsung SGH-F210 phone and earlier than that. I really miss that phone, it had just 1GB internal memory, so all my music had to go on the 2GB SD card.

I’m not sure I have the energy to work on this project tonight, I absolutely hate messing with the networking when I need internet access to search manuals and wikis. A second computer would have been god-sent. And while I do use phones from time to time when it becomes necessary, because I screw things up, I don’t like using phones because they keep locking up when I need them to stay awake so I can read the dang screen. Changing the settings to turn off the screen later, and back when I’m finished with it is a pain.

I’ll see if I can do anything while I listen to a downloaded podcast or something, I’ll need to open all the pages I need before I start breaking stuff.

2 Likes
5

I remember why I didn’t get OpenBSD to boot on the RkPr64. Tonight I’ve been messing with it, but when trying to mount OpenBSD’s partition from the SD card over on my Pi to view its contents, Linux didn’t know to read UFS, FFS!

Thankfully my old home lab came in handy, I made an OpenBSD VM and I’ll use that to mount the image and extract the u-boot.bin file and the dtbs for the RkPr64, hopefully I’ll make it work tonight.

2 Likes
6

I managed to find a 128GB emmc module for the Odroid N2 back in January, Samsung no longer makes those flash chips, but look around maybe you get lucky, emmc on it is like magic.

3 Likes
7

After my Odroid HC4 gets here, I’ll start doing netboot on them and have a NFS on the RAID1 HDDs that I’ll get. The only thing that needs local storage is the HC4 and the RkPr64. I just needed to test how the boot process goes on these things.

1 Like
8

I can’t, for the love of God, get OpenBSD to boot on the dang RockPro64. I don’t know what I’m doing wrong, it’s not apparent to me. The install for the supported boards doesn’t say anything about writing the u-boot.bin onto the microSD card after the fact.

And I can’t believe that Manjaro of all things has such a big support for ARM. Manjaro? And not Arch Linux ARM? ALARM doesn’t even have a RockPro64 image. At least ALARM has a build for the Odroid N2/+.

I found a github repo, z2d (zero to docker), which supposedly helps you get docker up and running on some ARM boards. Apparently it has support for getting Alpine generic to run on Odroid N2, so I’m going to be reading its source code (it’s just shell) and see how I can make it work without adding other non-necessary things, like docker on my install.

1 Like
9

I have read a bit into the z2d. I really don’t want to execute the scripts. Most of them are fine, just installs docker and other utilities, sets up cloudflare’s dns in resolv.conf, updates the system and a few other things. However, for one, the system uses Alpine rootfs instead of generic ARM, i.e. it’s installing alpine in disk mode, which I don’t want, otherwise I wouldn’t be running Alpine in the first place.

Secondly, this is the kind of stuff you have to lookout for in any random scripts on the internet:
curl -sSL https://www.dropbox.com/s/e0usa1hy934w9qp/linux-4.9.160-n2-169894-g300d0f15a969.tar.xz?dl=0 | tar --numeric-owner -xhJpf -

It doesn’t look too innocuous, it “just downloads” some binaries off of dropbox. They look like a kernel, libraries and modules. But you don’t know what those things have been compiled with.

Since z2d is off the table, and since I could not make OpenBSD to boot on the RockPro64, all I can do now is try to make Alpine work on it, at least to make sure the SBC itself is fine. I mean, it does get power and the Ethernet port LED stays active at all times (even without a cable in it). Actually, I don’t know if that’s a good sign, but considering that both the power and reset button work, the board itself should be fine, maybe.

I did download armbian (which was a pain to figure out how to download, because the armbian site is broken on the rkpr64 page), but the image didn’t fit on a 2GB microsd card, despite the compressed xz image being just 300 MB or so. Quite surprising how much space xz can save up.

I’ll continue working on it tomorrow. I realized today while reading the alpine wiki that I haven’t ever set the bootable flag on the first partition when creating any images. I read that u-boot defaults to the first partition of the SD card if there is no boot flag enabled. It didn’t even go through my mind that u-boot may need it. I format all my HDDs and SSDs as GPT, so I haven’t used DOS partition in ages and forgot about the boot flag. But that didn’t seem to help, at least in my first try. I’ll have to see tomorrow if Ii manage to get either the rkpr64 or the on2 to boot into alpine.

If I can make either one of them, I’d be happy, because nothing went as expected. This is also the reason why I created this thread in the first place, if you aren’t going to run a prebuilt image like armbian, debian, ubuntu, manjaro or ALARM, it is going to be a pain to set up.

I do still wonder if it’s worth my time waste, considering the fact that I will move my setup to net boot, which alpine does offer images for, which should ease my pain. But until then, I need to see at least alpine runs fine on the on2. I’m more annoyed at openbsd not booting on the rkpr64. Someone needs to draw to me how to do it, because it beats me that bad and the man (install.arm64 text file) is not helpful.

2 Likes
10

I remember why I didn’t get OpenBSD to boot on the RkPr64

Have you looked into this? https://tow-boot.org

It will flash itself into the SPI to give you a BIOS-like experience with boot menus (plus network boot) and everything that will allow you to boot any AARCH64 provided the proper hardware support exists.

4 Likes
11

There’s Armbian for RockPro64.

Not sure how hard would it be to get uboot to use tftp or http to load a kernel and initrd image and mount root over NFS.

(There’s probably instructions/uboot configs for RockPro64 floating somewhere online).

4 Likes
12

I didn’t have time to respond in the morning. The RockPro64 will have to use local storage, a microSD card is mandatory for it unfortunately, because it will be the first device to boot on my network: my router.

I just now installed Armbian on it to test that the board is working. I had it downloaded from yesterday, but I didn’t feel like playing with SD cards and plugging and unplugging my HDMI cable.

Anyway, Armbian booted nice, changed root password, created an account, now it’s up and running. I need to get OpenBSD to work on it. @ulzeraj believe you got it to run on your RockPro64, did you do anything special to get it to run? Or is it just me who’s doing something wrong, somehow? I followed the miniroot70.img instructions mot-a-mot, but the board didn’t boot. Or maybe if it booted, it didn’t display anything out on HDMI and only did on the serial console? Should I have used the install70.img? What are your thoughts?

I’m happy that both boards are working, now I just need to set the software on them properly and it’s been so frustrating trying to run Alpine or Void, since they don’t have official (or even unofficial) images for those boards. And I refuse to run Manjaro, Armbian, ALARM, Debian or Ubuntu on them. I may consider Fedora as a last option for the Odroid and Alpine as a last option for the RockPro64. But I would really want to get OpenBSD on the rkpr64 and Void on the ON2, eventually PXE or NetBooted.

Today my smart toaster has arrived, the Odroid HC4. I wasn’t expecting it, so I need to quickly get it up and running and buy some SSDs or HDDs for it to make a RAID mirror zpool.

I really started to grow on Alpine Linux, since I ran it on my Pi 3 as a router OS for so long now. Extending the lifespan of SD cards is always a nice bonus. I’m not sure if I should get Alpine or Void to run on the HC4. On one hand, Alpine would save the SD card that I would end up putting on it. On the other hand, it would be a bit harder to manage for me. I’ve ran Void for way longer and I know most of its quirks and I’m more comfortable on it. But ZFS is easy to get running on both Alpine and Void. I guess I’ll have to go with whatever I can make it to work and I have had more success with Void. Partial success of getting at least Void booted is better than no success in getting Alpine diskless mode to boot on the ON2.

If I had to pick and choose, I would say my priority right now would be the rkpr64 getting opensbd to run on it. I really want to get better internet speeds than 100 Mbps that my Pi 3 is currently running at. Even my phone has faster speeds through WiFi 5GHz. Seriously, I get 186 Mbps with peaks at 190 Mbps download speed to speedtest. I haven’t tried local iperf3 tests yet, I wouldn’t be surprised if I can go over that on WiFi. But even without the WiFi speeds, I still have a gigabit switch that I want to use to its fullest potential once I start moving data between vlans through my router.

One last thing that I got is a case for the ON2. I didn’t expect it to be so small, being dark it makes the ON2 look more compact than the exposed PCB (the GPIO pins definitely made it look bigger when close to it). The SD card hole is a bit too tight. The SD can go through it, but it cannot be removed without removing the case, which is annoying when I have to unplug the microSD card so many times. I may need to file it down, but I don’t want to ruin the glossy aesthetic, even if it would be hidden by the SD card.

2 Likes
13

I wasn’t using OpenBSD. It was FreeBSD 13, in which a convenient image is available for 13-RELEASE onwards. Since 13 they have committed to make AARCH64 a first class citizen in FreeBSD. Although I gave up because of SATA problems and the cryptographic accelerators which are not there yet - at least not for Samba and GNUTLS.

Honestly if you want a router because of PF I’d recommend giving FreeBSD 13 a try. I think you’ll get a lot more multithreaded performance out of the FreeBSD network stack - which uses a slightly modified PF stack along with some in house goodies like Netgraph. Please note however that neither FreeBSD and OpenBSD schedulers are big.little aware. That means they will use whatever core its free and disregard if they are A53 or A72.

As for OpenBSD the Pine wiki points towards this GitHub page: GitHub - jasperla/openbsd-rockpro64: OpenBSD/arm64 on PINE64 RockPro64. It seems you need to flash U-boot (or Tow-Boot) to the device SPI (this is not the sd card - its the SBC own firmware flash media) and copy the DTB files over. Its a 4 years old article so that file might not work. I’d try looking for a more recent rk3399-rockpro64.dtb.

2 Likes
14

I already saw that article, I was hoping there are other solutions out there. Armbian did boot up fine and interestingly enough, only had a single ext4 partition, no vfat ones, like it’s usually recommended.

1 Like
15

This right here suggested you’ve either skipped the part where you flash the u-boot into the SPI or the process didn’t went as planned. You should at least have seen some boot messages from u-boot.

Again, try this: PINE64 ROCKPro64 | Tow-Boot

Flash the bin file into the scard and boot from it. You’ll either reach a boot menu or press the key to bring it up during initialization. On the menu select boot from the sd card again and select the option to write the firmware. Repeat the process from the GitHub page starting from “Preparing the installation media”.

In case you want to roll back boot the tow-boot installer again to the same menu and select to erase the firmware.

3 Likes
16

I have to be doing something wrong, it’s impossible. I literally just downloaded pine64-rockpro64-2021.10-004.tar.xz and then

tar -xJvf pine64-rockpro64-2021.10-004.tar.xz -C pine64-rockpro64-2021.10-004
doas dd if=/path/to/spi.installer.img of=/dev/sdc bs=1M oflag=direct,sync status=progress

It wrote successfully, the RockPro64 won’t boot it anyway. I read that part where towboot says:

You may need to prevent default startup sources from being used to install using the Tow-Boot installer image.

However, I have no idea how to make the rkpr64 to boot straight from microSD instead of SPI, I’ll need to read up on it. But I expect I may need a serial adapter, which I don’t have on hand.

I just flashed Armbian on my SD card again and I’ll try to see if I can get Alpine or Void to boot on it. It shouldn’t be too hard, should it? I mean, I didn’t get the N2+ to work as intended yet, so…

1 Like
17

apt on the rockpro64 is dog slow. apt update and dist-upgrade were done in like 5 minutes total, that’s fine, but a simple apt search lxd has been going on for 30 minutes now. It goes 1% up every couple of minutes. I gets stuck at 51% and then slowly increases over time. Even after I rebooted. Ridiculous!

I changed the repo mirrors closer to me, it is still doing the same. I don’t get it… I guess I’ll just have to get rid of Armbian ASAP.

1 Like
18

After 2 episodes of the Level1 News and some more time, the apt search lxd reached 97%… htop reports the cpu is running at 1.8 GHz and 1.2 GHz (with fluctuations). So it’s likely apt going insane. I should have used time before running this…

The rockchip is kinda toasty though, running at 85 C for an average system load of 18%. I need to get a metal heatsink for it, to cool that puppy (and to give me something to hold the board in place when I add or remove stuff from it).

1 Like
19

Lmao, getting Void to run on the RockPro64 was the easiest thing in the world using the Armbian image. All I had to do was:

  • Write Armbian to the microSD card
  • Boot for the first time to let Armbian do its thing (expand FS, modify fstab and stuff)
  • Shutdown the board
  • Remove the SD card and insert it into another PC, mounted it, deleted everything except /boot and /etc/fstab
  • Copied over the Void aarch64-musl generic rootfs
  • Chrooted into it and made my user, symlinked sshd and dhcpd-eth0 to have runit start those when Void boots up
  • Exited chroot, unmounted the SD card, placed it back in the RockPro64

Done, got a running Void system. I would have really liked to have OpenBSD on it, but at least that’s a beginning. Obviously I can’t use the Linux bootloader to get into OpenBSD, so that will have to wait. As for Alpine, I’ll probably figure something out, as I really wish I won’t have too many writes on my SD card. I think that’s enough for the day, I’ll probably have to look into the Odroid HC4 tomorrow… or sometime.

Edit:

[oddmin@bikyrkpr64 ~]$ free -mh
               total        used        free      shared  buff/cache   available
Mem:           3.8Gi        73Mi       3.6Gi       0.0Ki       131Mi       3.6Gi

Wow, 73 MiB used after updated the OS.

3 Likes
20

My personal experience is that kind of weirdness is normally related to the SD card. Not sure of model/spec or if the sd card is just trash.

My first attempt was a Sandisk which basically either refused to boot or would boot but fail afterwards. I got a Samsung card that works perfectly. Failure rates probably being related to the data payload and timeout/tolerancy settings on the boot code.

3 Likes