Odroid hc4, another awfull arm experience

[ynfh26jy]

It took me almost 2 years to get over my pinebook pro disappointment. Pinebook turned out to be half baked, prototype device witch turned out to be completely useless. Desktop experience was hard to justify even for testing. Headless experience was also bad because SSDs kept disappearing from the OS. (probably because of power delivery issues). It was really frustrating.

Odroid hc4 interesting hardware bad software experience
Specs of this board are rally interesting. If you buy a HC4-P kit you get a basic NAS with real sata slots powered by asmedia sata controller.
Key specs:

• amlogic S905x3, 4 cortex-a55 cores
• 1Gb ethernet
• ASmedia ASM1061, 2 sata ports
• 4GB of ram

Base cost without shipping and taxes 88 USD. They ship via DHL, shipping is fast but expensive.
It’s a nice package. personally I’d prefer 2 ethernet ports for some redundancy but I could live with single port if everything else was fine…

I decided to give arm another shot. What could go wrong?

“Open source” hell
At first I’ve purchased single hc4 for testing. My goal was test how usable it. My goal was to get gentoo running on it.

Bootloaders

Odroid hc4 comes with interesting bootloader, petitboot. It’s an embedded linux distribution designed to act as bootloader your real linux distribution (it launches you krenel via kexec). It comes from ibm power9 architecture and sounds like a major improvement over crud u-boot. Sadly odroid petitboot has some strange issues.
Petitboot on hc4 can parse grub.cfg and boot.scr scripts and generate grub-style menu entries. Sadly it’s not very good at it. It often doesn’t detect your menu entries and I couldn’t find a way to debug config detection process. I’ve later managed to craft some grub.cfg according to forum examples.

Second petitboot issue: It doesn’t detect your menuentry if you don’t have initramfs. This is just bizarre, If you menuentry doesn’t contain initrd command petitboot won’t show this menuentry. You cannot boot automatically without initrd.

Final nail in the coffin. Odroid petitboot cannot boot mainline kernels? For some reason petitboot cannot boot mainline armbian kernels. In the same Odroid kernel can kexec odroid kernel. But odroid kernel cannot kexec mainline kernel. I’ve never experienced it on x86.
I’ve asked manufacturer about it but they didn’t help me.

Armbian images for HC4 exist and work really well. At least linux works well. Because of these petitboot issues armbian boots using basic u-boot on HC4. It can be done by pressing spi boot bypass button during boot or wiping petitboot from spi flash.
After dozens of boots with button pressed I’ve just wiped the spi.

OS image and software support
Odroid provides official ubuntu and debian images for their board. Images are customized to work on their boards and run on customized kernel images.
Odroid maintains their own kernel sources… they are not very up to date.
I like to avoid manufacturer linux images. Who knows what messed up? Kenrel updates and patches? I don’t think so.

Thankfully odroid hc4 works just fine on mainline u-boot and mainline linux. After some try and error and studying of armbian build scripts I’ve managed to compile my own u-boot, kernel and glued it all together into working SD card images. Success.
It was a bumpy road and manufacturer documentation wasn’t useful at all.
Official u-boot compilation guide suggests using some ancient precompiled GCC from 2013 and 2014 to compile their own branch of u-boot from 2015. I tried but encountered a lot of compile errors… using precompiled tools and custom branch.
Odroid wiki also doesn’t tell you that you need amlogic blobs to create working SD cards. This one took me a while to figure out. I should have started with u-boot wiki instead.

I don’t understand why hardware manufacturers are so bad at it. Why maintain your custom outdated kernel branches instead of supporting and fixing mainline? (money and laziness)

Back to the main topic.
I have odroid hc4 and I can reliably build gentoo images for it. What’s next?
I’ve considered using this board as a low-end ceph node or minio. It’s so much cheaper than my n100dc-itx node that I could forgive it’s lack of RAM and single ethernet port.

I’ve purchased 3 more HC4s and decided to start by setting up test minio instance on it. I’ve discovered some hardware issues…

1. Odroid board cannot reboot with some bigger SD cards
   If you wipe petitboot and boot using u-boot your system cannot cleanly reboot
   This affects official images and unofficial ones. I’ve discovered that its doesn’t happend when I’m booting from some no-name 4GB SD card. All my 32GB cards fail to reboot.
   Smells like a hardware issue.
   This issue was reported on the forum in 2022 but manufacturer disregarded it as armbian specific issue.

2. One of my 4 HC4s randomly crashes. Linux dies because IO with sd card stops. I’m still investigating it but it doesn’t look good.

3. One of my HC4 has silent data corruption on one of sata ports. This is quite bad. Sata port is just broken, checksuming filesystems fail on good drives connected to this port.
   I’ll have to RMA this board. Odroid has a bad RMA policy, they only give you 16 weeks warranty. You have to pay for RMA shipment to korea and they prioritize repair over replacing boards. This doesn’t make me confident into their product. If you don’t mind higher costs, I’d suggest buying their products from local distributors if your local customer protection laws are good.

I’m really salty in this post because I really like the ides of this product. It just didn’t live up to my expectations. Software support is bad, opensource friendlyness of the manufacturer is bad. Hardware issues are bad.
I’m disappointed.

I’m considering buying some RK3588 based board with 32GB of ram for some dev work but other than that I’m giving up on non-enterprise, non-uefi arm for foreseeable future. User experience and hardware inflexibility of these devices it just horrible.

[ack]

This needing a custom kernel for so much of the ARM SBC ecosystem is a really bummer, and why I’ve stuck been sticking to x86.

Really hoping we get through this phase, hardware vendor supplied distros and kernels just raise too many security questions, and are quite the downgrade over being able to install run vanilla Debian.

It’s a shame because the HC4 is a very cool affordable device on paper.

[ynfh26jy]

I’ve used enterprise arm servers with uefi. You can install ubuntu straight from ubuntu website on it. Only noticable difference in user experience was bad single thread performance.

Arm can be this seamless, but we’re stuck in this embedded hell.

[greatnull]

Did those server have uefi or uefi-like enviroment? Keeping to existing standarts would explain the ease of installation. Oh and ACPI, without it you are device tree hell and good bye universal compatibility.

There was some talk about standardizing Server_Base_System_Architecture just avoid exactly these issues in arm land years ago, but it seemingly hasn’t progressed anywhere beyond enterprise hardware.

Even those brand new snapdragon elite cpu are in some weird in between state, linux isos must be customized to each device.

Something about having uefi, but messed up nonstandard extensions.

EDIT: maybe not, but still bad

ACPI tables shipped by Qualcomm aren’t suitable for use outside of Windows, with Qualcomm drivers.

src: https://www.reddit.com/r/hardware/comments/ymh8uu/eli5_is_there_any_good_reason_why_arm_systems/

[ulzeraj]

@ynfh26jy

Base cost without shipping and taxes 88 USD. They ship via DHL, shipping is fast but expensive.
It’s a nice package. personally I’d prefer 2 ethernet ports for some redundancy but I could live with single port if everything else was fine…

I personally think their cases are weird, specially the toaster NAS. I wish the toaster NAS format would go away and be buried in the sand where researchers 1000 years from now will wonder what was that ancient civilization was thinking.

@ynfh26jy

My goal was to get gentoo running on it.

Ah, a pain connoisseur.

@ynfh26jy

Odroid hc4 comes with interesting bootloader, petitboot. It’s an embedded linux distribution designed to act as bootloader your real linux distribution (it launches you krenel via kexec). It comes from ibm power9 architecture and sounds like a major improvement over crud u-boot. Sadly odroid petitboot has some strange issues.

Interesting.

@ynfh26jy

Petitboot on hc4 can parse grub.cfg and boot.scr scripts and generate grub-style menu entries. Sadly it’s not very good at it. It often doesn’t detect your menu entries and I couldn’t find a way to debug config detection process. I’ve later managed to craft some grub.cfg according to forum examples.

ew… I prefer u-boot now.

@ynfh26jy

Second petitboot issue: It doesn’t detect your menuentry if you don’t have initramfs.

Can’t blame them. Other than being one less line to configure I can’t think of many reasons to include all modules on the kernel.

@ynfh26jy

Final nail in the coffin. Odroid petitboot cannot boot mainline kernels? For some reason petitboot cannot boot mainline armbian kernels. In the same Odroid kernel can kexec odroid kernel. But odroid kernel cannot kexec mainline kernel. I’ve never experienced it on x86.
I’ve asked manufacturer about it but they didn’t help me.

Device trees are the first thing that comes to mind. You need one to boot ARM and other architectures. There is no BIOS plug and play.

@ynfh26jy

Armbian images for HC4 exist and work really well. At least linux works well. Because of these petitboot issues armbian boots using basic u-boot on HC4. It can be done by pressing spi boot bypass button during boot or wiping petitboot from spi flash.

Maybe check if tow-boot supports your device? It’s an u-boot variant that pretends to be an UEFI firmware. On my RockPro64s I can install Linux distributions from their official iso images thanks to it.

@ynfh26jy

I don’t understand why hardware manufacturers are so bad at it. Why maintain your custom outdated kernel branches instead of supporting and fixing mainline? (money and laziness)

Probably because upstreaming the patches are hard. Normally they have a chip, assemble the PCB and want to sell as quick as possible to cash out.

@ynfh26jy

I’ve purchased 3 more HC4s and decided to start by setting up test minio instance on it. I’ve discovered some hardware issues…

You really like pain!

@ynfh26jy

I’m considering buying some RK3588 based board with 32GB of ram for some dev work but other than that I’m giving up on non-enterprise, non-uefi arm for foreseeable future. User experience and hardware inflexibility of these devices it just horrible.

Certified masochist.

My take on this. I have 2 RockPro64s which work really well but after a great deal of suffering and finding out the best possible configurations. I wouldn’t buy more Pine stuff neither the Rockchip stuff unless for a very specific project. Not worth it.

Mac minis are great if you can live with either the limitations of either MacOS or current Asahi. Meanwhile I heard on ETA Prime about the first SBCs using Qualcomm’s chips and that company seems to be doing things somewhat properly so if I were to buy more ARM stuff I’d watch the market for used M1s or wait for whatever Qualcomm is cooking.

[greatnull]

@ulzeraj

My take on this. I have 2 RockPro64s which work really well but after a great deal of suffering and finding out the best possible configurations. I wouldn’t buy more Pine stuff neither the Rockchip stuff unless for a very specific project. Not worth it.

How bad was it compared to like rpi? I have heard lot of warnings, but never specifics.

On paper its interesting and well performing hardware, but issue like that would be a dealbreaker.

[ulzeraj]

It’s not as bad now as it used to be few years ago. You can install tow-boot to the SP and have an UEFI-like environment where you can boot the Fedora or Manjaro installer and just next-next-finish your way. There is a warning about the boot loader on fedora installer but you can ignore it.

Make sure the adapter is the one with 5A if you are planning to use with peripherals.

PCI-e adapters can be finicky. Even when it works you might see an eventual error on dimesg about something going out to lunch.

Using a SATA PCI-E adapter 4 SATA SSDs will work fine with a power splitter. Spinning rust will probably use more energy than the board can provide.

Do not use BSD. With the exception of NetBSD none of the others know how to operate CPUs with different speeds. FreeBSD will just sync the speed of the 6 cores with the lower common denominator.

Finally I would advise against buying it unless you find used with a very attractive price. It’s a 2018-ish board full of weird quirks. YEs it has good support but unless you really want to use it as a NAS (because the metal case is nice) just stick with RPi5.

Here is the one with 1 SATA disk running relatively IO intensive applications. 25 days uptime, recent kernel fedora server 40.

dmesg is clean of errors.

This one is mostly idle with 4 SATA SSDs.

dmesg shows eventual SATA link resets. Doesn’t seem to affect the system overall. Just some SATA ports going for a split second lunch break I think.

Good think about these boards is the crypto. I can have full disk encryption and no bottleneck on the 1GB/s ethernet port. I could even go for 2.5Gb/s through an usb adapter I believe.

[ynfh26jy]

@ulzeraj

@ynfh26jy

Second petitboot issue: It doesn’t detect your menuentry if you don’t have initramfs.

Can’t blame them. Other than being one less line to configure I can’t think of many reasons to include all modules on the kernel.

I disagree, majority of linux setups doesn’t really require initrams.

You don’t need all modules built in to boot. Dracut by default won’t even include nvme module in initramfs if you’re building dracut initramfs on system without nvme drives.
Noone would notice if kernel was built with sata and nvme drivers built in.
Initrd needs only enough modules to mount rootfs. Other modules are can be loaded later from /lib/modules

@ulzeraj

@ynfh26jy

Final nail in the coffin. Odroid petitboot cannot boot mainline kernels? For some reason petitboot cannot boot mainline armbian kernels. In the same Odroid kernel can kexec odroid kernel. But odroid kernel cannot kexec mainline kernel. I’ve never experienced it on x86.
I’ve asked manufacturer about it but they didn’t help me.

Device trees are the first thing that comes to mind. You need one to boot ARM and other architectures. There is no BIOS plug and play.

It’s not that. Kexec on arm takes dtb as an argument. Issue is that their kernel images cannot kexec mainline kernels.

@ulzeraj

@ynfh26jy

Armbian images for HC4 exist and work really well. At least linux works well. Because of these petitboot issues armbian boots using basic u-boot on HC4. It can be done by pressing spi boot bypass button during boot or wiping petitboot from spi flash.

Maybe check if tow-boot supports your device? It’s an u-boot variant that pretends to be an UEFI firmware. On my RockPro64s I can install Linux distributions from their official iso images thanks to it.

Thanks, I’ll check it out. idk if I I’ll actually install it now that I have working u-boot and my patience is low.

U-boot itself now has some built-in efi booting. I haven’t tried it could also fill that role if I managed to create working SPI image with it. U-Boot Standard Boot — Das U-Boot unknown version documentation

@ulzeraj

Mac minis are great if you can live with either the limitations of either MacOS or current Asahi. Meanwhile I heard on ETA Prime about the first SBCs using Qualcomm’s chips and that company seems to be doing things somewhat properly so if I were to buy more ARM stuff I’d watch the market for used M1s or wait for whatever Qualcomm is cooking.

Heh, recent Qualcomm laptop adventures make me skeptical. But if they have competent efi I’ll consider it.
I haven’t really considered mac mini. Linux on apple arm situation is meh. Lack of any apple support for asahi deves and soldered SSDs and general apple bs are realy discouraging for me.

[ulzeraj]

@ynfh26jy

I disagree, majority of linux setups doesn’t really require initrams.

It’s still a weird way of doing it nowadays. Initramfs is not just for loading modules so they have to include it to support stuff like network support on booting and other stuff.

@ynfh26jy

Heh, recent Qualcomm laptop adventures make me skeptical. But if they have competent efi I’ll consider it.

The bad press is all about running Windows and its ecosystem. If there is interest to streamline it Qualcomm have more resources to submit patches than RockChip and Amlogic.

@ynfh26jy

I haven’t really considered mac mini. Linux on apple arm situation is meh. Lack of any apple support for asahi deves and soldered SSDs and general apple bs are realy discouraging for me.

Yeah it is what it is for now.

Just run MacOS. Brew or Darwin-Nix can get most stuff running. An UTM Linux VM will achieve better performance and reliability than your average arm board. Specially now with UTM server and UTM remote.

It’s not like you have a reliable option for attaching storage on ARM boards either. Thunderbolt is WIP on Asahi so if it ever gets done we might be able to do a lot more stuff on hopefully inexpensive M1 minis.

[ynfh26jy]

@greatnull

Did those server have uefi or uefi-like enviroment? Keeping to existing standarts would explain the ease of installation. Oh and ACPI, without it you are device tree hell and good bye universal compatibility.

These servers have real uefi and acpi. Ubuntu doesn’t need to provide device trees. Pxe boot just works under openstack Ironic.

[ulzeraj]

Ampere server boards are the closest we have to ARM Xeons that can be build out of the shelf. The issue with those is that they are running 2020 tech with DDR4. It seems to idle at 60-100W.

https://www.jeffgeerling.com/tags/ampere