Journey into SBCs (ThatGuyB in ARMland)

Hardware Hub
170

Nailed it. XGS1210-12.

Just the fact that OpenBSD is much smaller, with a focus on secure defaults. If I had performance constraints, I would probably run FreeBSD, because their driver stack and their pf implementation are much faster. But OpenBSD is the most secure OS, by its overly minimalist design, which is the only technical reason why I prefer OpenBSD on routers.

I like all BSDs honestly, but I don’t know if I would be able to manage all of them at once. Unlike Linux distros in general, I tend not to dislike any BSDs, I think, because of their design philosophy and development models, all the BSDs serve a purpose and are not completely useless, as some Linux distros are.

I like that FreeBSD gets the best out of what Solaris made and arguably make it better, DragonFly BSD is developing HAMMER (2), OpenBSD is secure by default and NetBSD is very portable. There are other smaller ones, but those are mostly desktop FreeBSD versions, which is fine.

I know that FreeBSD would probably work better on the RockPro64, because the scheduling is way better, OpenBSD will just move threads around from an A72 to an A53 core, instead of using the better performance cores constantly. Probably other stuff to consider as well. But I would still prefer OpenBSD if I can get it to work.

If I were to work for a company as a sysadmin again, I would like to manage an infrastructure of OpenBSD routers and FreeBSD servers. I find B-Hyve and Jails interesting.

1 Like
171

Today I spent a bunch of hours (probably since 9:00, so upwards of 8 hours with very small breaks) to make the RockPro64 my main router. It was such a pain, but I managed it. I used the Void install that I did a few days ago on the rkpr64. Unfortunately I could not get Alpine to boot on that thing.

Yesterday or two days ago, I used the same method I used with Void, wrote Armbian on the SD card, removed the armbian rootfs and kept some items in the boot folder, placed the alpine-u-boot generic image, but in addition, because alpine needs the modloop bootarg and a few other things in the bootloader, I added those in the… oh fugg, I realized I have not recompiled the boot script, I only modified the script (boot.cmd) that is used to compile the u-boot boot script (boot.scr) that is used at startup. FUGG, OMG, I’M SO RETARDED.

OH GOD IM FUCKING RETARDED

I will try Alpine again… BRUH, I spent 8 hours today to make Void work. And I had a lot of work, dhcpd, interfaces, bridges, vlans, I even locked myself out of my switches management interface, because I’m retarded and I switched the native vlan (unrelated), without changing the management vlan on the bloody thing first).

I need a break.

1 Like
172

Well, I have been redeemed. That script didn’t work. I can’t figure out u-boot’s boot process on the rkpr64. If I could, I would write Alpine’s u-boot on the SD card. Well, I already tried that, but I don’t know at what sector u-boot looks for the boot script or whatnot.

And it’s unfunny that tow-boot doesn’t work. I kinda liked it, because it made things a lot easier (well, not as easy as petitboot, but still, better than trying to figure out u-boot). Besides, I agree with them, the boot process of a SBC should not be handled by the distros, it’s a ton of work. Imagine distros on the desktop trying to support each Gigabyte, Asrock, Asus, MSI and other brands motherboards.

Anyway, I am somewhat sad that I’m too much of a brainlet to figure these out, but I am glad I got Void going on it. My work will pay off, TREMENDOUSLY:

2022-09-06_00-09-1662412691
2022-09-06_00-09-1662412691923×169 14.9 KB

RockPro64 = port 1
Pi 3 (previous router running Alpine) = port 2

I’m glad that this RTL8211F supports VLANs and all that. I hope it will be stable.

2 Likes
173

I did that yesterday morning when I managed to create a couple of pretty good broadcast storms which took down most of the network. I borked some vlan assignments which started it, and there’s obviously something I’m missing about how the networking of jails in TrueNAS works, because they hate me.

1 Like
174 
type or p<iframe width="560" height="315" src="https://www.youtube.com/embed/pAo887mTIZA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

@ThatGuyB, I am looking for an SBC based on the X86 platform. This board almost fits my use case. The only thing it is missing is two Intel Nic’s. Also, do you know of two ethernet ports, X86 Platform SBC?

175

I don’t have any x86 SBCs. I know of the DFI GHF51, but it’s expensive as all heck.

ZimaBoard.

It has 2x Realtek 8111H. But you can just add any Intel PCI-E NICs, just like you can with the RockPro64. The price is more swallowable than the DFI. What you pay for the DFI is the GPU, which you don’t need in a router, those boards are made for arcade games and slot machines.

The creators say that it’s compatible with pfsense, openwrt, plex and others, but not sure if they actually tested it prolonged running those.

1 Like
176

There a few “testimonial”/talks around regarding how secure it is but these are all rather dated and some arguments are questionable at best irregardless what you’re comparing it too. OpenBSD does have more strict settings which theoretically improves security but you can likely assume that you’re not a target in general and some of these settings will kill performance making it more or less impossible to use in practice “as-is”.

Given the amount of hoops you’ve done just to get Void installed, is it even possible to update it in a sane manner without breaking it?

177

Keep in mind that ZimaBoard is based around a rather old platform which might not be what you want…

1 Like
178

This should be split into 3 answers:

  • RockPro64: On the rkpr64, the only custom thing Void has is the driver for the realtek USB wifi 5 dongle from github. I can realistically just update it as usual and when I get a kernel update, just remember to rebuild the driver if necessary and use the deploy script. It’s just a kernel module, so probably doesn’t need to be rebuilt though, just deployed in the modules of the new kernels.

  • Odroid HC4: As of right now, I am still running the armbian kernel on it, because the board won’t boot with Void’s kernel. I got confirmation from a Void dev on IRC that apparently the kernel options to support it are not enabled, even if the dtbs and all the other stuff seems to be there. Which makes sense as to why I was unable to boot into it with the kernels from the repo. But I have a chroot environment, I can just update the chroot, and grab the new kernel. Only real problem is that I can’t seem to install ZFS even on Armbian when I run it bare metal. So right now I’m running BTRFS, although I want to move to ZFS. In some crazy turn of events, I may use NixOS on it, since apparently it is supported there… Just need to flash it to a SD card.

  • Odroid N2+: This one’s an oddball. I managed to boot with the kernel 5.15 from the repo when I was using u-boot from the SD card (same method as on the rkpr64, flashed armbian to a SD, ripped out the rootfs, placed void there, booted successfully, installed the kernel, switched some symlinks and it just worked). But when I moved to Petitboot, the armbian kernel still works to boot into, but not the void kernel. I wonder if it’s because I’m using root on NFS and netbooting, I may need to retest it. Worse case, I will just buy an eMMC for it and call it a day, what interests me is what will run on it, not what it is running on. NFS was just a good way to test kernels without having to continuous get out of bed and switch SD cards around, reflash them, or change symlinks or other configs.

So, to answer this more generally, I can update Void on RockPro64 and on HC4 successfully, but for now, the N2+ isn’t fully configured yet, so the last one may break regardless.

One other way I could test it, instead of using root on NFS, is to put the root on a SD card, but keep using Petitboot and netboot via tftp and only keep the /boot in a different location. Can save me some headaches when I mistakenly poweroff or reboot my threadripper box… (I used to have the rootfs on the Pi 2 on its SD card and later on a USB HDD, but the Pi 2 was too slow for more than 1 system to run on it, any update would work with a few KB/s because of the slow write speeds).

179

and odyssey X86 J4125! Quad Core Celeron J4125 Windows 10 Mini PC

3 Likes
180

Welcome to the forum!

Networking: Intel I211AT PCIe Gigabit LAN,

This may be worth the additional cost over the ZimaBoard if people don’t care about expansion, or if they’re fine with janky m.2 to pci-e adapters. I’m glad someone made the good decision to pay a bit more and get Intel network controllers.

181

Testing my Odroid’s best-case scenario speed for data transfer. Using NFS. Using 2x Crucial MX500 SSDs in BTRFS mirror.

[oddmin@bikypi4 tmp]$ rsync -v --progress hrmpf-x86_64-5.15.11_1-20211227.iso /mnt/2/hrmpf-x86_64-5.15.11_1-20211227.iso
  1,566,277,632 100%   72.31MB/s    0:00:20 (xfr#1, to-chk=0/1)

sent 1,566,660,132 bytes  received 35 bytes  66,666,390.09 bytes/sec
total size is 1,566,277,632  speedup is 1.00

66 MB/s write. Not looking good.

[oddmin@bikypi4 tmp]$ rsync -v --progress /mnt/2/hrmpf-x86_64-5.15.11_1-20211227.iso odroidhc4/hrmpf-x86_64-5.15.11_1-20211227.iso
  1,566,277,632 100%   44.50MB/s    0:00:33 (xfr#1, to-chk=0/1)

sent 1,566,660,132 bytes  received 35 bytes  46,765,975.13 bytes/sec
total size is 1,566,277,632  speedup is 1.00

~47 MB/s reads? u w8 m8 :question::exclamation:

Maybe my Pi 4 is not up to snuff. I am using a m.2 NVME SSD in a USB 3.0 enclosure. I still have about 88 GB left (out of 256), so it shouldn’t slow down that much, but the Pi doesn’t have the best CPU, nor the best USB controller, so there are tons of bottlenecks here.

Let’s switch things a little. My TR system has 2x Intel 670p 1TB QLC NVME SSDs in ZFS mirror, of which I only have about 106GB left (not ideal, as it’s not under the 80% mark to have better caching and stuff in MLC or TLC mode), but should be way better than my main PC (the Pi 4). Besides, it will probably cache the image in its massive 64GB of RAM anyway.

[oddmin@biky-tr ~]$ rsync -v --progress /media/hrmpf-x86_64-5.15.11_1-20211227.iso .
hrmpf-x86_64-5.15.11_1-20211227.iso
  1,566,277,632 100%  110.64MB/s    0:00:13 (xfr#1, to-chk=0/1)

sent 1,566,660,132 bytes  received 35 bytes  108,045,528.76 bytes/sec
total size is 1,566,277,632  speedup is 1.00

108 MB/s reads, now we’re talking!

[oddmin@biky-tr ~]$ rsync -v --progress hrmpf-x86_64-5.15.11_1-20211227.iso /media/
hrmpf-x86_64-5.15.11_1-20211227.iso
  1,566,277,632 100%  785.74MB/s    0:00:01 (xfr#1, to-chk=0/1)

sent 1,566,660,132 bytes  received 35 bytes  89,523,438.11 bytes/sec
total size is 1,566,277,632  speedup is 1.00

Wait, what the…?! 785 MB/s writes? That can’t be right. It took 1 second to transfer it, WTH? It’s a 1.5GB file! That is 6825 Mbps, that’s almost 7x faster than gigabit. I don’t trust it. The 89.5 MB/s sounds more realistic.

I did delete the file before transferring it, but just to make sure the result isn’t skewed by some kind of RAM caching or BTRFS on-disk recreation schenanigans, I just transferred a win10 iso from my PC to my TR system. Attempting to write this to the HC4 via NFS.

[oddmin@biky-tr ~]$ rsync -v --progress Win10_21H2_English_x64.iso /media/
Win10_21H2_English_x64.iso
  5,883,697,152 100%  815.57MB/s    0:00:06 (xfr#1, to-chk=0/1)

sent 5,885,133,702 bytes  received 35 bytes  97,274,937.80 bytes/sec
total size is 5,883,697,152  speedup is 1.00

I don’t understand how rsync does that thing with the initial read, I should probably RTFM. But 97 MB/s sounds realistic, I’ll take it.

I’m happy with the results for now for big files.

sent 1,046,287,563 bytes  received 44,867 bytes  15,055,142.88 bytes/sec
total size is 1,045,868,704  speedup is 1.00

This is my PC transferring 1GB of spicy memes to the HC4. All small files. 15MB/s write. Not great, but hey… small files. Besides, bottlenecks from the Pi. This thing really is kinda garbage, isn’t it (the Pi 4)?

Here’s what a real tiny chad transfer looks like:

sent 1,046,276,466 bytes  received 44,867 bytes  72,160,091.93 bytes/sec
total size is 1,045,868,704  speedup is 1.00

72MB/s reads from HC4 to TR. For small files.

How about we delete the memes on the HC4 and re-transfer them from the TR?

sent 1,046,287,730 bytes  received 44,867 bytes  16,222,210.81 bytes/sec
total size is 1,045,868,704  speedup is 1.00

16MB/s write. Ouch. I’d say it’s margin of error at this point. Still, a GB of memes. Ok, ok, fine, the HC4 isn’t the greatest in writing tons of small files at once. I wonder how it will perform when I will launch a package manager update for the containers that will run on the HC4. It will be an interesting experiment, although I kinda wish to use iSCSI and make new FS on it, instead of using NFS. From my experience with NFS on the Pi 2 on a USB 5400 RPM external HDD, I was not impressed with the speed my package manager was extracting and installing downloaded binaries.

Well, that’s a story for another time. That’s all for now.

182

The Linux kernel is using read and write caching which produces a lot of “odd” results and can lead to potential data loss depending on how its configured. If you want to bench rsync you probably also should use more data to normalize results, 30-50Gb would probably give you better results.

If you want to benchmark locally you should look into using fio

If you’re using NFS 4.2 (or higher) performance should be much better than 3.x but I’m not sure how well tuned it is in Linux. Keep in mind that if you use iSCSI on top of a file system you do get more overhead rather than using it directly on a block device. I do that on top of ZFS for one my of RockPro64 “build boxes” on a single SSD (SATA) and while performance isn’t great it’s still fast enough to not make it a bottleneck compared to the RK3399 SoC.

183

I just did a small test to see how it’s going to perform via the network. Nothing fancy. Using NFS 4.2 (as seen by mount command on the client).

I’m almost never going to transfer such a massive file. Maybe 5GB once a blue moon when downloading new isos for testing, but I doubt it. Most LXD rootfs are under 500MB, because they don’t have the cruft of the kernels and other junk, which is the main purpose for this NAS.

I don’t mind the overhead of iSCSI on a FS as long as I don’t hit the weird NFS bugs I hit with my HC4 with its rootfs on NFS. Although funnily enough, the HC4 as a client reports that its / is mounted using nfs v3, even though the server (my TR) is running 4.2. Probably a limitation of mounting the / on NFS.

I’ll give NFS a shot once I have my lab set up properly and see how that moves.

184

I got a decent 4 port SATA card. Same brand, but another model. Really good and gave new life to my RockPro64. Running 2x1TB RAID1 SSD on Linux encrypted and compressed. Considering buying another kit.

General question: can the 12v/5A power supply handle 4x8TB SATA SSDs? I’ve seen a post about someone running a mix of 2x3’5 spinning rust and 2x1TB 2’5 SSD with little to no modification.

1 Like
185

Wrong OS but oh well… :wink: How would you power all drives off the board? I guess you could use a splitter but you might be pushing it even with a quality PSU.

186

Only Linux and NetBSD can do proper big.LITTLE and I need something that supports current century stuff so NetBSD is out.

The store has an official 2 way splitter for 2 drives and I do use it on my current NAS case from Pine64 so nothing new here. An aftermarket 4 way split do exist. I would prefer to use some PicoPSU but I’m not a fan of homemade house fire arrangements.

187

There are complaints about the Pine64 rockpro64 PSU cooking itself to death. 2 users on this forum were complaining about it as well. Keep in mind the Odroid HC4 uses a 15V 5A brick for 2x HDDs. For the RockPro64, you might need one of these laptop style bricks, but 12v (because the rockpro64 doesn’t support 15v anymore IIRC), probably a 12v 7A or so should do.

1 Like
188

Can you point to those? I’ve found most complaints are from users using the 12v/3A version.

That’s the thing. That ugly AF transparent toaster NAS case (what were they thinking) is designed to support 2 3’5 spinning rust platers. I understand SSDs use very little in comparison.

189

Misread. Yeah, a quality 12v 5A PSU should do. I thought we were talking 4 HDDs.

The beauty of these boards is that they use standard barrel connector and you can swap most of their PSUs among the boards, as long as you don’t supply more voltage than the boards can take.

I plan on buying another rkpr64 myself, but I will likely get an amlogic brick instead (since I’m buying all of them from the same store).

I like my toaster, IRL it’s not too out of place on my desk.