Journey into SBCs (ThatGuyB in ARMland)

The only services running were nfs, zfs, sshd and iscsid. The only packages I installed were neovim, abduco, dvtm and htop. My system was very barebones. But I’m pretty sure lack of swap is what got it. I only had 2 nfs shares (via zfs sharenfs).

I should look into the arc stuff, I don’t need lots of cache.

For fun I have a RockPro64 using iSCSI as a client for anything (mount points) that’s write intensive which works fine until you start to load it heavily. Problem occurs when the kernel pushes ctld to swap and the SoC can’t keep up which results in a small hiccup as iSCSI is rather time sensitive. This causes the client to disconnect but since it’s at least partially swapped out it can’t reconnect and you end up with a loop :confused:

I think the SoC is fine for what I need it to do, the lack of ram is likely a problem. The lack of swap is probably worse than having time-sensitive services swapped, as being unable to do a clean reboot in case of a softhang is a major problem!

The emmc was ufs, but I do recall me adding something in rc.conf, that might have broken it. I think I initially had

mountd_flags="-r"

I tried to restart mountd, but it failed, because it was not added to rc.conf.

Then I added a line

mountd_enable="YES"

It worked when I restarted the service, but probably is now failing at boot.

I’m not sure if these were the exact options. I’m not in the mood to troubleshoot it and I feel now more than previously that I need a stabler system. I’m not saying FreeBSD is unstable, I’m pretty sure it would’ve happened to Linux too (especially with linux going nuts when it lacks swap), I’m saying that I need some better hardware, just to have an additional safety net. Overprovisioning / overspecing a system makes sense, up to a point. I’d think a quad core celeron and 16GB of RAM would be more than plenty (could probably get away with 8GB too).

I’ll be trying to resurrect the NAS before I just to buy stuff… it’s just that frustrations add up.

I’ve had no issue with swap over USB so SATA should be fine too. It’s a bit of a pity that RK3399 tries to use EMMC before SD otherwise it would be rather easy to boot it up in “rescue mode”. I’m not sure what you used NFS for but if you used it for swap that’s probably the reason why it ran out of memory (known limitation as far as I can tell → ZFS - ArchWiki - Paragraph 5.2 ). I understand your frustration though…

I’ve only used NFS for read only (and v3 only) so I can’t say how well or bad it behaves with v4 and writing.

I used NFS v3 for a /home and another one for a rootfs backup, with rsync. No swap on NFS. The rootfs backup froze in the middle, when copying a bunch of small files from /usr/src/kernel-*. But the first backup, of just /home alone (pun intended) worked fine. And I got quite a few small files there too (I have the entire void-packages git cloned locally, it has a bunch of templates).

I left the system off for the night, now I plugged it in… it just booted. And it showed a display output while at it.

It did give an error at start that / was not properly dismounted and that ada2 and ada3 have corrupt or invalid gpt tables. It’s probably fine, because I didn’t format a partition on the disk, I just gave the entire disk to ZFS, created by /dev/diskid/DISK-220* (whatever the two disk IDs were, it’s a mirror). zpool status shows everything fine.

I don’t get it, why would the system refuse to power on? I did try to leave it a minute without power, but it didn’t do jack :poop:, there was still no display output. After a night, it seems fine. Idk… I don’t even want to think about it.

Any tips on creating a swap partition? In the rootonzfs wiki, I can see that in the install guide, the recommendation for swap is to do it like this:

zfs create -V 2G -o org.freebsd:swap=on -o checksum=off -o compression=off -o dedup=off -o sync=disabled -o primarycache=none zroot/swap

What do you think? I’ll probably do an 8GB zvol, it’s unlikely it’s going to use more than 2GB of swap, but it’s fine.

To my knowledge using ZFS for swap is not recommended and may likely cause issues, based on that you’re probably better off using your eMMC storage (UFS) for that. I assume all space is already allocated so I’d suggest that you use a swapfile. FreeBSD Handbook | FreeBSD Documentation Portal

I’m guessing that the HDDs weren’t clean from GPT information when they were added to a ZFS which is why you’re seeing that message. The reason why you might want to use GPT (partitions) is because HDDs can vary in size and since ZFS requires at least the same amout of space of the replacement drive you may run into annoying issues where your replacement drive (if you need to get a another model/brand) don’t have the exact same amount of bytes available.

I have no idea why your system didn’t power up, since you have USB drives one guess is that you get some backfeed going on (Google usb backfeed rpi) which causes the SoC not to fully turn off. I don’t know if that’s an issue specific for the RockPro64 but I know its an issue on other SBCs.

I don’t know if it’s really right to ask this here, but I didn’t feel like this question was worth a new thread: has anyone managed to make ffmpeg work with the Pi4 GPU? I have a bunch of videos in my Photoprism container and I can’t even think of playing them back. Trying to decode them through the CPU just results in ffmpeg pinning the CPU to 100% for way too long.

And even if I get it to work, would it be worth doing or the Pi 4 GPU is not good enough to decode smartphone 4k video?

Note: I don’t have any USB drives in the rockpro64 yet. And I didn’t have even a keyboard. I guess it will stay a mystery.

Have not used ffmpeg for that, although technically yt-dlp uses it when converting or merging video and audio.

cat /boot/config.txt

gpu_mem=512

# Enable/Disable experimental desktop GL driver
# requires package: mesa-dri

# with full kms
dtoverlay=vc4-kms-v3d,cma-256

# with fake kms
#dtoverlay=vc4-fkms-v3d,cma-256

I am using kms for GPU acceleration in general and I have the mesa-dri package installed. Note I also added cma-256, but that was only because of some sway bug IIRC, where it wouldn’t start because it thought the gpu only had 128mb or something like that (although the vgpu is self-adjustable IIRC again). You can try the cma-256, or leave it out, it should be optional.

If you haven’t tried this, try it, then reboot your pi and see if you can make ffmpeg to do it. I’m not sure if the pi gpu can do 4K, it can barely play some highly compressed, low frame rate 1080p youtube and it struggles with high bitrate 720p. Sometimes I get better luck playing videos in firefox than mpv, which is weird. I’m not sure how encoding / decoding would would with gpu acceleration.

1 Like

I had that container running, but don’t remember doing any configuration on the base OS to make it work. I should look into that for sure!

Surely will, thanks for the pointer! I’m not trying to play back the files locally so I think the container is trying make some transcoding on the fly to serve the video. I need to check if there’s a way to disable that aswell. It’s not like I’m watching my own videos like youtube, just trying to figure out if they’re corrupted or not.

1 Like

@ThatGuyB
Ahh sorry… I read two different posts at the same time :confused:
Hardware acceleration is only used for decoding and encoding, not for muxing (what yt-dlp does).

Did it work out using a swap file?

1 Like

Haven’t tried yet, lmao. I will either today or tomorrow. Thank you for your tips! :slight_smile:

1 Like

Did you ever get around fixing it?

Sorry for taking my sweet time. I was just thinking today of doing it. Seeing your message made me jump on it.

Yes, just went with a swap file on emmc. That way I won’t be dependent on ZFS mounting the pools on boot and I can save some power from hdd spins (which is what I would have used for swap), or save some writes on the ssd pool.

Just finished adding the 2GB swap file in fstab now. I will give the rsync over nfs another go, hope it doesn’t break.
:man_shrugging:

1 Like

Given the lack of “noise” :wink: I guess its working now?

I haven’t really had the time to work on it. Yesterday I was trying to do some experiments and use nfs v3 and iscsi via wireguard on my LAN, just for lulz. But I didn’t set it up yet.

I had taken a second rootfs backup via rsync on nfs and it seemed fine (well, it was an incremental, but I also delete things that have been deleted from the source). About 200M of difference. I need to think of a backup strategy for this, but I haven’t put my mind to work. It’d probably be something like mount, rsync, unmount, snapshot, zfs send to the other pool.

You probably want to set CPUTYPE?= in /etc/make.conf and compile for your specific arch if you’re going to play around with crypto etc. I have latest patchversion of 13.1-RELEASE compiled and a curated ports collection if you’re interested.

1 Like

I will look into it, I had other things I prioritized, I didn’t have time to work on it since. I remembered FreeBSD has a bunch of example confs, so I grabbed the make.conf. But neither the example, nor the man make.conf show anything ARM related, the example is pentium3.

make.conf « etc « examples « share - src - FreeBSD source tree is what you’re looking for :slight_smile:

1 Like