The only services running were nfs, zfs, sshd and iscsid. The only packages I installed were neovim, abduco, dvtm and htop. My system was very barebones. But I’m pretty sure lack of swap is what got it. I only had 2 nfs shares (via zfs sharenfs).
I should look into the arc stuff, I don’t need lots of cache.
For fun I have a RockPro64 using iSCSI as a client for anything (mount points) that’s write intensive which works fine until you start to load it heavily. Problem occurs when the kernel pushes ctld to swap and the SoC can’t keep up which results in a small hiccup as iSCSI is rather time sensitive. This causes the client to disconnect but since it’s at least partially swapped out it can’t reconnect and you end up with a loop 
I think the SoC is fine for what I need it to do, the lack of ram is likely a problem. The lack of swap is probably worse than having time-sensitive services swapped, as being unable to do a clean reboot in case of a softhang is a major problem!
The emmc was ufs, but I do recall me adding something in rc.conf, that might have broken it. I think I initially had
mountd_flags="-r"
I tried to restart mountd, but it failed, because it was not added to rc.conf.
Then I added a line
mountd_enable="YES"
It worked when I restarted the service, but probably is now failing at boot.
I’m not sure if these were the exact options. I’m not in the mood to troubleshoot it and I feel now more than previously that I need a stabler system. I’m not saying FreeBSD is unstable, I’m pretty sure it would’ve happened to Linux too (especially with linux going nuts when it lacks swap), I’m saying that I need some better hardware, just to have an additional safety net. Overprovisioning / overspecing a system makes sense, up to a point. I’d think a quad core celeron and 16GB of RAM would be more than plenty (could probably get away with 8GB too).
I’ll be trying to resurrect the NAS before I just to buy stuff… it’s just that frustrations add up.
I’ve had no issue with swap over USB so SATA should be fine too. It’s a bit of a pity that RK3399 tries to use EMMC before SD otherwise it would be rather easy to boot it up in “rescue mode”. I’m not sure what you used NFS for but if you used it for swap that’s probably the reason why it ran out of memory (known limitation as far as I can tell → ZFS - ArchWiki - Paragraph 5.2 ). I understand your frustration though…
I’ve only used NFS for read only (and v3 only) so I can’t say how well or bad it behaves with v4 and writing.
I used NFS v3 for a /home and another one for a rootfs backup, with rsync. No swap on NFS. The rootfs backup froze in the middle, when copying a bunch of small files from /usr/src/kernel-*. But the first backup, of just /home alone (pun intended) worked fine. And I got quite a few small files there too (I have the entire void-packages git cloned locally, it has a bunch of templates).
I left the system off for the night, now I plugged it in… it just booted. And it showed a display output while at it.
It did give an error at start that / was not properly dismounted and that ada2 and ada3 have corrupt or invalid gpt tables. It’s probably fine, because I didn’t format a partition on the disk, I just gave the entire disk to ZFS, created by /dev/diskid/DISK-220* (whatever the two disk IDs were, it’s a mirror). zpool status shows everything fine.
I don’t get it, why would the system refuse to power on? I did try to leave it a minute without power, but it didn’t do jack 
, there was still no display output. After a night, it seems fine. Idk… I don’t even want to think about it.
Any tips on creating a swap partition? In the rootonzfs wiki, I can see that in the install guide, the recommendation for swap is to do it like this:
zfs create -V 2G -o org.freebsd:swap=on -o checksum=off -o compression=off -o dedup=off -o sync=disabled -o primarycache=none zroot/swap
What do you think? I’ll probably do an 8GB zvol, it’s unlikely it’s going to use more than 2GB of swap, but it’s fine.
To my knowledge using ZFS for swap is not recommended and may likely cause issues, based on that you’re probably better off using your eMMC storage (UFS) for that. I assume all space is already allocated so I’d suggest that you use a swapfile. FreeBSD Handbook | FreeBSD Documentation Portal
I’m guessing that the HDDs weren’t clean from GPT information when they were added to a ZFS which is why you’re seeing that message. The reason why you might want to use GPT (partitions) is because HDDs can vary in size and since ZFS requires at least the same amout of space of the replacement drive you may run into annoying issues where your replacement drive (if you need to get a another model/brand) don’t have the exact same amount of bytes available.
I have no idea why your system didn’t power up, since you have USB drives one guess is that you get some backfeed going on (Google usb backfeed rpi) which causes the SoC not to fully turn off. I don’t know if that’s an issue specific for the RockPro64 but I know its an issue on other SBCs.
I don’t know if it’s really right to ask this here, but I didn’t feel like this question was worth a new thread: has anyone managed to make ffmpeg work with the Pi4 GPU? I have a bunch of videos in my Photoprism container and I can’t even think of playing them back. Trying to decode them through the CPU just results in ffmpeg pinning the CPU to 100% for way too long.
And even if I get it to work, would it be worth doing or the Pi 4 GPU is not good enough to decode smartphone 4k video?
Note: I don’t have any USB drives in the rockpro64 yet. And I didn’t have even a keyboard. I guess it will stay a mystery.
Have not used ffmpeg for that, although technically yt-dlp uses it when converting or merging video and audio.
cat /boot/config.txt
gpu_mem=512
# Enable/Disable experimental desktop GL driver
# requires package: mesa-dri# with full kms
dtoverlay=vc4-kms-v3d,cma-256# with fake kms
#dtoverlay=vc4-fkms-v3d,cma-256
I am using kms for GPU acceleration in general and I have the mesa-dri package installed. Note I also added cma-256, but that was only because of some sway bug IIRC, where it wouldn’t start because it thought the gpu only had 128mb or something like that (although the vgpu is self-adjustable IIRC again). You can try the cma-256, or leave it out, it should be optional.
If you haven’t tried this, try it, then reboot your pi and see if you can make ffmpeg to do it. I’m not sure if the pi gpu can do 4K, it can barely play some highly compressed, low frame rate 1080p youtube and it struggles with high bitrate 720p. Sometimes I get better luck playing videos in firefox than mpv, which is weird. I’m not sure how encoding / decoding would would with gpu acceleration.
I had that container running, but don’t remember doing any configuration on the base OS to make it work. I should look into that for sure!
Surely will, thanks for the pointer! I’m not trying to play back the files locally so I think the container is trying make some transcoding on the fly to serve the video. I need to check if there’s a way to disable that aswell. It’s not like I’m watching my own videos like youtube, just trying to figure out if they’re corrupted or not.
@ThatGuyB
Ahh sorry… I read two different posts at the same time
Hardware acceleration is only used for decoding and encoding, not for muxing (what yt-dlp does).
Did it work out using a swap file?
Haven’t tried yet, lmao. I will either today or tomorrow. Thank you for your tips! 
Did you ever get around fixing it?
Sorry for taking my sweet time. I was just thinking today of doing it. Seeing your message made me jump on it.
Yes, just went with a swap file on emmc. That way I won’t be dependent on ZFS mounting the pools on boot and I can save some power from hdd spins (which is what I would have used for swap), or save some writes on the ssd pool.
Just finished adding the 2GB swap file in fstab now. I will give the rsync over nfs another go, hope it doesn’t break.
Given the lack of “noise” 
I guess its working now?
I haven’t really had the time to work on it. Yesterday I was trying to do some experiments and use nfs v3 and iscsi via wireguard on my LAN, just for lulz. But I didn’t set it up yet.
I had taken a second rootfs backup via rsync on nfs and it seemed fine (well, it was an incremental, but I also delete things that have been deleted from the source). About 200M of difference. I need to think of a backup strategy for this, but I haven’t put my mind to work. It’d probably be something like mount, rsync, unmount, snapshot, zfs send to the other pool.
You probably want to set CPUTYPE?= in /etc/make.conf and compile for your specific arch if you’re going to play around with crypto etc. I have latest patchversion of 13.1-RELEASE compiled and a curated ports collection if you’re interested.
I will look into it, I had other things I prioritized, I didn’t have time to work on it since. I remembered FreeBSD has a bunch of example confs, so I grabbed the make.conf. But neither the example, nor the man make.conf show anything ARM related, the example is pentium3.
Past week adventures.
Due to my stupidity, which I’m somewhat thankful for, weirdly enough, I moved my NAS full time from the Pi 2 with a USB HDD to the RockPro64 NAS, using NFS and the spinning rust pool. I’m surprised how quiet the seagate ironwolfs are during operation in the rkpr64 box, you can only hear them spin up on a random file access at times, usually videos, but after the initial scan, they are inaudible. I am in the same room with it, about 10 ft from it.
Ok, so what happened exactly? I moved my stuff around and redid my wiring, trying to make the power bar more portable with rubber reusable zip ties (and failed miserably). And in my massive geniusness, I forgot to plug my powered usb hub that powers the hdd. I rebooted the pi 2 a bunch of times, with and without the hdd plugged in. I thought a kernel update broke it, which was weird, because haven’t updated it in a while. I switched the hub to the pi 3, same thing. dmesg “spinning disk,” wutshappening.png. After replugging, some errors that the driver needed for it is not loaded or something.
I thought these were signs of failures or something (doesn’t help that I lightly dropped it on the table by mistake). I also saw some dmesg errors asking for a partition fsck, but that was likely because I powered off the pi so many times. I didn’t have the partition mounted (I couldn’t mount it), but it was still complaining.
I moved it to the rkpr64, but I could not figure out how to mount the bloody thing. mount /dev/da1p1 /mnt did not work (looking at geom part list it showed up as direct attached drive 1, which made sense, since the rest of the disks were serial ata drives). I gave up on that fast, I know freebsd supports ext4, but I was in a rush to recover data. It was unimportant data, just videos off the interwebs, but it would have been a shame to lose them and have to download them again (most of them I wouldn’t even know to redownload, it’s just my “watch later library” basically, after watching, I delete them anyway). No, I have not tried mount -t ext2fs /dev/da1p1 /mnt, not sure if the ext2fs applies to ext4, I only found out about this now, while casually reading the freebsd docs.
I plugged it to my pi 4 directly to a usb 3 port and lo and behold, it worked there without a fuss. I mounted it, left it overnight and woke up to a frozen pi. I was thinking 2 things, either the hdd hit a bad sector and it had some weird effect on the os, leading to a freeze, or the hdd wanted too much power, which the board could not supply. I also have a usb m.2 ssd connected to it. And it doesn’t help I’m using a 2.4A brick (it has 2 ports, a type-c and a type-a, but for some reason, the type-c does not power the pi at all, probably the big brain engineering at adafruit with their stupid usb power delivery hardware bug nonsense).
I had it set up like this before I used the pi 2 as the nas. With and without the usb hub, it was still hanging, so I left it to the power delivery (I also run a 2ghz oc). I plugged it to my TR straight to a USB 3 and it worked. Then I had some issues reading it, which I thought again that the drive might be it on its way out. My system froze when I was transferring files via rsync to nfs.
On reboot, my tr wouldn’t boot. I powered it off, unplugged the hdd, it took some time, but it started eventually. Plugged it back in and didn’t mount it, just did a dd imaging of it and sent it to the rkpr64 via nfs. Worked flawlessly. Then I tried mounting it again and continued the rsync that got stopped. It worked without a hitch this time. Thus my media library got moved to the HDD pool as of now.
Then I realized the usb hub was unplugged. You probably can’t imagine how much shame I felt during that moment, even though there was nobody, but myself watching.
I might be trying to move the scratch media back to the external hdd, only because I would like to save some write cycles on the 10tb drives. The usb one is 1tb and still has plenty of life left in it. I might just have to figure out how to mount it on freebsd. Maybe the usb 3 port on the rockpro64 didn’t provide enough power for it either, who knows. I trust pine64 to make a quality board as much as I trust adafruit, which is to say, not a lot. But they are cheap and I did kinda want to support them, had a pci-e connector and was supported by openbsd and freebsd, but I’m still leaning towards odroids in general.
And if freebsd can’t mount the drive as ext4, then fine, I’ll just format it to ufs, or maybe exfat for compatibility with other systems. Anything that requires to be kept and have its data integrity intact will be moved to the zfs pool. What I need now is to get a test sata drive or 2 and start getting the hc4 to work as a backup server.
As for the RockPro64 serving as a NFS server for 1 and a half client (my pi 4 and the tr whenever I power it on and do some testing), it has been rock solid, 10/10. I’m planning to implement nfs and iscsi over wireguard, hopefully the cpu can keep up with 3 or 4 clients. Just want to see how a trustless network in a diy way would look like. Using nfs v4, kerberos and ipsec is probably going to be more than I’m willing to put on the poor rkpr64, I’d think wireguard being so lightweight has the potential to work decently here. With high powered pizza boxes, it can probably scale way better, but I’m into low power sbcs for a reason and not just because I’m a cheapskate. Ok, it’s because I’m a cheapskate, I don’t want my hobby to become too expensive and especially not too cluttered, I want to be able to move easily until I get my own house.
Sound like a fun week, there’s a fuse driver in e2fsprogs (sysutils/e2fsprogs-core) for ext2/3/4 which may be of interest although I haven’t tried it myself. IPSec is probably also fine as you have hw crypto accl on RK3399 which RPi lacks.