Building an arm64 OpenBSD router using RockPro64

blind_gh0st · October 18, 2020, 4:58pm

My current internet setup is using PfSense in a Proxmox VM with IOMMU passthrough to a four-port gigabit ethernet card. That was annoyingly difficult to setup and I am still having regular issues with the re driver support for FreeBSD. I regularly get re#: watchdog timeout which causes my network to go down. Luckily, my Ubiquiti switch still allows routing to the hypervisor when that happens so I can address the issue, but I’ve learned that a virtualized router is a bad idea and pairing that networking card with PfSense (or FreeBSD in general) is also a bad idea. Yay learning!

So instead I wanted to switch to a low power solution but keep my networking stack flexible and extensible. Looking through SoC solutions I found that most small network appliances are fairly expensive (I see this one from Amazon mentioned a lot) and I wanted to keep the price more reasonable. I found the RockPro64 had a x4 PCI-E slot and is officially supported by OpenBSD. The RockPro64’s PCI-E slot is meant to pair with Pine64’s SATA card (at least that’s the only use I’ve found documented) but I figured, “Why not just throw a NIC in there and create an OpenBSD router?”

Well, easier said than done. So far I’ve successfully:

Installed OpenBSD 6.7 to an SD Card
…and that’s it

I’ve run into several problems along the way so I though it might be helpful to share. I’ve tried three network cards. The RealTek I already owned was the first attempt, but for some reason it didn’t show up in OpenBSD. The PCI device was there, but it just appeared as a generic device and didn’t actually show any of the NICs. I figured “I guess RealTek just sucks. I’ll try Intel.” I ordered this card with an Intel I350-T2 chip (supported by OpenBSD), plugged it in, booted up and…

panic: uvm_fault failed

Dang it. I tried doing some debugging, but there isn’t much to do if the system won’t boot. So I just returned the card and figured I’d try a different chip. Next I tried this card with an Intel 82571GB (also officially supported) and…

panic: uvm_fault failed

Well dang it again. Anyway, I’ll be trying yet another card, this time trying a RealTek driver. I guess we’ll see what happens.

Has anyone else tried this? Any other suggested cards to speed up my search? Hopefully someone out there can learn from my failures thus far.

EDIT:
So I thought this would be helpful for anyone looking to do this for themselves. This thread doesn’t really have anything on “I have OpenBSD installed, what now?”

I wanted to plug this great website dedicated to keeping an updated guide on how to setup a basic OpenBSD router. It’s a great place to start.

imhigh.today · October 19, 2020, 3:29am

I’m going to go out on a limb here and say the problem isn’t actually with your cards, but the RockPro64’s PCI slot.

I have never, ever experienced an instance of hardware on OpenBSD’s supported hardware list that didn’t work well out of the box.

Unfortunately, I don’t have hardware on hand to test.

Installed OpenBSD 6.7 to an SD Card

OpenBSD 6.8 is out as of today.

https://www.openbsd.org/68.html

It includes fixes for crashes in the re driver, as well as better Realtek support for certain variations of the card. It might be worth downloading and flashing to see if the issues have been improved in the past six months.

blind_gh0st · October 19, 2020, 12:19pm

If that’s true that would suck as I think I’d have to buy a new RockPro64. I don’t think Pine64 allows returns (or returns after this long).

My guess was that it was maybe a power issue. The first NIC has four ports and wasn’t recognized at all, the next two had two ports and caused a crash, so I thought I’d try this single port. I also saw updates to rge(4) so figured I could give that a shot. If this doesn’t work it’s likely the PCI-E slot and I’ll need to see about getting a new board.

blind_gh0st · October 26, 2020, 11:46pm

I got the rge(4) interface to be recognized by OpenBSD 6.8! I used the Rosewill 20001 2.5Gbit NIC and the card showed up during the OpenBSD 6.8 install as rge0 ready to be configured. I wish I could give some “techy” explanation for what I did to get it to work, but I didn’t do anything. It just worked this time. If I had to guess, I would say possibly power draw of multiple interfaces on a single card. If anyone has a more educated guess I’m sure those reading this thread would appreciate it.

So this is a bit of a bummer because I’ll only be able to run a single subnet (since I only have two interfaces total) but I’ll see if I can’t make do. Next steps will be configuration, installation, and benchmarks. I will return with a well documented network diagram, benchmarks, and my anecdotal experience for those interested. If someone else tries this and is able to get a NIC with multiple interfaces running let me know, but for the moment “bird in the hand”.

Mastic_Warrior · October 27, 2020, 6:32am

Nice, I would like to use an ARM device with PCIe slot to run as a 10Gbit fiber router but I would rather use BSD’s network stack for better control and throughput. The list of supported boards for BSD with PCIe is slim.

blind_gh0st · November 15, 2020, 7:35pm

Sorry for the hiatus everyone. I have been very caught up in work and life and wasn’t able to get around to setting everything up. I’m back now and I have successfully installed the RockPro64 OpenBSD router for my home network. I promised some benchmarks and review and so I shall deliver.

Currently my router is SUPER basic following (more or less) OpenBSD’s official guide to creating a router. Currently my network is just a star, no VLANs or subnets (yet) just so I could get the basics working. I won’t post a network diagram because I’m pretty sure everyone knows what a star is. I did not configure WiFi because I already have UniFi APs handling that.

On to the benchmarks! Disclaimer, I’m not a professional benchmark-er so I just ran some basic tests that anyone can run at home. It’s mostly download tests because I don’t really have anywhere I upload things so take what you will from this.

Steam

Average: 75 MB/s Peak: 79 MB/s

I figured a good real world test would be to download a game off Steam. Bioshock Infinite seemed beefy enough to get a consistent reading of bandwidth and it was steady around 75MB/s which is excellent in my opinion. There were no sudden drops or disconnects, just a smooth and steady download.

SpeedTest.net

Average: 645.28 Mb/s Peak: 650 Mb/s

For a more traditional test I went with SpeedTest.net. Again, very steady performance and consistent with what I was seeing in the Steam downloads.

Fast.com

Average: 660Mb/s Peak: 660Mb/s

For even more consistency I tested using Netflix’s Fast.com. I was rather surprised running this as it hit 660Mb/s and locked there for the remainder of the test. I ran it a couple more times and it stayed right at 660Mb/s so I suppose that’s that.

HTTP Download

Average: 30MB/s Peak: 35MB/s

Screenshot 2020-11-15 140923
SpeedTest and Fast are cool, but I wanted some more real world tests. I often times need to download ISOs for various OSs so I figured that might also be a good test. I chose to download Ubuntu since it’s one of the more popular distro’s out there and is large enough to get a good reading on download speed. Most of the download hovered between 28-31MB/s with a few times peaking around 35-36MB/s. Not as fast as the above speed tests suggest, but reasonable as I’m sure Ubuntu’s servers are always under load.

Torrent

Average: 32MB/s Peak: 36MB/s

My last test was running a torrent to try and get a read on download speeds. My (fully legal) choice for this was (again) the Ubuntu ISO. It is common enough to have plenty of seeds to get a reading. The torrent seemed to go slightly faster than the HTTP download (expected) averaging around 32MB/s but peaking around 36MB/s.

Conclusion

I’m not Level1Techs or LTT, but hopefully this thread gives people enough information about running an OpenBSD router on a RockPro64. Hopefully others have better luck than I finding a good NIC to use, but OpenBSD is a solid OS and is fairly easy to configure once you learn the syntax. If anyone has any questions on my setup or just how to set up and OpenBSD router in general let me know!

ThatGuyB · October 28, 2021, 12:47am

I just became a Necromancer

I also intend to build an arm64 OpenBSD router with the RockPro64, so I guess it’s worth a shot reviving this thread. How has it been thus far (if you are still using it)? @blind_gh0st

At first, I was thinking of getting a quad port Intel NIC, but then I realized I could just get a 10 Gbps NIC and do VLANs on it, just have to connect it to a switch with at least a 10 G uplink port (I will need a switch with around 20x 1 Gbps ports). But seeing issues with booting when using different cards seems a bit scary. Well, considering the OpenBSD support for the RockPro64 increased massively in the past year, now with OpenBSD 7 already released, I think it’s worth a shot trying (worst case scenario, I’ll just repurpose it for something else, as I’ll probably order quite a few RockPro64 for my future small cluster).

However, now I don’t know if I want to go 2x 2.5 Gbps or single 10 Gbps. I guess pricing of the switch and NIC will be the ultimate factor, but I wish I could know beforehand which card is working on the RockPro64 before ordering stuff.

Have you tried pushing the RockPro64 harder? Like, multiple VLANs on that NIC, running DNS, DHCP and NTP servers and maybe a VPN client or VPN server.

Mastic_Warrior · October 28, 2021, 2:46pm

Usually for a minimal price increase, you can get a dual port Intel Fiber NIC. The Intel kits are usually open source drivered and have best compatibility among the unices. This will give you the ability to setup redundant paths.

Unfortunately, I have never use the RockPro kits and I only am familiar with the Amlogic stuff and RaspberryPi stuff. The VIM4 looks promising. I am hoping an OdroidN comes out with the new Amlogic chip.

ucav117 · October 28, 2021, 3:49pm

Check this out @ThatGuyB

ThatGuyB · October 28, 2021, 4:59pm

Not interested in kickstarter projects…

But if I wanted a high-end router and for some reason did not want to stick with x86, I’d probably go with a Honeycomb LX2K for the 4x SFP+ ports.

Any experience with Odroid-HC4? Or any idea about Amlogic S905X3 support in the Linux kernel? I’d like to experiment with toaster NAS boxes in my future homelab (for Ceph).

ucav117 · October 28, 2021, 5:26pm

$750

whew

That Honeycomb is cool but dayum they are proud of it.

ThatGuyB · October 28, 2021, 9:08pm

Yeah, it’s an ARM workstation (16 ARM A72 cores) in a Mini-ITX form factor and comes with UEFI, so you can boot any random arm linux image (AFAIK). The platform it uses is based on a network chip (NXP LX2160A SoC). It supports up to 64GB of RAM, has m.2, sata and PCI-E. Supposedly a system with it can get as low as 15w at idle (with a GPU, m.2 and 2 fans).

It is juicy, but I kinda want one, just to flex (and use as a web browsing and email desktop). Might also be a potential replacement for a whole low-powered cluster (with obviously a SPOF), but I kinda want to spread workloads around and also manage network traffic, as opposed to just managing a single beefy server.

blind_gh0st · October 29, 2021, 2:55pm

Hey I’m here and still using my OpenBSD router! I have upgraded to OpenBSD 7.0 and unfortunately the driver support has not gotten better with the cards I have.

I have tried doing VLANs but it seems there is a driver problem with rge. Software-wise OpenBSD reports packets to be tagged, however the driver enables hardware VLAN tagging and I’ve found the hardware doesn’t actually tag the traffic. OpenBSD receives VLAN traffic just fine, but it cannot send it with that driver. Unfortunately I don’t know enough about the hardware layer to debug that issue. I confirmed that behavior by connecting a Mac directly to the NIC and running tcpdump.

The other NICs still have the behavior of either crashing the OS or the driver not loading properly. I have opened a couple bug email threads here and here but devs haven’t been very communicative. If anyone here has experience debugging hardware I’d be happy to help! Any further information gathered would likely be very helpful for the OpenBSD team.

ThatGuyB · October 29, 2021, 3:07pm

That’s a little disappointing. I guess I’ll have to use other options at my disposal, either run an ARM Linux router/firewall, or an x86 BSD router/firewall. I have experience with Linux firewalls and networking, but I wanted to get into BSD on ARM.

@oO.o do you have any experience with debugging hardware related issues on OpenBSD? Or know someone on the forum who does and is willing to help?

oO.o · October 29, 2021, 6:16pm

Not like this. If it’s a problem with the driver communicating with the NIC, then it’s just a bug and you’d need to wait for a fix, unless there’s a sysctl setting that can turn off vlan hardware offload.

From my experience with openbsd, my guess is they think something about the hardware implementation is flawed and won’t write a workaround into the driver because they just refuse to support hardware that doesn’t operate within their expectations. More generally, I’ve seen anecdotal advice to try to stick with Intel NICs as much as possible. I am running one re interface on my home router that hasn’t been a problem yet, but otherwise everything I use is em or ix.

ulzeraj · October 29, 2021, 9:17pm

OpenBSD devs are just being OpenBSD devs. They ship their product to run on proper computers (those are not my words btw) and to be honest Rockchip hardware is a bit sketchy. Good thing that FreeBSD is making an effort to treat Aarch64 as a first class citizen.

I’m using the RockPro64 with FreeBSD 13-RELEASE. Works nicely except for the ASMedia SATA controller which I had to force on SATA 2 mode. Sad but more than enough for my use case. Not a NAS but a software plus busy database. I’ve been running jails on top of a ZFS mirror of 2 SATA SSDs with encryption and ZSTD compression. The aforementioned problem with the SATA controller caused me some headaches related to data corruption. I’m still undecided if I like the board or not. Its the one ARM SBC that is more expandable and includes support for AES unlike the Pi4.

Have you ever considered trying the NanoPi R4S? It is based on the same chip as the Rockpro64 and has 2 Gigabit ethernet ports one being from the SoC and the other using PCI-E. Might have better luck with people trying to build OpenBSD specifically for that hardware.

ThatGuyB · October 29, 2021, 10:07pm

Doesn’t sound like it fits my use case, I need at least 3 LANs and a WAN (and I’ll probably end up with a lot more if I get vLAN support). That’s how I set all my home networks. I could maybe get away with 2 Gbps ports, but I want to do some more bandwidth consuming inter-vlan routing (storage on separate VLAN, but since the SBC I will use will only have 1 eth port, I’ll route everything through the router, it’ll be a fun experiment for me - in a data center, I’d have a separate storage vlan with dedicated ethernet ports to it, for speed and to avoid passing traffic through the router). I don’t know, I might or might not get away with 2 ports, but I’d rather not take my chances.

Awesome!

The RockPro64 seems to be the most extensible, with a close second being the Rock Pi 4 (not a Pine64 SBC), which still has the RK339, but has an m.2 port, which would require fancy adapters and maybe custom cases. The RPi CM4 would probably be way more expensive after I get all the add-ons for it (carrier board and stuff) and I doubt OpenBSD would behave much different on a RPi than on RockPro64.

If the attempt to run the RockPro64 as a router will fail, I’ll probably go with the already battle tested Asrock J3455M or a similar / newer board.

Not sure how to find cards that use em or ix drivers. On my old pfSense box (the aforementioned J3455M), I’ve been using re (onboard NIC) for WAN and 3x igb (from a 4x 1Gbps HP Intel NIC) for LAN. Had no issues with either, but obviously, that is x86. I don’t remember when I built it, probably in 2017 or 2018 and has been working since.

oO.o · October 29, 2021, 10:28pm

igb is FreeBSD iirc? I don’t see it in openbsd.

ThatGuyB · October 29, 2021, 10:34pm

Probably, yeah, since pfSense is based on FreeBSD.

Mastic_Warrior · November 1, 2021, 11:49am

I use the Odroid-N2 and Odroid-N2+. The 905 is a stripped down version of the 922 (311) that is in the N2. Nice kit. The work in the mainline kernel is all there and the GPU is fully supported as well.