Building an arm64 OpenBSD router using RockPro64

Alright, so from the looks of it, I’ll have to order a RockPro64, an Odroid N2+, a 12 port switch (8 gigabit + potentially 4 sfp+ and ethernet modules) and preorder an Odroid HC4. And all their power adapters. And 2 WiFi USB cards. From the looks of it, it seems pretty reasonable, just like I want it to be.

Later down the line, I will probably buy a Khadas VIM4 when it gets released, I’m still impressed by the specs and my main desktop could use an upgrade.

I will connect them via ethernet.

Nah, that’s what the Odroid HC4s are for. :slight_smile:

Plenty!

hass.io being the home assistant alternative? Alright, if it runs Alpine, I guess it should be fine.

Just what I want.

Don’t really care about the HATs, I just want compute and network from the N2+ and network storage from HC4.

RK3399 has its longevity going for it, so it’s well supported and the RockPro64 should be supported by Pine64 until 2023 or so, which gives me enough time to buy the second board when I need it.

Jeff has been my inspiration when he was using CM3s, along with Novaspirit Tech and Techno Tim, arguably the Oracle devs who made a cluster of RPi 3s too, but that’s so insane that a 5 ARM servers would have made more sense.

Thanks everyone for the recommendation and for reading all that.

His channel is so slept on.

HASSIO, or what ever they are calling it now are the official image built by the HA team. Alpine is the OS, Home AssistantCore is the containerization release (docker image), and mainline kernel support with HA tweaks.

1 Like

Just ordered a RockPro64 4GB, Odroid N2+ 4GB, Pine64 RockPro64 12V/3A charger, Odroid HC4 12/4A charger and 2 WiFi AC USB 3 adapters. And pre-ordered Odroid HC4 base (no display), Odroid N2+ 12V/2A charger and a Odroid N2+ case. The guys over at ameridroid didn’t have those in stock.

Now I need an 8 port 1 Gbps and 4 port SFP+ switch, and SFP+ ethernet modules and a PCI-E dual 2.5 Gbps NIC. I read that the Intel i225 really sucks ass due to some hardware issues that makes it run at 1-8 Mbps at times.

I need recommendations for a PCI-E x1 dual 2.5 Gbps NIC that can run on OpenBSD.

Am I going insane, or are switch prices kinda high? And I can’t seem to find something that fits my needs! I need help.

The only thing I found that checks most of the boxes is a fs.com S3900-24T4S. 24 Gbps ports and 4 SFP+ ports, managed switch and it’s stackable for $769. But it’s not passively cooled.

There’s the MikroTik CRS326-24G-2S+IN, CSS326-24G-2S+RM or CRS326-24G-2S+RM that unfortunately only have 2 SFP+ ports, I don’t want to connect the switches with just 1 Gbps port, even though the price feels right at $200, $159 and $209 respectively, and are passively cooled. Having 24 ports, I could connect 1 VLAN per port and I slightly doubt I will need a lot of communication between switches (it’s more likely I’ll jump through the router), but I still want something decent. If I wasn’t going for switch redundancy, I would definitely get this one, but it’s not as compelling as the FS one.

And I can’t seem to find a 12-16 port 2.5 Gbps managed switch anywhere. Actually, I can’t find any 12 Ethernet port switches with 4 SFP+ ports.

Found a DLink DIS-300G-12SW, 8 gigabit ethernet, 4 sfp+… but it’s $2000, I ain’t paying that much on a switch.

Found Aruba (HPE)) 1950 12XGT 4SFP+ switch. 12x 10 GbE copper and 4 SFP+ ports… $1266. REEEEE

QNAP QSW-M408-4C looks like an ideal fit. 4-Port 10GbE SFP+/RJ45 Combo and 8-Port Gigabit, so I don’t even need to get SFP+ modules. $459, so decently priced. But not sure what “web managed” means. If it only has web management, I don’t want it.

Holy crap, that QNAP hits all the hardware boxes, I can’t believe it. Anyone has any experience with QSS? Can you manage it via SSH?

Almost all my tech buddies say universally burn QNAP and all its products and pieces of software with fire. If that’s much to go on I’d say its terrible

IMHO you should invest in a switch its fine. A solid switch can last a long long time dude. Divide its price over the months used. Is it really that bad after that?

1 Like

Stay away from Qnap

2 Likes

2 opinions to stay away from QNAP. Mkay. I mean, I knew their NAS servers were junk (both consumer and enterprise), but I wasn’t sure about switches.

@ucav117 I already saw that switch. It has lots of ports, but I need 4 SFP+, or even 2.5 Gbps RJ45. Detailed above, I need a switch with 8 to at most 12 gigabit ports and 3 to 4 2.5 Gbps ports (I don’t care if it’s SFP+ or RJ45, I’ll convert to copper anyway). If I can find a managed 12x 2.5 Gbps Ethernet switch, I’d probably buy that, to avoid all the other headaches.

The thing is that I want power efficiency more than I want expandability. I can buy a 24 or 48 port switch, but those will easily eat more than 80W. Also, I want my build to be reproducible and somewhat affordable, so that people can start small and upgrade later.

Found a Zyxel 12 port switch. XGS1210-12. I don’t shy away from Zyxel, we used them before. They have a tendency to have their ports die, but what can I say, 8 gigabit ports, 2 2.5 Gbps ports and 2 SFP+ ports and 5 year warranty.

For $209! That’s the best deal I could find yet, wew, and it’s not QNAP.

1 Like

That is a neat unit. Also I know you want ARM but check this guy out.

1 Like

Where did you find it for $209?

I know the GHF51, saw it on Novaspirit Tech (is that the video?), but I’m more into ARM. If I’m going to x86, I’ll probably get something with a lot of expansion, or a NUC-style PC.

Amazon (sold by Zyxel) and another website that I forgot.

About real time clock. I’ve bought Koszyk na baterie 2xAAA z wtykiem JST-PH Botland - Sklep dla robotyków in hopes it would work but the connector is slightly different. I wonder if its safe to adapt the one I got by cutting a piece of the plastic connector.

1 Like

The RTC on RockPro64 uses either 2x AAA or CR-2032.
https://wiki.pine64.org/wiki/ROCKPro64

To me, the AAA holder doesn’t look like it has any capacitor or resistor, at least on the surface.

Lithium CR2032s give 3V, AAA alkaline batteries give 1.5V (3V when in series).

I see no reason why modifying the battery pack holder you got would be unsafe, as long as the polarity is ok and you aren’t using unsafe batteries.

I just noticed OpenSSL and GNUTLS performance on FreeBSD is very disappointing on FreeBSD 13 running on a RockPro64.

Yesterday I’ve finished setting a Samba file server. Setting server signing to mandatory yields an atrocious 5MB/s transfer speed over gigabit network. Yikes. Although we are talking about Samba this will definitely affect VPN software like OpenVPN.

SCP which is normally slower than Samba can transfer to the same machine with 20MB/s. Still not good but its SSH. I’ve managed 70MB/s with unencrypted rsync protocol. The armv8crypto seems to be loaded into kernel not as a module but in kernel.

So my guess is that the software cryptographic libraries are not using ARMv8 crypto extensions?

Samba 4.13 uses AES-128-CCM for signing and transport. It also uses GNUTLS instead of OpenSSL.

# smbstatus
Samba version 4.13.17
PID     Username     Group        Machine                                   Protocol Version  Encryption           Signing              
----------------------------------------------------------------------------------------------------------------------------------------
85606   ulzeraj      unixadmins   REDACTED (ipv4:REDACTED:61683)    SMB3_11           AES-128-CCM          AES-128-CMAC         

Service      pid     Machine       Connected at                     Encryption   Signing     
---------------------------------------------------------------------------------------------
IPC$         85606   REDACTED  Fri Mar 11 08:57:38 2022 UTC     AES-128-CCM  AES-128-CMAC
tank         85606   REDACTED  Fri Mar 11 08:57:05 2022 UTC     AES-128-CCM  AES-128-CMAC

This script should work in every OS with OpenSSL and sh:

#!/bin/sh
for ALG in aes-128-ccm aes-128-gcm; do
    openssl speed -evp ${ALG} -bytes 1500 2> /dev/null | grep "^${ALG}"
done

Here are the RockPro64 results (also adding GCM for comparison):

aes-128-gcm      15808.85k
aes-128-ccm      13117.70k

I have a “replica” machine running literally the same software and Operating system versions but instead on amd64. Its an old APU2C2 with a AMD GX-412TC SOC and 2GB of RAM. This machine runs on a 2012 4 core embedded processor that scores worse than a Raspberry Pi 4 in raw power.

aes-128-gcm     319367.06k
aes-128-ccm     190703.62k

GASP is 25-ish times faster!!! OpenSSL runs on FreeBSD like crap! I should mention however that both systems are running encrypted AES256 disks with ZSTD compression and the RockPro64 outperforms the APU in every aspect.

It looks like there is something missing on both OpenSSL and GNUTLS for FreeBSD. I’m looking at the versions:

RockPro: 
OpenSSL 1.1.1k-freebsd  24 Aug 2021
gnutls-3.6.16
nettle-3.7.3
APU2C2: 
OpenSSL 1.1.1k-freebsd  24 Aug 2021
gnutls-3.6.16
nettle-3.7.3

Here are the results on various machines for comparison:

Pop_OS 21.10 Ryzen 9 5900HX:

aes-128-gcm    5137056.00k
aes-128-ccm    1612059.00k

Ubuntu 20.04 i7-9750H NUC

aes-128-gcm    4880986.00k
aes-128-ccm    1435525.00k

macOS Monterey M1:
This is OpenSSL on ARM. GCM is faster on M1 while CCM is slower than the AMD64 machines. Had to slightly modify the script because Mac OpenSSL sends stdout to stderr.

AES-128-GCM    5901342.64k
AES-128-CCM    1044495.00k

I will try to compile the OpenSSL from ports to check if it makes a difference. For samba however its a hopeless case for now since the latest version on pkg and ports doesn’t seem to support those features.

EDIT: OpenSSL from pkg has KTLS support:

aes-128-ccm     108811.86k
aes-128-gcm     246024.50k

Still kinda left behind on CCM but better on GCM. Considering the M1 results is ARM bad at AES-128-CCM?

EDIT2: Just discovered gnutls-cli --benchmark-tls-ciphers. There results are consistent with my transfer speeds:

APU2C2:

# gnutls-cli --benchmark-tls-ciphers
Testing throughput in cipher/MAC combinations (payload: 1400 bytes)
                   AES-128-GCM - TLS1.2  74.29 MB/sec
                   AES-128-GCM - TLS1.3  67.16 MB/sec
                   AES-128-CCM - TLS1.2  29.43 MB/sec
                   AES-128-CCM - TLS1.3  28.55 MB/sec
             CHACHA20-POLY1305 - TLS1.2  23.21 MB/sec
             CHACHA20-POLY1305 - TLS1.3  21.89 MB/sec
                   AES-128-CBC - TLS1.0  20.25 MB/sec
              CAMELLIA-128-CBC - TLS1.0  8.69 MB/sec
           GOST28147-TC26Z-CNT - TLS1.2  3.79 MB/sec

Testing throughput in cipher/MAC combinations (payload: 16384 bytes)
                   AES-128-GCM - TLS1.2  131.10 MB/sec
                   AES-128-GCM - TLS1.3  127.85 MB/sec
                   AES-128-CCM - TLS1.2  36.79 MB/sec
                   AES-128-CCM - TLS1.3  35.60 MB/sec
             CHACHA20-POLY1305 - TLS1.2  27.30 MB/sec
             CHACHA20-POLY1305 - TLS1.3  26.94 MB/sec
                   AES-128-CBC - TLS1.0  31.21 MB/sec
              CAMELLIA-128-CBC - TLS1.0  11.26 MB/sec
           GOST28147-TC26Z-CNT - TLS1.2  4.01 MB/sec

RockPro64:

# gnutls-cli --benchmark-tls-ciphers
Testing throughput in cipher/MAC combinations (payload: 1400 bytes)
                   AES-128-GCM - TLS1.2  6.67 MB/sec
                   AES-128-GCM - TLS1.3  7.91 MB/sec
                   AES-128-CCM - TLS1.2  6.12 MB/sec
                   AES-128-CCM - TLS1.3  5.77 MB/sec
             CHACHA20-POLY1305 - TLS1.2  14.24 MB/sec
             CHACHA20-POLY1305 - TLS1.3  14.29 MB/sec
                   AES-128-CBC - TLS1.0  7.76 MB/sec
              CAMELLIA-128-CBC - TLS1.0  6.59 MB/sec
           GOST28147-TC26Z-CNT - TLS1.2  3.18 MB/sec

Testing throughput in cipher/MAC combinations (payload: 16384 bytes)
                   AES-128-GCM - TLS1.2  7.08 MB/sec
                   AES-128-GCM - TLS1.3  8.36 MB/sec
                   AES-128-CCM - TLS1.2  5.64 MB/sec
                   AES-128-CCM - TLS1.3  5.98 MB/sec
             CHACHA20-POLY1305 - TLS1.2  15.79 MB/sec
             CHACHA20-POLY1305 - TLS1.3  15.59 MB/sec
                   AES-128-CBC - TLS1.0  8.30 MB/sec
              CAMELLIA-128-CBC - TLS1.0  6.95 MB/sec
           GOST28147-TC26Z-CNT - TLS1.2  3.28 MB/sec

Meanwhile, can someone check my benchmark script on a RockPro running Linux or OpenBSD? If Also gnutls-cli --benchmark-tls-ciphers if you already have it installed. Thanks.

2 Likes

I only have Raspberry Pis and AmLogic boards. But this is interesting.

1 Like

I would be interested to know how these perform on other OS and boards. If any of this has proved anything is that hardware means nothing if the support isn’t there.

I ran a few more tests:

FreeBSD VM on the M1:

OpenSSL from base:

aes-128-ccm     169061.26k
aes-128-gcm     157656.00k

OpenSSL from pkg:

aes-128-ccm    1011978.00k
aes-128-gcm    4268362.86k

GNUTLS benchmark:

gnutls-cli --benchmark-tls-ciphers
Testing throughput in cipher/MAC combinations (payload: 1400 bytes)
                   AES-128-GCM - TLS1.2  87.56 MB/sec
                   AES-128-GCM - TLS1.3  87.37 MB/sec
                   AES-128-CCM - TLS1.2  75.69 MB/sec
                   AES-128-CCM - TLS1.3  75.71 MB/sec
             CHACHA20-POLY1305 - TLS1.2  172.27 MB/sec
             CHACHA20-POLY1305 - TLS1.3  171.13 MB/sec
                   AES-128-CBC - TLS1.0  96.21 MB/sec
              CAMELLIA-128-CBC - TLS1.0  59.41 MB/sec
           GOST28147-TC26Z-CNT - TLS1.2  23.07 MB/sec

Testing throughput in cipher/MAC combinations (payload: 16384 bytes)
                   AES-128-GCM - TLS1.2  90.89 MB/sec
                   AES-128-GCM - TLS1.3  90.77 MB/sec
                   AES-128-CCM - TLS1.2  78.60 MB/sec
                   AES-128-CCM - TLS1.3  78.57 MB/sec
             CHACHA20-POLY1305 - TLS1.2  186.75 MB/sec
             CHACHA20-POLY1305 - TLS1.3  185.87 MB/sec
                   AES-128-CBC - TLS1.0  103.75 MB/sec
              CAMELLIA-128-CBC - TLS1.0  62.15 MB/sec
           GOST28147-TC26Z-CNT - TLS1.2  23.45 MB/sec

Arch Linux ARM VM on the M1:

OpenSSL:

aes-128-ccm     920152.00k
aes-128-gcm    4123021.91k

GNUTLS benchmark:

gnutls-cli --benchmark-tls-ciphers
Testing throughput in cipher/MAC combinations (payload: 1400 bytes)
                   AES-128-GCM - TLS1.2  0.72 GB/sec
                   AES-128-GCM - TLS1.3  0.70 GB/sec
                   AES-128-CCM - TLS1.2  0.35 GB/sec
                   AES-128-CCM - TLS1.3  0.34 GB/sec
             CHACHA20-POLY1305 - TLS1.2  178.56 MB/sec
             CHACHA20-POLY1305 - TLS1.3  177.00 MB/sec
                   AES-128-CBC - TLS1.0  0.36 GB/sec
              CAMELLIA-128-CBC - TLS1.0  63.72 MB/sec
           GOST28147-TC26Z-CNT - TLS1.2  23.59 MB/sec

Testing throughput in cipher/MAC combinations (payload: 16384 bytes)
                   AES-128-GCM - TLS1.2  0.87 GB/sec
                   AES-128-GCM - TLS1.3  0.87 GB/sec
                   AES-128-CCM - TLS1.2  0.38 GB/sec
                   AES-128-CCM - TLS1.3  0.37 GB/sec
             CHACHA20-POLY1305 - TLS1.2  193.52 MB/sec
             CHACHA20-POLY1305 - TLS1.3  192.91 MB/sec
                   AES-128-CBC - TLS1.0  0.56 GB/sec
              CAMELLIA-128-CBC - TLS1.0  65.77 MB/sec
           GOST28147-TC26Z-CNT - TLS1.2  23.98 MB/sec
[root@alarm ~]# gnutls-cli --version
gnutls-cli 3.6.12

It definitely looks like the ARMv8 AES extensions are not being used by GNUTLS on FreeBSD and there is no option to set those on Nettle or GNUTLS.

1 Like

Possibly they were not compiled in the binary and you need to roll our own? I would imagine that the support should be standard across the aarch64 chips if they have the extensions. I know the RPis do not but the Amlogic chips do.

I will give it a look see when I get home.

1 Like

> be me
> new to ARM world other than RPi
> buy a RockPro64
> download OpenBSD
> dd onto an SD card
> plug SD in RockPro64
> plug power in
> see red LED turn on
> no display out
> “why is it not working?”
> rewrite the SD card
> give up
> use the SD card to test something else
> 3 days later
> take a glance at the rkpr64 on the table
> see 2 buttons
> take a very close look
> “Power Button”
> open the pine64 wiki on rkpr64
> “IT HAS A FLIPPIN’ POWER BUTTON!”
> feel ashamed for not RTFM
> feelsbadman.png

Unbelievable…

2 Likes

That’s seems to be the case. According to my thread on the FreeBSD Forums ARMv8 features are often not detected by default and most of the time relies on setting CPU flags like CPUTYPE?=cortex-a53+crc+crypto during compilation time.

Having said that recompiling GNUTLS and Nettle with those flags didn’t made a difference so I’m building a custom kernel and world image with -mcpu=cortex-a53+crc+crypto and hopefully I can unlock the accelerated crypto extensions. Thankfully I can compile everything from the M1 without having to worry about dreadful CPU speeds on the RockPro64 or the hassle of cross compiling.

Meanwhile someone suggested me this:

GNUTLS_CPUID_OVERRIDE=0x24 gnutls-cli --benchmark-tls-ciphers
Testing throughput in cipher/MAC combinations (payload: 1400 bytes)
                   AES-128-GCM - TLS1.2  35.37 MB/sec
                   AES-128-GCM - TLS1.3  26.27 MB/sec
                   AES-128-CCM - TLS1.2  19.80 MB/sec
                   AES-128-CCM - TLS1.3  20.49 MB/sec
             CHACHA20-POLY1305 - TLS1.2  14.79 MB/sec
             CHACHA20-POLY1305 - TLS1.3  14.51 MB/sec
                   AES-128-CBC - TLS1.0  19.59 MB/sec
              CAMELLIA-128-CBC - TLS1.0  7.50 MB/sec
           GOST28147-TC26Z-CNT - TLS1.2  3.28 MB/sec

Testing throughput in cipher/MAC combinations (payload: 16384 bytes)
                   AES-128-GCM - TLS1.2  42.84 MB/sec
                   AES-128-GCM - TLS1.3  43.46 MB/sec
                   AES-128-CCM - TLS1.2  25.65 MB/sec
                   AES-128-CCM - TLS1.3  34.18 MB/sec
             CHACHA20-POLY1305 - TLS1.2  16.37 MB/sec
             CHACHA20-POLY1305 - TLS1.3  16.18 MB/sec
                   AES-128-CBC - TLS1.0  25.10 MB/sec
              CAMELLIA-128-CBC - TLS1.0  7.14 MB/sec
           GOST28147-TC26Z-CNT - TLS1.2  3.31 MB/sec

Setting this CPUID_OVERRIDE flag produced way better results! I suspect these results are as far as the RockPro64 can go in encrypted I/O data and that’s good enough to me. I don’t need to encrypt every share.

I’m really curious to see OpenBSD results.

2 Likes