Advice for a Linux-based router, beyond 10g

For a number of years, I’ve been using a router based on CentOS 7 to separate most wired devices on my network from all of the wireless devices. The router connected to my cable modem was an UnTangle instance running on Qotom hardware, to which I connected to wireless access points, a switch for devices such as streaming boxes, blu-ray players, televisions and such, and my CentOS 7 computer. The hardware is fairly old, but I keep it around because it supports a decent number of PCIe lanes and would be fairly expensive to replace. I don’t get great speed out of it. Even if copying from NVMe to NVMe, I normally don’t get any better than 5Gbps. An iperf3 test shows my bandwidth is about 28Gbps. Still, it’s better than gigabit, but I may have been better off just sticking to 10Gbps direct-attach-copper. Behind it, I’m using two desktop PCs and two UnRaid servers, each with a 40Gbps link.

MSI X99S Gaming 7
Intel® Core™ i7-7820X (40 PCIe lanes)
16GB of memory (currently)
OS on a 2565GB Samsung NVMe (soon switching to sata, for reasons)
Two dual-port 40Gbps Mellanox ConnectX-3 cards
One 4-port Intel gigabit card
Soon adding a dual-port 10Gbps Mellanox ConnectX-3 card

My problems started when I switched from cable to fiber. After the switch, replacing my Qotom hardware with a Calix Linspire U6 from Allo, I lost the ability to SSH into my DNS servers that are on the WAN side of my CentOS router. Nothing changed on CentOS at all, so there must be something different about the Calix security settings that are preventing the connectivity. I had to set up port forwarding in the CentOS firewall to allow me to access the web interface of the Calix router and likely have to do the same thing to be able to get to my DNS servers and services, but whatever.

My next challenge was from trying to add a dual port 2.5G Intel network adapter in the CentOS router. While the hardware is detected, the OS doesn’t see it as a network interface. From my understanding, it’s a kernel limitation, and yeah, I need to update to a newer OS anyway. That’s where I start to question things. If I use an NVMe SSD on the motherboard, I lose access to the lowest x16 slot on the motherboard because the two devices use the same lanes, so I took the existing NVMe out and put in a SATA M.2 and performed an install of Oracle Linux 9. That process kind of worked, but has been pretty painful so far. The OS doesn’t use any of the configuration files in /etc/sysconfig/network-scripts, an instead seems to do everything through NetworkManager, which seems odd because when I kickstart an OL9 server at work, it seems to use the configuration files without much fuss, but I’m not typically doing anything as complicated as routing for my job. That computer has access to the Internet just fine and connectivity between the bridged devices on the LAN side also work fine, but I couldn’t get it to route properly. Forwarding is enabled, but I don’t know if I’m missing something in firewalld to make it work or if I’m missing a route to make it work properly, or maybe dnsmasq needs to be set up to get things to work properly. After several hours of mucking around, I decided it was enough for the weekend and put the old NVMe back in to get back to a known stable condition.

Has pfSense or OPNsense matured enough in the last few years to support 40gbps ConnectX-3 cards? Would I get better throughput using one of those two, or should I try using UnTangle again? I looked for some tutorials but didn’t find anything that was quite what I was looking for. I’m sure that CentOS or Oracle Linux aren’t the best choices for handling router duties but I’m not sure what the best direction would be. I could also look at a network device that would handle these tasks, but without switching out all of my cards to something else, I’d likely be looking at $650+ for a device that would handle at least four qsfp+ connections.

Welcome to the forum!

Typically I advise people to keep using what they are used to and this is no different. I’d probably go with Rocky Linux 9 though, but I can see the appeal in Oracle Linux because of UEK, which is typically a newer kernel.

I think FreeBSD should probably work. It does have the mlx4en driver for ConnectX-3 100G and 40G.

Can’t speak for pfSense / OPNsense, you’d have to test it.

I had that happen to me with CentOS 8 (RIP) and it was just as easy as disabling it and using the option that the sysconfig parts should not be managed by NM. I see no reason why this would change in RHEL 9.

Read Hardware recommendations for 40gb internet, 100gb lan | Netgate Forum
You might get away by using ipfw(2) in FreeBSD but you’ll need to do manual tuning.

1 Like

I vote for VyOs
It’s Linux based, it has specific router capabilities baked into the standard image, it handles backups/upgrades router style (with a config file) but it requires learning its own configuration language Cisco IOS like) as opposed to having to deal with the raw linux package config files

As for OEL9 and network config files, you want to read this:

and unlike RHEL8 you cannot revert back to the traditional style of config
(Why are scripts under /etc/sysconfig/network-scripts directory are gone on rhel8/rhel9? - Red Hat Customer Portal)
posted Verbatim as it is behind registration/paywall

The gist of it is:

On RHEL9, the network-scripts package has been removed. 
Use NetworkManager to configure connections.

Why are scripts under /etc/sysconfig/network-scripts directory are gone on rhel8/rhel9?
 SOLUTION VERIFIED - Updated April 28 2023 at 11:55 AM - English 
Red Hat Enterprise Linux (RHEL) 8 and 9
NetworkManager, version 1.22.8 and later
ifcfg-rh scripts
Why are scripts under /etc/sysconfig/network-scripts directory gone in rhel8/9?
How to install scripts under /etc/sysconfig/network-scripts directory in rhel8/9?
Why are files in /etc/sysconfig/network-scripts ignored? Where are interface configs now?
My directory /etc/sysconfig/network-scripts is empty, but apparently from somewhere my system is picking up network interface configs?
I changed a config in an ifcfg-file in directory /etc/sysconfig/network-scripts, why are these changes apparently ignored by the system?
On both RHEL8 and RHEL9, as best practice, each interface should just get a config file in either directory, so either in /etc/NetworkManager/system-connections or in /etc/sysconfig/network-scripts. The first of these 2 directories, so the newer one, should be preferred.

For easier management, it is recommended to not spread config files over both of these directories, but only use system-connections.

Only on RHEL8, if the /etc/sysconfig/network-scripts scripts are required, the installation of the deprecated network scripts in the system is still possible with the following command:

# yum install network-scripts
With that package installed, the ifup and ifdown scripts link to the installed legacy network scripts. Calling the legacy network scripts shows a warning about their deprecation.

On RHEL9, the network-scripts package has been removed. Use NetworkManager to configure connections.

Root Cause
As of RHEL8, the network scripts are deprecated are no longer provided by default. On RHEL8 and later, the default installation provides a new version of theifup and ifdown scripts which call the NetworkManager service through the nmcli tool. On RHEL8, to run the ifup and ifdown scripts, NetworkManager must be running. Note that custom commands in /sbin/ifup-local,ifdown-pre-local and ifdown-local scripts are not executed.

With RHEL9, as per release notes, the /etc/sysconfig/network-scripts, aka ifcfg-rh, is no longer the primary source for network interfaces.

[root@rhel9a ~]# NetworkManager --print-config
# NetworkManager configuration: /etc/NetworkManager/NetworkManager.conf

# plugins=keyfile,ifcfg-rh
[root@rhel9a ~]#
With these defaults, NetworkManager is

First looking for files in /etc/NetworkManager/system-connections, these should be in keyfile format
Then looking for files in /etc/sysconfig/network-scripts, in the classic ifcfg-rh format
When files for an interface get created in both of these directories, then /etc/NetworkManager/system-connections gets priority and is used. Thus, changes in the interfaces config file in the other directory will not be picked up.

A simple keyfile might look like this:

id=Main eth0


Changes to keyfiles can be notified to NetworkManager using nmcli con reload.

Routing performance is probably similar to what FreeBSD achieves, what kills performance is firewall (packet filtering).

your cpu does old pcie 3.0.(3,9 GByte/s) 4.0 (7,8 GByte/s)doubles it. those nvme seems to be bored? when ive put a m2/sata adapter into my rig, the interface named changed to enp4s0 instead of enp3s0, networkctl didnt seemed to manage it, it messed alot. e.g. setting link local und ula to wrong ipv6-identifier etc.
what ive found out, iproute2 is the underlying stuff when systemd doenst do its job.
/etc/network/interfaces(.d) took the job and systemctl restart network-manager on my debian 12 system.

its a mess that u need to go to xeons/threadripper to get enough lanes as u can wish.

I have no complaints using Arch Linux for my router and NAS boxes. No unnecessary firewalld/NetworkManager layers on top of netfilter/iproute2. My ConnectX-4/5/6’s work fine, SR-IOV too

Not part of the issues I’m having with my CentOS router, but I ordered a Mikrotik CSS326-24G-2S+RM to replace an aging D-Link DGS-1210-28, into which my access points and wired data-consumption devices are connected. MikroTick has a switch/router device that has four qsfp28 ports that can be used as-is, or broken down to as many as sixteen sfp28 interfaces. I suspect that going with a device like that would give me better performance than what I’m trying to do with CentOS/Oracle Linux.

+1 for vyOS, they do a phenomenal job! Also I advise against the BSDs for that kinds of speeds. Even when bridging, Linux bridges burn less cpu cycles and are faster. Also the firewall(s) in BSD are now (much) slower (in some circumstances) than netfilter in recent Linux kernels. Linux just has much more time and money invested.

My RHEL9 ARM based router (Mochabin) works great btw. I’ve switched from NetworkManager to systemd-networkd and could not be happier. It’s great. I need it of v6 PD and it’s just much easier to handle. It can even create WireGuard tunnels, enable masquerading for certain interfaces and act as a DHCP server for both v4 and v6. And for the firewalling I used nftables directly. It’s much easier to read than plain IPtables. It really doesn’t take much work to get into.

I also don’t want to advise against Rocky Linux but the project (or it’s leadership) has done some really crappy marketing and payed for advertisements on Google for Alma Linux searches. That’s just bad. They also declined interviews and coverage on some Linux podcast… The alma Linux team on the other hand was very active in promoting and explaining their organization. They also kick out releases faster. :wink: Oh and yes they even had some nice words to say about Rocky even though they didn’t have to. Don’t know why they have such a big mind share…

Do you have any actual numbers backing up your claims? Last time (years ago) I looked netfilter was very slow and nftables peformed even worse. I’m a bit curious about your claims in general as netfilter is more or less deprecated everywhere infavour of nftables.

Benchmarking nftables | Red Hat Developer (old)

Looking at this I’d say that at least FreeBSD appears to perform pretty good.

Off-topic rant

Wow, that feels like such a poor decision from user perspective. I understand why as a business RHEL would deprecate it altogether, since they don’t want to support something old, but I find that ridiculous (they could just not support it and tell people to use NM instead of ifcfg files).

I’m certainly glad I was forced to switch over to using ip scripts for initializing the system. ip is present in all linux system, so basically universal (unlike NM). I say forced, because I was lacking /etc/network/interfaces ala Debian or Alpine, netplan ala Ubuntu, nor did I have access to /etc/sysconfig/network-scripts like RHEL systems. And I sure as heck did not want to use NM or ConnMan for a static IP configuration that (almost) never changes.

So ip it was. Was hard to get over to bridging and made use of the deprecated brctl for a while, but eventually switched over to ip as well. Man, I’m not looking forward to work with RHEL 9, but at least I’m somewhat used to nmcli from Fedora (when I actually had a use for it, since I had to switch between wifi connections, wired connections to different subnets with different profiles and connect to a vpn).

I find NM completely useless for servers, which are basically set and forget, never touch it again or you’ll break it kinda deal. For laptops, a connection manager is useful (although I still use ip, combined with dhcpcd and wpa_supplicant now).

@charlesshoults if you aren’t used to network manager, I’m not saying a switch to FreeBSD would be a similar learning curve, because (unless you are using pfSense or OPNsense, which are pretty intuitive) you have to learn a lot more than just how to make the network connections, but I still see it as a better option for (high-speed) routers. Just my $0.02.

I’m unfortunately unable to find my source again. It was a test with some dpdk based packet generator, though.