Hoping for some advice on turning my NAS into a Forbidden Router

[Susanna]

So this is what my NAS currently looks like. It may seem like just a messy gaming computer on the inside, because it is, but the exterior is all business. Rack, hotswap bays, even a lock on the front panel. I just happened to have a lot of gaming computer and watercooling parts laying around.

• ASUS B550F Gaming Wifi II
• Ryzen 5600G
• 64GBs of mismatched RAM
• Aquantia AQC107 10GbE NIC
• 2x Double Realtek RTL8125 2.5GbE NICs on M.2 (the things with all the black cables at the top left of the photo)
• Onboard Intel I225-V 2.5GbE NIC
• Onboard Mediatek MT7921K WiFi 6E NIC
• Two optanes, an SSD, and four HDDs

As a NAS, it’s been wonderful. High performance, superb reliability, no noise apart from the HDDs. I’m hoping to make it my new router, too. The problem I’m running into is my lack of knowledge when it comes to anything networking related. I’m assuming the default advice of “run the router in a VM with the NICs passed through” is out of security concerns, but while each of the M.2 2.5GbE NICs gets their own IOMMU group, the 10GbE and the motherboard 2.5GbE both share groups with things I can’t pass through (an optane and the SATA controller respectively). Unfortunately “pcie_acs_override=downstream,multifunction” didn’t help.
In a pinch I can survive without the onboard NIC, but the 10GbE is unfortunately very necessary. Can I instead give the VM virtual NICs bridged to the real NICs, or would that be hideously insecure? The only port I’m leaving open in the firewall for incoming connections from the internet is my wireguard port, so could I conceivably do this without a VM at all, instead just letting the NAS route for me, or would that be an awful idea? If not, what daemons would I need apart from DHCP, DNS, and NAT? dnsmasq seems like an easy way to handle the former two, and I think I know how to set up iptables to handle forwarding and NAT.

[risk]

@Susanna

Can I instead give the VM virtual NICs bridged to the real NICs, or would that be hideously insecure?

Naaah, it’d fine, your firewall takes care it.

@Susanna

… so could I conceivably do this without a VM at all, instead just letting the NAS route for me, or would that be an awful idea

Your router VM would probably be just another OS with another firewall anyway. What OS are you running as your NAS?

…

With respect to dnsmasq, it doesn’t support a bunch of fancy-shmancy DNS security features. I run it with adguard home on the side which adds those, but there’s folks who use unbound with dnsmasq or use pihole altogether

[Susanna]

@risk

Naaah, it’d fine, your firewall takes care it.

So something like running OPNsense in a VM would be done for convenience more than for security?

@risk

Your router VM would probably be just another OS with another firewall anyway. What OS are you running as your NAS?

I’m using NixOS, so adding a router component should be fairly simple.

@risk

With respect to dnsmasq, it doesn’t support a bunch of fancy-shmancy DNS security features. I run it with adguard home on the side which adds those, but there’s folks who use unbound with dnsmasq or use pihole altogether

Well, I don’t really know what I need. I’ve read up, and as far as I can tell my needs are pretty basic. No pihole for example, I just use an ad blocking extension in Edge. I need two wireless APs on separate channels (one the onboard wifi, the other a standalone AP on the other side of the house), and three subnets (the office, guests, and the CCTV). The NAS needs to be accessible from the office and CCTV subnets, but guests should only get internet access.

[diizzy]

Given the less than ideal NICs for solid networking why not just grab a small box SBC/or whatever and run OpenWrt or a full fledged os on separate hardware? It’ll very likely be more reliable and save you a lot of headaches.

[Susanna]

@diizzy

less than ideal NICs

Do you mean any card in particular, or just that they’re from aliexpress rather than pulled out of enterprise ewaste? I know the Aquantia has a bad rep but I’ve been using it for three years without issues, so I think mine is a solid sample. The M.2 NICs are a bit janky, but they’re actually a lot better than the junk 4x1GbE card they replaced. I ran some benchmarks with them after I installed them, and they seem to perform alright. Solid transfer speeds and no packet drops.

@diizzy

why not just grab a small box SBC/or whatever and run OpenWrt or a full fledged os on separate hardware? It’ll very likely be more reliable and save you a lot of headaches.

The main point of this exercise is to give me a way to learn networking hands on. I knew nothing about computers when I first started using Linux, and I learned a lot very fast, so I was hoping to learn networking by building my own infrastructure. I could run the router on separate hardware, I have a bunch of RPis and old laptops, but I feel like then I might as well just use a commercial router. I already felt that OPNsense on a VM was a bit like cheating, I just assumed it was for firewall reasons or something like that. While I want to learn, I don’t want to get hacked. This is for my home theatre and home office, if things break I can just plug my old router back in and work over wifi, fun is more important than reliability.

[alkafrazin]

RLT8125 based nics are just very flaky. So are i225, though less so.
Not sure why, but comparatively, 2.5gbe nics are just very prone to sudden failure/dropouts. I haven’t had a lot of problems with my i225v3s, but I’ve heard horror stories about them, and my 8125b went dead short after a year or so without warning; suddenly, the machine it was in locked up, and refused to give any signs of life until the card was removed. Exact same symptoms as having a copper heatsink bridging a circuit on the motherboard after being knocked off. Same symptoms in every system it plugs into.

It’s not necessarily the case that you’ll have problems, but if your router suddenly locks up and seems to be DOA, check the network cards first, rather than the apparent board or power supply. It might save you a day’s worth of headache.

Aside from that 8125 and earlier i225 have compatibility issues with some modern intel and AMD chipsets. X570, notably, because a lot of boards came with advertised 8125 based nics that were effectively 200% useless, something that was never fixed. I hear more recent Xlake chipsets had similar problems. Hope your B550 fares better; for what it’s worth, X470 didn’t seem to have the same problem, and both are Asmedia chipset based.

[MadMatt]

@Susanna

I need two wireless APs on separate channels (one the onboard wifi

That … Do yourself a favour and use a proper access point …

@Susanna

so could I conceivably do this without a VM at all, instead just letting the NAS route for me, or would that be an awful idea?

It would be an awful idea , you would be putting all your eggs in one basket, and risk giving access to your data to whoever…

Do you have a switch that can handle VLANs?

@Susanna

So something like running OPNsense in a VM would be done for convenience more than for security?

Running in a VM presents a wider attack security surface, because of the VM (if someone breaks into your firewall then it could theoretically spill over to the hypervisor) and because you will have your internet physical cable connected directly to your hypervisor, unless you pass the nic through (and even then…) So when discussing of ideal best scenarios having a separate hardware firewall is deemed more secure. Whether you need this ‘more securedness’ or not is up to you and how much you value the additional benefit versus the cost, space, power and having another thing that can break in your setup

@Susanna

I have a bunch of RPis and old laptops, but I feel like then I might as well just use a commercial router. I already felt that OPNsense on a VM was a bit like cheating, I just assumed it was for firewall reasons or something like that

You need to balance the time you want to spend learning against the speed you want the firewall to be up and running and doing what you need … You can do everything op sense does with a plain Linux VM, nftables and additional packages, it will take you a lot more time … is all …
With a virtualized setup nothing stops you running op sense for your ‘productivity’ tasks and a parallel setup to do the learning …

[risk]

@Susanna

So something like running OPNsense in a VM would be done for convenience more than for security?

Usually yes.

@Susanna

I’m using NixOS, so adding a router component should be fairly simple.

I haven’t played with NixOS.

I can tell you my current router is just Debian with a bunch of network interfaces (pair of physical and a bunch of virtual bridges and macvlans and tunnels).

I use systemd-networkd for most of the configuration, and I have a few simple hand written iptables rules.

I have dnsmasq doing DHCP and DNS for the local domain, but I’ve pointed my individual lan machines to Adguard - mostly so I get rid of ads on the phone.

It can definitely work fine with no VMs, there’s no faux pas you’d be commiting by not running a distro with a more convenient home-router UI.

[Susanna]

@alkafrazin

It’s not necessarily the case that you’ll have problems, but if your router suddenly locks up and seems to be DOA, check the network cards first, rather than the apparent board or power supply. It might save you a day’s worth of headache.

Hmm, good to know. The onboard NIC has been good so far, but I’ve only been using this motherboard for a few months, so it could still develop problems. Disappointed the Realteks are apparently bad too, but they were always going to be my top suspects if there were hardware issues. The PCBs arrived bent, and the header pins soldered quite poorly. But they recognised and did some file transfers just fine, so they get to stay in.
Ideally I would have gotten 10GbE on all of them, 2.5 is such a silly stopgap when 10 has been a thing for literally over a decade, but I would have needed a bunch of extra switches since they only have one port per.

@MadMatt

That … Do yourself a favour and use a proper access point …

Okay, I’ll get an extra AP so I won’t need to use the onboard wifi. I guess it’s just a matter of plugging them into the router, opening the IP address in a browser, and setting the same SSID and encryption on both, and which bands they should operate on?

@MadMatt

Do you have a switch that can handle VLANs?

You mean a managed switch? While I have plenty of managed switches (I bought them for the PoE), none will handle ten gigabit, which the office computers need. I can buy one though, if you’ve got any recommendations? I’d want four ports minimum (but I’m willing to pay extra for eight ports, so I can expand) at least two of which support 10GbE over copper. So basically the topology would be this?

That’s actually what it already looks like, just the NAS and the Computers are behind a second switch for 10GbE, and none of the switches are currently managed even though the CCTV ones technically support it.
Would the Zyxel XGS1250-12 be a decent candidate? Seems to have four 10GbE ports (though one is SFP+) and eight normal gigabit ports.

@MadMatt

Running in a VM presents a wider attack security surface, because of the VM (if someone breaks into your firewall then it could theoretically spill over to the hypervisor) and because you will have your internet physical cable connected directly to your hypervisor, unless you pass the nic through (and even then…) So when discussing of ideal best scenarios having a separate hardware firewall is deemed more secure. Whether you need this ‘more securedness’ or not is up to you and how much you value the additional benefit versus the cost, space, power and having another thing that can break in your setup

The alternative would be running it on the NAS directly, so I guess the VM is still the safer option. Since you’d need to break both iptables and kvm to get in, instead of just iptables. Breaking out of a hypervisor sounds like basically zero day level stuff, I think those are usually pretty locked down.

@risk

I haven’t played with NixOS.

I can tell you my current router is just Debian with a bunch of network interfaces (pair of physical and a bunch of virtual bridges and macvlans and tunnels).

I use systemd-networkd for most of the configuration, and I have a few simple hand written iptables rules.

I have dnsmasq doing DHCP and DNS for the local domain, but I’ve pointed my individual lan machines to Adguard - mostly so I get rid of ads on the phone.

It can definitely work fine with no VMs, there’s no faux pas you’d be commiting by not running a distro with a more convenient home-router UI.

Yeah, all of this should be reasonably easy to do with NixOS. It’s basically just a normal Linux distro with weird package management and configuration. Thanks for the list, it gives me a great place to start. The only thing my phones use the internet for is downloading music and ebooks, so I can probably do without the adguard.

[MadMatt]

@Susanna

Would the Zyxel XGS1250-12 be a decent candidate? Seems to have four 10GbE ports (though one is SFP+) and eight normal gigabit ports.

Yes, it supports VLANs and has a very good price point for 4x10GB ports …

@Susanna

Okay, I’ll get an extra AP so I won’t need to use the onboard wifi.

Since you want a Guest VLAN, make sure the APs support it … mesh ones will require a controller and let you configure things once (TP link deco, omada, Unifi), non mesh ones will need to be configured using the same settings but you’ll probably not get seamless roaming between them …

@Susanna

The alternative would be running it on the NAS directly, so I guess the VM is still the safer option.

The VM will be the easier to manage option … when changing stuff in the firewall you will not impact things happening on the NAS (think applying the wrong firewall rule and cutting out access to the wrong subnet to the NAS, maybe your pc included …). You will be able to test different firewall options (plain linux, openbsd, freebsd)

As for what option is best for you it really depends on how much manual config you want to get yourself into versus the time it will take to get up and running, and also how much you are willing to configure manually when you want to add services/functionality and how much you want to manually set up things

If you go the manual way, I’d suggest looking into nftables instead of iptables
If you want a middle ground, have a look at vyos, it’s linux based (debian), no gui, you can still have a look at all the config files, gives you a lot of options when you grow past your current simple needs (BGP/VRRP router, IPV6, wan load balancing) but it still requires you to get into the how things are working in order to get it running properly
If you want to feel like you are cheating (and not understand fully what’s happening) then pfsense/opnsense have a nice gui an plenty of ‘tutorials’

As an example, this is my current setup using Vyos (inside a Proxmox hypervisor, used to be truenas scale until two weeks ago):

vyos@vyos:~$ sh int
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
eth0             172.30.2.1/24                     u/u  LAN
                 beef:beef:beef: beef:beef: beef:beef:beef/64

eth1             192.168.101.144/24                u/u  TIM-TEST
eth2             xxx.xxx.xxx.xxx/10                 u/u  STARLINK
                 beef:beef:beef: beef:beef: beef:beef:beef/64

eth3             192.168.200.1/24                  u/u  STORAGE
eth4             172.30.3.1/24                     u/u  IOT
                 beef:beef:beef: beef:beef: beef:beef:beef/64

eth5             172.30.4.1/24                     u/u  GUEST
lo               127.0.0.1/8                       u/u
                 ::1/128
wg0              10.223.220.2/24                   u/u  VPN-to-OCI

• 4 internal VLANS (LAN - STORAGE-IOT-GUEST)
• 2 WAN interfaces (TIM-TEST-STARLINK)
• 1 Wireguard interface to a cloud based VPS (VPN-To-OCI)

Starlink provides an IPV6 routable network, so I am using that to assign IPV6 addresses to all my internal devices that support it

Outgoing traffic is load balanced across the two WANS by default, but I ghave some policies to direct specific IPs over one or the other:

vyos@vyos:~$ sh wan
Interface:  eth1
  Status:  active
  Last Status Change:  Mon Apr 17 19:54:23 2023
  +Test:  user  Script: /config/scripts/tim_check.sh
    Last Interface Success:  0s
    Last Interface Failure:  n/a
    # Interface Failure(s):  0

Interface:  eth2
  Status:  active
  Last Status Change:  Mon Apr 17 19:54:23 2023
  +Test:  user  Script: /config/scripts/starlink_check.sh
    Last Interface Success:  0s
    Last Interface Failure:  n/a
    # Interface Failure(s):  0

vyos@vyos:~$ sh conf com | match wan
set load-balancing wan rule 20 description 'docker per packet lb'
set load-balancing wan rule 20 inbound-interface 'eth0'
set load-balancing wan rule 20 interface eth1 weight '1'
set load-balancing wan rule 20 interface eth2 weight '1'
set load-balancing wan rule 20 per-packet-balancing
set load-balancing wan rule 20 protocol 'all'
set load-balancing wan rule 20 source address '172.30.2.176'
set load-balancing wan rule 50 description 'Docker prefer TIM for DNS'
set load-balancing wan rule 50 destination address '8.8.4.4'

I still have access to all the low level information (and then some) the linux subsystem has been configured according to the Vyos cli commands:

vyos@vyos:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

S>* 0.0.0.0/0 [210/0] via x, eth2, weight 1, 00:28:50
  *                   via 192.168.101.1, eth1, weight 1, 00:28:50
S>* 10.128.8.0/24 [1/0] is directly connected, wg0, weight 1, 00:28:50
C>* 10.223.220.0/24 is directly connected, wg0, 00:28:51
C>* x/10 is directly connected, eth2, 00:28:52
C>* 172.30.2.0/24 is directly connected, eth0, 00:28:55
C>* 172.30.3.0/24 is directly connected, eth4, 00:28:54
C>* 172.30.4.0/24 is directly connected, eth5, 00:28:55
S>* 192.168.100.0/24 [1/0] is directly connected, eth2, weight 1, 00:28:50
C>* 192.168.101.0/24 is directly connected, eth1, 00:28:54
C>* 192.168.200.0/24 is directly connected, eth3, 00:28:54

[Susanna]

@MadMatt

Yes, it supports VLANs and has a very good price point for 4x10GB ports …

Cool.

@MadMatt

Since you want a Guest VLAN, make sure the APs support it … mesh ones will require a controller and let you configure things once (TP link deco, omada, Unifi), non mesh ones will need to be configured using the same settings but you’ll probably not get seamless roaming between them …

Guest VLAN in this case just means I don’t want these devices to see anything but the internet. My laptops will waste time sending their backups to my ISP and then back home (wireguard), but that’s happening regardless since I want them to be able to backup even while travelling. I’ve got 1Gb/s fibre so it’s not a big problem. Definitely going to make sure they support seamless roaming though, I’d assumed they all could do that but I’ll make sure they’re mesh things.

@MadMatt

The VM will be the easier to manage option … when changing stuff in the firewall you will not impact things happening on the NAS (think applying the wrong firewall rule and cutting out access to the wrong subnet to the NAS, maybe your pc included …). You will be able to test different firewall options (plain linux, openbsd, freebsd)

The nice thing about NixOS is that I can easily rollback any changes I make. I track changes to the configs with my gitlab and can rollback the Nix config with one command. I guess I’ll start by trying to do it without a VM, I think it’ll be easier for me to manage everything through NixOS. It’s just convenient to have all my computers configurations available in a single project directory, and I can make and apply changes easily to everything that way.

Are you using Starlink as a backup WAN somehow? One thing I have wanted to try, is making my wireguard interface the backup. Like, if the NAS is available on the local network, it will connect directly, but if it isn’t it’ll connect through the wireguard. If I could get that working, I’d want the APs to do mac filtering so my laptops end up on the trusted VLAN and can connect directly to the NAS, and any other devices get only internet.

[MadMatt]

@Susanna

Are you using Starlink as a backup WAN somehow? One thing I have wanted to try

I am using the two WANs at the same time in load balancing mode, for the clients/subnets that are allowed, the total bandwidth is the sum of the two links (I d o not have gigabit,yet), if one of the two link fails, the other is used as backup. This is transparent to the internal clients

@Susanna

Like, if the NAS is available on the local network, it will connect directly, but if it isn’t it’ll connect through the wireguard

You can do that using a fqdn instead of an ip for the nas, and the firewall as the DNS for the local lan so that it remaps the fqdn to the internal ip for local clients …

@Susanna

If I could get that working, I’d want the APs to do mac filtering so my laptops end up on the trusted VLAN and can connect directly to the NAS, and any other devices get only internet

Usuallly you want to do that at the SSID level (two SSIDs, Home and Guest) … mac addresses can be easily spoofed so assigning a VLAN based on them would be insecure at best and would only cause you maintenance headaches. If you want to use a single SSID and dynamically assign VLANs you want to implement 802.1x auth but that is a compeletely different can of worms and segmenting by using different SSIDs will be way less complicated …

[Susanna]

@MadMatt

I am using the two WANs at the same time in load balancing mode, for the clients/subnets that are allowed, the total bandwidth is the sum of the two links (I d o not have gigabit,yet), if one of the two link fails, the other is used as backup. This is transparent to the internal clients

I see, that sounds very interesting. You don’t run into any weirdness from the difference in latency between Starlink and your other connection?

@MadMatt

You can do that using a fqdn instead of an ip for the nas, and the firewall as the DNS for the local lan so that it remaps the fqdn to the internal ip for local clients …

I see, that makes sense. So I would mount the share with the public Wireguard address, and the router+firewall will alias the Wireguard address to the local one? The issue I can see with that is that my Wireguard address only has the Wireguard port open. I don’t want the smb port open to the internet, for obvious reasons. As I’m doing it now, each laptop runs a Wireguard client that connects them to the NAS, and mount the smb share with the Wireguard IP of the server. Could you give an example of how I’d translate that into a domain name? I do have a static IP, it’s how I’m pointing my domain name home already.

@MadMatt

Usuallly you want to do that at the SSID level (two SSIDs, Home and Guest) … mac addresses can be easily spoofed so assigning a VLAN based on them would be insecure at best and would only cause you maintenance headaches. If you want to use a single SSID and dynamically assign VLANs you want to implement 802.1x auth but that is a compeletely different can of worms and segmenting by using different SSIDs will be way less complicated …

Yeah, that’s a really good point. I have a bunch of lawnmowers and vacuum cleaners that don’t support anything fancier than wpa2, so I guess that does limit what I can do. Two SSIDs isn’t what I had in mind, but you’re right, using the MAC address wouldn’t be enough on its own.

I think I’ll just skip all that complexity and keep the laptops on the guest network. I may just upgrade to ten gigabit internet instead, it’ll let the laptops use the full bandwidth and be less of a hassle to set up. Gives me more bandwidth for RDP too.

[MadMatt]

@Susanna

So I would mount the share with the public Wireguard address, and the router+firewall will alias the Wireguard address to the local one

No,
you either keep things as they are now (lan ip annd wireguard ip) and set up things using a dns like nas.my.domain, and configure your local dns to resolve that to you LAN nas ip, and your remote dns when you are connected to wireguard to your wireguard ip or, even better, you always use your NAS local lan IP and set up proper routing in wireguard …

[Susanna]

Oh, so I would have to pipe all traffic through the Wireguard interface? Yeah, that’s not going to be convenient. I do that when I travel to China, which to be fair is often, but in my country I generally don’t use VPN for all traffic, only the traffic explicitly going to my home.