Grant's Forbidden Router Adventure WIP

wendell · November 3, 2025, 4:24am

Background

TODO

Getting Started

The first step is configuring the switch. Even (especially) if you have a pre-existing router.

Our goal is to configure some of the ports on the switch to be on a separate vlan. This is very close to the same setup as it would be if your ISP’s equipment (modem, ont bridge, whatever) was plugged into a dumb switch and then into your router.

Before we do that, we should take stock of the current (working?) configuration.

Understand that only about half of ISP equipment is designed to talk to one device at a time. The other half is both a bridge to the ISP’s network and a router.. so you could plug a dumb switch into your ISP equipment to have more wired ethernet ports. If your ISP equipment also has wireless, chances are it’s a bridge+router and you’re going to want to put it in bridge mode.

Generally, though, we want the “real” IP from your ISP assigned to the router virtual machine.

Before making any changes try to assess what situation you’re in by looking at your existing router’s configuration.

Make a note of:

Do you see a public IP on your router?
Does it match what you get from i.e. ipchicken.com
Save the MAC address on the WAN side of your router (it often shows in the gui)

Getting Started - VLANs

With that mindset, and notes made, it should be safe to configure the switch so that some ports are on their own VLANs.

Mikrotik Edition

We showed two models in the video
CRS326-4C+20G+2Q+RM and

For the fancier mikrotik you can connect the console port with a serial cable. I’m just doing it on one of the Proxmox hosts. apt install minicom if you need to.

The goal is to create a new vlan100 and combo1 will be untagged to this vlan. combo2 will be tagged vlan1 and vlan100 into our proxmox machine.

If we had a second ISP/connection, that should be vlan200 and the other port pairs should be similar.

If your ISP speed is under 2.5gb, it makes more sense to use one of the 2.5g copper ports like ether1 for your untagged vlan ISP connection port.

Our Mikrotik with 8x 2.5gbe and 2x10g sfp+ does not have a console port like the other one does, which is fine. Winbox can also be used, or go to the IP to configure it.

it has
ether1-ether8 and sfp-sfpplus1 and sfp-sfpplus2

similarly, we configure
ether1 for the ISP – vlan100 only
ether2 is for our “small” proxmox host.. it doesn’t have a fast interface
sfp-sfpplus1 is for our 10g computer or "other’ 10g switch
sfp-sfpplus2 is for our 10g nas “big” proxmox node as we showed in the video

Also note vlan 1 is “special” – it is a best practice not to use it for this kind of trunking.

Here are the commands I used to setup the mikrotik switch

# show details of the vlans config to start 
/interface/bridge/vlan/print detail

# and port details
/interface/ethernet/print

My thinking is:

ether1 = VLAN 100 (untagged)
ether3–8 = VLAN 10 untagged
ether2 and sfp-sfpplus2 = trunk (admit-all, PVID 1, effectively vlan100 and vlan10 are “both” on these ports )

# 0) Safer to edit with filtering off
/interface/bridge set bridgeLocal vlan-filtering=no protocol-mode=none

# 1) Bridge ports: set access vs trunk roles
/interface/bridge/port
# VLAN100 access
set [find interface=ether1] pvid=100 frame-types=admit-only-untagged-and-priority-tagged

# VLAN10 access on ether3..8
:for i from=3 to=8 do={
  set [find interface=("ether".$i)] pvid=10 frame-types=admit-only-untagged-and-priority-tagged
}

# VLAN10 access on sfp-sfpplus1
set [find interface=sfp-sfpplus1] pvid=10 frame-types=admit-only-untagged-and-priority-tagged

# Trunks on ether2 and sfp-sfpplus2 (carry 1,10,100); PVID 1 for untagged ingress
set [find interface=ether2] pvid=1 frame-types=admit-all
set [find interface=sfp-sfpplus2] pvid=1 frame-types=admit-all

# 2) Bridge VLAN table (who is tagged/untagged per VLAN)
/interface/bridge/vlan remove [find]
/interface/bridge/vlan
# VLAN 1 (management core / native on trunks)
add bridge=bridgeLocal vlan-ids=1 \
    tagged=bridgeLocal,ether2,sfp-sfpplus2 \
    untagged=
# VLAN 10 (user/access on ether3-8 + sfp1)
add bridge=bridgeLocal vlan-ids=10 \
    tagged=bridgeLocal,ether2,sfp-sfpplus2 \
    untagged=ether3,ether4,ether5,ether6,ether7,ether8,sfp-sfpplus1
# VLAN 100 (WAN/other access on ether1)
add bridge=bridgeLocal vlan-ids=100 \
    tagged=bridgeLocal,ether2,sfp-sfpplus2 \
    untagged=ether1

# 3) Management SVIs
# VLAN1: DHCP client (primary mgmt)
/interface/vlan add name=mgmt-vlan1 interface=bridgeLocal vlan-id=1
/ip/dhcp-client add interface=mgmt-vlan1 use-peer-dns=yes use-peer-ntp=yes add-default-route=yes

# VLAN10 + VLAN100: add IPs for reachability (pick static addresses or use DHCP if you have servers there)
/interface/vlan add name=mgmt-vlan10 interface=bridgeLocal vlan-id=10
/interface/vlan add name=mgmt-vlan100 interface=bridgeLocal vlan-id=100
# Example static IPs (change to your subnets or swap to DHCP)
/ip/address add interface=mgmt-vlan10 address=192.168.10.2/24 comment="Mgmt on VLAN10"
/ip/address add interface=mgmt-vlan100 address=192.168.100.2/24 comment="Mgmt on VLAN100"
# (Optional) If you prefer DHCP on those too, use instead:
# /ip/dhcp-client add interface=mgmt-vlan10 use-peer-dns=no use-peer-ntp=no add-default-route=no
# /ip/dhcp-client add interface=mgmt-vlan100 use-peer-dns=no use-peer-ntp=no add-default-route=no

# 4) Lock management services to your mgmt subnets (adjust networks)
/ip/service
set [find name=winbox] address=192.168.1.0/24,192.168.10.0/24,192.168.100.0/24
set [find name=ssh]    address=192.168.1.0/24,192.168.10.0/24,192.168.100.0/24
set [find name=www]    address=192.168.1.0/24,192.168.10.0/24,192.168.100.0/24

# 5) Enable filtering (programs hardware tables)
/interface/bridge set bridgeLocal vlan-filtering=yes

Avaya E-waste Edition

TODO how to do the same thing on the 24 gbe + 2 sfp+ Avayia

Proxmox side of things

I am assuming the proxmox install and cluster setup are not needed as part of this tutorial. There are some great resources for that here, and elsewhere on the internet. If you have trouble for that, feel free to make a separate post in the appropriate section of this forum.

Post-Install

I showed these scripts in the video for helping Grant get a quick start with immich, nextcloud, calibre book server, jellyfin and many other cool proxmox bits.

Before we get to that we need to do the networking configuration

Proxmox Networking config for Vlans

As I mentioned in the video “ideally” in a proxmox cluster you have a couple network interfaces. You can use a 1gb or “slow” interface for management and the cluster sync “corosync” services, and the “fast” interface for synchronization, vm migration and that sort of thing. The main reason for this is that proxmox doesn’t prioritize corosync traffic and loss/delay of sync traffic between nodes while a migration is going on can cause wonky behavior on the cluster.

In general this danger is somewhat overblown imho on a “home” cluster like this. For Grant’s setup I experimented with both vmbr0 and vmbr1, but did a lot of “torture testing” with vmbr1 as the only interface.

Ultimately, that’s our setup here.

Two Node Cluster?

This requires a little bit of reason, and it would be a good idea to watch the video.

In “production” environments one wants a 3-node cluster, ideally. The least-bad option for a 2-node cluster is to have a special quorum voting instance called a qdevice. This is a part of the cluster, and can help the remaining node know if it’s in a split-brain situation (meaning both cluster nodes are up but they can’t see each other for some communication reason) or if a node really is down.

With replication, and a down node, the remaining node will boot up its last-synced VM. That could be hours old. That might be really bad! You, the administrator, can decide what to do.

The ‘best’ option here for Grant’s setup was actually to rely on OPNSense (or pfSense) built-in HA. In this case there is no need to replicate that VM between hosts directly – the HA is managed by having two ‘router’ VMs that are running all the time and software features in the routing software manage HA. This is also the way to handle mysql/mariadb and postgresql type databases.

Corosync does have a ‘special’ two-node mode that is usable as a third option. It’s not recommended and seems frowned-upon to even mention.

# /etc/pve/corosync.conf -- I added these two lines 
 two_node: 1
 wait_for_all: 0

Reference Man Page for Corosync vote quorum.

xmicks · December 8, 2025, 9:49pm

Hey thanks for sharing. I just watched the yt video and got inspired!

I have a TrueNAS box and in that im running a Ubuntu VM and with Docker running in that. In the setup example from this thread Proxmox is doing the vitualization stuff so then im gussing now, where does my preexisting NAS fit into this setup? The main OS on the NAS is Proxmox?
I assume it is, should I then go ahead and replace the TrueNAS OS with Proxmos? My followup question then is, what piece of software would then be doing my NAS stuff?

superben · December 8, 2025, 10:18pm

It would depend what you are doing for NAS stuff. You could virtualize truenas within proxmox and then have both if you needed both things. You could passthrough certain disks to a truenas VM to control if you have hardware that can do passthrough. I just created a NFS share on the zfs pool on proxmox for my setup though.

sneer · December 8, 2025, 10:49pm

Just a semi-OT question: why router needs to be virtualized? Wouldn’t it all go down if hardware fails anyway?

wendell · December 8, 2025, 11:24pm

so thats covered in the video. when you setup proxmox HA or HA in the router itself, it doesn’t go down. because its running on the nas and the N305 mini pc. or two nases, or whatever. the router being a VM means its easy to migrate it between hardware and do maintenance or swap things with nothing going down. whereas if you update your router firmware it’ll be down for a few minutes while that runs.

here you could update opnsense on one side, then the other, and no one would know anything happened really

HanSolo · December 8, 2025, 11:53pm

I just wanted to take my hat off to Grant. I don’t recall him having been in front of camera before (although I may well be wrong) but honestly, I wouldn’t have known it if I hadn’t not seen him in L1T videos before… if that makes sense. As someone who doesn’t exactly relish public speaking, I did find it rather impressive that he seamed so at ease.

I can’t really address the rest of your comment as I think I’m too tired to give you a proper answer. In regards to the question of “NAS stuff” though… personally I use Turnkey Fileserver for handling the access to my ZFS datasets. Which are mounted into the Turnkey LXC using bindmounts. There is a little bit of a learning curve to this approach, especially in regards to bindmount permissions mapping but there’s some good resource online that can help with that.

BeepDog · December 9, 2025, 2:47am

One piece of hardware I didn’t see in the video, that I run myself:

(I’m not allowed to post links) https ://protectli.com/products/

I have their 4 port: https ://protectli.com/product/vp2440/

Coreboot, it’s as open as it can be. No fans, so there’s no hardware to fail. They sell a DC battery “UPS” which can get you free runtime there. Comes with opnsense installed, and TBH, I could’ve put proxmox on it, and VM’d out an opnsense, because nobody needs an opnsense with 32gb of ram

strauss2 · December 9, 2025, 2:47pm

I’m a big fan of the forbidden router. I’m using ryzen 5650g pro with x570-d4u running proxmox. I get 13Gb/s through opnsense using iperf3. I am running an arr stack on a mix of sata sdd’s and hdd’s. Everything is running perfectly but I am scared for what happens if my host goes down. I would love to see a video going more into depth on disaster plans for when stuff goes wrong. Maybe simulating the proxmox host drive failing or the opnsense going down without having HA or a cluster. Cool hardware, looks like the dream setup for quiet home server with good performance!

rexnebula · December 9, 2025, 3:51pm

Just a couple of comments/suggestions. I’ve done forbidden routers for a looooong time now even though the most recent iteration of my home-not-lab I went back to physical boxen. I’ve done forbidden routers both with and without HA (using CARP on openbsd and later OPNsense) as well as with and without failover (using proxmox ZFS replication to another node in the cluster - I don’t call that full HA if it’s not on a clustered filesystem).

The biggest problem you’re going to have enabling HA with two separate OPNsense VMs on a proxmox cluster is the fact that most residential internet connections provide a single IP address via DHCP. Getting HA/CARP working with a single dynamic IP address is not for the faint of heart, requiring a custom script to be run on the standby node that disables the WAN connection when the primary is up, and re-enables it when the primary is down. The best way to get around this is to pay your ISP to get multiple static IP addresses (a /30 block is nice but you can work around the 3 public IP CARP problem if you only can get a single IP). Unfortunately not all ISPs will provide that, or if they do it can be prohibitively expensive. You can also have your ISP modem/gateway device act as a router and hand out private IPs to your forbidden VM router, which you then use as your WAN IP. Unfortunately this introduces the double NAT problem which is often not fun if you’re passing through a lot of external IPs to various internal services.

Second, I think in the video especially, and also in this guide it’s a little unclear to the layman the VLAN configuration, why you need it in this configuration, and what exactly that looks like at the Proxmox and VM levels especially for the OPNsense VM. There are multiple places you can end the tagging, but it looks like you’re passing it through all the way to the OPNsense VM level so you only have one interface assigned to the OPNsense VM in proxmox, and then configure multiple VLAN interfaces in OPNsense itself.

strauss2 · December 9, 2025, 3:51pm

I am using a cockpit LXC just for smb shares. It can also do nfs but nfs was easier to manage on the host. And as echoed by others the permissions are done through idmap on the bind mount. It took me forever to figure it out because I originally needed to use it for gpu sharing to an lxc. I manage all my zfs snapshots and replication with sanoid in the command line on the host. That is also kind of a pain to set up compared to truenas.
The easy button for all this on proxmox is running a truenas VM, it is just highly recommended to make sure you are passing through your sata controller or HBA. On my particular setup, I get to save 8gb ram and save 10w idle by not using my HBA and managing storage on the host. I can’t say for sure if it was worth all the time I spent setting it up.

DoubleA · December 10, 2025, 11:46am

Been looking into doing forbidden router setup for a while now.

I have physical opnsense, proxmox, and truenas builds. I will try to do something similar soon with an opnsense vm on proxmox and pairing to my physical opnsense. Maybe even attempt doing a 2 node proxmox cluster (with Rpi as Qdevice for qorum), but with the 2nd proxmox node being a vm on truenas.

James_M · December 14, 2025, 6:35pm

My FR is on a N150 near-fanless MiniPCs that has dual SFP+, and 3x 2.5GE. WAN terminates into one of the 2.5GE on vmbr7, and LAN is trunked to the dual SFP+ in a LACP bond going to a mokerlink L3-Lite mixed GB switch (8x2.5GE-POE, 2x10G-RJ45, 4xSFP+), then UBNT WAPs hang off the 2.5GE ports on the core and routing is OSPF between the OPNSense VM on the N150 and the Mokerlink.

PVE’s management for the FR VM is a VLAN hanging off the vmbr above the bond. Works quite nice.

The main reason I had to go FR and not on metal, whenever I updated OPNSense it would kill the SFP+ bond and take the firewall offline and need to be redeployed, updated, then network rebuilt. Running it in a VM fixed that.

marf8 · June 17, 2026, 2:11pm

So if you are able to access the mikroswitch web ui through any of the ethernet ports, is there a security concern that using some of the ethernet ports for the WAN can expose the mikrotik switch web ui? Does mikrotik allow a way to lock down the switch management?

Strawberry · June 17, 2026, 2:27pm

They do. You can restrict it to specific ports, specific VLANs etc. This is an important point to note.