Tips on creating a secure home k3s cluster with pfsense

So I have a 2 ethernet port sbc that I’m going to install pfsense on it. From a topology stand point what I’m thinking of doing is: modem–>router–>pfsense(eth1)–>pfsense(eth2)–>16 port switch–>cluster.

The idea here is the phsense box is acting as a means to vpn into the cluster and a firewall for the cluster. So if I host a website, I can access it without needing to vpn in, but if I want to ssh into any of the nodes, I VPN into the network. Is this doable? Also is any of this doable without having to mess with the router?

Fairly simple, though you will need to mess with the router if you want the website(s) to be accessable from the internet. You port-forward web server to the outside of pfSense. You still need to forward that same port to the outside of the router for it to be seen from the internet. Same issue with the VPN. You run whatever VPN (wireguard, OpenVPN, ect) onpfSense, then forward that port on the router so it can be reached from the internet. If you want to minimize configuration you can set the pfSense’s IP address as the DMZ in the router. Or, if you’re allowed, you can drop the router from this entirely. That would be my preferred option.

Is there a way to rearrange things where I don’t need to touch the router. Can I do something where phsense connects to the modem and switch and the router connects on that switch?

That’s what I ended up doing with my network. The router became just a wireless access point (it doesn’t do PAT, routing, DHCP).
The network becomes:
Internet > Modem > pfSense > switch > AP
Just so we’re clear, when you say router you mean like a home wireless router, and not like… business grade router-router, right?
The only problem with this is you still end up needing to touch the router to turn off DHCP, if you want the wireless AP clients on the same subnet as everybody else. If you’re okay with them sitting in their own little subnet this isn’t an issue.

So yes, this is just your basic wireless router.

So if I understand things correctly, If I don’t mind it being on a separate subnet; I can leave the router untouched and not needing to disable DHCP?

Now I’m on the fence on the matter. The whole issue is I don’t want to reset it since I forgot the password X_X. So long as I keep the name and access password the same, can previous device reconnect to it?

Yes. So with the router you have the WAN (internet) and LAN (local) side. Right now you have the router attached to the modem on the WAN port. So the modem is assigning that port an IP address, then the router is assigning an IP address to devices which connect on the LAN ports or wireless. If you hang the router off the switch the router’s WAN port will be assigned an IP by pfSense, and then the router will continue to serve IP address to its LAN and wireless clients. This is mostly okay. The only issue is you won’t be able to connect from devices on the pfSense LAN to devices on the router’s LAN. Going the other direction will be fine.