Resilient Networking exploration madness - join me on a journey

Networking
1

I’m abount to try a complex homelab setup - anyone want to join in the fun as I document and provide suggestions? If so read on (quite long!)

I watched a YT video by Network Chuck (DO NOT design your network like this!! ) discussing network architectures and thinking this is not something easy to practice on a home network. Then I ended up on this page - OPNsense setup with bridged interfaces, VLAN and RTSP not worki- and thought I could try that!

I have an Opnsense router, 2 Mikrotik routers and a Netgear 10Gb switch, so I want to try a toplogy like the example on this page: Topology of a STP enabled network

SW1 would be Opnsense, SW2 and SW3 the Mikrotik, and SW4 the Netgear. The topology is easy - just plug wires in ports. The configuration is the harder, fun part…

Constraints:

  1. Avoid bridges if possible (Opnsense document suggestion)
  2. Keep Internet access open (as little downtime as possible) fo2 Wi-Fi and wired PC and wired NAS.
  3. The dual links in the toplogy will be 802.3ad LAGGs.
  4. The Opnsense connects to the Internet. This is a single point of failure. Cost constraints mean this has to stay as s SPOF.
  5. No more hardware purchases.
  6. Mikrotik must be in RouterOS (not SwOS) - just because passwords aren’t strong enough under SwOS.

Stretch Target - I have 2 Engenius EWS377AP currently as separate SSIDs on different ports of the Opnsense. Can I make this more resilient in itself, and as resilient as possible from the wired network.

As I work my way through this I will document the adventure, but feel free to join in at anytime!

Documentation

Opnsense: Supermicro SYS-E300-9A with 4 1Gb ports (2 used for WAN LAGG) and 6 10Gb ports (4 available, 2 unusable, but not in use)
2xMikrotik CRS312-4C+8XG-RM (12 10Gb ports, 4 will be used for interconnecting LAGG)
Netgear XS508 (LAGG, VLAN support web interface)
Engenius EWS377AP support 2,5Gb, so use 10Gb ports if available, otherwise 1Gb.

Initial plan:

  1. Initial configuration of Opnsense with WAN port and 1x10Gb to connect to switches. Daisy chain switches SW2-3-4 without return to SW1 at present. Configure switch web interfaces (passwords etc.) and check for Internet access. Connect wired devices.
2

would you happen to have a pair of proxmox hosts in your homelab?

you virtualize your router using something like this: [OpenWrt Wiki] High availability

3

Thanks for the suggestion. I’ve never really got into Proxmox, but the idea is interesting so I may look at that in future.

The LAN side of this will end up “very resilient” in some aspects, but not at all really as it is all in the same rack. I know the SPOF is there, but my ISP is far less reliable than my kit :slight_smile:

The aim of the exercise is to learn about network aspects I never really use (such as STP, maybe bridges, more complex routing approaches) and the different approaches to configuration between Opnsense and Mikrotik.

Hence the madness…

4

Dumb (fun though) suggestion: Use routing where possible.
Just have all your switches talk to each other via a different network (if your home network is 192.168.1.0, then 172.16.0.0 for your switches) and enable OSPF (or similar).