I’m building an OPNSense baremetal box for my home/home office network, and have a dual 10GbE SFP+ card installed.
My plan was to use one of the ports for WAN-In (1Gbps symmetrical fiber, max 2Gbps if I ever upgrade). The second port was going to go to my core switch on a full duplex 10GbE port. I’m going to have some 10GbE machines in my network on their own VLANs, so I wanted to plan ahead for that, and have the capability for full-speed inter-VLAN routing if I can’t avoid it.
Then I started looking around and saw a lot of people LACP’ing their 2x10GbE SFP+ ports together to get a 20 GbE LAG on the LAN-side.
I’m trying to figure out what I’d have to be doing to need to do that. I’m not a pro network admin, so I don’t have much experience to draw on here. This is the first 10G network I’ve ever built.
Could you elaborate a bit on how it benefits inter-VLAN routing? I’ve not yet actually set up VLANs, so my understanding of inter-VLAN routing isn’t fully developed yet.
Link aggregate only works if both sides of the connection are set that way. Is your switch capable of link aggregation? If not, no matter what you try on the OPNSense side, it won’t work.
even so, given that it’s your first deep foray into networking, best keep it simple and stick to a single connection until you either need it or have gained the knowledge to perform link-aggregation in your network.
Well, instead of sending all your inter-VLAN traffic over a single 10gbe link now it can use two, so it will be up to twice as fast assuming everything else can keep up.
I’m using QNAP managed switches. Even on the higher-end core switch, setting up LACP is easy and I’ve already done it for a couple of downlink switches. In fact, setting up LACP is probably the one “advanced networking” thing I’ve really done so far.
I am definitely embracing the keep-it-simple vibe right now. I’d prefer to stick to a single connection in/out, so I appreciate the reassurance.
I guess I’m trying to figure out how to tell if I need to LACP 2x10Gbps on the LAN side. Like, what should I be looking for? Bottlenecking at the core switch’s uplink to the firewall? (QNAP switches make it very easy to monitor how much traffic is going over a specific port in Gbps, so it’ll be pretty easy to see if I’m getting clogged up there.)
I suppose my real question, then, is what sort of thing(s) I’d need to be doing to generate enough inter-VLAN traffic that I’d really need to be using a 2x10GbE pipe (with two 10 GbE lanes, not one big 20 GbE lane).
You would need a lot of traffic between many devices for it to be useful. But it can also prevent your internet from slowing down if you saturate the single 10gb link with VLAN traffic.
As above it could be useful for inter VLAN routing but I would argue that if you’re trying to push >10 gig through a router to local destinations on your home network you’re probably doing it wrong.
Use a layer 3 switch for that volume of local VLAN routing.
A port channel (LACP for example) is useful for link redundancy (eg all my workgroup switches at day job are port channeled back to my core) but even then with 100+ ports per uplink (office of ~400 users plus printers, wifi etc across 6 workgroup switches/stacks) I’m not seeing bandwidth contention on even one leg of the 10 gig port channel in the real world. Most people simply don’t push that much data at the same time all day.
Obviously in a telco/carrier network you’d see more traffic but I’m running an office campus.
Also worth noting that just because a firewall/router may have dual 10 gig in it, doesn’t mean you can route that fast. It costs a lot of cpu (and adds latency) to push that much data through a software router with firewall rule set etc.
This is why layer 3 switches are normally used for that. Because they’re routing Ethernet to Ethernet without detailed packet inspection they can take some short cuts for speed.
Thanks, everybody. I really appreciate all the advice. I’m aiming to minimize Inter-VLAN routing, but I want the 10GbE+ link from the core switch to the firewall just in case I can’t avoid it for something, and also to avoid any unnecessary bottleneck with things like Crowdsec or Suricata.
Also I want to run a local iperf3 server on the firewall that can do 10GbE so I can test my connections at full throttle.
What I’m taking away from this is that I probably don’t need 2x10GbE LACP now but going ahead and setting it up won’t hurt anything. I’ve got more 10GbE devices coming online soon, so who knows…I might actually want it later. I’ve never set up a LACP interface in OPNSense and it sounds it won’t hurt anything, so it’ll be good practice, besides. Worst case, my box just won’t be able to push enough traffic to take full advantage of it.
Everyone recommends Dell Optiplex 5040s for sub-$100 10GbE firewalls, and I grabbed this one with 16GB RAM and with the SFP+ card for about $125, so I’m interested to see how far I can push it. I’m already very pleased with how quiet it is and (bizarrely enough) how easy the Dell 2015-era BIOS/UEFI (which got updated April 2022, surprisingly) is to use. At least if I outgrow it at some point (or more likely, it dies because it’s nearly 10 years old), I can reuse the cards.
I picked up an Intel chipset 2.5GbE NIC to use to connect the firewall to the WAN, so I can have both SFP+ ports available for the LAN. This one: QXG-2G2T-I225 | Flexible, fast, economical | QNAP (US) . It’s got the Intel Ethernet Controller I225-LM chipset.
That’ll also free up the 1 GbE port on the motherboard, so I can do an initial setup with 2.5GbE WAN uplink and 1 GbE to the core switch, just to get OPNSense installed. Once everything is set up and working (after I’ve migrated from the Topton I’m using now), I’ll set up the SFP+ ports in LACP and switch the WAN over to that. That should let me get away with minimal downtime.
Haven’t done LACP in pfsense, but generally speaking you configure LACP on your switch/OS and it uses LACP frames to negotiate the building of the port channel.
There’s not a HEAP to it, the main thing will be determine the configuration terminology for pfsense/opnsense vs. your switch (or Windows or whatever) for configuring LACP.
for a Cisco switch for example (e.g., the device you’d maybe have your pfsense building the LACP port channel to), LACP is as easy as this (3 interface example - this example is from a stack of Cisco 9200s back to a 4507 chassis switch as per the interface description):
On cisco if you configure the member interfaces as trunks first, and then channel-group X, when you update the port-channel1 interface it will apply the VLAN trunk permissions to the interfaces that are a member of it.
The “channel-group 1 mode active” statement binds an interface to the port-channel1 interface and sets “active” mode which in Cisco parlance means to actively use LACP without detecting the other end is LACP. Cisco has other options as per:
per-9200-stack1(config-if)#channel-group 1 mode ?
active Enable LACP unconditionally
auto Enable PAgP only if a PAgP device is detected
desirable Enable PAgP unconditionally
on Enable Etherchannel only
passive Enable LACP only if a LACP device is detected
PAgP is from memory a cisco proprietary alternative from before LACP was a standard.
By design, if only one of your interfaces works, the port channel should be “up”, as it is designed to tolerate individual link failures.
Lacp link aggregation will not double the speed for a single connection (client).
For instance if you have an lacp connection of 2x10g, the maximum throughput a single client will be able to see is the throughput of one link, so 10gig.
However a second client will be able to use that other 10g available, up to the maximum speed of a single connection.
It’s useful if you have many many clients, but even then unless you’re a carrier probably more likely useful for fault tolerance.
I use port channels as standard practice on all my workgroup switches to account for SFP failure, fibre patch lead failure/dislodgement, etc. Much nicer to just get an alert that an interface is down than have 100-150 users stop working.
Not so much for speed.
For reference, on my switches/stacks of ~120-150 office users each, the average port channel bandwidth back to the core on a dual 10 gigabit ether channel is peaking around 200 megabytes/sec over 5 minute polling cycles. I.e., 2 gigabit uplink seems to cover 100-150 “general office users” just fine. Their workload is very bursty and unlikely that many users are hitting the network at the same time.
Of course, YMMV (a bunch of video editors will be 100% different for example) - but if its for things like storage over IP (iSCSI, NFS ,etc.) then avoid routing it in the first place. Create new interfaces on the same VLAN dedicated to storage for example.
Thanks again, everyone. I ended up spending today chasing down poor performance (I was only getting 5 Gbps with iperf3) before realizing I needed to enable MTU 9000 on OPNSense’s LAN interface and my 10Gbps machine to actually get correct results, but at least I know it’s all working.
My intent is to use VLAN tagging to make sure the core switch will be able to handle LAN 10Gbps traffic without involving the firewall, but I wanted the 10Gbps LAN link just in case I ended up doing inter-VLAN switching because something was set up wrong or didn’t support VLANs or whatever.
Everything is working and I’m tired of messing with it, so I’ll try LACP again next weekend just to see if I can make it work. Still think I probably don’t need it. 10Gbps LAN traffic shouldn’t hit the firewall ever (unless I’m running a speedtest from a client to the firewall’s iPerf3 server) if I do it right.
Inter vlan routing is great if you want to route at full link speed between subnets (if using a capable enough layer 3 switch of course)
Just be mindful that you are effectively bypassing your firewall between those subnets when using your switch as a gateway, unless of course you setup ACL rules on the switch.
This is correct. If you’re using both a firewall and a switch to do such things don’t give your switch a VLAN interface directly on the networks that shouldn’t have high speed access to the storage network for example.
Create a point to point subnet between the switch and firewall and make such traffic do two hops through both firewall and router.
This sort of thing is why in the bad old days you would have seperate physical stitches for storage etc.
In fact, setting up LACP is probably the one “advanced networking” thing I’ve really done so far.
