So, I am about to start on the homelab path so wish me luck. However, had a thought which led to a question:
If I am doing a pfSense build and I have multiple networks going up to my pfSense machine, if I am making that my gateway/firewall for all of my networks isn’t that going to choke the traffic? Even if I bond ports together (1 or 2.5gb) won’t that still be a bottleneck to all of my inter-vlan traffic?
For example: Say I have a TrueNAS machine running with dual 10GB connections (I have an Arista 10x48, 6x40gb switch), if it is on a different lan than my client PC machines its going to have to truck all the way up to the pfSense machine, get routed back down the tiny trunks and over to the client? Granted, I probably would have the management interface for the TrueNAS on a different net/nic but basically my point/question is won’t this choke the intervlan traffic?
Yes it would for anything that needs to be routed. If you are moving lots of data between different VLANs then you’re going to need a router with interfaces that are sized to handle it. Also keep in mind that if you are going to be applying firewall rules or other processing on the traffic, you’ll need quite a bit of CPU power to handle that piece as well.
One of my side projects is going to be digging into vmware more so this would be a good excuse to do just that. May some micro segmentation with vmware NSX as well. Regardless, thanks all
Welcome to the circus, you just got to the proverbial can of worms
The answer is ‘it depends’
The main question you need to answer yourself before everything else, as @Ghan said, is: ’ do I need to apply firewall rules to my inter vlan traffic’?
If the answer is ‘no’, or ‘I can get by with access lists’ , and your Arista can do layer 3 routing, then you don’t send your vlans to the pfsense, and have the switch route between internal VLANS, while applying access lists, and only use pfsense to handle NAT/traffic towards the WAN
If on the other hand your answer is ‘Yes, I want to packet filter between my VLANS, and maybe apply IDS/IPS/TLS transparent decryption and such’ then you pfsense won’t be able to do that at 10Gbit/s, ever, and you need to go either with a super powerful hypervisor host onto which you install a routing platform that can do firewalling (Vyos) and then apply to it the IDS/IPS part using whatever appliance you want, or with a super expensive dedicated hardware (enterprise gear that can firewall at 10Gbps starts at 5K, and quickly goes over 10-20K plus licenses if you want to add IDS/IPS) …
Most of the homelabbers (@wendell probably being the exception, but he has access to a different level of hardware) just route between VLANS that need high speed traffic, and firewall the rest …
The thing is, being a homelab and working in IT myself, I can’t see myself ever NOT doing something with this. I am going to want to do everything with it. Even if I am not getting my full speed out of my network I still plan on doing things with it eventually.
Pretty much everything. My budget can’t afford “everything” so I will probably end up with running things off of a underpowered vmware host and hating the results for about 2 years until I can buy or build what I really want.
No you’re not ,
over the years the only thing that improved for me is some rationality over the amount of money I allow myself to spend in ‘research projects’ of this kind, and have learnt not to take too many hard stances, like ‘never avirtaul firewall’ or ‘never use routed vlans’ or ‘only buy datacenter grade equipment to make my homelab work’
That has eased the pain in the wallet, somewhat.
Specifically to the 10Gbit routing/firewalling … been there/done that last year/this spring, and went with a virtual vyos firewall/router that can do 20Gbps routed, 6/7Gbps firewalled, but kept the 10Gbit stuff on a single VLAN as I do want to apply firewall rules between my VLANs (home-secure,IOT,home-insecure, WAN, STARLINK and homelab) …
This bottle neck exists, and there are really three ways to work around it.
Put all your high speed devices in one subnet so your router won’t be the bottle neck (NAS and desktop for example).
Run a layer 3 switch for fast routing, with OSPF routing to pfsense.
Upgrade your firewall hardware with a 10gig port, and use tagged vlans to your switch.
I’ll also say that you don’t need to get it right at the start. One of the best things about a homelab is no business requirements, so you can make compromises to get a better setup.
We lost power yesterday for 30 minutes (they’re digging to expand the nearby intersection to add bicycle lanes), and wifi went out and didn’t come back until an hour later due to motherboard UEFI getting reset. (It’s complicated)
I’ll be hearing about this for a month.
This got me thinking, is there a decent protocol for 4-tuple routing?
If not, could pfsense announce source based routes for whitelisted set of stuff into the Arista to use Arista as an accelerator?
(… or make pfsense log when it allows a flow, … and have something syphon the log record and send it to some fib on Arista… however the Arista API looks like)
PBR would be the most relevant thing here. Essentially an access list, where the action is setting a new gateway. But I’m not sure how many layer 3 switches would support PBR.
That kind of sounds like what Software Defined Networking was actually supposed to be. But realistically if you want a device to route a flow, it needs to go through that device. Basically all networking is built that way.
Personally I’d say put a 10G interface on the firewall. 10Gbit is symmetrical, so you can route one flow at 10 Gbit succesfully. Trunking all the vlans through means that the 10 Gbit bandwidth is shared between all the vlans too.
I guess you could also use stateless access lists with layer 3 switches. They mostly work, but they’re a pain when you want one way access, especially for UDP.
,