I am building a new Proxmox server for my “home lab” and pfSense will be one of the VM’s. It will serve as my main router/firewall. I am hoping to achieve something that I am not seeing examples of and just want to see if anyone can confirm or deny if it is possible.
Ok here it goes…I want pfSense to have more than 2 nic ports where the additional ports would essentially work as a switch. Much like it would on a traditional consumer router with 4-8 ports.
So I would have:
-1 x 1Gb port for wan
-1 x 1Gb port for lan
-2 x 10Gb fiber ports as “switch ports” (Mellanox ConnectX-3)
-2 x 1Gb port as “switch ports” (Intel I340-T4 or T2)
The goal is to eliminate the need for another switch that can handle 10Gb and allow me to connect my work station (has 10Gb fiber) to the proxmox server. I also have a Truenas server with 10Gb fiber that would connect to the Proxmox server.
So my work station and Truenas servers would be connected to the 2 10Gb fiber ports which would be handled by pfSense.
1 of the Gb “switch ports” would go to one of my switches and the LAN port to another with one “switch port” to spare.
Is there any reason to think this wouldn’t work as I’m expecting?
I’m also unclear on if I should passthrough NICs from Proxmox to pfSense. Would that provide the best performance between Proxmox, Truenas, and workstation? I want my workstation and Truenas to be able to have a 10Gb connection to all VMs on Proxmox.
Well, the reason you want the switch is because these devices perform the “switching” (moving data between ports, evaluating firewall rules, monitoring, etc.) in ASIC as opposed to generic CPU hardware. This is much faster and should be more power efficient (even using decade old enterprise switches).
You’ll find that the hw on the pfSense box may be the limiting factor when switching 10gb network speeds.
I hope folks on this forum with more hands-on experience than me can chime in with examples, recommendations and such.
Thanks! That’s something I hadn’t considered. I knew that 10gb would likely tax the pfsense vm harder than my old pfsense vm with only 1gb but I never thought it would be that much more work. I assumed maybe 10x CPU usage over the old one.
With my new CPU Xeon E5-2650Lv3, I would be surprised if it can’t handle it. Of course the idea of this was actually to save power and a bit of money. If it’s really pushing the CPU then that may defeat the purpose.
You can’t really passthrough network ports, you can passthrough devices which could be network devices, and if your host supports sr-iov you can take a physical device in proxmox and have it create additional pcie devices, that you can then passthrough separately to VMs.
FWIW, unless you want filtering on your switch, I’d keep the bridge on proxmox, and would hookup pfSense VM to that bridge using just a regular virtual nic.
Anyway, if your 10G nic supports bridging ports in hardware, do that, and then in proxmox bridge the 10G interface, and the two 1G interfaces, and a virtual nic for pfSense. At least that’s what I’d do.
Thanks for the input. I really appreciate it everyone!
@risk Thank you so much for this. You probably saved me hours of trying different configurations and testing performance. I had a sneaking suspicion that maybe it would make more sense to do the bridge through Proxmox and add that bridge to the pfSense VM.
Presumably I can passthrough my network device(s) to pfSense. I’ve got a Mellanox ConnectX-3 (dual port) for the 10G and can use either a Intel I340-T4 or Intel I340-T2 or both as well.
I’m guessing that if I passthrough a NIC to pfSense that traffic going over that device can’t pass internally between VMs. So if I were to passthrough the Mellanox (10G) to pfSense then the Trunas server and workstation would be able to communicate at 10G but Proxmox wouldn’t be able to talk to either at 10G because it would have to go over 1Gb connection.
This is something I never considered a possibility. Looks like I have something new to research.
Just wanted to report that I got this setup and it is working exactly as I had hoped. Thank you everyone!
I ended up creating three new bridges in Proxmox 1 for the two ports on the 10Gb Mellenox card and another for 2 ports on an intel i340-t4 (4 ports) that I ended up using. 1 port on the i340 was put in its own bridge for the WAN on pfsense. The remaining port on the i340 was added the the initial bridge that Proxmox sets up by default.
This allowed me to have i340 port for WAN. The LAN then became a switch of 2 x i340 ports + 2 10gb Mellanox ports in pfsense.
I created separate bridges in Proxmox for the 2 ports on i340 and 10Gb Mellanox because I wanted to leave the i340 with default packet size and have the 10Gb use Jumbo packets. Not sure if this was necessary but seemed like it may be a more stable approach.
The remaining i340 port was added to initial Proxmox bridge to add redundancy.