I have a cable modem providing WAN via DHCP to my ProxMox / pfSense box with 6 onboard NICs at the front of my network. pfSense then feeds my Netgear WAX206 AP.
NIC 1 is an offline management port (NOT passed through to pfSense) with a statically mapped ip address so I can access ProxMox (via ethernet cable plugged directly from my laptop to NIC 1) when I screw something up.
I have NICs 2 thru 6 passed through directly to pfSense via PCI.
What I cannot seem to understand / figure out is how to connect ProxMox to the internet through pfSense so ProxMox can still get updates and access time servers, etc. without being exposed to the open internet or going un-filtered.
Nearly all of the guides and examples online use virtual linux bridges in ProxMox with bonded ports. I cannot seem to grasp how to achieve the same secure, segmented network with EVERYTHING going through pfSense while using direct PCI passthrough AND no other physical hardware switches in the mix.
It depends on how you want to do it. You can create a new bridge and bridge it to a virtual interface that you add to the pfSense VM. Then in pfSense, you make a new subnet, assign an IP on the virtual interface in pfSense, it should show up as OPT5 or OPT6 and you enable DHCP on it. Proxmox will then grab a DHCP address via that bridge if you enable DHCP on that bridge in /etc/network/interfaces.
In VMs, you can also bridge their virtual interfaces to that proxmox bridge, so they can also get IP addresses from pfSense and you will have a completely segmented network for proxmox.
However, that is going to allow VMs to access Proxmox too, which you probably do not want. Or you enable the Proxmox firewall, so you block access to it from the VMs, but then you have to maintain 2 firewalls.
What I would do is give port 2 back to proxmox and only use ports 3 to 6 in pfSense. Then you wire up port 2 to your switch. You can create a trunk, so that you get all the vlans on the switch on proxmox as well and you can assign VMs to vlans where proxmox does not have an IP address.
Let me know if you need a more detailed explanation, or if you got the gist of it and can manage.
CURRENT SETUP:
NIC1 → PxMx MAN
NIC2 → pf WAN
NIC3 → pf LAN
NIC4 → OPT1 (un-used, miniPC)
NIC5 → OPT2 (un-used, pocketPC)
NIC6 → OPT3 (un-used, rasberry pi)
NOTE:
I also have an HDMI output going to my main 4k monitor with PiP and a bluetooth keyboard for CLI access / fixes in my ProxMox host as well.
correct:
I do NOT want any VMs I set up to have any sort of access to my ProxMox host
I do NOT want to have to mess with ProxMox’s built-in firewall either
OK, so that would mean NIC3 becomes my pf WAN?
I do not have any actual (physical hardware) network switches. I am assuming this would be a virtual device (VLAN?) on ProxMox, correct?
If I am following you correctly, the NIC2 that I take back from pfSense and give to ProxMox will become bonded to a “linux bridge” in ProxMox with neither an ip address nor a gateway assigned to it. From there, pfSense will receive its WAN connection via VLAN and the remainder of my network segmentation will occur via VLANs?
Dang it, I read Netgear and I did not finish the line, I didn’t see it was an AP, not a switch. This changes things. Unless you are willing to buy a switch.
I would say you probably should get one, you will learn more about layer 2 security and vlans, which is nice. A used enterprise managed switch with 24 Gigabit ports should not land you more than $50.
Scrap that for now, we’ll come back to it once you buy a managed switch.
Ok, so you only have router ports. At this point, without a switch, the only way forward I can see this going is as follows, besides leaving everything as it is:
make a Linux bridge in Proxmox
add 2 VLANs to it: vlan2 management and vlan3 vm-net, you can add more vlans later
create a virtual interface in proxmox under the pfSense VM and bridge it to the Bridge interface, not to the VLAN interfaces
in pfsense, set that virtual interface for vlans as well, configure 2 vlans on it, same vlan ids 2 and 3
for each vlan interface in pfsense, assign a subnet
go back to proxmox and assign an ip address to one of the vlan interfaces based on the subnets assigned in pfsense on the same vlan id, in this case 2 for management
whenever you create new vms, you add their virtual NIC to the vlan 3 interface and all vms should automatically be in vlan 3 and not have access to the ip address that proxmox has on the management vlan interface
I am currently downloading proxmox, I will make a proxmox VM on my hypervisor and try to guide you there. I kinda needed a test platform to show others where to go, I’ll make a pfsense vm inside the proxmox vm to emulate your environment, hope it works.
Ok, have been testing a bit. I’m using virt-manager, but proxmox is using the same qemu+kvm in the backend, so they should have similar quirks.
You can’t add a virtual interface to a vlan interface directly. But the solution is simple, create another bridge and make the vlan the master of the bridge, or in other nomenclature, probably in proxmox too, bridge the bridge to the vlan interface. Then whatever VMs you want to assign to vlan3, you bridge their interfaces with the bridge connected to vlan3. That’s what I did when I added proxmox in my separate subnet.
I have vlans coming to my server, but I have not taken the time to migrate my VMs to separate networks. I will configure my interfaces later. In the meantime, proxmox is installing.
I look forward to seeing what you are able to come up with in your experiments and appreciate you taking the time to explore this with me. I am new to a lot of this, but enjoy learning about it.
I have to get to bed and ready for work tomorrow, but I will be around and reading up on whatever I might see posted here. TGIF! If nothing else, I hope to get this sorted over the weekend when I have more than an hour to sit down with this.
I will talk to the budget troll about getting a switch, but I am about out of power plugs, splitters, and extensions too, lol.
As a quick aside, if my pfSense network scope is from 192.168.X.2/24 to 192.168.X.254/24 and I have my NIC1 PxMxMAN port statically mapped to 192.168.X.1/24 (no gateway) …
Is that a security issue?
Is it possible to set the gateway of the PxMxMAN port to the pfLAN 192.168.X.2 and allow traffic to pass via firewall rules, or (I presume) is that the same as exposing my host ProxMox system to the internet with no protection?
There is no security issue as long as there are no malicious hosts on the network… the layer 3 (networking level) when it comes to a single subnet is not really securable, that has to be done via local firewall rules on hosts. Which is why subnetting and router firewalls are done in the first place, you kinda want to avoid mixing potentially malicious clients with things that you trust.
I am not sure I understand what you are saying. Let me check:
you have Proxmox port 1’s IP address manually set to 192.168.x.2
you have the pfsense LAN’s port set to 192.168.x.1
Assuming X is equal, i.e. the same on both interfaces, it’s not a security issue at all. The 2 ports are split at the layer 2 (data link level). Port 1 cannot access port 2.
This question makes me so confused.
I will explain how you can set it up after I finish setting up a client to use to connect to pfsense.
The way it works logically is as follows:
internet <—> pfsense WAN - 69.69.69.69/32
pfsense <—> port 3 = LAN1 - 192.168.ab.1 in 192.168.ab.0/24
pfsense <—> port 4 = LAN2 - 192.168.cd.1 in 192.168.cd.0/24
…
pfsense <—> virtual port = split into 2 VLANs - no IP
virtual port <—> vport vlan2 (management) = LAN N - 192.168.mn.1 in 192.168.mn.0/24
virtual port <—> vport vlan3 (vm-net) = LAN N+1 - 192.168.op.1 in 192.168.op.0/24
Proxmox will be staying in LAN N and will pretty much have its own network, split from everything else, just like you have OPT1 for miniPC and OPT3 for Raspberry Pi. Each of the ports (NIC 4 and NIC6) have their own IP address set on pfSense, and their own subnets, so each device connected to this port will get a different subnet.
If you leave everything to DHCP on all interfaces, your devices would look something like this:
miniPC = 192.168.cd.2
pocketPC = 192.168.ef.2
raspberry pi = 192.168.gh.2
proxmox = 192.168.mn.2
vm1 under proxmox = 192.168.op.2
vm 2 under proxmox = 192.168.op.3
vm 3 under proxmox = 192.168.op.4
See the pattern? The way you configure pfSense then would be to allow or restrict traffic between those ab, cd, ef etc. subnets. Because you are using a router and not a switch, you cannot place the miniPC, pocket PC, RPi and Proxmox on the same subnet. But using VLANs (bridging to be precise, which is basically what a switch is, a bridge to a “pool,” a.k.a. a subnet), you can have the VMs under the same subnet.
But man, is pfsense such a junk OS. They make handholding be very g**. I only created 1 virtual interface to give it, but I knew from the get-go to make 3 vlans, because pfsense won’t allow me to not set a WAN, so I set vlan4 to WAN. But the script was so dumb that it just got stuck waiting for WAN to be configured. No pfsense, that’s not what I want. I don’t have DHCP or anything on the WAN, just pretend it’s another LAN. I want a goddamn local network, segregated from the internet, stop thinking that everything needs a wan! I am so glad I am not using those handholding OSes anymore.
Anyway, excuse the mini-rant, I had to start with shttalking pfsense. Now to continue shttalking proxmox.
Apparently proxmox is dumbly programmed to force people to set vmbr as the name of virtual bridges. I didn’t realize this before, because long ago, I was just using /etc/network/interfaces, because it’s been just easier in my environment. I tried using the GUI, so to eliminate variables and help you configure it. But this is stupid. Sure, we have comments, which I made sure to use here, but that should not be a problem to begin with. Anyway, onto the real tutorial now.
Ignore vmbr0. Pretend that it’s the bridge to your port 1 for out-of-band management.
Create a new bridge, make it VLAN aware. I called it vmbr10, because proxmox doesn’t allow me to name it something sane. Do not attach it to anything, it’s just a bridge floating in the sky.
Create a vlan and name it exactly after the bridge name, that is the convention that you should respect. I named it vmbr10.2, so it is bridge vmbr10 tagging the VLAN 2 tag onto layer 2 frames. Then I created a new vlan for vmbr10.3 for VLAN 3.
After that, because you cannot attach VLAN interfaces to VMs, I have created a new bridge for each VLAN. I named them vmbr22 and vmbr33 and set them to bridge vmbr10.2 and respectively vmbr10.3. Try to follow along and use the screenshot for sanity check.
Note that on vmbr22 I have set a static IP address. You want that for the management VLAN, so that you can access proxmox from other vlans connected under pfsense. You do not have a DHCP option for those in the GUI, so static IP is your friend (besides, you really don’t want the proxmox IP to change willy-nilly).
Then you go to your pfsense VM and create a virtual NIC and bridge it to the main bridge, the one floating in the sky (vmbr10).
Congratulations, you just bridged proxmox to pfsense. But don’t celebrate yet, it’s just a switch right now, with no routing. We need to set that up as a trunk port in pfsense, so pfsense will tag the layer 2 frames with the VLAN tag.
You go in pfsense under Interfaces → assign → vlans and you +add new vlans to the freshly created virtual interfaces. This will set up tagging on frames like mentioned above.
Screenshot for sanity check. My virtual interface is named vtnet0, I suspect it would be the same for you if you use the default virtio network interface driver, which should give you a virtual 10GbE.
Screenshot just for the lulz, the wan is another virtual nic, vtnet1, I had to reinstall pfsense because this junk was stuck in an infinite loop waiting for an IP for WAN that would never come.
After you finished with the vlans, go to interfaces → assignments → interface assignment, select the vtnet0.2 and hit +add. Repeat for vlan3.
Then you go to system → interfaces → OPT N (whatever you get for vlan2, I got LAN as you can see). Enable the interface, set an IP address and a subnet (or leave it to /24) and hit save. Go to OPT N+1 for vlan2 and do the same. Then hit Apply Changes.
You are pretty much set now. Here’s a test VM placed in vmbr33.
As you can see, VM has 192.168.3.100 IP address, gotten from DHCP (I keep my pool between 100-200, because why the heck not, I usually reserve at least the first and last 10 IPs, but in test networks with less than 20 VMs, I reserve the first 99 and last 54 IP addresses - if I want to assign a static IP, I do it from the pools of reserved IPs for non-DHCP / static IP / DHCP MAC mapping assignation).
The reason why I cannot ping it is because, by default, a new OPT has no rules, so it can’t go anywhere, so all are blocked by default. I added some sane rules on the VM subnet for things that you would normally not trust.
block from vlan3 any to vlan2 any
allow from vlan3 to firewal (self) udp port 68
block from vlan3 to firewall (self) any
allow from vlan3 to any (note that without other restrictions, this will allow vlan3 to hop onto other subnets, like the pi subnet for example, but also allow it to go to the internet)
Cheat: you can go to http : / / pfsense / status.php#pfFirewallRules to view a summary or a lot of stuff on a single page.
Now I feel reminiscent when I was a noob myself. Gotta read my old posts tomorrow, want to look back on my cringe clueless moments. I still remember when I was asking for a solution to get my 2 router ports to be part of the same subnet (without realizing that later on, I would love that segregation). Back then, I didn’t know about bridging, so I was asking how to do it using VLANs. Big OOF.
That pfsense box is still going strong today, I am very sad that my ISP is such a horrendous company and I can’t use my old router (which is still way overkill, more powerful than my current one for sure) for other purposes, because I can’t put the ISP router in bridge mode (because of their buggy PPPoE server) and because I’m locked out of the management interface (with the router being unable to factory reset after requesting it be put back in NAT mode). Enough rants for the day, I need to get some sleep, I also work tomorrow.
Here is what I started with when I first posted this topic. pfSense was providing internet to my AP and our home devices were working fine, but ProxMox had no access to the internet for updates, etc.
I was asking if ProxMox could reach the internet, through pfSense, by adding a gateway address of 192.168.48.2 to the MAN port enp1s0 (in the picture above)?
This makes sense to me.
This makes sense to me as well. Though I do actually have all of my permanent home devices statically mapped in pfSense since I was getting annoying app notifications of devices going online and offline multiple times per day. The recommendation of statically mapping them fixed the annoyances.
Makes sense, still following.
Should I use NIC1’s actual real hardware MAC address here?
My ProxMox box’s real hardware NIC addresses are:
NIC1: xx:xx:xx:xx:xx:0A
NIC2: xx:xx:xx:xx:xx:0B
NIC3: xx:xx:xx:xx:xx:0C
NIC4: xx:xx:xx:xx:xx:0D
NIC5: xx:xx:xx:xx:xx:0E
NIC6: xx:xx:xx:xx:xx:0F
For any duplicate / virtual devices I bump the final number by 1 and keep the same ending letter so I can keep track of which physical NIC port the virtual / fake MAC addresses are meant to represent.
I should not have to mess with adding a temporary vtnet1 because my cable modem automatically issues a DHCP WAN address to my pfSense’s WAN connection, correct?
I suppose I better post this insane reply now, just in case something doesn’t work out exactly right, and my wife knocks me out for taking down the internet (yet again), .
I can’t thank you enough for your (warm welcome) time and effort you put into quickly responding to my noob questions that pretty much anyone else would have just told me to stick with the ProxMox virtual adapters rather than bothering with PCI passthrough on a nonsensical setup such as mine, but damn it I am a stubborn OCD type that seems to enjoy doing things the hard way whenever the opportunity presents itself. /s
Correct, that’s what I had to do because pfsense would not allow me to ignore the WAN setup.
No, just use a random MAC address. You do not want MAC duplication on your network, or at least in the same subnet, that would be bad.
The interface assignment looks right.
The VLAN2 and VLAN3 interfaces, you want to assign an IP address and a subnet to each of them, otherwise devices in those VLANs will not be able to get out of them. Just check your own LAN (igb1) configuration and emulate it. Just do a different subnet.
It cannot ping to the host bridge ip address (same as yours) but it seems to be able to reach DNS at 1.1.1.1
However, it just stalls when trying to ping google.com
Correct. The port will be in “mode access” (meaning no vlan tags are allowed, just like a dumb switch connected to the pfsense vNIC) and pfsense will take care of firewalling. Again, keep in mind that devices in the same subnet can talk to one-another, so you want to segregate them in things you trust or want to be in the same subnet, or if not segregated, have a local firewall on each of them (technically, Proxmox’s firewall helps you centralize all those rules that would be on, say, iptables or windows firewall on each VM). I never use the firewall option in Proxmox, I have a sane network segregation.
So it has access to the internet (good), it doesn’t have access to proxmox (good), but doesn’t seem to have DNS (bad). Configure a DNS on the VM (idk, 1.1.1.1), or better, you should get DNS from DHCP, so set a primary and secondary DNS server in your DHCP server on pfsense and allow Firewall from VM-net (I suppose vlan3) to This Firewall (self) port 53 (DNS port).
I suspect your VM can’t access DNS on the pfsense and pfsense is set to be the DNS in your DHCP server. If you set another DNS server locally on the VM, you should be able to ping by domain names. But best to just allow DNS from vlan3 to pfsense.
You can ping the internet, 1.1.1.1, can’t you? You have access to the internet, you just can’t query the DNS server configured on the VM, which is most likely Firewall (Self).
So before the rule Block from VLAN3 to Firewall (self) ANY, add Allow from VLAN3 to Firewall UDP port 53.
@maleko48, don’t be so hard on yourself; if networking were easy, companies wouldn’t have to pay network system admin good money; they could get a secretary to do their job and pay them a lot less.
Thanks for creating this thread; I will be installing Proxmox pr Vmware EXSI on my spare computer as soon as I can replace damaged parts when that computer’s power supply explodes.
Unfortunately this did not fix my ProxMox host not having internet access. I suppose I need additional rules similar to VLAN3 eh?
I will be back to tinker more shortly… I gotta go grab dinner for the family now.
I did make this nifty diagram of my home network though. Any input is welcome, it’s my first network diagram ever, lol. Definitely took me way too long to make.
Your diagram looks better than my first network diagram. Well, I can use fancy software, but I just love to keep it cringe and use (kolour)paint, lmao.
Keep in mind that when doing any new OPT network, pfsense has a deny by default, with the only exception being LAN. So if you don’t add rules to allow stuff to get out, you are confined to that subnet.
I would rename the OPTs in pfsense to something more decent, like “wifi,” “vm-net,” “management,” “proxmox” etc.
After staring at my network diagram (updated above) and pondering it in my sleep, I was able to finally get ProxMox out to the internet for updates / etc by adding a gateway to vmbr22:
I figured it wouldn’t hurt to give both bridge interfaces a static ip addresses? Hopefully someone will tell me if this is a bad idea. (Since internet connectivity required at least 1 of the bridges to have a CIDR address defined in ProxMox).
I had thought that just having a VLAN gateway and no CIDR address defined should have been enough to allow pfSense’s DHCP server to issue an ip address from its pool to ProxMox host and VMs, but I guess not.
Even though there can only be a single gateway for ProxMox, I tried flipping the gateway to be the vm-net’s gateway of 192.168.240.1 on vmbr33 and that seemed to work as well, maintaining internet access for both VLANs.
Though I suppose there might be some changes to the firewall rules required to address that? I need to ponder this all some more.
I am still mostly just poking and prodding, messing around a bit to see what does what and breaks what and who can talk to who.
My ProxMox install still appears to have its firewall switched “on”, though I have not touched anything in there:
pfSense’s ProxMox firewall is switched “off”:
All said, my pfSense’s hardware tab ended up as such:
Now onto the next goal:
getting my mPCIe Atheros AR9280 WiFi card to work in pfSense.
Right now if I try to include it in pfSense’s hardware tab, pfSense fails to start and gives me exit code 1. I have read a little bit about it, and it seems I need to blacklist ProxMox from initializing the WiFi card so I can fully hand it over to pfSense, does that sound right?
You supposedly don’t want your VMs to access Proxmox. So why would you put proxmox inside the VM-net?
Better to remove that static IP from vlan 3.
Missed that. Yep, makes sense.
No, it should be enough. Look at my settings, I had no IP address on vlan 3, yet the Fedora VM got an IP just fine while in vlan 3.
It’s fine, won’t impact anything as long as you don’t enable it at the VM’s vNIC settings.
Sounds about right. Use the lspci to find the ID of the PCI-E card. Might be a problem if the PCI-E is connected to your motherboard’s chipset though.
I don’t have a guide for proxmox, I don’t remember how to blacklist it in mkinitramfs.
If ProxMox’s management bridge and the vm-net bridge are both in separate networks (192.168.28.X/24 vs 192.168.240.X/24) and also separate VLANs, why does that pose a risk exactly?
(I went ahead and removed the static ip at your suggestion, just curious.)
Yeah, my VMs seem to be able to get their DHCP ip addresses without a CIDR address set for vmbr33, but my ProxMox host was not able to get out to the internet without both a static CIDR ip address defined for vmbr22 AND at least 1 interface containing a valid gateway.
and finally I ran update-initramfs -u -k all as instructed then rebooted
according to PCI(e) Passthrough - Proxmox VE since lspci -nnk is showing Kernel driver in use: vfio-pci I should be able to pass it through to pfSense as a PCIe device but pfSense won’t start and gave the following error:
kvm: -device vfio-pci,host=0000:07:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,rombar=0: vfio 0000:07:00.0: failed to add PCI capability 0x11[0x70]@0x90: table & pba overlap, or they don't fit in BARs, or don't align
TASK ERROR: start failed: QEMU exited with code 1
I have not yet attempted any PCI device ID overrides yet, switching from SeaBIOS to OVMF, allowing unsafe interrupts, etc. but I did find some threads with similar issues as mine:
so naturally I added args: -set device.hotspci5.x-msix-relocation=bar2 to my /etc/pve/qemu-server/100.conf file after adding the wireless card to pfSense’s hardware tab as a PCIe device, which allowed pfSense to boot up with the wireless card added to the hardware tab…
Here is my /etc/pve/qemu-server/100.conf file’s contents:
VLAN 2 and 3 are separate, so no risk there if you restrict traffic on your router from vlan 3 to vlan 2. The problem is that, if you add an IP address to proxmox on vlan3, the same vlan that your VMs are using, then the point of having a separate management vlan is moot, because now in vlan 3, VMs and proxmox are communicating directly, bypassing the router.
Well, in a home network, depending on what you planning to run in your VM network, it’s not that big of a deal, but it’s a nice learning experience.
Yes, this of course doesn’t work. You need both IP and a gateway on vlan2. So on brige22, set the IP and gateway for the management vlan. So configure 28.2 as the IP on vmbr22 and the IP 28.1 as gateway.
What I told you to remove was the IP 240.2 from vmbr33, as that was in the same subnet as your VMs.
The passthrough seems fine on the proxmox side. Never seen that bug before, not fitting in BAR or aligning.
You need to keep using q35 if you want PCI-E passthrough. And likely UEFI boot.
Never mind, as long as it works.
Go into shell in pfsense and run dmesg and see if you see any complaints about drivers. Your WiFi card might not be supported in FreeBSD yet.
Should I use NIC1’s actual real hardware MAC address here?
Boy do I feel dumb rn. 
