[SOLVED] Pass VLAN tags between pfSense VM and Managed switch

Hardware Hub Networking
, ,
1

My router (pfSense) is inside a VM on my server. I have 2 NICs. One of them is PCI passed through to the pfSense VM (this is where the WAN is plugged) and the other (enp5s1) is bridged with with all the VMs (including pfSense) (this is the LAN one that is connected to manged switch TL-SG108E).
So from pfSense POV I have 2 NICs. 1 passed through that the internet comes from (WAN) and 1 virtual (that is bridged with enp5s1) which goes to the switch (LAN).

Now I created a VLAN in pfSense and added it in Interface Assignments
VLAN Interfaces:

vtnet0 (lan) 	11 		Gaming VPN Network

Interface Assignments:

Interface            Network Port
WAN                  re0 (98:de:d0:83:00:3e)
LAN                  vnet0 (fe:54:00:ae:b2:2c)
VLAN11               VLAN11 on vtnet0 - lan (Gaming VPN Network)

In the normal use case where pfSense is running on it’s own device my switch would pick the VLAN (when I select the VLAN number it should display it’s name meaning it detected it) but in my use case it doesn’t.

How should I configure my host OS (Rocky Linux) so that the VLAN get passed through the bridge (so that my switch detects them)?

Here is my host net conf:
ip link (MACs are obfuscared buf if they’re the same originally they are the same here)

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: enp5s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP mode DEFAULT group default qlen 1000
    link/ether 90:e2:ba:11:ff:32 brd ff:ff:ff:ff:ff:ff
4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 90:e2:ba:11:ff:32 brd ff:ff:ff:ff:ff:ff
5: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:00:51:df brd ff:ff:ff:ff:ff:ff
6: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:00:51:df brd ff:ff:ff:ff:ff:ff
7: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether fe:54:00:bd:11:2d brd ff:ff:ff:ff:ff:ff
8: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether fe:54:00:3e:d1:82 brd ff:ff:ff:ff:ff:ff
9: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether fe:54:00:ae:b2:2c brd ff:ff:ff:ff:ff:ff
10: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether fe:54:00:cc:22:3d brd ff:ff:ff:ff:ff:ff

ip addr (MACs are obfuscared buf if they’re the same originally they are the same here)

3: enp5s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000
    link/ether 90:e2:ba:11:ff:32 brd ff:ff:ff:ff:ff:ff
4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 90:e2:ba:11:ff:32 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.50/24 brd 192.168.0.255 scope global noprefixroute br0
       valid_lft forever preferred_lft forever
    inet6 ...
5: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:00:51:df ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
6: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:00:51:df brd ff:ff:ff:ff:ff:ff
7: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:bd:11:2d brd ff:ff:ff:ff:ff:ff
    inet6 ...
8: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:3e:d1:82 brd ff:ff:ff:ff:ff:ff
    inet6 ...
9: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:ae:b2:2c brd ff:ff:ff:ff:ff:ff
    inet6 ...
10: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:cc:22:3d brd ff:ff:ff:ff:ff:ff
    inet6 ...

I found this post that suggests:

creating vlan sub interface attach to a bridge and connect to the pfsense respective vlan interface

Now if I understood it correctly I should do nmcli con add type vlan con-name enp5s1.77 dev enp5s1 id 11 master br0 connection.autoconnect yes. But when I do that all the devices connected to the switch (i.e. everything connected to enp5s1) lose connection. The server (pfSense VM host) is OK because it’s connected to pfSense VM via bridge (vnet2 → br0)

2

Ah, … so the trouble is there’s various options.

You could configure the VLAN “sub-interface” with a physical interface on the host, make a new bridge, add the VLAN sub-interface to the new bridge, make a new tap, add it to your VM (or point whatever you’re using to configure KVM at this new bridge).

The above is the “simple” approach.

There’s all kinds of macvlan and macvtap that you can do stuff with, I’m guessing you probably don’t care about performance minutia and isolation too much in this case and these other methods are more complicated. Linux bridges also support VLANs natively which can also be complicated.

3

Let’s pretend that your VMs are in vlan 77, then we do the following:

  1. Create bridge br0 with interface enp5s1 (already done).
  2. Add bridge interface br0.77 as type vlan with vlan id 77 in network manager.
  3. Move the IP 192.168.0.50/24 from br0 to br0.77, br0 will have no IP.
  4. Modify all your normal VMs to use the new bridge br0.77.
  5. Give your pfsense to use br0.
  6. Setup the switch port going to enp5s1 as vlan tagged.

The basic idea of the above is that br0 now carries VLAN-tagged frames, and in order to give them to normal VMs we need to create bridge sub-interfaces that tag the respective vlan. However pfsense is special in that it gets all the VLAN-tagged packets, like a regular router.

(I’d suggest saying away from macvlan and macvtap, there are some gotchas that make things annoying. Bridges are all you need.)

4

My idea was everything to be outside VLAN or in VLAN1 (this is a default VLAN that gets created by my switch when I enable VLANs with every port marked as Untagged). Then I’ll have 1 VM that will host OpenVPN server be on VLAN77 (for example). My pfSense firewall settings will block VLAN77 from accessing everything on default network in practice making an isolated VLAN (VLAN77) that anyone can connect to (even from outside via port forwarding) and communicate with other devices connected to that VLAN but unable to access the rest of my home network. And devices on VLAN77 will have some other network ofc like 192.168.10.0.
Won’t your proposed setup put everything on the network in VLAN77?

5

My proposed setup was an example, I didn’t realise you were using the untagged vlan. (FYI vlan 1 can be tagged, so be careful calling it that). In that case you don’t need to modify anything about your setup. The br0 bridge will forward any tagged frames directly to Pfsense from your switch, and you just create a new sub-interface in pfsense for vlan 77.

On your switch you’ll need to create a vlan 77, modify the port to be a trunk, set 77 as tagged, and vlan 1 as untagged + pvid/default.

Use “tcpdump -i br0 -n -n vlan id 77 -c 10” to test if br0 is receiving packets tagged with vlan 77.

In general though I don’t recommend mixing tagged and untagged vlans on a hypervisor, but that’s mainly when you have dedicated nics for hypervisor management and VM traffic. I’d also suggest not using vlan 1 at all, but no reason to change your setup if its working for you.

6

So I managed to get pfSense to work with the VLANs.
I set rules in pfSense for VLAN77 to allow traffic to destination VLAN77 network. In my switch (port 1 - pfSense LAN (NIC bridged to br0), port 3 - my PC (not the server that runs all the VMs incl pfSense), port 7-8 - test ports) I set:

  • VLAN1

    • port 1-8 - Untagged

  • VLAN77

    • port 1 - Tagged
    • port 3,7,8 - Untagged
    • other - Not member

  • PVID

    • port 1-6 - 1
    • port 7,8 - 77

In CentOS (VM host) I set on br0 (this has my IP):

  • bridge.vlan-filtering: yes
  • bridge.vlan-default-pvid: 1
  • bridge.vlans: 1-4094

Now the physical devices work as intended with VLANS. If I plug devices in port 7/8 they get IP in the 192.168.10.0 network and can ping other devices in that network but nothing outside. A device connected to port 3 (which is in 192.168.0.0 network) can ping devices connected to port 7/8 (in 192.168.10.0 network).

However do you know how I can create a VM (KVM/Libvirt) that is in VLAN77 and in 192.168.10.0 network? If I create a VM connected to br0 it connects to 192.168.0.0 network. I want all my VMs to be in 192.168.0.0 network (as they are now) except the VM I’m gonna host OpenVPN in (which should be in VLAN77/192.168.10.0). That way anyone can connect to OpenVPN but they can only talk to eachother.

UPDATE: One suggestion I keep finding is to connect the VM to br0 which should make the VLAN trunk available in the VM and there I should create a VLAN Interface but it doesn’t work: (this VM is Debian)

allow-hotplug enp1s0.77
iface enp1s0.77 inet static
    address 192.168.10.123/24
    vlan-raw-device enp1s0

allow-hotplug enp1s0.77
iface enp1s0.77 inet dhcp
    vlan-raw-device enp1s0

(Both cfg don’t get an IP)

7

You can create a new sub-interface enp5s1.77 on enp5s1 for vlan id 77. Then create a new second bridge br0.77, and attach it to enp5s1.77.

8

So I got it working. Now I have to figure why when I connect to the VLAN through OpenVPN it doesn’t work but I’ll open another thread for the OpenVPN config problem.

Here is my network config that works:
I disabled NetworkManager everywhere!

Server (Rocky Linux) / pfSense VM host

/sys/class/net/br0/bridge/vlan_filtering

0

/etc/sysconfig/network-scripts/ifcfg-enp5s1

DEVICE=enp5s1
UUID=5c593283-788c-4c66-a590-6fcc3a0e2a49
TYPE=Ethernet
NAME=enp5s1
ONBOOT=yes
NM_MANAGED=no
BRIDGE=br0

/etc/sysconfig/network-scripts/ifcfg-br0

TYPE=Bridge
DEVICE=br0
NAME="Bridge br0"
UUID=d2d68553-f97e-7549-7a26-b34a26f29318
ONBOOT=yes
NM_MANAGED=no
IPADDR=192.168.0.50
PREFIX=24
GATEWAY=192.168.0.1
DNS1=192.168.0.20
DNS2=1.1.1.1
BOOTPROTO=static
DEFROUTE=yes

OpenVPN VM (Rocky Linux): (Debian doesn’t work for some reason)

This can be 1 interface without the bridge. I just need the bridge to bridge enp1s0.77 with a TAP interface for OpenVPN.
/etc/sysconfig/network-scripts/ifcfg-enp1s0.77

DEVICE=enp1s0.77
ONBOOT=yes
BOOTPROTO=none
VLAN=yes
BRIDGE=br0

/etc/sysconfig/network-scripts/ifcfg-br0

TYPE=Bridge
DEVICE=br0
NAME="Bridge br0"
ONBOOT=yes
BOOTPROTO=none
NM_MANAGED=no
IPADDR=192.168.10.2
NETMASK=24
GATEWAY=192.168.10.1
DNS1=1.1.1.1

Switch

Port 1 - server w/ {pfSense}/{other VMs}
Port 3 - my PC
Port 7,8 - test ports for VLAN (auto get connected to VLAN77 when DHCP)

VLAN77:

Tagged/Trunk: Port 1,3
Untagged/Access: Port 7,8
Not member: Port 2,4,5,6

PVID:

Port 1-6: 1
Port 7-8: 77

pfSense

VLAN Interfaces :

Interface VLAN tag Priority Descirption
vtnet0 (lan) 77 Gaming VPN Network

Interface Assignments:

Interface Network Port
WAN re0 (98:de:d0:83:00:3e)
LAN vnet0 (fe:54:00:ae:b2:2c)
VLAN77 VLAN77 on vtnet0 - lan (Gaming VPN Network)

Rules: (basically allow all for now for testing, but later I’ll disable rule 1 and enable rule 2)

VLAN77: (E: Enable [V] / Disable [X]; A/B: [A] - allow / [B] - block)

E A/B Protocol Source Port Dest Port Gateway
V A IPv4 VLAN77 Net * * * *
X B IPv4+6 * * This Firewall RouterSwitch Management_Ports *
V A IPv4+6 * * VLAN77 Net * *