A few days ago I decided to setup a VLAN (VLAN77
) and an OpenVPN layer2/tap server in it for gaming. The goal is to be able to connect from my gaming PC(192.168.0.101
) to that VLAN using OpenVPN and clients from WAN to also be able to connect to that VLAN using OpenVPN and be able to play games while everyone is confined in the 192.168.10.0
network therefore not being able to access my local stuff that is on 192.168.0.0
. My setup is:
- 1 server PC (connected to trunk port) (
192.168.0.50
)- pfSense VM (
192.168.0.1
) - OpenVPN VM (
192.168.10.2
) - Other VMs
- pfSense VM (
- 1 main/gaming PC running Linux (connected to trunk port) (
192.168.0.101
)- Windows VM
- Test VMs for direct connection to
VLAN77
(running RockyLinux)netconfig for 1 of them (/etc/sysconfig/network-scripts/ifcfg-enp1s0.77)
DEVICE=enp1s0.77 ONBOOT=yes VLAN=yes NM_MANAGED=no IPADDR=192.168.10.40 NETMASK=24 GATEWAY=192.168.10.1
- 1 laptop for testing (connected to untagged/access port with PVID
77
) -gets IP by DHCP inVLAN77
net
My VLAN config (switch, router, server)
Server (Rocky Linux) / pfSense VM host
/sys/class/net/br0/bridge/vlan_filtering
0
/etc/sysconfig/network-scripts/ifcfg-enp5s1
DEVICE=enp5s1
UUID=5c593283-788c-4c66-a590-6fcc3a0e2a49
TYPE=Ethernet
NAME=enp5s1
ONBOOT=yes
NM_MANAGED=no
BRIDGE=br0
/etc/sysconfig/network-scripts/ifcfg-br0
TYPE=Bridge
DEVICE=br0
NAME="Bridge br0"
UUID=d2d68553-f97e-7549-7a26-b34a26f29318
ONBOOT=yes
NM_MANAGED=no
IPADDR=192.168.0.50
PREFIX=24
GATEWAY=192.168.0.1
DNS1=192.168.0.20
DNS2=1.1.1.1
BOOTPROTO=static
DEFROUTE=yes
OpenVPN VM (Rocky Linux): (Debian doesn’t work for some reason)
This can be 1 interface without the bridge. I just need the bridge to bridge enp1s0.77
with a TAP interface for OpenVPN.
/etc/sysconfig/network-scripts/ifcfg-enp1s0.77
DEVICE=enp1s0.77
ONBOOT=yes
BOOTPROTO=none
VLAN=yes
BRIDGE=br0
/etc/sysconfig/network-scripts/ifcfg-br0
TYPE=Bridge
DEVICE=br0
NAME="Bridge br0"
ONBOOT=yes
BOOTPROTO=none
NM_MANAGED=no
IPADDR=192.168.10.2
NETMASK=24
GATEWAY=192.168.10.1
DNS1=1.1.1.1
Switch
Port 1 - server w/ {pfSense}/{other VMs}
Port 3 - my PC
Port 7,8 - test ports for VLAN (auto get connected to VLAN77 when DHCP)
VLAN77:
Tagged/Trunk: Port 1
,3
Untagged/Access: Port 7
,8
Not member: Port 2
,4
,5
,6
PVID:
Port 1-6: 1
Port 7-8: 77
pfSense
VLAN Interfaces :
Interface | VLAN tag | Priority | Descirption |
---|---|---|---|
vtnet0 (lan) | 77 | Gaming VPN Network |
Interface Assignments:
Interface | Network Port |
---|---|
WAN | re0 (98:de:d0:83:00:3e) |
LAN | vnet0 (52:54:00:ae:b2:2c) |
VLAN77 | VLAN77 on vtnet0 - lan (Gaming VPN Network) |
Rules: (basically allow all for now for testing, but later I’ll disable rule 1 and enable rule 2)
VLAN77: (E: Enable [V] / Disable [X]; A/B: [A] - allow / [B] - block)
E | A/B | Protocol | Source | Port | Dest | Port | Gateway |
---|---|---|---|---|---|---|---|
V | A | IPv4 | VLAN77 Net | * | * | * | * |
X | B | IPv4+6 | * | * | This Firewall | RouterSwitch Management_Ports | * |
V | A | IPv4+6 | * | * | VLAN77 Net | * | * |
VLANs are working correctly. (The laptop and the test VMs in my main PC are in 192.168.10.0
network and can ping each other but can’t ping my main LAN when I disable the rule that allows traffic from VLAN77
to LAN
in pfSense for VLAN77
(The first rule in the last table in the post I linked above) . So VLAN works.)
However when I connect to the OpenVPN server I CAN’T ping anyone and CAN’T connect to anyone in VLAN77
. I CAN connect with everyone outside of VLAN77
(because I haven’t blocked it yet). The directly connected VMs and laptop also CAN’T ping anyone connected to VLAN77
through OpenVPN. I even tried hosting web server on 1 of the directly connected to the VLAN VMs and tried connecting to it from my main PC with an active OpenVPN connection to check if the issue was only with ping.
Here is my OpenVPN server conf (tap0
is statically added interface):
server.conf
verb 3
port 1194
proto udp
dev tap0
server-bridge 192.168.10.1 255.255.255.0 192.168.10.100 192.168.10.254
ifconfig-pool-persist /etc/openvpn/ipp.txt
# Auth
cert /etc/openvpn/server_vdWvipFvZXBYLr3N.crt
key /etc/openvpn/server_vdWvipFvZXBYLr3N.key
ca /etc/openvpn/ca.crt
dh /etc/openvpn/dh.pem
auth SHA512
#tls-auth /etc/openvpn/ta.key 0
sndbuf 0
rcvbuf 0
client-to-client
duplicate-cn
keepalive 10 120
cipher AES-256-CBC
comp-lzo
user nobody
#group nobody
persist-key
persist-tun
status status-tap.log
crl-verify /etc/openvpn/crl.pem
# The line below allows us to run scripts.
script-security 2
/etc/sysconfig/network-scripts/ifcfg-openvpn_tap
DEVICE=tap0
ONBOOT=yes
BOOTPROTO=none
TYPE=Tap
BRIDGE=br0
I disabled Firewalld and SELinux so that’s not the problem. I also installed iptables and tried this sudo iptables -t nat -A POSTROUTING -s 192.168.10.0/24 -o br0 -j MASQUERADE
(every chain is policy ACCEPT
). BTW what is the alternative firewalld command?
OpenVPN client log
2022-07-03 08:31:17 WARNING: Compression for receiving enabled. Compression has been used in the past to break encryption. Sent packets are not compressed unless “allow-compression yes” is also set.
2022-07-03 08:31:17 DEPRECATED OPTION: --cipher set to ‘AES-256-CBC’ but missing in --data-ciphers (AES-256-GCM:AES-128-GCM). Future OpenVPN version will ignore --cipher for cipher negotiations. Add ‘AES-256-CBC’ to --data-ciphers or change --cipher ‘AES-256-CBC’ to --data-ciphers-fallback ‘AES-256-CBC’ to silence this warning.
2022-07-03 08:31:17 OpenVPN 2.5.7 [git:makepkg/a0f9a3e9404c8321+] x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on May 31 2022
2022-07-03 08:31:17 library versions: OpenSSL 1.1.1p 21 Jun 2022, LZO 2.10
2022-07-03 08:31:17 WARNING: No server certificate verification method has been enabled. See How To Guide: Set Up & Configure OpenVPN Client/server VPN | OpenVPN for more info.
Enter Private Key Password: ****************************************************************
2022-07-03 08:31:20 WARNING: this configuration may cache passwords in memory – use the auth-nocache option to prevent this
2022-07-03 08:31:20 TCP/UDP: Preserving recently used remote address: [AF_INET]192.168.10.2:1194
2022-07-03 08:31:20 UDP link local: (not bound)
2022-07-03 08:31:20 UDP link remote: [AF_INET]192.168.10.2:1194
2022-07-03 08:31:20 [server_vdWvipFvZXBYLr3N] Peer Connection Initiated with [AF_INET]192.168.10.2:1194
2022-07-03 08:31:21 TUN/TAP device tap0 opened
2022-07-03 08:31:21 net_iface_mtu_set: mtu 1500 for tap0
2022-07-03 08:31:21 net_iface_up: set tap0 up
2022-07-03 08:31:21 net_addr_v4_add: 192.168.10.100/24 dev tap0
2022-07-03 08:31:21 Initialization Sequence Completed
^C2022-07-03 08:31:44 event_wait : Interrupted system call (code=4)
2022-07-03 08:31:44 net_addr_v4_del: 192.168.10.100 dev tap0
2022-07-03 08:31:44 SIGINT[hard,] received, process exiting
OpenVPN server log
Jul 03 08:30:21 gamingvpn systemd[1]: Starting OpenVPN service for server…
Jul 03 08:30:21 gamingvpn openvpn[4021]: OpenVPN 2.4.12 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Mar 17 2022
Jul 03 08:30:21 gamingvpn openvpn[4021]: library versions: OpenSSL 1.1.1k FIPS 25 Mar 2021, LZO 2.08
Jul 03 08:30:21 gamingvpn openvpn[4021]: NOTE: when bridging your LAN adapter with the TAP adapter, note that the new bridge adapter will often take on its own IP address that is different from what the LAN adapter was previously set to
Jul 03 08:30:21 gamingvpn openvpn[4021]: WARNING: --ifconfig-pool-persist will not work with --duplicate-cn
Jul 03 08:30:21 gamingvpn systemd[1]: Started OpenVPN service for server.
Jul 03 08:30:21 gamingvpn openvpn[4021]: Diffie-Hellman initialized with 2048 bit key
Jul 03 08:30:21 gamingvpn openvpn[4021]: CRL: loaded 1 CRLs from file /etc/openvpn/crl.pem
Jul 03 08:30:21 gamingvpn openvpn[4021]: TUN/TAP device tap0 opened
Jul 03 08:30:21 gamingvpn openvpn[4021]: TUN/TAP TX queue length set to 100
Jul 03 08:30:21 gamingvpn openvpn[4021]: Could not determine IPv4/IPv6 protocol. Using AF_INET
Jul 03 08:30:21 gamingvpn openvpn[4021]: Socket Buffers: R=[212992->212992] S=[212992->212992]
Jul 03 08:30:21 gamingvpn openvpn[4021]: UDPv4 link local (bound): [AF_INET][undef]:1194
Jul 03 08:30:21 gamingvpn openvpn[4021]: UDPv4 link remote: [AF_UNSPEC]
Jul 03 08:30:21 gamingvpn openvpn[4021]: UID set to nobody
Jul 03 08:30:21 gamingvpn openvpn[4021]: MULTI: multi_init called, r=256 v=256
Jul 03 08:30:21 gamingvpn openvpn[4021]: IFCONFIG POOL: base=192.168.10.100 size=155, ipv6=0
Jul 03 08:30:21 gamingvpn openvpn[4021]: IFCONFIG POOL LIST
Jul 03 08:30:21 gamingvpn openvpn[4021]: Initialization Sequence Completed
Jul 03 08:31:20 gamingvpn openvpn[4021]: 192.168.0.101:50127 TLS: Initial packet from [AF_INET]192.168.0.101:50127, sid=ed4f6d24 2c9add82
Jul 03 08:31:20 gamingvpn openvpn[4021]: 192.168.0.101:50127 VERIFY OK: depth=1, CN=cn_Q7Y5kdxk139Eon81
Jul 03 08:31:20 gamingvpn openvpn[4021]: 192.168.0.101:50127 VERIFY OK: depth=0, CN=FieryRider
Jul 03 08:31:20 gamingvpn openvpn[4021]: 192.168.0.101:50127 peer info: IV_VER=2.5.7
Jul 03 08:31:20 gamingvpn openvpn[4021]: 192.168.0.101:50127 peer info: IV_PLAT=linux
Jul 03 08:31:20 gamingvpn openvpn[4021]: 192.168.0.101:50127 peer info: IV_PROTO=6
Jul 03 08:31:20 gamingvpn openvpn[4021]: 192.168.0.101:50127 peer info: IV_NCP=2
Jul 03 08:31:20 gamingvpn openvpn[4021]: 192.168.0.101:50127 peer info: IV_CIPHERS=AES-256-GCM:AES-128-GCM:AES-256-CBC
Jul 03 08:31:20 gamingvpn openvpn[4021]: 192.168.0.101:50127 peer info: IV_LZ4=1
Jul 03 08:31:20 gamingvpn openvpn[4021]: 192.168.0.101:50127 peer info: IV_LZ4v2=1
Jul 03 08:31:20 gamingvpn openvpn[4021]: 192.168.0.101:50127 peer info: IV_LZO=1
Jul 03 08:31:20 gamingvpn openvpn[4021]: 192.168.0.101:50127 peer info: IV_COMP_STUB=1
Jul 03 08:31:20 gamingvpn openvpn[4021]: 192.168.0.101:50127 peer info: IV_COMP_STUBv2=1
Jul 03 08:31:20 gamingvpn openvpn[4021]: 192.168.0.101:50127 peer info: IV_TCPNL=1
Jul 03 08:31:20 gamingvpn openvpn[4021]: 192.168.0.101:50127 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 256 bit EC, curve: prime256v1
Jul 03 08:31:20 gamingvpn openvpn[4021]: 192.168.0.101:50127 [FieryRider] Peer Connection Initiated with [AF_INET]192.168.0.101:50127
Jul 03 08:31:20 gamingvpn openvpn[4021]: FieryRider/192.168.0.101:50127 MULTI_sva: pool returned IPv4=192.168.10.100, IPv6=(Not enabled)
Jul 03 08:31:21 gamingvpn openvpn[4021]: FieryRider/192.168.0.101:50127 PUSH: Received control message: ‘PUSH_REQUEST’
Jul 03 08:31:21 gamingvpn openvpn[4021]: FieryRider/192.168.0.101:50127 SENT CONTROL [FieryRider]: ‘PUSH_REPLY,route-gateway 192.168.10.1,ping 10,ping-restart 120,ifconfig 192.168.10.100 255.255.255.0,peer-id 0,cipher AES-256-GCM’ (status>
Jul 03 08:31:21 gamingvpn openvpn[4021]: FieryRider/192.168.0.101:50127 Data Channel: using negotiated cipher ‘AES-256-GCM’
Jul 03 08:31:21 gamingvpn openvpn[4021]: FieryRider/192.168.0.101:50127 Outgoing Data Channel: Cipher ‘AES-256-GCM’ initialized with 256 bit key
Jul 03 08:31:21 gamingvpn openvpn[4021]: FieryRider/192.168.0.101:50127 Incoming Data Channel: Cipher ‘AES-256-GCM’ initialized with 256 bit key
Jul 03 08:31:46 gamingvpn openvpn[4021]: read UDPv4 [ECONNREFUSED]: Connection refused (code=111)
I executed 3 commands while I was connected:
ping 192.168.0.101
ping 192.168.10.1
ping 192.168.10.2
The last 2 failed!
When I pinged 192.168.10.1
while connected to OpenVPN and failed I monitored the bridge on the VM host (that hosts pfSense and OpenVPN) that bridges VM virtual nics with enp5s1
with sudo tcpdump -i br0 -nn src net 192.168.0.0/24 and icmp
and got:
01:01:24.589756 IP 192.168.0.101 > 192.168.10.2: ICMP 192.168.0.101 udp port 50041 unreachable, length 77
01:01:24.589788 IP 192.168.0.101 > 192.168.10.2: ICMP 192.168.0.101 udp port 50041 unreachable, length 77
01:01:30.665772 IP 192.168.0.101 > 192.168.10.2: ICMP 192.168.0.101 udp port 50041 unreachable, length 200
01:01:30.665840 IP 192.168.0.101 > 192.168.10.2: ICMP 192.168.0.101 udp port 50041 unreachable, length 200