Hi everyone, I recently tried to set up my own OPNSense router following a guide I was not allowed to post. Essentially, I have OPNsense running in a vm, and the LAN interface is in a lanbridge that has the hardware ports on it. On one of the ports, the host OS has a DHCP server with forwarding to the lanbridge, so that I can access the host OS even if the router is down.
The setup seemed to be working fine, and I installed Docker to my host OS (Debian). There I am running pihole and some other services. As per the guide, I have a WAN port, a management port and 4 LAN ports. These 4 lan ports are the problem. The management port works flawlessly, I can access both the host system and the LAN network, and all my Docker services are reachable.
However, the LAN ports behave weirdly. When I connect a device, and it tries to get an IP address via DHCP, either it works immediately, or the port on the host machine does not receive a single packet. Then, on the premise that DHCP worked, every following package seems to arrive. However, if I try to resolve any domain via my pihole, the response comes from a random port. I dont have this issue over the management port, so Docker seems to be fine? Also, when I try too fetch a website hosted on Docker via HTTP, the initial connection seems to work, however then it says âconnection reset by peerâ. This also works on the management port.
I really donât get whatâs going on here, can anyone help?
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
#auto lo
#iface lo inet loopback
# The primary network interface
#allow-hotplug enp1s0
#iface enp1s0 inet dhcp
# This is an autoconfigured IPv6 interface
#iface enp1s0 inet6 auto
# WAN interface
allow-hotplug enp1s0
iface enp1s0 inet manual
iface enp1s0 inet6 manual
# WAN bridge
auto wanbr
iface wanbr inet manual
bridge-ports enp1s0
bridge-waitport 0
bridge-fd 0
bridge-maxwait 0
# LAN interfaces
allow-hotplug enp3s0
iface enp3s0 inet manual
allow-hotplug enp4s0
iface enp4s0 inet manual
allow-hotplug enp5s0
iface enp5s0 inet manual
allow-hotplug enp6s0
iface enp6s0 inet manual
#iface enp6s0 inet dhcp
# LAN Bridge
auto lanbr
iface lanbr inet manual
pre-up ip link add veth0 type veth peer name veth1
pre-up ip addr add 192.168.1.2/24 dev veth0
pre-up ip link set veth0 up
pre-up ip link set veth1 up
pre-up ip route add default via 192.168.1.1 dev veth0
pre-up iptables -t nat -A POSTROUTING ! -s 192.168.1.2 -o veth0 -j MASQUERADE
bridge_ports enp3s0 enp4s0 enp5s0 enp6s0 veth1
# bridge_ports enp3s0 enp4s0 enp5s0 veth1
bridge_stp off
bridge_waitport 0 # no delay before a port becomes available
bridge_fd 0 # no forwarding delay
bridge_maxwait 0
bridge_stp on
# Managing Network Interface
allow-hotplug enp2s0
iface enp2s0 inet static
address 10.10.10.1
netmask 255.255.255.0
# dhcpd.conf
#
# Sample configuration file for ISC dhcpd
#
# option definitions common to all supported networks...
option domain-name "mgmt.local";
# IP of host OS with pihole running
option domain-name-servers 192.168.1.2;
default-lease-time 600;
max-lease-time 7200;
# The ddns-updates-style parameter controls whether or not the server will
# attempt to do a DNS update when a lease is confirmed. We default to the
# behavior of the version 2 packages ('none', since DHCP v2 didn't
# have support for DDNS.)
ddns-update-style none;
# If this DHCP server is the official DHCP server for the local
# network, the authoritative directive should be uncommented.
authoritative;
# Use this to send dhcp log messages to a different log file (you also
# have to hack syslog.conf to complete the redirection).
#log-facility local7;
# No service will be given on this subnet, but declaring it helps the
# DHCP server to understand the network topology.
#subnet 10.152.187.0 netmask 255.255.255.0 {
#}
# This is a very basic subnet declaration.
#subnet 10.254.239.0 netmask 255.255.255.224 {
# range 10.254.239.10 10.254.239.20;
# option routers rtr-239-0-1.example.org, rtr-239-0-2.example.org;
#}
# This declaration allows BOOTP clients to get dynamic addresses,
# which we don't really recommend.
#subnet 10.254.239.32 netmask 255.255.255.224 {
# range dynamic-bootp 10.254.239.40 10.254.239.60;
# option broadcast-address 10.254.239.31;
# option routers rtr-239-32-1.example.org;
#}
# A slightly different configuration for an internal subnet.
#subnet 10.5.5.0 netmask 255.255.255.224 {
# range 10.5.5.26 10.5.5.30;
# option domain-name-servers ns1.internal.example.org;
# option domain-name "internal.example.org";
# option routers 10.5.5.1;
# option broadcast-address 10.5.5.31;
# default-lease-time 600;
# max-lease-time 7200;
#}
# Hosts which require special configuration options can be listed in
# host statements. If no address is specified, the address will be
# allocated dynamically (if possible), but the host-specific information
# will still come from the host declaration.
#host passacaglia {
# hardware ethernet 0:0:c0:5d:bd:95;
# filename "vmunix.passacaglia";
# server-name "toccata.example.com";
#}
# Fixed IP addresses can also be specified for hosts. These addresses
# should not also be listed as being available for dynamic assignment.
# Hosts for which fixed IP addresses have been specified can boot using
# BOOTP or DHCP. Hosts for which no fixed address is specified can only
# be booted with DHCP, unless there is an address range on the subnet
# to which a BOOTP client is connected which has the dynamic-bootp flag
# set.
#host fantasia {
# hardware ethernet 08:00:07:26:c0:a5;
# fixed-address fantasia.example.com;
#}
# You can declare a class of clients and then do address allocation
# based on that. The example below shows a case where all clients
# in a certain class get addresses on the 10.17.224/24 subnet, and all
# other clients get addresses on the 10.0.29/24 subnet.
#class "foo" {
# match if substring (option vendor-class-identifier, 0, 4) = "SUNW";
#}
#shared-network 224-29 {
# subnet 10.17.224.0 netmask 255.255.255.0 {
# option routers rtr-224.example.org;
# }
# subnet 10.0.29.0 netmask 255.255.255.0 {
# option routers rtr-29.example.org;
# }
# pool {
# allow members of "foo";
# range 10.17.224.10 10.17.224.250;
# }
# pool {
# deny members of "foo";
# range 10.0.29.10 10.0.29.230;
# }
#}
subnet 10.10.10.0 netmask 255.255.255.0 {
range 10.10.10.24 10.10.10.128;
option routers 10.10.10.1;
}
To clear it up for me: are the LAN ports youâre complaining about physical ports or virtual ones via the same physical port as the mgt-port? If so, install a network card to separate physical ports.
There is a single virtual lan port coming from the VM that is on the network bridge (lanbr). On this bridge are 4 physical ports (I called those lan ports) and a link (veth0/veth1) over which the management port runs (a fifth physical port)
OK, I see. The symptoms you describe point towards a DHCP conflict. That is, thereâs another DHCP server on the network which captures requests and answers them before âyourâ DHCP can respond and thus your clients connect to that server instead of yours. You may need to rewire your network.
Thank you. I donât see how this could be the issue. The only DHCP servers in the network are the OPNsense one and the one serving the management port. The latter one should only serve that one port (set in /etc/default/isc-dhcp-server). Could I diagnose this somehow? Maybe with tcpdump?
Disable the one serving the management port and see if the behaviour youâve seen continues. If it does, the problem is likely something else.
Btw, management ports should really have a static IP address to keep them accessible at a known location. Setting a static IP address isnât difficult.
the VM host OS is proxmox or libvirt or qemu shell script or âŚ
dhcp is a layer 2 protocol by default, are you talking about one dhcp server proxying to another dhcp server?
Ok, this is old Debian ifupdown package, so maybe Proxmox? But proxmox has UIs, ⌠if another distro, systemd-networkd is kind of the default standard these days, it evolved closely with the kernel, other stuff is there for compatibility mostly or corner cases, or for UI porcelain.
Your lanbr has a pair of veth ⌠donât do that, assign that IP to a bridge directly - youâll probably want stp off on all of it, life is simpler that way at home.
I donât know about _waitport, _fd ⌠you can probably remove those.
Your VM/qemu will plug in a tap interface into lanbr when it starts up, you could make a permanent one ahead of time and parametrize lanbr-vmtap using fancy tools, but I donât think you need to.
Those small cleanup tweaks are where Iâd start,
No, youâre bridging 2 bridges together, just remove the veth, leave management port separate from any bridge, you need to manage the VM host you donât need to involve lanbr into it.