Ping works but no internet through virtual bridge via pfSense

[Derkades]

I run pfSense in a VM in my server. A physical 2 port gigabit NIC is passed through to the VM for WAN and LAN. My server is supposed to get an internet connection through a virtual bridge, that is added to the pfSense VM as well.

Maybe this helps to explain it (I made it a while ago, the server IP is actually 10.0.1.2 and I am using VLAN 2 for guest):

Everything works on the LAN interface.

Behavior on server (connected to OPT1)

• I can access pfSense (on 10.0.1.254 but also on 10.0.0.254 and 10.0.2.254)
• I can do DNS lookups

root@server:~# nslookup google.com
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
Name:	google.com
Address: 216.58.211.110
Name:	google.com
Address: 2a00:1450:400e:809::200e

• I can ping devices on LAN:

root@server:~# ping 10.0.0.101
PING 10.0.0.101 (10.0.0.101) 56(84) bytes of data.
64 bytes from 10.0.0.101: icmp_seq=1 ttl=63 time=0.319 ms
64 bytes from 10.0.0.101: icmp_seq=2 ttl=63 time=0.546 ms
64 bytes from 10.0.0.101: icmp_seq=3 ttl=63 time=0.463 ms

• I can ping stuff on the internet

root@server:~# ping example.com
PING example.com (93.184.216.34) 56(84) bytes of data.
64 bytes from 93.184.216.34 (93.184.216.34): icmp_seq=1 ttl=54 time=92.3 ms
64 bytes from 93.184.216.34 (93.184.216.34): icmp_seq=2 ttl=54 time=92.5 ms
64 bytes from 93.184.216.34 (93.184.216.34): icmp_seq=3 ttl=54 time=92.1 ms

• I can’t do any TCP requests. Browser doesn’t work, neither does telnet google.com 80.
• I can’t SSH into the server from LAN (ssh [email protected]).
• I can ping the server from LAN. (ping 10.0.1.2).
• I can SSH into the server from WAN, after port forwarding (ssh root@myexternalip -p 2222).

Firewall rules

DHCP server

LAN: Enabled: 10.0.0.1-10.0.0.253
SERVER: Enabled: 10.0.1.2-10.0.1.253

IP Route

On my desktop (LAN):

robin@pop-os:~$ ip route
default via 10.0.0.254 dev enp33s0 proto dhcp metric 100 
10.0.0.0/24 dev enp33s0 proto kernel scope link src 10.0.0.101 metric 100 
169.254.0.0/16 dev virbr0 scope link metric 1000 linkdown 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown

On my server (OPT1):

root@server:~# ip route
default via 10.0.1.254 dev virbr0 proto dhcp src 10.0.1.2 metric 100 
10.0.1.0/24 dev virbr0 proto kernel scope link src 10.0.1.2 
10.0.1.254 dev virbr0 proto dhcp scope link src 10.0.1.2 metric 100 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
172.18.0.0/16 dev br-cd18ef605502 proto kernel scope link src 172.18.0.1 
172.19.0.0/16 dev pterodactyl0 proto kernel scope link src 172.19.0.1 linkdown 
172.20.0.0/16 dev br-7477a87eb961 proto kernel scope link src 172.20.0.1 linkdown

[Derkades]

The network is configured using netplan:

network:
  version: 2
  renderer: networkd
  ethernets:
    # enp34s0:
    virbr0:
      dhcp4: yes

I get an IP address:

5: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:b8:64:19 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.2/24 brd 10.0.1.255 scope global dynamic virbr0
       valid_lft 3079sec preferred_lft 3079sec

[Derkades]

I am now using a static configuration, host IP 10.0.1.1. It’s still the same: traceroute works, ping works, nslookup works, tcp doesn’t work. The routes look exactly the same as on my laptop, so I am starting to think it is a pfsense issue.

What do I need to do in pfSense to get internet access from an interface? I have:

• Enabled OPT1 interface
• Added a firewall rule to the OPT1 interface that allows every protocol from every source to every destination
• Enabled DHCP server on OPT1

[RageBone]

i have a few dead ports on a Ubiquiti switch.
Ports light up and blink, pinging works most of the time, DHCP doesn’t really work and nothing else works.
Sounds similar.

Haven’t really investigated the Switch jet, but can you rule out the hardware being the issue?

[Derkades]

Turns out hardware checksum offload needs to be disabled in pfSense when using virtio drivers.

thanks to ericson007 https://www.linuxquestions.org/questions/linux-networking-3/ping-ok-dig-ok-tcp-fail-kvm-bridged-network-4175550312/
https://docs.netgate.com/pfsense/en/latest/hardware/troubleshooting-lost-traffic-or-disappearing-packets.html