PFSense Wireguard "Client" not able to ping other Peers

PWRxPSYCHO · December 12, 2023, 11:14pm

TLDR: I’m running into an issue where other peers connected to my Wireguard VPN are unable to ping each other.

Current Setup:

Ubuntu
Cloud server as Wireguard server.
eth0: (1.2.3.4)
wg0: 192.168.80.1/24

wg0.conf

[Interface]
ListenPort = 51820
PrivateKey = {PrivateKey}

PostUp = ufw route allow in on wg0 out on eth0
PostUp = iptables -t nat -I POSTROUTING -o eth0 -j MASQUERADE
PostUp = ip6tables -t nat -I POSTROUTING -o eth0 -j MASQUERADE
PreDown = ufw route delete allow in on wg0 out on eth0
PreDown = iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
PreDown = ip6tables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

# Peer1
[Peer]
PublicKey = {Public key from Peer1}
AllowedIPs = 192.168.80.7/32, 192.168.2.0/24

# PFsense
[Peer]
PublicKey = {Public Key from PFSense}
AllowedIPs = 192.168.80.8/32, 192.168.2.0/24

PFsense

Pfsense.conf:

# PFSense
[Interface]
PrivateKey = {Private Key}

# Server
[Peer]
PublicKey = {Server public Key}
AllowedIPs = 0.0.0.0/0
PersistentKeepalive = 0

“client” to Ubuntu:

Tunnel Config:
Private/Public Keys generated
Assigned Interface

Interface Config:
Static IPv4
No Ipv6

Static IPv4 Config:
Gateway created: 192.168.80.8/32

Peer:
Tunnel Assigned
Endpoint (1.2.3.4:51820)
PublicKey: From Ubuntu Server
AllowedIps: 0.0.0.0/0

NAT:
Outbound Hybrid
IPv4+Ipv6
Source: VLAN1
Translation → Address WG address

Firewall
Allow one IP on VLAN1 to use VPN connection

VLAN1:
Pass
Interface: VLAN1
Protocol: Any
Source: 192.168.2.5
Gateway: WG Gateway (192.168.80.8)

Peer1

Peer1.conf

# Peer1
[Interface]
PrivateKey={PrivateKey}
PublicKey={PubKey}
Addresses= 192.168.80.7/32

# Server
[Peer]
PublicKey={Server Public Key}
Endpoint=1.2.3.4:51820
AllowedIps= 0.0.0.0/0, ::/0

Diagram:

When trying to ping the server on VLAN1 I never get successful packets. Even when trying the PFSense interface (192.168.80.8).
I have a feeling its related to NAT or Firewall rules but I’m not sure what to start at.

Running sudo tcpdump -tttnei wg0 icmp on the Wireguard server I am able to see that the server behind PFSense is able to ping out and get a response but when the peer outside of PFSense reaches out the packets stop at PFSense.

Would it be a correct assumption this could be a firewall problem with PFSense?

Dexter_Kane · December 13, 2023, 1:04am

You can search for the IPs in the pfsense firewall log. Make sure you have the correct rules on the appropriate interfaces. You may also have to manually configure the outbound NAT rules as I don’t think wireguard interfaces get added automatically, although this should only effect gateway access.

It could also be that you’re /32 addresses, try with /24 and see if that helps.

PWRxPSYCHO · December 13, 2023, 1:18am

After checking the firewall logs it looks like my WAN interface was blocking the packets coming in from the Ubuntu server. Though the source IP was the eth0 of the Ubuntu server.

I made a specific allow rule but I’m not sure how to tell PFSense to take those packets and move them over Wireguard.

Dexter_Kane · December 13, 2023, 1:23am

You wouldn’t be getting any tunnelled traffic on the WAN interface so that is likely the actual wireguard traffic.

PWRxPSYCHO · December 13, 2023, 1:31am

So then if it’s coming in, I’m not sure why the traffic isn’t being received by the peer.

If I ping out from the server behind PFSense the connections are okay. It’s just reverse it has issues

risk · December 13, 2023, 10:24am

Using ping tool to troubleshoot network issues across many hops, is the first step of a “divide and conquer” troubleshooting strategy.

When you’re attempting to use ping running on peer1 (192.168.20.8), and trying to ping (192.168.2.5). You should see a ping request (icmp echo request) packet going through the wireguard link to the cloud server, over the cloudserver-pfsense link and over the pfsense-vlan1server link … and you should a ping reply (icmp echo reply) going back over the same series of links in reverse.

IP is based on end to end addressing, every host gets an IP, and each host and router knows where to direct packets based on destination IP. echo request will have a destination IP of 192.168.2.5 … and src ip address of 192.168.80.7, and reply will be reversed. Each host will look at destination IPs and will be making decisions.

tcpdump helps you look at each link, from each host, and thus knowing how you expect packets to flow, you can troubleshoot and discover all possible misbehaving nodes (hosts and routers).

If you’re not seeing icmp echo requests for 192.168.2.5 exiting 192.168.80.7 on wg interface, it’s likely that its routing table is misconfigured.

wireguard tools are typically used to install routes based on AllowedIPs in the config, but also there’s a king of light simple stateless filtering going on, for safety and to prevent misconfigurations, think of AllowedIPs as you telling wireguard “what’s on the other side”, so it could do the right thing.

If you don’t want wireguard tools managing routing, you can tell the config to not do that, and only filtering will be happening.

So if your peer1 has AllowedIPs set to 0/0 (default gateway routes), as a result, when the tools bring up the interface, you should see such a route added to peer1, and all packets coming in with a src address being anything, will be allowed in.

in the cloud server, for peer1 you have AllowedIPs 192.168.2.0/24 … this makes no sense, no packets will be coming in from 192.168.2.x via that inteface, only for or to those IPs … remove that.

also, it’d be better if your wireguard interface on cloud server would have an interface address, as that would cause the useful local routes on cloud server to be installed.

on pfsense, add a wireguard interface address, and add 192.168.80.0/24 as AllowedIPs.

I don’t know wireguard on pfsense well, route management and address management being done by a gazillion things can lead to problems, you should verify that a reasonable looking address is being assigned to each interface involved.

In conclusion, make those couple of changes to AllowedIP and Addresses I suggest above, … look at everything with tcpdump on one host at a time, … fix one host at a time, and on each host one interface at a time, firewall, wireguard config, another firewall, routing table.

there’s no reason why your Diagram shouldn’t end up working, probably just need to check AllowedIPs and routing tables. Only place you have to do NAT is when you’d have packets going to the internet with a private address in a source ip field otherwise.

PWRxPSYCHO · December 13, 2023, 9:00pm

So after making those suggested changes:

removing 192.168.2.0/24 from the server wg0.conf of peer 1 & PFsense
adding a wireguard interface address to the server (192.168.80.1/32)
adding 192.168.80.0/24 to Allowed Ips for the allowed ips of the Cloud Server on PFsense.

PFSense.conf

# PFSense
[Interface]
PrivateKey = {Private Key}

# Server
[Peer]
PublicKey = {Server public Key}
AllowedIPs = 0.0.0.0/0, 192.168.80.0/24
PersistentKeepalive = 0

I am still unable to ping from Peer 1 to the VLAN1 server behind PFsense.

When pinging from Peer 1 to VLAN 1 Server and doing a TCP dump I see the following

Could server: echo request
PFsense WG_Interface: echo request
PFsense VLAN1_Interface: no packets

Which leads me to believe the packets are either being stopped by the firewall or don’t know where to go after they hit the PFSense WG interface.

I have a firewall rule up on the VLAN1 interface that makes all packets from VLAN 1 Server go out the WG Gateway (192.168.80.8)

But I guess the reverse is blocked somewhere.

Also here are the PFSense configs:

Dexter_Kane · December 13, 2023, 11:21pm

You probably don’t want this. For rules allowing traffic between local networks (including VPN networks) you want to use the default gateway. If you want to send your Internet traffic through the VPN tunnel you would then create an additional rule for internet traffic using the VPN gateway.

You can either have the local rules on top and then an allow to any rule at the bottom with the VPN gateway or you can create an alias with your local networks and use the invert checkbox to create a rule for everything which isn’t a local network.

I also think this should be /24

You will also need to create static routes on the server if you haven’t already so that it knows what networks are behind pfsense.

PWRxPSYCHO · December 14, 2023, 12:09am

So then here, I would delete the WGGW and keep the Interface with the IP 192.168.80.8 but no gateway? Or do I set it to WAN and have an allow rule for inbound traffic on the port?

What would this look like? I’m not familiar with doing it directly on Linux.

sudo ip route add 192.168.2.0/24 dev wg0 ? or would it need other information

Dexter_Kane · December 14, 2023, 12:37am

No keep your gateway settings but in the firewall rules for traffic between networks (not internet traffic) use the default gateway. Use the wireguard gateway for rules where you want to send internet traffic over the VPN tunnel, you will also need to make sure you have outbound NAT rules set as well as sometimes the automatic mode doesn’t work with VPN gateways.

I’m not really sure but yes it would be something like that, you want to configure the routes on the server so that it knows that traffic for things behind pfsense have to go over the wireguard connection.

risk · December 14, 2023, 2:04pm

No, on cloud server configs, keep it on pfSense peer, remove from peer1 peer. As I mentioned before just read “AllowedIPs=…” as “IPs I’d expect to find living at this peer’s house and/or further behind this peer”.

but then again, I’m not sure why cloud server would be forwarding dstip=192.168.2.x packets over to pfsense…

… maybe you have some routes left over from last reboot.

pfsense

# PFSense
[Interface]
PrivateKey = ...
Address = 192.168.80.8/24

# Cloud Server
[Peer]
PublicKey = ...
AllowedIPs = 192.168.80.0/24
Endpoint = 1.2.3.4:51820
PersistentKeepalive = 23

cloud server

# Cloud server
[Interface]
PrivateKey = ...
Address = 192.168.80.8/24
ListenPort = 51820
PersistentKeepalive = 43

# pfsense
[Peer]
PublicKey = ...
AllowedIPs = 192.168.80.8/32, 192.168.2.0/24
PersistentKeepalive = 31

# peer1
[Peer]
PublicKey = ...
AllowedIPs = 192.168.80.7/32
PersistentKeepalive = 19

peer1

# peer1
[Interface]
PrivateKey = ...
Address = 192.168.80.7/24

# Cloud Server
[Peer]
PublicKey = ...
AllowedIPs = 192.168.80.0/24, 192.168.2.0/24
Endpoint = 1.2.3.4:51820
PersistentKeepalive = 29

Something like these three configs should get you a bit closer to what you need, … except for the firewalls.

anon7678104 · December 15, 2023, 10:47am

check your wireguard certs are in the rite folder and that you have certs.

i recently tried installing wireguard on cyberghost and it works on windows, but the linux installer failed to install the certs.
end result i couldn’t get a connection past my router.