[Solved] Can't send WG traffic from peer in site-to-site setup

Hardware Hub Networking
1

Hi people,
have a weird problem, which I can debug, but where I’m not sure how to fix it.
I have three sites all connected to each other with site-to-site setups. Meaning at all subnets can reach all other subnets. Wireguard runs in Docker on each site.

Subnets:

192.168.8.0/24		WG-Host: 192.168.8.5/24	
192.168.16.0/24		WG-Host: 192.168.16.31/24
192.168.24.0/24		WG-Host: 192.168.24.6/24

I have configured routing and the hosts properly, everthing else works fine. Except I can’t reach the subnets from the hosts where WG is running.

Example:

  • When i try to ping (not WG-Host) ==> (not WG-Host) (in another subnet ofc) it works.
#Host 192.168.16.91
➜  ~ ping 192.168.8.1
PING 192.168.8.1 (192.168.8.1) 56(84) bytes of data.
From 192.168.16.1 icmp_seq=1 Redirect Host(New nexthop: 192.168.16.31)
64 bytes from 192.168.8.1: icmp_seq=1 ttl=62 time=20.3 ms
64 bytes from 192.168.8.1: icmp_seq=2 ttl=62 time=20.6 ms
64 bytes from 192.168.8.1: icmp_seq=3 ttl=62 time=20.1 ms
^C
--- 192.168.8.1 ping statistics ---
3 packets transmitted, 3 received, +1 errors, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 20.113/20.322/20.601/0.205 ms
  • When i try to ping (not WG-Host) ==> (to WG-Host) it works.
#Host 192.168.16.91
➜  ~ ping 192.168.8.5
PING 192.168.8.5 (192.168.8.5) 56(84) bytes of data.
From 192.168.16.1 icmp_seq=1 Redirect Host(New nexthop: 192.168.16.31)
64 bytes from 192.168.8.5: icmp_seq=1 ttl=63 time=19.0 ms
64 bytes from 192.168.8.5: icmp_seq=2 ttl=63 time=19.1 ms
64 bytes from 192.168.8.5: icmp_seq=3 ttl=63 time=19.3 ms
^C
--- 192.168.8.5 ping statistics ---
3 packets transmitted, 3 received, +1 errors, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 19.031/19.156/19.327/0.125 ms

When I try to Ping (WG-Host) ==> (not WG-Host) it does not work.

#Host 192.168.16.31
root@docker2:/docker/promgraf# ping 192.168.8.1
PING 192.168.8.1 (192.168.8.1) 56(84) bytes of data.
^C
--- 192.168.8.1 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3057ms

I’m pretty sure it has to do something with the routes on the WG-Hosts.

I did some tcp dumping and saw that, in the cases where traffic works, the source IP is the right one (192.168.X.X)
When I watch the traffic in a case where it doesn’t work, the Source-IP is directly 10.0.0.X.
So my suspition is, that the packets are directly sent to the WG-interface, without traversing through the host interface.

Here are the routes from one host (192.168.16.31)

root@docker2:/docker/promgraf# ip r s
default via 192.168.16.1 dev ens3 proto static 
10.0.0.0/24 dev wg0 proto kernel scope link src 10.0.0.2 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
172.18.0.0/16 dev br-d8b6936d1a8c proto kernel scope link src 172.18.0.1 linkdown 
172.20.0.0/16 dev br-64627e6f93b1 proto kernel scope link src 172.20.0.1 
172.21.0.0/16 dev br-4a2f194092e0 proto kernel scope link src 172.21.0.1 linkdown 
172.24.0.0/16 dev br-4e89d3b18e4e proto kernel scope link src 172.24.0.1 
172.25.0.0/16 dev br-9ce31703595b proto kernel scope link src 172.25.0.1 
172.28.0.0/16 dev br-9f75bcdd1f79 proto kernel scope link src 172.28.0.1 
172.29.0.0/16 dev br-6733bb4fa579 proto kernel scope link src 172.29.0.1 
172.30.0.0/16 dev br-1812446a9708 proto kernel scope link src 172.30.0.1 
172.31.0.0/16 dev br-cc0379dc2ab0 proto kernel scope link src 172.31.0.1 
192.168.8.0/24 dev wg0 scope link 
192.168.16.0/24 dev ens3 proto kernel scope link src 192.168.16.31 
192.168.24.0/24 dev wg0 scope link 
192.168.32.0/20 dev br-75c60b839d17 proto kernel scope link src 192.168.32.1 
192.168.48.0/20 dev br-a9f14bea449c proto kernel scope link src 192.168.48.1

Example config:

#Host 192.168.16.31
root@docker2:/docker/wg-p2p# cat config/wg0.conf 
[Interface]
Address = 10.0.0.2/24
PrivateKey = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
ListenPort = 37589

[Peer]
PublicKey = KeDJbutugXNbMDNRI/useD0QtMmOLxedOCN5boqN7EY=
Endpoint = XXXXXXXXXXXXX:37589
AllowedIPs = 10.0.0.0/24, 192.168.8.0/24

[Peer]
PublicKey = NuOoBWlMEJx6mb2NfCKLjs6nRFdTYj69IKia8wFEzVg=
Endpoint = XXXXXXXXXX:37589
AllowedIPs = 10.0.0.0/24, 192.168.24.0/24
2

I’m suspicious of how wireguard is keeping track of 10. routes.

There was something I read recently about how Wireguard is emulating broadcast segments internally that’s making me suspicious, i don’t know how to dump that state.

Can you ip route show table all and post it here inside [details] ? Maybe also ip rule show

Also, btw, you can use /32 as peer/interface addresses just fine if you want to , for wg host interfaces

1 Like
3

Thanks for your reply!
Heres the full table, quite long bc docker interfaces. (Can I make this collapsible somehow?)

root@docker2:/docker# ip r s table all
default via 192.168.16.1 dev ens3 proto static 
10.0.0.0/24 dev wg0 proto kernel scope link src 10.0.0.2 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
172.18.0.0/16 dev br-d8b6936d1a8c proto kernel scope link src 172.18.0.1 linkdown 
172.20.0.0/16 dev br-64627e6f93b1 proto kernel scope link src 172.20.0.1 
172.21.0.0/16 dev br-4a2f194092e0 proto kernel scope link src 172.21.0.1 linkdown 
172.24.0.0/16 dev br-4e89d3b18e4e proto kernel scope link src 172.24.0.1 
172.25.0.0/16 dev br-9ce31703595b proto kernel scope link src 172.25.0.1 
172.28.0.0/16 dev br-9f75bcdd1f79 proto kernel scope link src 172.28.0.1 
172.29.0.0/16 dev br-6733bb4fa579 proto kernel scope link src 172.29.0.1 
172.30.0.0/16 dev br-1812446a9708 proto kernel scope link src 172.30.0.1 
172.31.0.0/16 dev br-cc0379dc2ab0 proto kernel scope link src 172.31.0.1 
192.168.8.0/24 dev wg0 scope link 
192.168.16.0/24 dev ens3 proto kernel scope link src 192.168.16.31 
192.168.24.0/24 dev wg0 scope link 
192.168.32.0/20 dev br-75c60b839d17 proto kernel scope link src 192.168.32.1 
broadcast 10.0.0.0 dev wg0 table local proto kernel scope link src 10.0.0.2 
local 10.0.0.2 dev wg0 table local proto kernel scope host src 10.0.0.2 
broadcast 10.0.0.255 dev wg0 table local proto kernel scope link src 10.0.0.2 
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1 
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1 
broadcast 172.17.0.0 dev docker0 table local proto kernel scope link src 172.17.0.1 linkdown 
local 172.17.0.1 dev docker0 table local proto kernel scope host src 172.17.0.1 
broadcast 172.17.255.255 dev docker0 table local proto kernel scope link src 172.17.0.1 linkdown 
broadcast 172.18.0.0 dev br-d8b6936d1a8c table local proto kernel scope link src 172.18.0.1 linkdown 
local 172.18.0.1 dev br-d8b6936d1a8c table local proto kernel scope host src 172.18.0.1 
broadcast 172.18.255.255 dev br-d8b6936d1a8c table local proto kernel scope link src 172.18.0.1 linkdown 
broadcast 172.20.0.0 dev br-64627e6f93b1 table local proto kernel scope link src 172.20.0.1 
local 172.20.0.1 dev br-64627e6f93b1 table local proto kernel scope host src 172.20.0.1 
broadcast 172.20.255.255 dev br-64627e6f93b1 table local proto kernel scope link src 172.20.0.1 
broadcast 172.21.0.0 dev br-4a2f194092e0 table local proto kernel scope link src 172.21.0.1 linkdown 
local 172.21.0.1 dev br-4a2f194092e0 table local proto kernel scope host src 172.21.0.1 
broadcast 172.21.255.255 dev br-4a2f194092e0 table local proto kernel scope link src 172.21.0.1 linkdown 
broadcast 172.24.0.0 dev br-4e89d3b18e4e table local proto kernel scope link src 172.24.0.1 
local 172.24.0.1 dev br-4e89d3b18e4e table local proto kernel scope host src 172.24.0.1 
broadcast 172.24.255.255 dev br-4e89d3b18e4e table local proto kernel scope link src 172.24.0.1 
broadcast 172.25.0.0 dev br-9ce31703595b table local proto kernel scope link src 172.25.0.1 
local 172.25.0.1 dev br-9ce31703595b table local proto kernel scope host src 172.25.0.1 
broadcast 172.25.255.255 dev br-9ce31703595b table local proto kernel scope link src 172.25.0.1 
broadcast 172.28.0.0 dev br-9f75bcdd1f79 table local proto kernel scope link src 172.28.0.1 
local 172.28.0.1 dev br-9f75bcdd1f79 table local proto kernel scope host src 172.28.0.1 
broadcast 172.28.255.255 dev br-9f75bcdd1f79 table local proto kernel scope link src 172.28.0.1 
broadcast 172.29.0.0 dev br-6733bb4fa579 table local proto kernel scope link src 172.29.0.1 
local 172.29.0.1 dev br-6733bb4fa579 table local proto kernel scope host src 172.29.0.1 
broadcast 172.29.255.255 dev br-6733bb4fa579 table local proto kernel scope link src 172.29.0.1 
broadcast 172.30.0.0 dev br-1812446a9708 table local proto kernel scope link src 172.30.0.1 
local 172.30.0.1 dev br-1812446a9708 table local proto kernel scope host src 172.30.0.1 
broadcast 172.30.255.255 dev br-1812446a9708 table local proto kernel scope link src 172.30.0.1 
broadcast 172.31.0.0 dev br-cc0379dc2ab0 table local proto kernel scope link src 172.31.0.1 
local 172.31.0.1 dev br-cc0379dc2ab0 table local proto kernel scope host src 172.31.0.1 
broadcast 172.31.255.255 dev br-cc0379dc2ab0 table local proto kernel scope link src 172.31.0.1 
broadcast 192.168.16.0 dev ens3 table local proto kernel scope link src 192.168.16.31 
local 192.168.16.31 dev ens3 table local proto kernel scope host src 192.168.16.31 
broadcast 192.168.16.255 dev ens3 table local proto kernel scope link src 192.168.16.31 
broadcast 192.168.32.0 dev br-75c60b839d17 table local proto kernel scope link src 192.168.32.1 
local 192.168.32.1 dev br-75c60b839d17 table local proto kernel scope host src 192.168.32.1 
broadcast 192.168.47.255 dev br-75c60b839d17 table local proto kernel scope link src 192.168.32.1 
local ::1 dev lo proto kernel metric 256 pref medium
fd24:c98e:fe47:4364::/64 dev ens3 proto ra metric 1024 pref medium
fd62:d8c1:79dd::/64 via fe80::800:7f4d:6cf3:2bfa dev ens3 proto ra metric 1024 pref medium
fe80::/64 dev ens3 proto kernel metric 256 pref medium
fe80::/64 dev br-1812446a9708 proto kernel metric 256 pref medium
fe80::/64 dev br-6733bb4fa579 proto kernel metric 256 pref medium
fe80::/64 dev br-64627e6f93b1 proto kernel metric 256 pref medium
fe80::/64 dev br-9f75bcdd1f79 proto kernel metric 256 pref medium
fe80::/64 dev br-4e89d3b18e4e proto kernel metric 256 pref medium
fe80::/64 dev br-9ce31703595b proto kernel metric 256 pref medium
fe80::/64 dev veth6274c00 proto kernel metric 256 pref medium
fe80::/64 dev veth992e819 proto kernel metric 256 pref medium
fe80::/64 dev veth41a3087 proto kernel metric 256 pref medium
fe80::/64 dev vetha893197 proto kernel metric 256 pref medium
fe80::/64 dev veth2545614 proto kernel metric 256 pref medium
fe80::/64 dev veth8bdb05a proto kernel metric 256 pref medium
fe80::/64 dev veth4ce6439 proto kernel metric 256 pref medium
fe80::/64 dev veth3e0bb7e proto kernel metric 256 pref medium
fe80::/64 dev vethc71151b proto kernel metric 256 pref medium
fe80::/64 dev vethcb335b4 proto kernel metric 256 pref medium
fe80::/64 dev vethac0ab77 proto kernel metric 256 pref medium
fe80::/64 dev veth1167a73 proto kernel metric 256 pref medium
fe80::/64 dev vethb4ba1b4 proto kernel metric 256 pref medium
fe80::/64 dev vethb659c99 proto kernel metric 256 pref medium
fe80::/64 dev veth5d8bda3 proto kernel metric 256 pref medium
fe80::/64 dev vethae5e673 proto kernel metric 256 pref medium
fe80::/64 dev veth92f53c0 proto kernel metric 256 pref medium
fe80::/64 dev veth0a7c3bb proto kernel metric 256 pref medium
fe80::/64 dev veth1ccd46f proto kernel metric 256 pref medium
fe80::/64 dev veth024bf27 proto kernel metric 256 pref medium
fe80::/64 dev veth11e6729 proto kernel metric 256 pref medium
fe80::/64 dev vethf624d6a proto kernel metric 256 pref medium
fe80::/64 dev br-cc0379dc2ab0 proto kernel metric 256 pref medium
fe80::/64 dev veth95938f0 proto kernel metric 256 pref medium
fe80::/64 dev veth01f1808 proto kernel metric 256 pref medium
fe80::/64 dev vethd2552e8 proto kernel metric 256 pref medium
fe80::/64 dev docker0 proto kernel metric 256 linkdown pref medium
fe80::/64 dev veth450eb07 proto kernel metric 256 pref medium
fe80::/64 dev br-75c60b839d17 proto kernel metric 256 pref medium
fe80::/64 dev veth548e694 proto kernel metric 256 pref medium
fe80::/64 dev veth483cca5 proto kernel metric 256 pref medium
fe80::/64 dev veth9822486 proto kernel metric 256 pref medium
fe80::/64 dev vethe5b7ea2 proto kernel metric 256 pref medium
fe80::/64 dev vethbad4ebc proto kernel metric 256 pref medium
fe80::/64 dev veth6d56e61 proto kernel metric 256 pref medium
fe80::/64 dev veth68b8804 proto kernel metric 256 pref medium
fe80::/64 dev veth205667e proto kernel metric 256 pref medium
fe80::/64 dev veth6811383 proto kernel metric 256 pref medium
fe80::/64 dev vethc879917 proto kernel metric 256 pref medium
fe80::/64 dev veth4632310 proto kernel metric 256 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
local fd24:c98e:fe47:4364:5054:ff:feb5:ec59 dev ens3 table local proto kernel metric 0 pref medium
local fe80::21:26ff:fe32:e8bc dev veth6d56e61 table local proto kernel metric 0 pref medium
local fe80::42:29ff:fe67:a993 dev br-6733bb4fa579 table local proto kernel metric 0 pref medium
local fe80::42:31ff:fe3b:16e9 dev docker0 table local proto kernel metric 0 linkdown pref medium
local fe80::42:3bff:fe0e:57bf dev br-1812446a9708 table local proto kernel metric 0 pref medium
local fe80::42:49ff:fe68:69cd dev br-9ce31703595b table local proto kernel metric 0 pref medium
local fe80::42:68ff:fed7:c201 dev br-cc0379dc2ab0 table local proto kernel metric 0 pref medium
local fe80::42:77ff:fe2b:9b49 dev br-75c60b839d17 table local proto kernel metric 0 pref medium
local fe80::42:7bff:fe1a:a1e9 dev br-4e89d3b18e4e table local proto kernel metric 0 pref medium
local fe80::42:acff:fe7c:c2da dev br-9f75bcdd1f79 table local proto kernel metric 0 pref medium
local fe80::42:d1ff:febd:bf8d dev br-64627e6f93b1 table local proto kernel metric 0 pref medium
local fe80::48c:d8ff:fe4e:62f6 dev veth1167a73 table local proto kernel metric 0 pref medium
local fe80::10bf:20ff:fe58:62ef dev veth992e819 table local proto kernel metric 0 pref medium
local fe80::18b8:eeff:fe9d:ea75 dev veth450eb07 table local proto kernel metric 0 pref medium
local fe80::205e:d0ff:fef6:edf3 dev veth92f53c0 table local proto kernel metric 0 pref medium
local fe80::2414:84ff:fe0e:d0da dev veth11e6729 table local proto kernel metric 0 pref medium
local fe80::2498:d7ff:feb3:58e9 dev veth2545614 table local proto kernel metric 0 pref medium
local fe80::5054:ff:feb5:ec59 dev ens3 table local proto kernel metric 0 pref medium
local fe80::545a:acff:feff:7427 dev veth4632310 table local proto kernel metric 0 pref medium
local fe80::5c10:25ff:fe91:1ea5 dev vethe5b7ea2 table local proto kernel metric 0 pref medium
local fe80::5c26:8ff:fe38:d144 dev vethf624d6a table local proto kernel metric 0 pref medium
local fe80::5c9e:91ff:fe49:5540 dev vetha893197 table local proto kernel metric 0 pref medium
local fe80::5cd8:74ff:fe4d:9157 dev veth5d8bda3 table local proto kernel metric 0 pref medium
local fe80::6424:96ff:fe30:e057 dev vethb659c99 table local proto kernel metric 0 pref medium
local fe80::6829:9ff:fe88:ac8e dev veth41a3087 table local proto kernel metric 0 pref medium
local fe80::68b3:2cff:fe68:fb5c dev veth1ccd46f table local proto kernel metric 0 pref medium
local fe80::7432:5eff:fe63:a93c dev veth6274c00 table local proto kernel metric 0 pref medium
local fe80::74f5:c8ff:febf:cdc1 dev vethcb335b4 table local proto kernel metric 0 pref medium
local fe80::8051:f3ff:feb8:11bb dev vethc879917 table local proto kernel metric 0 pref medium
local fe80::8053:74ff:fe48:bc9a dev veth6811383 table local proto kernel metric 0 pref medium
local fe80::8075:abff:fe58:2bda dev veth483cca5 table local proto kernel metric 0 pref medium
local fe80::8807:d0ff:fe59:44ea dev vethbad4ebc table local proto kernel metric 0 pref medium
local fe80::88b2:15ff:feef:af1d dev veth548e694 table local proto kernel metric 0 pref medium
local fe80::88d8:baff:fe84:5fb0 dev vethb4ba1b4 table local proto kernel metric 0 pref medium
local fe80::8ca4:bdff:fe29:799d dev veth9822486 table local proto kernel metric 0 pref medium
local fe80::9870:a5ff:fe3e:5fbe dev veth01f1808 table local proto kernel metric 0 pref medium
local fe80::a8a2:12ff:fe75:601a dev vethac0ab77 table local proto kernel metric 0 pref medium
local fe80::acb8:94ff:fec5:1b9f dev vethd2552e8 table local proto kernel metric 0 pref medium
local fe80::b438:e4ff:fe66:d208 dev veth3e0bb7e table local proto kernel metric 0 pref medium
local fe80::c40b:34ff:fe8a:4bff dev veth8bdb05a table local proto kernel metric 0 pref medium
local fe80::c420:18ff:fece:5980 dev vethae5e673 table local proto kernel metric 0 pref medium
local fe80::cc21:3bff:fe1d:fd63 dev vethc71151b table local proto kernel metric 0 pref medium
local fe80::d42b:50ff:fe83:81dd dev veth205667e table local proto kernel metric 0 pref medium
local fe80::d4ef:ebff:fe25:d971 dev veth95938f0 table local proto kernel metric 0 pref medium
local fe80::dcee:22ff:fe42:2fb1 dev veth0a7c3bb table local proto kernel metric 0 pref medium
local fe80::f43a:1bff:fe3d:d112 dev veth68b8804 table local proto kernel metric 0 pref medium
local fe80::fc1e:edff:fe13:1270 dev veth024bf27 table local proto kernel metric 0 pref medium
local fe80::fc52:39ff:fec5:e762 dev veth4ce6439 table local proto kernel metric 0 pref medium
ff00::/8 dev ens3 table local proto kernel metric 256 pref medium
ff00::/8 dev br-1812446a9708 table local proto kernel metric 256 pref medium
ff00::/8 dev br-6733bb4fa579 table local proto kernel metric 256 pref medium
ff00::/8 dev br-64627e6f93b1 table local proto kernel metric 256 pref medium
ff00::/8 dev br-9f75bcdd1f79 table local proto kernel metric 256 pref medium
ff00::/8 dev br-4e89d3b18e4e table local proto kernel metric 256 pref medium
ff00::/8 dev wg0 table local proto kernel metric 256 pref medium
ff00::/8 dev br-9ce31703595b table local proto kernel metric 256 pref medium
ff00::/8 dev veth6274c00 table local proto kernel metric 256 pref medium
ff00::/8 dev veth992e819 table local proto kernel metric 256 pref medium
ff00::/8 dev veth41a3087 table local proto kernel metric 256 pref medium
ff00::/8 dev vetha893197 table local proto kernel metric 256 pref medium
ff00::/8 dev veth2545614 table local proto kernel metric 256 pref medium
ff00::/8 dev veth8bdb05a table local proto kernel metric 256 pref medium
ff00::/8 dev veth4ce6439 table local proto kernel metric 256 pref medium
ff00::/8 dev veth3e0bb7e table local proto kernel metric 256 pref medium
ff00::/8 dev vethc71151b table local proto kernel metric 256 pref medium
ff00::/8 dev vethcb335b4 table local proto kernel metric 256 pref medium
ff00::/8 dev vethac0ab77 table local proto kernel metric 256 pref medium
ff00::/8 dev veth1167a73 table local proto kernel metric 256 pref medium
ff00::/8 dev vethb4ba1b4 table local proto kernel metric 256 pref medium
ff00::/8 dev vethb659c99 table local proto kernel metric 256 pref medium
ff00::/8 dev veth5d8bda3 table local proto kernel metric 256 pref medium
ff00::/8 dev vethae5e673 table local proto kernel metric 256 pref medium
ff00::/8 dev veth92f53c0 table local proto kernel metric 256 pref medium
ff00::/8 dev veth0a7c3bb table local proto kernel metric 256 pref medium
ff00::/8 dev veth1ccd46f table local proto kernel metric 256 pref medium
ff00::/8 dev veth024bf27 table local proto kernel metric 256 pref medium
ff00::/8 dev veth11e6729 table local proto kernel metric 256 pref medium
ff00::/8 dev vethf624d6a table local proto kernel metric 256 pref medium
ff00::/8 dev br-cc0379dc2ab0 table local proto kernel metric 256 pref medium
ff00::/8 dev veth95938f0 table local proto kernel metric 256 pref medium
ff00::/8 dev veth01f1808 table local proto kernel metric 256 pref medium
ff00::/8 dev vethd2552e8 table local proto kernel metric 256 pref medium
ff00::/8 dev docker0 table local proto kernel metric 256 linkdown pref medium
ff00::/8 dev veth450eb07 table local proto kernel metric 256 pref medium
ff00::/8 dev br-75c60b839d17 table local proto kernel metric 256 pref medium
ff00::/8 dev veth548e694 table local proto kernel metric 256 pref medium
ff00::/8 dev veth483cca5 table local proto kernel metric 256 pref medium
ff00::/8 dev veth9822486 table local proto kernel metric 256 pref medium
ff00::/8 dev vethe5b7ea2 table local proto kernel metric 256 pref medium
ff00::/8 dev vethbad4ebc table local proto kernel metric 256 pref medium
ff00::/8 dev veth6d56e61 table local proto kernel metric 256 pref medium
ff00::/8 dev veth68b8804 table local proto kernel metric 256 pref medium
ff00::/8 dev veth205667e table local proto kernel metric 256 pref medium
ff00::/8 dev veth6811383 table local proto kernel metric 256 pref medium
ff00::/8 dev vethc879917 table local proto kernel metric 256 pref medium
ff00::/8 dev veth4632310 table local proto kernel metric 256 pref medium

root@docker2:/docker# ip rule show
0:	from all lookup local 
32766:	from all lookup main 
32767:	from all lookup default

I never really understood the peer interface configuration. Do they not all need to be in the same subnet? Can i just pick random addresses?

4

You have “interface address”, Try setting them to 10.0.0.8/32, 10.0.0.16/32 ; 10.0.0.24/32 … also in allowed IPs for peers.

I suspect somethings broken in internal wireguard routing because you have overlapping 10. subnets on multiple peers.

wg-quick tool will add the peer routes into the normal kernel routing tables based on allowed IPs. So 192.168.24.0/24 will go to wg0 ; but so will anything in 192.168.8.0/24 … once your pings reach wireguard wg0 something gets scrambled, this is why I’m suggesting you try /32 above - it might make wireguard’s job easier.

It just occurred to me, Occam’s razor… have you checked with tcpdump whether the packets are coming into the firewall?

Maybe you have masquerade/nat enabled somewhere unexpectedly?

1 Like
5

Thanks again for your help, unfortunately that did not do the trick. Firewalls are all off.

I added the two TCP-Dumps whilst ping was running. First is, where it works (request and reply), and it shows the correct ip-addresses.

root@docker2:/docker/wg-p2p# tcpdump -ni any  icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
11:35:25.820935 IP 192.168.16.182 > 192.168.8.5: ICMP echo request, id 45784, seq 0, length 64
11:35:25.820977 IP 192.168.16.182 > 192.168.8.5: ICMP echo request, id 45784, seq 0, length 64
11:35:25.840567 IP 192.168.8.5 > 192.168.16.182: ICMP echo reply, id 45784, seq 0, length 64
11:35:25.840592 IP 192.168.8.5 > 192.168.16.182: ICMP echo reply, id 45784, seq 0, length 64
11:35:26.826387 IP 192.168.16.182 > 192.168.8.5: ICMP echo request, id 45784, seq 1, length 64
11:35:26.826445 IP 192.168.16.182 > 192.168.8.5: ICMP echo request, id 45784, seq 1, length 64
11:35:26.845544 IP 192.168.8.5 > 192.168.16.182: ICMP echo reply, id 45784, seq 1, length 64
11:35:26.845588 IP 192.168.8.5 > 192.168.16.182: ICMP echo reply, id 45784, seq 1, length 64
11:35:30.328057 IP 192.168.16.30 > 192.168.24.6: ICMP echo request, id 12644, seq 1, length 64
11:35:30.328122 IP 192.168.16.30 > 192.168.24.6: ICMP echo request, id 12644, seq 1, length 64
11:35:30.334057 IP 192.168.24.6 > 192.168.16.30: ICMP echo reply, id 12644, seq 1, length 64
11:35:30.334082 IP 192.168.24.6 > 192.168.16.30: ICMP echo reply, id 12644, seq 1, length 64
11:35:54.325562 IP 192.168.16.30 > 192.168.8.5: ICMP echo request, id 12660, seq 1, length 64
11:35:54.325627 IP 192.168.16.30 > 192.168.8.5: ICMP echo request, id 12660, seq 1, length 64
11:35:54.344902 IP 192.168.8.5 > 192.168.16.30: ICMP echo reply, id 12660, seq 1, length 64
11:35:54.344940 IP 192.168.8.5 > 192.168.16.30: ICMP echo reply, id 12660, seq 1, length 64
^C
16 packets captured
16 packets received by filter
0 packets dropped by kernel

Second one it shows right away the internal wg interface ip and only echo requests. Shouldnt it also be from the host source (192.168.16.30)?

root@docker2:/docker/wg-p2p# tcpdump -ni any  icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
11:35:01.999467 IP 10.0.0.16 > 192.168.8.1: ICMP echo request, id 24085, seq 1, length 64
11:35:03.024265 IP 10.0.0.16 > 192.168.8.1: ICMP echo request, id 24085, seq 2, length 64
11:35:04.044290 IP 10.0.0.16 > 192.168.8.1: ICMP echo request, id 24085, seq 3, length 64
11:35:05.068263 IP 10.0.0.16 > 192.168.8.1: ICMP echo request, id 24085, seq 4, length 64
11:35:06.092264 IP 10.0.0.16 > 192.168.8.1: ICMP echo request, id 24085, seq 5, length 64
11:35:07.116285 IP 10.0.0.16 > 192.168.8.1: ICMP echo request, id 24085, seq 6, length 64
11:35:08.140317 IP 10.0.0.16 > 192.168.8.1: ICMP echo request, id 24085, seq 7, length 64
11:35:09.164263 IP 10.0.0.16 > 192.168.8.1: ICMP echo request, id 24085, seq 8, length 64
^C
8 packets captured
8 packets received by filter
0 packets dropped by kernel
6

ok… and when you’re pinging 10.0.0.16 > 192.168.8.1

what does running tcpdump on 10.0.0.8 show is traversing over its wg0?
what does tcpdump on the 192.168.8.1 show?
can you ping 10.0.0.8 from 10.0.0.16?

Second one it shows right away the internal wg interface ip and only echo requests. Shouldnt it also be from the host source (192.168.16.30)?

If you don’t bind a socket to a specific ip or interface the ‘src’ hint from the routing table is what gets used. Intutively I like to think about the IP “closest” to destination being used.

7

Again, thanks so much, really appreciate it.

I also tried to force ping through the host interface of the VM to no effect:

root@docker2:/docker/wg-p2p# ping -I ens3 192.168.8.1
PING 192.168.8.1 (192.168.8.1) from 192.168.16.31 ens3: 56(84) bytes of data.
From 192.168.16.1: icmp_seq=1 Redirect Host(New nexthop: 192.168.16.31)
^C
--- 192.168.8.1 ping statistics ---
16 packets transmitted, 0 received, 100% packet loss, time 15345ms

When i enter the wireguard container and try to reach a subnet “on the other site”, it works:
Ping target is on 192.168.8.5

# Container runs on 192.168.16.31
bash-5.0# ping 10.0.0.8
PING 10.0.0.8 (10.0.0.8): 56 data bytes
64 bytes from 10.0.0.8: seq=0 ttl=64 time=19.452 ms
64 bytes from 10.0.0.8: seq=1 ttl=64 time=19.496 ms
64 bytes from 10.0.0.8: seq=2 ttl=64 time=19.251 ms
64 bytes from 10.0.0.8: seq=3 ttl=64 time=19.477 ms
^C
--- 10.0.0.8 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 19.251/19.419/19.496 ms

Now, when I run TCPdump on the receiving end (container host, not inside container) of the wireguard tunnel.
This is what it looks like when it works:

root@mampi-server:/docker/wireguard# tcpdump -ni wg0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wg0, link-type RAW (Raw IP), capture size 262144 bytes
15:39:48.153719 IP 192.168.16.91 > 192.168.8.1: ICMP echo request, id 6, seq 1, length 64
15:39:48.154421 IP 192.168.8.1 > 192.168.16.91: ICMP echo reply, id 6, seq 1, length 64
15:39:49.155196 IP 192.168.16.91 > 192.168.8.1: ICMP echo request, id 6, seq 2, length 64
15:39:49.155762 IP 192.168.8.1 > 192.168.16.91: ICMP echo reply, id 6, seq 2, length 64
15:39:50.156274 IP 192.168.16.91 > 192.168.8.1: ICMP echo request, id 6, seq 3, length 64
15:39:50.156893 IP 192.168.8.1 > 192.168.16.91: ICMP echo reply, id 6, seq 3, length 64
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel

This is what it looks like when it doesn’t

root@mampi-server:/docker/wireguard# tcpdump -ni wg0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wg0, link-type RAW (Raw IP), capture size 262144 bytes
15:40:52.565315 IP 10.0.0.16 > 192.168.8.1: ICMP echo request, id 10862, seq 6, length 64
15:40:53.589189 IP 10.0.0.16 > 192.168.8.1: ICMP echo request, id 10862, seq 7, length 64
15:40:54.613347 IP 10.0.0.16 > 192.168.8.1: ICMP echo request, id 10862, seq 8, length 64
15:40:55.637314 IP 10.0.0.16 > 192.168.8.1: ICMP echo request, id 10862, seq 9, length 64
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel

=> Again shows the internal network address as source, which is why i assume it does not get routed back.

what does tcpdump on the 192.168.8.1 show?
No possible unfortunately, its a simple consumer router.

I can try a different target where i can run tcpdump.
This is from 192.168.16.91 > 192.168.8.37.
This is when it works:

root@odroid:~# tcpdump -ni any icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
15:45:02.158083 IP 192.168.16.91 > 192.168.8.37: ICMP echo request, id 11, seq 6, length 64
15:45:02.158143 IP 192.168.8.37 > 192.168.16.91: ICMP echo reply, id 11, seq 6, length 64
15:45:03.158597 IP 192.168.16.91 > 192.168.8.37: ICMP echo request, id 11, seq 7, length 64
15:45:03.158646 IP 192.168.8.37 > 192.168.16.91: ICMP echo reply, id 11, seq 7, length 64
15:45:04.159575 IP 192.168.16.91 > 192.168.8.37: ICMP echo request, id 11, seq 8, length 64
15:45:04.159617 IP 192.168.8.37 > 192.168.16.91: ICMP echo reply, id 11, seq 8, length 64
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel

This is from 192.168.16.32 > 192.168.8.37.
This is what it looks like when it doesn’t work

root@odroid:~# tcpdump -ni any icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
15:46:04.010763 IP 10.0.0.16 > 192.168.8.37: ICMP echo request, id 15428, seq 1, length 64
15:46:04.010838 IP 192.168.8.37 > 10.0.0.16: ICMP echo reply, id 15428, seq 1, length 64
15:46:05.018371 IP 10.0.0.16 > 192.168.8.37: ICMP echo request, id 15428, seq 2, length 64
15:46:05.018426 IP 192.168.8.37 > 10.0.0.16: ICMP echo reply, id 15428, seq 2, length 64
15:46:05.581456 IP 192.168.8.5 > 192.168.8.37: ICMP echo request, id 2156, seq 1, length 64
15:46:05.581496 IP 192.168.8.37 > 192.168.8.5: ICMP echo reply, id 2156, seq 1, length 64
15:46:06.038763 IP 10.0.0.16 > 192.168.8.37: ICMP echo request, id 15428, seq 3, length 64
15:46:06.038803 IP 192.168.8.37 > 10.0.0.16: ICMP echo reply, id 15428, seq 3, length 64
^C
8 packets captured
8 packets received by filter
0 packets dropped by kernel

Same picture really, it doesn’t set the right source address…

8

I’d skipped the br-xxxx parts of routes before, it didn’t occur to me you might be running wireguard in containers, … docker does NAT-ing by default, it’s probably what’s rewriting your IPs.

(but it could just be stupid fire alarm beeping at 3am for being low on battery - I’m sure that’s not helping me keep focus).

Are you using wireguard from linuxerver.io in docker-compose?

there’s some stuff from 2018 here, Is it possible to disable nat in docker-compose? - Compose - Docker Community Forums … might be worth checking documentation and the output of iptables-save from within the wireguard container, as well as on the host, … look for NAT there.

1 Like
9

Yeah, all in Docker.
No I use more lightweight and not so heavily overengineered docker image. The one from linuxserver.io I can’t get to do meshed site-to-site.
This one: GitHub - masipcat/wireguard-go-docker: Wireguard docker image

compose file:

root@docker2:/docker/wg-p2p# cat docker-compose.yaml 
version: '3.3'
services:
  wireguard:
    container_name: wg-p2p
    image: masipcat/wireguard-go:latest
    cap_add:
     - NET_ADMIN
    volumes:
     # This is what lets us create a wg interface without kernel module on host 
     - /dev/net/tun:/dev/net/tun
     # Folder with 'publickey', 'privatekey' and 'wg0.conf'
     - ./config:/etc/wireguard
    environment:
     - WG_COLOR_MODE=always
     - LOG_LEVEL=debug
    network_mode: host
    restart: unless-stopped

iptables-save from the host (I suck with iptables, so can’t really read anything from it:

root@docker2:/docker/wg-p2p# iptables-save 
# Generated by iptables-save v1.6.1 on Fri Jan  6 21:03:10 2023
*filter
:INPUT ACCEPT [1731755:1695605126]
:FORWARD ACCEPT [54845:25617873]
:OUTPUT ACCEPT [1335950:2972980831]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o br-75c60b839d17 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-75c60b839d17 -j DOCKER
-A FORWARD -i br-75c60b839d17 ! -o br-75c60b839d17 -j ACCEPT
-A FORWARD -i br-75c60b839d17 -o br-75c60b839d17 -j ACCEPT
-A FORWARD -o br-cc0379dc2ab0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-cc0379dc2ab0 -j DOCKER
-A FORWARD -i br-cc0379dc2ab0 ! -o br-cc0379dc2ab0 -j ACCEPT
-A FORWARD -i br-cc0379dc2ab0 -o br-cc0379dc2ab0 -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o br-9ce31703595b -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-9ce31703595b -j DOCKER
-A FORWARD -i br-9ce31703595b ! -o br-9ce31703595b -j ACCEPT
-A FORWARD -i br-9ce31703595b -o br-9ce31703595b -j ACCEPT
-A FORWARD -o br-d8b6936d1a8c -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-d8b6936d1a8c -j DOCKER
-A FORWARD -i br-d8b6936d1a8c ! -o br-d8b6936d1a8c -j ACCEPT
-A FORWARD -i br-d8b6936d1a8c -o br-d8b6936d1a8c -j ACCEPT
-A FORWARD -o br-6733bb4fa579 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-6733bb4fa579 -j DOCKER
-A FORWARD -i br-6733bb4fa579 ! -o br-6733bb4fa579 -j ACCEPT
-A FORWARD -i br-6733bb4fa579 -o br-6733bb4fa579 -j ACCEPT
-A FORWARD -o br-64627e6f93b1 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-64627e6f93b1 -j DOCKER
-A FORWARD -i br-64627e6f93b1 ! -o br-64627e6f93b1 -j ACCEPT
-A FORWARD -i br-64627e6f93b1 -o br-64627e6f93b1 -j ACCEPT
-A FORWARD -o br-4a2f194092e0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-4a2f194092e0 -j DOCKER
-A FORWARD -i br-4a2f194092e0 ! -o br-4a2f194092e0 -j ACCEPT
-A FORWARD -i br-4a2f194092e0 -o br-4a2f194092e0 -j ACCEPT
-A FORWARD -i br-1812446a9708 -o br-1812446a9708 -j ACCEPT
-A FORWARD -o br-9f75bcdd1f79 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-9f75bcdd1f79 -j DOCKER
-A FORWARD -i br-9f75bcdd1f79 ! -o br-9f75bcdd1f79 -j ACCEPT
-A FORWARD -i br-9f75bcdd1f79 -o br-9f75bcdd1f79 -j ACCEPT
-A FORWARD -o br-4e89d3b18e4e -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-4e89d3b18e4e -j DOCKER
-A FORWARD -i br-4e89d3b18e4e ! -o br-4e89d3b18e4e -j ACCEPT
-A FORWARD -i br-4e89d3b18e4e -o br-4e89d3b18e4e -j ACCEPT
-A DOCKER -d 172.20.0.3/32 ! -i br-64627e6f93b1 -o br-64627e6f93b1 -p tcp -m tcp --dport 3000 -j ACCEPT
-A DOCKER -d 172.29.0.2/32 ! -i br-6733bb4fa579 -o br-6733bb4fa579 -p tcp -m tcp --dport 8443 -j ACCEPT
-A DOCKER -d 172.29.0.2/32 ! -i br-6733bb4fa579 -o br-6733bb4fa579 -p tcp -m tcp --dport 8080 -j ACCEPT
-A DOCKER -d 172.28.0.2/32 ! -i br-9f75bcdd1f79 -o br-9f75bcdd1f79 -p tcp -m tcp --dport 9001 -j ACCEPT
-A DOCKER -d 172.31.0.4/32 ! -i br-cc0379dc2ab0 -o br-cc0379dc2ab0 -p tcp -m tcp --dport 8000 -j ACCEPT
-A DOCKER -d 192.168.32.2/32 ! -i br-75c60b839d17 -o br-75c60b839d17 -p tcp -m tcp --dport 9787 -j ACCEPT
-A DOCKER -d 172.24.0.2/32 ! -i br-4e89d3b18e4e -o br-4e89d3b18e4e -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER -d 172.24.0.2/32 ! -i br-4e89d3b18e4e -o br-4e89d3b18e4e -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER -d 172.24.0.5/32 ! -i br-4e89d3b18e4e -o br-4e89d3b18e4e -p udp -m udp --dport 10000 -j ACCEPT
-A DOCKER -d 172.24.0.5/32 ! -i br-4e89d3b18e4e -o br-4e89d3b18e4e -p tcp -m tcp --dport 4443 -j ACCEPT
-A DOCKER -d 172.25.0.8/32 ! -i br-9ce31703595b -o br-9ce31703595b -p tcp -m tcp --dport 9093 -j ACCEPT
-A DOCKER -d 172.25.0.8/32 ! -i br-9ce31703595b -o br-9ce31703595b -p tcp -m tcp --dport 9091 -j ACCEPT
-A DOCKER -d 172.25.0.8/32 ! -i br-9ce31703595b -o br-9ce31703595b -p tcp -m tcp --dport 9090 -j ACCEPT
-A DOCKER -d 172.25.0.8/32 ! -i br-9ce31703595b -o br-9ce31703595b -p tcp -m tcp --dport 3000 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-75c60b839d17 ! -o br-75c60b839d17 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-cc0379dc2ab0 ! -o br-cc0379dc2ab0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-9ce31703595b ! -o br-9ce31703595b -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-6733bb4fa579 ! -o br-6733bb4fa579 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-64627e6f93b1 ! -o br-64627e6f93b1 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-4a2f194092e0 ! -o br-4a2f194092e0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 ! -s 172.30.0.0/16 -o br-1812446a9708 -j DROP
-A DOCKER-ISOLATION-STAGE-1 ! -d 172.30.0.0/16 -i br-1812446a9708 -j DROP
-A DOCKER-ISOLATION-STAGE-1 -i br-9f75bcdd1f79 ! -o br-9f75bcdd1f79 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-d8b6936d1a8c ! -o br-d8b6936d1a8c -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-4e89d3b18e4e ! -o br-4e89d3b18e4e -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o br-75c60b839d17 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-cc0379dc2ab0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-9ce31703595b -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-6733bb4fa579 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-64627e6f93b1 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-4a2f194092e0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-9f75bcdd1f79 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-d8b6936d1a8c -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-4e89d3b18e4e -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Fri Jan  6 21:03:10 2023
# Generated by iptables-save v1.6.1 on Fri Jan  6 21:03:10 2023
*nat
:PREROUTING ACCEPT [82877:6442768]
:INPUT ACCEPT [17958:2526776]
:OUTPUT ACCEPT [26584:1795409]
:POSTROUTING ACCEPT [39619:2640965]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 192.168.32.0/20 ! -o br-75c60b839d17 -j MASQUERADE
-A POSTROUTING -s 172.31.0.0/16 ! -o br-cc0379dc2ab0 -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.25.0.0/16 ! -o br-9ce31703595b -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o br-d8b6936d1a8c -j MASQUERADE
-A POSTROUTING -s 172.29.0.0/16 ! -o br-6733bb4fa579 -j MASQUERADE
-A POSTROUTING -s 172.20.0.0/16 ! -o br-64627e6f93b1 -j MASQUERADE
-A POSTROUTING -s 172.21.0.0/16 ! -o br-4a2f194092e0 -j MASQUERADE
-A POSTROUTING -s 172.28.0.0/16 ! -o br-9f75bcdd1f79 -j MASQUERADE
-A POSTROUTING -s 172.24.0.0/16 ! -o br-4e89d3b18e4e -j MASQUERADE
-A POSTROUTING -s 172.20.0.3/32 -d 172.20.0.3/32 -p tcp -m tcp --dport 3000 -j MASQUERADE
-A POSTROUTING -s 172.29.0.2/32 -d 172.29.0.2/32 -p tcp -m tcp --dport 8443 -j MASQUERADE
-A POSTROUTING -s 172.29.0.2/32 -d 172.29.0.2/32 -p tcp -m tcp --dport 8080 -j MASQUERADE
-A POSTROUTING -s 172.28.0.2/32 -d 172.28.0.2/32 -p tcp -m tcp --dport 9001 -j MASQUERADE
-A POSTROUTING -s 172.31.0.4/32 -d 172.31.0.4/32 -p tcp -m tcp --dport 8000 -j MASQUERADE
-A POSTROUTING -s 192.168.32.2/32 -d 192.168.32.2/32 -p tcp -m tcp --dport 9787 -j MASQUERADE
-A POSTROUTING -s 172.24.0.2/32 -d 172.24.0.2/32 -p tcp -m tcp --dport 443 -j MASQUERADE
-A POSTROUTING -s 172.24.0.2/32 -d 172.24.0.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A POSTROUTING -s 172.24.0.5/32 -d 172.24.0.5/32 -p udp -m udp --dport 10000 -j MASQUERADE
-A POSTROUTING -s 172.24.0.5/32 -d 172.24.0.5/32 -p tcp -m tcp --dport 4443 -j MASQUERADE
-A POSTROUTING -s 172.25.0.8/32 -d 172.25.0.8/32 -p tcp -m tcp --dport 9093 -j MASQUERADE
-A POSTROUTING -s 172.25.0.8/32 -d 172.25.0.8/32 -p tcp -m tcp --dport 9091 -j MASQUERADE
-A POSTROUTING -s 172.25.0.8/32 -d 172.25.0.8/32 -p tcp -m tcp --dport 9090 -j MASQUERADE
-A POSTROUTING -s 172.25.0.8/32 -d 172.25.0.8/32 -p tcp -m tcp --dport 3000 -j MASQUERADE
-A DOCKER -i br-75c60b839d17 -j RETURN
-A DOCKER -i br-cc0379dc2ab0 -j RETURN
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i br-9ce31703595b -j RETURN
-A DOCKER -i br-6733bb4fa579 -j RETURN
-A DOCKER -i br-64627e6f93b1 -j RETURN
-A DOCKER -i br-4a2f194092e0 -j RETURN
-A DOCKER -i br-9f75bcdd1f79 -j RETURN
-A DOCKER -i br-d8b6936d1a8c -j RETURN
-A DOCKER -i br-4e89d3b18e4e -j RETURN
-A DOCKER ! -i br-64627e6f93b1 -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.20.0.3:3000
-A DOCKER ! -i br-6733bb4fa579 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 172.29.0.2:8443
-A DOCKER ! -i br-6733bb4fa579 -p tcp -m tcp --dport 8080 -j DNAT --to-destination 172.29.0.2:8080
-A DOCKER ! -i br-9f75bcdd1f79 -p tcp -m tcp --dport 9001 -j DNAT --to-destination 172.28.0.2:9001
-A DOCKER ! -i br-cc0379dc2ab0 -p tcp -m tcp --dport 8000 -j DNAT --to-destination 172.31.0.4:8000
-A DOCKER ! -i br-75c60b839d17 -p tcp -m tcp --dport 9787 -j DNAT --to-destination 192.168.32.2:9787
-A DOCKER ! -i br-4e89d3b18e4e -p tcp -m tcp --dport 7654 -j DNAT --to-destination 172.24.0.2:443
-A DOCKER ! -i br-4e89d3b18e4e -p tcp -m tcp --dport 6000 -j DNAT --to-destination 172.24.0.2:80
-A DOCKER ! -i br-4e89d3b18e4e -p udp -m udp --dport 10000 -j DNAT --to-destination 172.24.0.5:10000
-A DOCKER ! -i br-4e89d3b18e4e -p tcp -m tcp --dport 4443 -j DNAT --to-destination 172.24.0.5:4443
-A DOCKER ! -i br-9ce31703595b -p tcp -m tcp --dport 9093 -j DNAT --to-destination 172.25.0.8:9093
-A DOCKER ! -i br-9ce31703595b -p tcp -m tcp --dport 9091 -j DNAT --to-destination 172.25.0.8:9091
-A DOCKER ! -i br-9ce31703595b -p tcp -m tcp --dport 9090 -j DNAT --to-destination 172.25.0.8:9090
-A DOCKER ! -i br-9ce31703595b -p tcp -m tcp --dport 3000 -j DNAT --to-destination 172.25.0.8:3000
COMMIT
# Completed on Fri Jan  6 21:03:10 2023

iptables-save from container (looks the same to me at glance ?!)

# Generated by iptables-save v1.8.4 on Fri Jan  6 20:05:04 2023
*filter
:INPUT ACCEPT [1749843:1704393566]
:FORWARD ACCEPT [55012:25711407]
:OUTPUT ACCEPT [1352292:2976735814]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o br-75c60b839d17 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-75c60b839d17 -j DOCKER
-A FORWARD -i br-75c60b839d17 ! -o br-75c60b839d17 -j ACCEPT
-A FORWARD -i br-75c60b839d17 -o br-75c60b839d17 -j ACCEPT
-A FORWARD -o br-cc0379dc2ab0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-cc0379dc2ab0 -j DOCKER
-A FORWARD -i br-cc0379dc2ab0 ! -o br-cc0379dc2ab0 -j ACCEPT
-A FORWARD -i br-cc0379dc2ab0 -o br-cc0379dc2ab0 -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o br-9ce31703595b -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-9ce31703595b -j DOCKER
-A FORWARD -i br-9ce31703595b ! -o br-9ce31703595b -j ACCEPT
-A FORWARD -i br-9ce31703595b -o br-9ce31703595b -j ACCEPT
-A FORWARD -o br-d8b6936d1a8c -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-d8b6936d1a8c -j DOCKER
-A FORWARD -i br-d8b6936d1a8c ! -o br-d8b6936d1a8c -j ACCEPT
-A FORWARD -i br-d8b6936d1a8c -o br-d8b6936d1a8c -j ACCEPT
-A FORWARD -o br-6733bb4fa579 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-6733bb4fa579 -j DOCKER
-A FORWARD -i br-6733bb4fa579 ! -o br-6733bb4fa579 -j ACCEPT
-A FORWARD -i br-6733bb4fa579 -o br-6733bb4fa579 -j ACCEPT
-A FORWARD -o br-64627e6f93b1 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-64627e6f93b1 -j DOCKER
-A FORWARD -i br-64627e6f93b1 ! -o br-64627e6f93b1 -j ACCEPT
-A FORWARD -i br-64627e6f93b1 -o br-64627e6f93b1 -j ACCEPT
-A FORWARD -o br-4a2f194092e0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-4a2f194092e0 -j DOCKER
-A FORWARD -i br-4a2f194092e0 ! -o br-4a2f194092e0 -j ACCEPT
-A FORWARD -i br-4a2f194092e0 -o br-4a2f194092e0 -j ACCEPT
-A FORWARD -i br-1812446a9708 -o br-1812446a9708 -j ACCEPT
-A FORWARD -o br-9f75bcdd1f79 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-9f75bcdd1f79 -j DOCKER
-A FORWARD -i br-9f75bcdd1f79 ! -o br-9f75bcdd1f79 -j ACCEPT
-A FORWARD -i br-9f75bcdd1f79 -o br-9f75bcdd1f79 -j ACCEPT
-A FORWARD -o br-4e89d3b18e4e -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-4e89d3b18e4e -j DOCKER
-A FORWARD -i br-4e89d3b18e4e ! -o br-4e89d3b18e4e -j ACCEPT
-A FORWARD -i br-4e89d3b18e4e -o br-4e89d3b18e4e -j ACCEPT
-A DOCKER -d 172.20.0.3/32 ! -i br-64627e6f93b1 -o br-64627e6f93b1 -p tcp -m tcp --dport 3000 -j ACCEPT
-A DOCKER -d 172.29.0.2/32 ! -i br-6733bb4fa579 -o br-6733bb4fa579 -p tcp -m tcp --dport 8443 -j ACCEPT
-A DOCKER -d 172.29.0.2/32 ! -i br-6733bb4fa579 -o br-6733bb4fa579 -p tcp -m tcp --dport 8080 -j ACCEPT
-A DOCKER -d 172.28.0.2/32 ! -i br-9f75bcdd1f79 -o br-9f75bcdd1f79 -p tcp -m tcp --dport 9001 -j ACCEPT
-A DOCKER -d 172.31.0.4/32 ! -i br-cc0379dc2ab0 -o br-cc0379dc2ab0 -p tcp -m tcp --dport 8000 -j ACCEPT
-A DOCKER -d 192.168.32.2/32 ! -i br-75c60b839d17 -o br-75c60b839d17 -p tcp -m tcp --dport 9787 -j ACCEPT
-A DOCKER -d 172.24.0.2/32 ! -i br-4e89d3b18e4e -o br-4e89d3b18e4e -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER -d 172.24.0.2/32 ! -i br-4e89d3b18e4e -o br-4e89d3b18e4e -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER -d 172.24.0.5/32 ! -i br-4e89d3b18e4e -o br-4e89d3b18e4e -p udp -m udp --dport 10000 -j ACCEPT
-A DOCKER -d 172.24.0.5/32 ! -i br-4e89d3b18e4e -o br-4e89d3b18e4e -p tcp -m tcp --dport 4443 -j ACCEPT
-A DOCKER -d 172.25.0.8/32 ! -i br-9ce31703595b -o br-9ce31703595b -p tcp -m tcp --dport 9093 -j ACCEPT
-A DOCKER -d 172.25.0.8/32 ! -i br-9ce31703595b -o br-9ce31703595b -p tcp -m tcp --dport 9091 -j ACCEPT
-A DOCKER -d 172.25.0.8/32 ! -i br-9ce31703595b -o br-9ce31703595b -p tcp -m tcp --dport 9090 -j ACCEPT
-A DOCKER -d 172.25.0.8/32 ! -i br-9ce31703595b -o br-9ce31703595b -p tcp -m tcp --dport 3000 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-75c60b839d17 ! -o br-75c60b839d17 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-cc0379dc2ab0 ! -o br-cc0379dc2ab0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-9ce31703595b ! -o br-9ce31703595b -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-6733bb4fa579 ! -o br-6733bb4fa579 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-64627e6f93b1 ! -o br-64627e6f93b1 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-4a2f194092e0 ! -o br-4a2f194092e0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 ! -s 172.30.0.0/16 -o br-1812446a9708 -j DROP
-A DOCKER-ISOLATION-STAGE-1 ! -d 172.30.0.0/16 -i br-1812446a9708 -j DROP
-A DOCKER-ISOLATION-STAGE-1 -i br-9f75bcdd1f79 ! -o br-9f75bcdd1f79 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-d8b6936d1a8c ! -o br-d8b6936d1a8c -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-4e89d3b18e4e ! -o br-4e89d3b18e4e -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o br-75c60b839d17 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-cc0379dc2ab0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-9ce31703595b -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-6733bb4fa579 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-64627e6f93b1 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-4a2f194092e0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-9f75bcdd1f79 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-d8b6936d1a8c -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-4e89d3b18e4e -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Fri Jan  6 20:05:04 2023
# Generated by iptables-save v1.8.4 on Fri Jan  6 20:05:04 2023
*nat
:PREROUTING ACCEPT [82985:6451470]
:INPUT ACCEPT [17986:2530582]
:OUTPUT ACCEPT [26622:1798192]
:POSTROUTING ACCEPT [39676:2644984]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 192.168.32.0/20 ! -o br-75c60b839d17 -j MASQUERADE
-A POSTROUTING -s 172.31.0.0/16 ! -o br-cc0379dc2ab0 -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.25.0.0/16 ! -o br-9ce31703595b -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o br-d8b6936d1a8c -j MASQUERADE
-A POSTROUTING -s 172.29.0.0/16 ! -o br-6733bb4fa579 -j MASQUERADE
-A POSTROUTING -s 172.20.0.0/16 ! -o br-64627e6f93b1 -j MASQUERADE
-A POSTROUTING -s 172.21.0.0/16 ! -o br-4a2f194092e0 -j MASQUERADE
-A POSTROUTING -s 172.28.0.0/16 ! -o br-9f75bcdd1f79 -j MASQUERADE
-A POSTROUTING -s 172.24.0.0/16 ! -o br-4e89d3b18e4e -j MASQUERADE
-A POSTROUTING -s 172.20.0.3/32 -d 172.20.0.3/32 -p tcp -m tcp --dport 3000 -j MASQUERADE
-A POSTROUTING -s 172.29.0.2/32 -d 172.29.0.2/32 -p tcp -m tcp --dport 8443 -j MASQUERADE
-A POSTROUTING -s 172.29.0.2/32 -d 172.29.0.2/32 -p tcp -m tcp --dport 8080 -j MASQUERADE
-A POSTROUTING -s 172.28.0.2/32 -d 172.28.0.2/32 -p tcp -m tcp --dport 9001 -j MASQUERADE
-A POSTROUTING -s 172.31.0.4/32 -d 172.31.0.4/32 -p tcp -m tcp --dport 8000 -j MASQUERADE
-A POSTROUTING -s 192.168.32.2/32 -d 192.168.32.2/32 -p tcp -m tcp --dport 9787 -j MASQUERADE
-A POSTROUTING -s 172.24.0.2/32 -d 172.24.0.2/32 -p tcp -m tcp --dport 443 -j MASQUERADE
-A POSTROUTING -s 172.24.0.2/32 -d 172.24.0.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A POSTROUTING -s 172.24.0.5/32 -d 172.24.0.5/32 -p udp -m udp --dport 10000 -j MASQUERADE
-A POSTROUTING -s 172.24.0.5/32 -d 172.24.0.5/32 -p tcp -m tcp --dport 4443 -j MASQUERADE
-A POSTROUTING -s 172.25.0.8/32 -d 172.25.0.8/32 -p tcp -m tcp --dport 9093 -j MASQUERADE
-A POSTROUTING -s 172.25.0.8/32 -d 172.25.0.8/32 -p tcp -m tcp --dport 9091 -j MASQUERADE
-A POSTROUTING -s 172.25.0.8/32 -d 172.25.0.8/32 -p tcp -m tcp --dport 9090 -j MASQUERADE
-A POSTROUTING -s 172.25.0.8/32 -d 172.25.0.8/32 -p tcp -m tcp --dport 3000 -j MASQUERADE
-A DOCKER -i br-75c60b839d17 -j RETURN
-A DOCKER -i br-cc0379dc2ab0 -j RETURN
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i br-9ce31703595b -j RETURN
-A DOCKER -i br-6733bb4fa579 -j RETURN
-A DOCKER -i br-64627e6f93b1 -j RETURN
-A DOCKER -i br-4a2f194092e0 -j RETURN
-A DOCKER -i br-9f75bcdd1f79 -j RETURN
-A DOCKER -i br-d8b6936d1a8c -j RETURN
-A DOCKER -i br-4e89d3b18e4e -j RETURN
-A DOCKER ! -i br-64627e6f93b1 -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.20.0.3:3000
-A DOCKER ! -i br-6733bb4fa579 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 172.29.0.2:8443
-A DOCKER ! -i br-6733bb4fa579 -p tcp -m tcp --dport 8080 -j DNAT --to-destination 172.29.0.2:8080
-A DOCKER ! -i br-9f75bcdd1f79 -p tcp -m tcp --dport 9001 -j DNAT --to-destination 172.28.0.2:9001
-A DOCKER ! -i br-cc0379dc2ab0 -p tcp -m tcp --dport 8000 -j DNAT --to-destination 172.31.0.4:8000
-A DOCKER ! -i br-75c60b839d17 -p tcp -m tcp --dport 9787 -j DNAT --to-destination 192.168.32.2:9787
-A DOCKER ! -i br-4e89d3b18e4e -p tcp -m tcp --dport 7654 -j DNAT --to-destination 172.24.0.2:443
-A DOCKER ! -i br-4e89d3b18e4e -p tcp -m tcp --dport 6000 -j DNAT --to-destination 172.24.0.2:80
-A DOCKER ! -i br-4e89d3b18e4e -p udp -m udp --dport 10000 -j DNAT --to-destination 172.24.0.5:10000
-A DOCKER ! -i br-4e89d3b18e4e -p tcp -m tcp --dport 4443 -j DNAT --to-destination 172.24.0.5:4443
-A DOCKER ! -i br-9ce31703595b -p tcp -m tcp --dport 9093 -j DNAT --to-destination 172.25.0.8:9093
-A DOCKER ! -i br-9ce31703595b -p tcp -m tcp --dport 9091 -j DNAT --to-destination 172.25.0.8:9091
-A DOCKER ! -i br-9ce31703595b -p tcp -m tcp --dport 9090 -j DNAT --to-destination 172.25.0.8:9090
-A DOCKER ! -i br-9ce31703595b -p tcp -m tcp --dport 3000 -j DNAT --to-destination 172.25.0.8:3000
COMMIT
# Completed on Fri Jan  6 20:05:04 2023

Hope you get the firealarm fixed :sweat_smile:

10

Ahhh, progress…
Link the repo, prompted me, to go through it again, and i cound in the example config post and preup lines:

PostUp = wg set wg0 private-key /etc/wireguard/privatekey && iptables -t nat -A POSTROUTING -s 10.99.0.0/24 -o eth0 -j MASQUERADE
PostDown = iptables -t nat -D POSTROUTING -s 10.99.0.0/24 -o eth0 -j MASQUERADE

the part with wg set wg0 private-key /etc/wireguard/privatekey doesn’t really work and isnt needed but the other has and effect.

(See Edit) Basically the reverse happens, with these rules applied. I can pring the other subnet from the wireguard peer but not from anything other host in network.
TCP Dump looks exact same.

I’ll try to do some more digging.

Edit:
Was a bit too quick. I can still ping non wg host from other non-wg hosts.
I can also now ping wghost to wghost through the tunnel.
what doesn’t work is wghost to non wghost through tunnel.

Final Edit:
Made it work. In the end it was the missing lines for post- and preup.
What i tried initally tried:
iptables -t nat -A POSTROUTING -s 10.0.0.8/32 -o eth0 -j MASQUERADE
This will not work for the other subnets.
Solution is:
iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE

Shout to @risk for guiding me through and pointing me in the right direction!