[SOLVED] Pihole rejecting requests from wg0 clients

Four0Four · January 9, 2022, 6:59pm

So I’ve been beating my head up against the wall since last night on this one. I’m trying to setup a new Wireguard + Pihole instance on Oracle Cloud in the same way I’ve done many times on AWS, Digital Ocean, and Vultr. There were no issues installing Wireguard, handshaking, or forwarding client traffic through the VPS… everything works as expected with wireguard.

The second half of the setup is to install pihole using the automated installation method ( curl -sSL https://install.pi-hole.net | bash). I choose wg0 for the interface and accept all the other recommended defaults. Installation completes without any errors and I’m able to access the pihole admin console with ssh port forwarding.

However my wireguard client isn’t able to access either the pihole admin console nor make DNS lookups to pihole. It’s receiving ICMP host 10.6.0.1 unreachable - admin prohibited but SSH and ping work fine to the server (10.6.0.1). Does this sound like an an issue with the pihole service?

Things I’ve Tried So Far:

Disabling UFW
Selecting “Permit all origins” in pihole under Settings → DNS → Interface settings
Starting from scratch at least 4 times ← Definition of insanity.
Checking IP tables for any obvious drop rules (although I lack experience here)
Adding a rule to allow any from wg0 to wg0

Ubuntu 20.04.3 LTS (GNU/Linux 5.11.0-1023-oracle aarch64)
Wireguard Server (wg0) : 10.6.0.1
Wireguard Client: 10.6.0.2

Pi-hole admin console and DNS blocked

15:09:08.384240 IP 10.6.0.2.64043 > 10.6.0.1.80: Flags [SEW], seq 2044417045, win 65535, options [mss 1240,nop,wscale 6,nop,nop,TS val 1116977908 ecr 0,sackOK,eol], length 0
15:09:08.384277 IP 10.6.0.1 > 10.6.0.2: ICMP host 10.6.0.1 unreachable - admin prohibited, length 72
18:08:45.874138 IP 10.6.0.2.60674 > 10.6.0.1.53: 60560+ PTR? lb._dns-sd._udp.254.141.0.10.in-addr.arpa. (59)
18:08:45.874174 IP 10.6.0.1 > 10.6.0.2: ICMP host 10.6.0.1 unreachable - admin prohibited, length 95

But SSH and ICMP ping work fine to the same host

18:04:05.372086 IP 10.6.0.2.53986 > 10.6.0.1.22: Flags [SEW], seq 197032587, win 65535, options [mss 1240,nop,wscale 6,nop,nop,TS val 4278353387 ecr 0,sackOK,eol], length 0
18:04:05.372286 IP 10.6.0.1.22 > 10.6.0.2.53986: Flags [S.E], seq 161738644, ack 197032588, win 64296, options [mss 1380,sackOK,TS val 281199899 ecr 4278353387,nop,wscale 7], length 0
15:12:04.542437 IP 10.6.0.2 > 10.6.0.1: ICMP echo request, id 52036, seq 1, length 64
15:12:04.542470 IP 10.6.0.1 > 10.6.0.2: ICMP echo reply, id 52036, seq 1, length 64

Routes

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.0.0.1        0.0.0.0         UG    100    0        0 enp0s3
10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 enp0s3
10.6.0.0        0.0.0.0         255.255.255.0   U     0      0        0 wg0
link-local      0.0.0.0         255.255.0.0     U     100    0        0 enp0s3

Netstat

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:53              0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:4711          0.0.0.0:*               LISTEN     
tcp        0    316 10.0.0.32:22            redacted:33572          ESTABLISHED
tcp        0      0 10.0.0.32:43860         169.254.169.254:80      ESTABLISHED
tcp6       0      0 :::111                  :::*                    LISTEN     
tcp6       0      0 :::80                   :::*                    LISTEN     
tcp6       0      0 :::53                   :::*                    LISTEN     
tcp6       0      0 ::1:4711                :::*                    LISTEN     
udp        0      0 0.0.0.0:9494            0.0.0.0:*                          
udp        0      0 0.0.0.0:53              0.0.0.0:*                          
udp        0      0 10.0.0.32:68            0.0.0.0:*                          
udp        0      0 0.0.0.0:111             0.0.0.0:*                          
udp6       0      0 :::9494                 :::*                               
udp6       0      0 :::53                   :::*                               
udp6       0      0 :::111                  :::*

IP Tables

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     udp  --  anywhere             anywhere             udp dpt:9494 /* wireguard-input-rule */
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             anywhere             udp spt:ntp
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:ssh
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited
ufw-before-logging-input  all  --  anywhere             anywhere            
ufw-before-input  all  --  anywhere             anywhere            
ufw-after-input  all  --  anywhere             anywhere            
ufw-after-logging-input  all  --  anywhere             anywhere            
ufw-reject-input  all  --  anywhere             anywhere            
ufw-track-input  all  --  anywhere             anywhere            

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             10.6.0.0/24          ctstate RELATED,ESTABLISHED /* wireguard-forward-rule */
ACCEPT     all  --  10.6.0.0/24          anywhere             /* wireguard-forward-rule */
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited
ufw-before-logging-forward  all  --  anywhere             anywhere            
ufw-before-forward  all  --  anywhere             anywhere            
ufw-after-forward  all  --  anywhere             anywhere            
ufw-after-logging-forward  all  --  anywhere             anywhere            
ufw-reject-forward  all  --  anywhere             anywhere            
ufw-track-forward  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
InstanceServices  all  --  anywhere             169.254.0.0/16      
ufw-before-logging-output  all  --  anywhere             anywhere            
ufw-before-output  all  --  anywhere             anywhere            
ufw-after-output  all  --  anywhere             anywhere            
ufw-after-logging-output  all  --  anywhere             anywhere            
ufw-reject-output  all  --  anywhere             anywhere            
ufw-track-output  all  --  anywhere             anywhere            

Chain InstanceServices (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             169.254.0.2          owner UID match root tcp dpt:iscsi-target /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     tcp  --  anywhere             169.254.2.0/24       owner UID match root tcp dpt:iscsi-target /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     tcp  --  anywhere             169.254.4.0/24       owner UID match root tcp dpt:iscsi-target /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     tcp  --  anywhere             169.254.5.0/24       owner UID match root tcp dpt:iscsi-target /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     tcp  --  anywhere             169.254.0.2          tcp dpt:http /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     udp  --  anywhere             169.254.169.254      udp dpt:domain /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     tcp  --  anywhere             169.254.169.254      tcp dpt:domain /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     tcp  --  anywhere             169.254.0.3          owner UID match root tcp dpt:http /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     tcp  --  anywhere             169.254.0.4          tcp dpt:http /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     tcp  --  anywhere             169.254.169.254      tcp dpt:http /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     udp  --  anywhere             169.254.169.254      udp dpt:bootps /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     udp  --  anywhere             169.254.169.254      udp dpt:tftp /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     udp  --  anywhere             169.254.169.254      udp dpt:ntp /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
REJECT     tcp  --  anywhere             169.254.0.0/16       tcp /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */ reject-with tcp-reset
REJECT     udp  --  anywhere             169.254.0.0/16       udp /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */ reject-with icmp-port-unreachable

Chain ufw-after-forward (1 references)
target     prot opt source               destination         

Chain ufw-after-input (1 references)
target     prot opt source               destination         

Chain ufw-after-logging-forward (1 references)
target     prot opt source               destination         

Chain ufw-after-logging-input (1 references)
target     prot opt source               destination         

Chain ufw-after-logging-output (1 references)
target     prot opt source               destination         

Chain ufw-after-output (1 references)
target     prot opt source               destination         

Chain ufw-before-forward (1 references)
target     prot opt source               destination         

Chain ufw-before-input (1 references)
target     prot opt source               destination         

Chain ufw-before-logging-forward (1 references)
target     prot opt source               destination         

Chain ufw-before-logging-input (1 references)
target     prot opt source               destination         

Chain ufw-before-logging-output (1 references)
target     prot opt source               destination         

Chain ufw-before-output (1 references)
target     prot opt source               destination         

Chain ufw-reject-forward (1 references)
target     prot opt source               destination         

Chain ufw-reject-input (1 references)
target     prot opt source               destination         

Chain ufw-reject-output (1 references)
target     prot opt source               destination         

Chain ufw-track-forward (1 references)
target     prot opt source               destination         

Chain ufw-track-input (1 references)
target     prot opt source               destination         

Chain ufw-track-output (1 references)
target     prot opt source               destination

UFW

$ sudo ufw status
Status: inactive

Thanks!

risk · January 9, 2022, 7:21pm

It’s your firewall:

REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

You can maybe do iptables -I INPUT 1 -i wg0 -j ACCEPT if you trust Wireguard…

… I don’t know how to use UFW sadly.

Four0Four · January 9, 2022, 8:20pm

Thank you. Guess I need to learn what I’m looking at in ip tables someday. I thought rule #4 in INPUT was an allow all.

ACCEPT     all  --  anywhere             anywhere

And now I think I see the issue that I’ve re-enabled UFW. Were the UFW rules not working because they are ordered below the REJECT rule?

Chain INPUT (policy DROP)
target     prot opt source               destination         
ACCEPT     udp  --  anywhere             anywhere             udp dpt:9494 /* wireguard-input-rule */
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             anywhere             udp spt:ntp
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:ssh
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited
ufw-before-logging-input  all  --  anywhere             anywhere            
ufw-before-input  all  --  anywhere             anywhere            
ufw-after-input  all  --  anywhere             anywhere            
ufw-after-logging-input  all  --  anywhere             anywhere            
ufw-reject-input  all  --  anywhere             anywhere            
ufw-track-input  all  --  anywhere             anywhere

Chain ufw-user-input (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh
ACCEPT     udp  --  anywhere             anywhere             udp dpt:9494
ACCEPT     tcp  --  10.6.0.0/24          anywhere             tcp dpt:domain
ACCEPT     udp  --  10.6.0.0/24          anywhere             udp dpt:domain
ACCEPT     tcp  --  10.6.0.0/24          anywhere             tcp dpt:http
ACCEPT     udp  --  10.6.0.0/24          anywhere             udp dpt:80

Chain ufw-user-forward (1 references)
target     prot opt source               destination         
ACCEPT     all  --  10.6.0.0/24          anywhere

Chain ufw-before-input (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ufw-logging-deny  all  --  anywhere             anywhere             ctstate INVALID
DROP       all  --  anywhere             anywhere             ctstate INVALID
ACCEPT     icmp --  anywhere             anywhere             icmp destination-unreachable
ACCEPT     icmp --  anywhere             anywhere             icmp time-exceeded
ACCEPT     icmp --  anywhere             anywhere             icmp parameter-problem
ACCEPT     icmp --  anywhere             anywhere             icmp echo-request
ACCEPT     udp  --  anywhere             anywhere             udp spt:bootps dpt:bootpc
ufw-not-local  all  --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             224.0.0.251          udp dpt:mdns
ACCEPT     udp  --  anywhere             239.255.255.250      udp dpt:1900
ufw-user-input  all  --  anywhere             anywhere

Thanks again

risk · January 9, 2022, 9:06pm

Crash course:

Look at output of iptables-save, it might make more sense than iptables -L

You can put that into a file, edit it, and try to load it atomically with iptables-restore

There’s several “tables” like “filter” and “nat” and there’s several built-in standard chains like “INPUT” “FORWARD” and “OUTPUT”, in each. Overall there’s like 10 built-in rule chains, and they all have a default policy, in your case ACCEPT.

In each rule chain, “first match” determines the action, and further processing stops.

In each rule -j is followed by an action name, there’s a few built-in ones like REJECT/ACCEPT/DROP … but -j could also be followed by a name of a custom chain… UFW apparently makes custom chains, you can see those in the output of iptables-save.

Each rule can optionally have match expressions, if it doesn’t the rule applies to all packets.

There’s some built-in matchers, like -i for input interface or -o for output interface. … and there’s other matchers that come as various kernel modules you can reference with -m matcher_name and then configure the matcher with arguments.

For example:

-A input -i enp3s0 -m multiport --dports 22,44,222,2222 -m comment --comment "open various ssh ports" -j ACCEPT

Uses multiple named matchers, and a default matcher.

There’s a ton of diagrams illustrating the ordering in which these tables and rule chains are applied e.g. this one:

Just type in iptables diagram into Google

Firewall can interact with connection tracking via matchers, so for example a common thing is

-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -j REJECT

… firewall will check whether the packet belongs to a known connection and will allow it; …
… in this case, if you couple the rule above with -A OUTPUT -j ACCEPT you end up with a machine that will only accept incoming packets for connections it itself initiatied, and will drop other incoming network packets.

Four0Four · January 9, 2022, 9:28pm

So fun fact. UFW is broken out of the box Oracle’s Ubuntu 20.04 aarch64 image. It comes preinstalled but the rule ordering places it below -A INPUT -j REJECT --reject-with icmp-host-prohibited

Looking at iptables-save I see that the #4 allow statement was only for the loopback. Thanks again for answering such a basic question. All I’ve ever looked at was iptables -L and now I understand why it’s always been so confusing. I’ll stop using UFW and force myself to use iptables moving forward… but maybe save nftables for next week though

Trooper_ish · January 10, 2022, 11:52am

Huh, that is odd; I thought UFW was just a front end for implementing iptables rules in a more “user friendly “ way.
It should make iptables-save etc unnecessary.

But if the rules it is setting in iptables are out of order, it is literally not doing the one job it has…

Is it just Oracles’s spin of UFW? (As far as you’ve seen?)

Four0Four · January 10, 2022, 1:13pm

Yes, Oracle is the only provider I’ve used which has predefined firewall rules. And those rules take precedence over ufw when enabled. Every other provider I’ve used just has a default allow (and a default drop on the network-based firewall).

Here's what iptables looks like on a new instance

# Generated by iptables-save v1.8.4 on Mon Jan 10 12:50:41 2022
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [187:27223]
:InstanceServices - [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p udp -m udp --sport 123 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -d 169.254.0.0/16 -j InstanceServices
-A InstanceServices -d 169.254.0.2/32 -p tcp -m owner --uid-owner 0 -m tcp --dport 3260 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.2.0/24 -p tcp -m owner --uid-owner 0 -m tcp --dport 3260 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.4.0/24 -p tcp -m owner --uid-owner 0 -m tcp --dport 3260 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.5.0/24 -p tcp -m owner --uid-owner 0 -m tcp --dport 3260 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.0.2/32 -p tcp -m tcp --dport 80 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p udp -m udp --dport 53 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p tcp -m tcp --dport 53 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.0.3/32 -p tcp -m owner --uid-owner 0 -m tcp --dport 80 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.0.4/32 -p tcp -m tcp --dport 80 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p udp -m udp --dport 67 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p udp -m udp --dport 69 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p udp -m udp --dport 123 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.0.0/16 -p tcp -m tcp -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j REJECT --reject-with tcp-reset
-A InstanceServices -d 169.254.0.0/16 -p udp -m udp -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j REJECT --reject-with icmp-port-unreachable
COMMIT
# Completed on Mon Jan 10 12:50:41 2022

Trooper_ish · January 10, 2022, 1:25pm

Nice, and it comes pre-installed with that? Looks like they have an intentional reason.

Thanks for sharing