Docker Container/Network Problems on Raspberry Pi (DNS over TLS; unbound+stubby)

Operating Systems & Open Source Linux
1

Hello,
I have a problem with my docker setup on a Raspberry Pi 3 Model B with Raspbian Stretch Lite.

I wanted to setup a local dns forwarder with DNS over TLS.

So I have basically 2(+1) containers:
stubby-main
unbound-main
alpine-test (for testing purposes only)

and 2 docker networks:
dns-main (bridge - 192.168.1.0/24)
macvlan (main net is 10.0.0.0/23)

When I run stubby-main and run bash inside to check whether it is working or not, all seems fine. ‘drill @127.0.0.1 -p 8053 google.com’ works and with netstat I can see that stubby listens on all interfaces (0.0.0.0:8053). The container is in the dns-main network and has the ip 192.168.1.5.
To test the network, I ran a little alpine container in the same network with the ip 192.168.1.3, attached to it and ran ‘drill @192.168.1.5 -p 8053 google.com’ which gave me the error: ‘Could not send or receive, because of network error’. So I tried to ping both with ip and hostname of the stubby-main container and it worked fine. I even ran nmap to see if the port was open, which it was.

Can someone help me get to the bottom of this?

Output of ‘docker inspect dns-main’:

    "Containers": {
        "1f54075cceebd2be70675623a3e8c4bd12265b0c8a84a8385c234baebaf48a28": {
            "Name": "stubby-main",
            "EndpointID": "7bea21087db0a21f20bd92ac3e3a51658140b1dea4b74796d9128e2599094e52",
            "MacAddress": "02:42:c0:a8:01:05",
            "IPv4Address": "192.168.1.5/24",
            "IPv6Address": ""
        },
        "47014c4122261c707369e38b37c8a0f7721660a607397db3edecc6626b8ac459": {
            "Name": "alpine-test",
            "EndpointID": "06fcfdc6b17298c1f08e158ad7e69f4de31e3501ca48cd2e15db4f5daa370be3",
            "MacAddress": "02:42:c0:a8:01:03",
            "IPv4Address": "192.168.1.3/24",
            "IPv6Address": ""
        }
    }

Sincerly
MrNightfloor

2

stubby (https://github.com/getdnsapi/stubby) describes itself as a " local DNS Privacy stub resolver", are you sure it will proxy remote requests?

3

I will do some further testing tomorrow, it is late where I am. But I believe it should work, I got the idea from someone else and only modified the Dockerfile for the Raspberry Pi.
The original docker-compose.yml was also supposed to start 2 containers, one for stubby and one for unbound. DNS queries should go to unbound and unbound should use stubby as a forwarder.
What I will do tomorrow, is compile the same stubby inside a chroot and try to connect to it from another machine.
Also, I am unsure what should happen when I try to query unbound for a DNS request and unbound can’t resolve that because it has no working forwarders. But I had the unbound container up and running, connected to both the dns-main network and the macvlan network with the ip address 10.0.0.2 exposed in my local network. I could ping the ip just fine but didn’t get a response at all for dns queries with e.g. dig from another machine on the network.
I think something with the containers networking isn’t working as it’s supposed to. But I can’t figure out what exactly…

4

You’re right. Stubby will proxy connections. I created a setup analogous to yours, with an lxc container on my router with hostname stubby-container (it was assigned the local ip fd00:8801:2909:6002::22 and global ip 2600:8801:2907:b02::22), installed stubby, and set it to listen on port 8053. The only change I made to the default config was to make it listen on all interfaces instead of just the loop back address:

listen_addresses:                                                                                                                                                             
#  - 127.0.0.1
#  - 0::1                                                                                  
  -  0::0@8053

And on another host, I sent the request:

clifford@Ubuntu-Laptop:~$ nslookup -type=AAAA -port=8053 ipv6.danger-rocket.com fd00:8801:2909:6002::22

which returned:

Server:		fd00:8801:2909:6002::22
Address:	fd00:8801:2909:6002::22#8053

Non-authoritative answer:
Name:	ipv6.danger-rocket.com
Address: 2600:1f14:229:a900:d027:2ea9:f122:e0eb

And I ran tcpdump on the bridge interface for the router’s lxc containers:

clifford@router:~$ sudo tcpdump -n -i br2 port 8053 or port 853

And tcpdump dutifully captured the unencrypted udp request from my laptop to stubby’s socket listening on port 8053:

11:59:13.796137 IP6 fd00:8801:2909:6001:f75d:252:4257:ea05.44808 > fd00:8801:2909:6002::22.8053: UDP, length 40

then stubby’s tls conversation with remote dns server’s listening interface on port 853:

11:59:13.950150 IP6 2600:8801:2907:b02::22.38232 > 2001:610:1:40ba:145:100:185:15.853: Flags [.], ack 1, win 225, options [nop,nop,TS val 89407790 ecr 777406519], length 0
11:59:13.950247 IP6 2600:8801:2907:b02::22.38232 > 2001:610:1:40ba:145:100:185:15.853: Flags [P.], seq 1:390, ack 1, win 225, options [nop,nop,TS val 89407790 ecr 777406519], length 389
11:59:14.106212 IP6 2001:610:1:40ba:145:100:185:15.853 > 2600:8801:2907:b02::22.38232: Flags [.], seq 1:1429, ack 390, win 232, options [nop,nop,TS val 777406557 ecr 89407790], length 1428
11:59:14.106233 IP6 2600:8801:2907:b02::22.38232 > 2001:610:1:40ba:145:100:185:15.853: Flags [.], ack 1429, win 248, options [nop,nop,TS val 89407829 ecr 777406557], length 0
11:59:14.107196 IP6 2001:610:1:40ba:145:100:185:15.853 > 2600:8801:2907:b02::22.38232: Flags [P.], seq 1429:2994, ack 390, win 232, options [nop,nop,TS val 777406557 ecr 89407790], length 1565
11:59:14.107214 IP6 2600:8801:2907:b02::22.38232 > 2001:610:1:40ba:145:100:185:15.853: Flags [.], ack 2994, win 272, options [nop,nop,TS val 89407830 ecr 777406557], length 0
11:59:14.107842 IP6 2600:8801:2907:b02::22.38232 > 2001:610:1:40ba:145:100:185:15.853: Flags [P.], seq 390:516, ack 2994, win 272, options [nop,nop,TS val 89407830 ecr 777406557], length 126
11:59:14.260542 IP6 2001:610:1:40ba:145:100:185:15.853 > 2600:8801:2907:b02::22.38232: Flags [P.], seq 2994:3252, ack 516, win 232, options [nop,nop,TS val 777406596 ecr 89407830], length 258
11:59:14.260676 IP6 2600:8801:2907:b02::22.38232 > 2001:610:1:40ba:145:100:185:15.853: Flags [P.], seq 516:675, ack 3252, win 295, options [nop,nop,TS val 89407868 ecr 777406596], length 159
11:59:14.426571 IP6 2001:610:1:40ba:145:100:185:15.853 > 2600:8801:2907:b02::22.38232: Flags [P.], seq 3252:3751, ack 675, win 240, options [nop,nop,TS val 777406638 ecr 89407868], length 499

and then stubby’s unencrypted udp response back to my laptop:

11:59:14.426702 IP6 fd00:8801:2909:6002::22.8053 > fd00:8801:2909:6001:f75d:252:4257:ea05.44808: UDP, length 490

I also ran tcpdump on the stubby-container:

clifford@stubby-container:/etc/stubby$ sudo tcpdump -n -i eth0 port 853 or port 8053

It captured the same traffic:

11:59:13.796141 IP6 fd00:8801:2909:6001:f75d:252:4257:ea05.44808 > fd00:8801:2909:6002::22.8053: UDP, length 40
11:59:13.796261 IP6 2600:8801:2907:b02::22.38232 > 2001:610:1:40ba:145:100:185:15.853: Flags [S], seq 1860576603, win 28800, options [mss 1440,sackOK,TS val 89407752 ecr 0,nop,wscale 7], length 0
11:59:13.950129 IP6 2001:610:1:40ba:145:100:185:15.853 > 2600:8801:2907:b02::22.38232: Flags [S.], seq 622492633, ack 1860576604, win 28560, options [mss 1440,sackOK,TS val 777406519 ecr 89407752,nop,wscale 7], length 0
11:59:13.950148 IP6 2600:8801:2907:b02::22.38232 > 2001:610:1:40ba:145:100:185:15.853: Flags [.], ack 1, win 225, options [nop,nop,TS val 89407790 ecr 777406519], length 0
11:59:13.950244 IP6 2600:8801:2907:b02::22.38232 > 2001:610:1:40ba:145:100:185:15.853: Flags [P.], seq 1:390, ack 1, win 225, options [nop,nop,TS val 89407790 ecr 777406519], length 389
11:59:14.106216 IP6 2001:610:1:40ba:145:100:185:15.853 > 2600:8801:2907:b02::22.38232: Flags [.], seq 1:1429, ack 390, win 232, options [nop,nop,TS val 777406557 ecr 89407790], length 1428
11:59:14.106232 IP6 2600:8801:2907:b02::22.38232 > 2001:610:1:40ba:145:100:185:15.853: Flags [.], ack 1429, win 248, options [nop,nop,TS val 89407829 ecr 777406557], length 0
11:59:14.107200 IP6 2001:610:1:40ba:145:100:185:15.853 > 2600:8801:2907:b02::22.38232: Flags [P.], seq 1429:2994, ack 390, win 232, options [nop,nop,TS val 777406557 ecr 89407790], length 1565
11:59:14.107213 IP6 2600:8801:2907:b02::22.38232 > 2001:610:1:40ba:145:100:185:15.853: Flags [.], ack 2994, win 272, options [nop,nop,TS val 89407830 ecr 777406557], length 0
11:59:14.107841 IP6 2600:8801:2907:b02::22.38232 > 2001:610:1:40ba:145:100:185:15.853: Flags [P.], seq 390:516, ack 2994, win 272, options [nop,nop,TS val 89407830 ecr 777406557], length 126
11:59:14.260545 IP6 2001:610:1:40ba:145:100:185:15.853 > 2600:8801:2907:b02::22.38232: Flags [P.], seq 2994:3252, ack 516, win 232, options [nop,nop,TS val 777406596 ecr 89407830], length 258
11:59:14.260674 IP6 2600:8801:2907:b02::22.38232 > 2001:610:1:40ba:145:100:185:15.853: Flags [P.], seq 516:675, ack 3252, win 295, options [nop,nop,TS val 89407868 ecr 777406596], length 159
11:59:14.426574 IP6 2001:610:1:40ba:145:100:185:15.853 > 2600:8801:2907:b02::22.38232: Flags [P.], seq 3252:3751, ack 675, win 240, options [nop,nop,TS val 777406638 ecr 89407868], length 499
11:59:14.426699 IP6 fd00:8801:2909:6002::22.8053 > fd00:8801:2909:6001:f75d:252:4257:ea05.44808: UDP, length 490

That is probably your best bet to find where your traffic being dropped. Run tcpdump on your container’s bridge interface and also on the container’s interface. Then run remote dns query. The output from tcpdump will show you exactly where traffic is being dropped and you can adjust your firewall rules accordingly.

5

Okay, I am lost.

Here is what I did:

  • start the stubby-main container
  • ran tcpdump inside this container
  • start the alpine-test container inside the same docker network
  • ran tcpdump on the host with the docker network as interface
  • ping from alpine-test to stubby-main: Success!
  • ‘drill @192.168.1.5 -p 8053’ (stubby-main): Failed!

tcpdump showed nothing on both host and stubby-main from the alpine-test ip 192.168.1.3

  • ‘netstat -plnt’ inside stubby-main just to be sure:

root@stubby-main:/opt/stubby# netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.11:37283 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:8053 0.0.0.0:* LISTEN -

  • edited the stubby.yml to include 192.168.1.5@8053 specifically
  • restarted the stubby-main container

tried above steps again and nothing showed up in tcpdump

  • compiled and installed the exact same stubby from the docker container inside a chroot with Rasbian Stretch Lite installed
  • started stubby with the exact same config file
  • looked at the output of ‘netstat -plnt’

root@network-manager:~# netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:8053 0.0.0.0:* LISTEN 14433/stubby
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 402/sshd
tcp6 0 0 :::22 :::* LISTEN 402/sshd

  • ran ‘drill @10.0.0.5 -p 8053 cloudflare.com’ from another machine in the home network and:

root@SIMONS-GAME-PC:~# drill @10.0.0.5 -p 8053 cloudflare.com
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 15925
;; flags: qr rd ra ; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; %only5linksallowed%. IN A
;; ANSWER SECTION:
%only5linksallowed%. 347 IN A 198.41.214.162
%only5linksallowed%. 347 IN A 198.41.215.162
%only5linksallowed%. 347 IN RRSIG A 13 2 600 20190215113745 20190213093745 34505 %only5linksallowed%. C8jL5tC1BasBcnAjYzhAxdWbTXFR+NjXWC39YA1qeBY1XyI2wWcUhP9TDzU945XsZ5aS7edzeFrl5VXtArY/9w==
;; AUTHORITY SECTION:
;; ADDITIONAL SECTION:
;; Query time: 159 msec
;; EDNS: version 0; flags: do ; udp: 1452
;; SERVER: 10.0.0.5
;; WHEN: Thu Feb 14 11:41:56 2019
;; MSG SIZE rcvd: 227

So in conclusion:
It is not the config file and it is not the version of stubby that I compiled.
Ping always works, but the actual dns query does not.

I am no linux pro, nor am I experienced with docker so I have no idea what to do next.

I really hope I don’t have to give this project up…

PS: I don’t know if docker has some pre configured firewall, but I didnt configure anything and the Raspberry Pi is freshly installed with the only modification of installing network-manager and purging openresolv and dhcpcd5.
If it helps I can post every step I made from flashing the Raspbian image on the sdcard and also upload the Dockerfiles and such…

6

So I just tested to connect the stubby-main container to the macvlan network, giving it the home network ip 10.0.0.2 and drilling from another machine.

This works!

Edit:
I kinda feel stupid now…
Turns out when I start another stubby container just for the purpose of having something else than alpine, I can indeed query the first stubby container over the docker network.

So I did further testing and tried to go inside the unbound container.
Funny thing, the unbound container never started successfully. No idea how I missed this…

I tired the original version without any modifications from https://hub.docker.com/r/mvance/unbound/ which also failed to start.

If anybody can assist me in fixing the Dockerfile, that would be great!

The Dockerfile I am using (modified version of mvances):

  • some formatting got lost in this post but you can get the idea!

FROM debian:stretch as openssl
LABEL maintainer=“MrNightfloor”

ENV version_openssl=openssl-1.1.1a
sha256_openssl=fc20130f8b7cbd2fb918b2f14e2f429e109c31ddd0fb38fc5d71d9ffed3f9f41
source_openssl=https://www.openssl.org/source/

WORKDIR /tmp/src

RUN set -e -x &&
build_deps=“build-essential ca-certificates curl dirmngr gnupg libidn2-0-dev libssl-dev” &&
debian_frontend=noninteractive apt-get update && apt-get install -y --no-install-recommends
$build_deps &&
curl -L “${source_openssl}${version_openssl}.tar.gz” -o openssl.tar.gz &&
echo “${sha256_openssl} ./openssl.tar.gz” | sha256sum -c - &&
curl -L “${source_openssl}${version_openssl}.tar.gz.asc” -o openssl.tar.gz.asc &&
GNUPGHOME="$(mktemp -d)" &&
export GNUPGHOME &&
tar xzf openssl.tar.gz &&
cd “$version_openssl” &&
./config --prefix=/opt/openssl no-weak-ssl-ciphers no-ssl3 no-shared -DOPENSSL_NO_HEARTBEATS -fstack-protector-strong &&
make depend &&
make -j 4 &&
make install_sw &&
apt-get purge -y --auto-remove
$build_deps &&
rm -rf
/tmp/*
/var/tmp/*
/var/lib/apt/lists/*

FROM debian:stretch as unbound
LABEL maintainer=“MrNightfloor”

ENV unbound_version=1.9.0
unbound_sha256=415af94b8392bc6b2c52e44ac8f17935cc6ddf2cc81edfb47c5be4ad205ab917
unbound_download_url=“https://nlnetlabs.nl/downloads/unbound/unbound-1.9.0.tar.gz”

WORKDIR /tmp/src

COPY --from=openssl /opt/openssl /opt/openssl

RUN build_deps=“ca-certificates curl gcc libc-dev libevent-dev libexpat1-dev make” &&
set -x &&
debian_frontend=noninteractive apt-get update && apt-get install -y --no-install-recommends
$build_deps
bsdmainutils
ldnsutils
libevent-2.0
libexpat1 &&
curl -sSL “${unbound_download_url}” -o unbound.tar.gz &&
echo "${unbound_sha256} unbound.tar.gz" | sha256sum -c - &&
tar xzf unbound.tar.gz &&
rm -f unbound.tar.gz &&
cd unbound-"${unbound_version}" &&
groupadd _unbound &&
useradd -g _unbound -s /etc -d /dev/null _unbound &&
./configure
–disable-dependency-tracking
–prefix=/opt/unbound
–with-pthreads
–with-username=_unbound
–with-ssl=/opt/openssl
–with-libevent
–enable-event-api &&
make install &&
mv /opt/unbound/etc/unbound/unbound.conf /opt/unbound/etc/unbound/unbound.conf.example &&
apt-get purge -y --auto-remove
$build_deps &&
rm -fr
/opt/unbound/share/man
/tmp/
/var/tmp/*
/var/lib/apt/lists/*

FROM debian:stretch
LABEL maintainer=“MrNightfloor”

ENV name=unbound
unbound_version=1.9.0
version=1.1

ENV summary="${name} is a validating, recursive, and caching DNS resolver."
description="${name} is a validating, recursive, and caching DNS resolver."

LABEL summary="${summary}"
description="${description}"
io.k8s.description="${description}"
io.k8s.display-name=“Unbound ${unbound_version}”
name=“mrnightfloor/${name}”
maintainer=“MrNightfloor”

WORKDIR /tmp/src

COPY --from=unbound /opt/ /opt/

RUN set -x &&
debian_frontend=noninteractive apt-get update && apt-get install -y --no-install-recommends
bsdmainutils
ldnsutils
libevent-2.0
libexpat1 &&
groupadd _unbound &&
useradd -g _unbound -s /etc -d /dev/null _unbound &&
rm -fr
/tmp/*
/var/tmp/*
/var/lib/apt/lists/*

COPY a-records.conf /opt/unbound/etc/unbound/
COPY unbound.sh /

RUN chmod +x /unbound.sh

WORKDIR /opt/unbound/

ENV PATH /opt/unbound/sbin:$PATH

EXPOSE 53

HEALTHCHECK --interval=5s --timeout=3s --start-period=5s CMD drill @127.0.0.1 cloudflare.com || exit 1

CMD ["/unbound.sh"]

I already tried removing the HEALTHCHECK but the container continues to restart even without it.
My run commands are:

docker run -d --network dns-main --name unbound-main --hostname unbound-main --ip 192.168.1.2 --restart unless-stopped --read-only -v /opt/dns/unbound/a-records.conf:/opt/unbound/etc/unbound/a-records.conf mrnightfloor/unbound-main:1.9.0-stubby

docker network connect --ip 10.0.0.2 --ip6 fd00::2 macvlan unbound-main

The original project this is based off:

7

I did some further testing.

As I mentioned, when I start a second stubby container I can do dns queries to the first one. So I tried to reproduce this with another container.

I found a docker container for unbound similar to the one I had but with alpine as a base system instead of debian. In the vanilla config, this container ran just fine.
But even after I disabled every iptables rule docker added , I was unable to perform dns queries to the stubby container.

Because this container was starting correctly I tried a different config file with the forward address set to stubby and the container didn’t start anymore.

I thought to myself, what if I modify my unbound container to run the same config as this one. After all I was able to perform dns queries from a debian based container.
But after I recompiled my container, it unfortunetly wouldn’t start. Even though this time, unbound was supposed to query public dns resolvers as forward address and didn’t need the stubby container at all.

Somehow both the dns-main network I created and the unbound container are broken, at least sometimes as far as the network is concerned.

This is frustrating… I need to take a break.
If someone has any more tips for me, I would really appreciate it!