Infrastructure Series -- Taking DNS One Step Further - Full DNS Server infrastructure

table of contents

Introduction

So we are all familiar with my other post: Infrastructure Series -- Recursive DNS and Adblocking DNS over TLS w/NGINX

Obligatory shill of blog stream post: Phaselockedloopable- PLL’s continued exploration of networking, self-hosting and decoupling from big tech

As always check for updates in the second post :wink:

DoT is great but it can be messed with or blocked by denying port 853. So what can we do to get around this and what are our options.

Well lets make a full set of architecture. Lets do DoH and normal DNS via our NGINX server! Why not right? We already have the setup and the backbone.

Standard DNS

A standard DNS server is easy to host given our DoT setup. It really is as simple as created a entry for port 53 and TCP streaming it via the stream block. (please refer to DoT post)

   # Standard DNS Server
server {
   listen 53;
   listen [::]:53;
   error_log  /var/log/nginx/dns53.log info;
   proxy_pass dns;
}

Open the port on your node and DNS will be functioning when you point at its addresses after you restart nginx. It really is that simple, but you will get DDOSd so here is the mitigation

Please note this is a very rough way to do this and its better to just force TCP replies and upgrades as well as using fail to ban. You will find this in violation of RFC. You can find this in my discussion post in the first setup DNS post of these guides. It will be the one discussing the blocking of certain services and why I do certain things

Mitigating port 53 attacks.

If you want your linode to proxy 53 dns you need to TCP dump where the floods are coming from…

Establish throttling:

iptables -A INPUT -p udp -m hashlimit --hashlimit-srcmask 24 --hashlimit-mode srcip --hashlimit-upto 120/m --hashlimit-burst 30 --hashlimit-name DNSTHROTTLE --dport 53 -j ACCEPT
iptables -A INPUT -p udp -m udp --dport 53 -j DROP

Protect against string patterns that are actively flooding your server which are usually ICP and RIPE:

iptables -A INPUT -p udp -m string --hex-string "|00000000000103697363036f726700|" --algo bm --to 65535 --dport 53 -j DROP
iptables -A INPUT -p udp -m string --hex-string "|0000000000010472697065036e6574|" --algo bm --to 65535 --dport 53 -j DROP 

Limit Any queries and throttle to 120 per minute. If someone is pulling an ANY query more than a few times a minute they are fucking with you. its rarely needed

iptables -A INPUT -p udp --dport 53 -m string --from 50 --algo bm --hex-string '|0000FF0001|' -m recent --set --name dnsanyquery
iptables -A INPUT -p udp --dport 53 -m string --from 50 --algo bm --hex-string '|0000FF0001|' -m recent --name dnsanyquery --rcheck --seconds 60 --hitcount 4 -j DROP

New Pizza SEO Attacks

iptables -I INPUT 1 -d DNS_SERVER_IP_ADDRESS -p udp -m udp --dport 53 -m string --string "pizzaseo" --algo kmp --from 41 --to 48 -j NFLOG --nflog-prefix  "DROP PizzaSEO DDoS ATTACK Requests"
iptables -I OUTPUT 1 -s DNS_SERVER_IP_ADDRESS -p udp -m udp --sport 53 -m u32 --u32 "0x0>>0x16&[email protected]&0x8005=0x8005" -j NFLOG --nflog-prefix  "DROP OUTBOUND DNS Query Refused Response"

PEACECORPS and ISC ANY Amplification attacks

iptables -A INPUT -i eth0 -p udp -m udp --dport 53 -m string --hex-string "|03697363036f72670000ff|" --algo bm --to 65535 -m comment --comment "Block isc.org ANY" -j DROP
iptables -A INPUT -i eth0 -p udp -m udp --dport 53 -m string --hex-string "|107065616365636f72707303676f760000ff|" --algo bm --to 65535 -m comment --comment "Block peacecorps.gov ANY" -j DROP

Disclaimer: *You should always apply to INPUT before FORWARD or OUTPUT

You will find those rules mixed amongst my other rules in my IPTABLES. This is just a dump: (it could be explained in a newer thread). There are some custom rules here for certain attacks ive experienced. These would otherwise resume if I didnt keep them. YMMV

Mangle

*mangle
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,RST FIN,RST -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,ACK FIN -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags ACK,URG URG -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags PSH,ACK PSH -j DROP
-A PREROUTING -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -j DROP
-A PREROUTING -p tcp -m conntrack --ctstate NEW -m tcpmss ! --mss 536:65535 -j DROP
-A PREROUTING -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP
-A PREROUTING -m conntrack --ctstate INVALID -j DROP
-A PREROUTING -p icmp -j DROP
COMMIT

Raw

*raw
-A PREROUTING -p tcp -m tcp --dport 53 -m string --hex-string "|000010|" --algo bm --to 65535 -m hashlimit --hashlimit-above 6/sec --hashlimit-burst 30 --hashlimit-mode srcip --hashlimit-name DNS_LIMIT_TXT_TCP --hashlimit-htable-expire 10000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT TXT TCP ." -j DROP
-A PREROUTING -p tcp -m tcp --dport 53 -m string --hex-string "|000010|" --algo bm --to 65535 -m hashlimit --hashlimit-above 1/sec --hashlimit-burst 60 --hashlimit-mode srcip --hashlimit-name DNS_LIMIT_TXT_TCP2 --hashlimit-htable-expire 120000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT TXT TCP ." -j DROP
-A PREROUTING -p tcp -m tcp --dport 53 -m string --hex-string "|000010|" --algo bm --to 65535 -m hashlimit --hashlimit-above 24/min --hashlimit-burst 90 --hashlimit-mode srcip --hashlimit-name DNS_LIMIT_TXT_TCP3 --hashlimit-htable-expire 10000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT TXT TCP ." -j DROP
-A PREROUTING -p udp -m udp --dport 53 -m string --hex-string "|000010|" --algo bm --to 65535 -m hashlimit --hashlimit-above 1/sec --hashlimit-burst 3 --hashlimit-mode srcip --hashlimit-name DNS_LIMIT_TXT_UDP --hashlimit-htable-expire 10000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT TXT UDP ." -j DROP
-A PREROUTING -p udp -m udp --dport 53 -m string --hex-string "|000010|" --algo bm --to 65535 -m hashlimit --hashlimit-above 5/min --hashlimit-burst 15 --hashlimit-mode srcip --hashlimit-name DNS_LIMIT_TXT_UDP2 --hashlimit-htable-expire 120000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT TXT UDP ." -j DROP
-A PREROUTING -p udp -m udp --dport 53 -m string --hex-string "|000010|" --algo bm --to 65535 -m hashlimit --hashlimit-above 1/min --hashlimit-burst 3 --hashlimit-mode srcip --hashlimit-name DNS_LIMIT_TXT_UDP3 --hashlimit-htable-expire 240000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT UDP TXT ." -j DROP
-A PREROUTING -p tcp -m tcp --dport 53 -m string --hex-string "|0000ff|" --algo bm --to 65535 -m hashlimit --hashlimit-above 6/sec --hashlimit-burst 30 --hashlimit-mode srcip --hashlimit-name DNS_LIMIT_ANY_TCP --hashlimit-htable-expire 10000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT ANY TCP ." -j DROP
-A PREROUTING -p tcp -m tcp --dport 53 -m string --hex-string "|0000ff|" --algo bm --to 65535 -m hashlimit --hashlimit-above 1/sec --hashlimit-burst 60 --hashlimit-mode srcip --hashlimit-name DNS_LIMIT_ANY_TCP2 --hashlimit-htable-expire 120000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT ANY TCP ." -j DROP
-A PREROUTING -p tcp -m tcp --dport 53 -m string --hex-string "|0000ff|" --algo bm --to 65535 -m hashlimit --hashlimit-above 24/min --hashlimit-burst 90 --hashlimit-mode srcip --hashlimit-name DNS_LIMIT_ANY_TCP3 --hashlimit-htable-expire 240000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT ANY TCP ." -j DROP
-A PREROUTING -p udp -m udp --dport 53 -m string --hex-string "|0000ff|" --algo bm --to 65535 -m hashlimit --hashlimit-above 1/sec --hashlimit-burst 3 --hashlimit-mode srcip --hashlimit-name DNS_LIMIT_ANY_UDP --hashlimit-htable-expire 10000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT ANY UDP ." -j DROP
-A PREROUTING -p udp -m udp --dport 53 -m string --hex-string "|0000ff|" --algo bm --to 65535 -m hashlimit --hashlimit-above 5/min --hashlimit-burst 15 --hashlimit-mode srcip --hashlimit-name DNS_LIMIT_ANY_UDP2 --hashlimit-htable-expire 120000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT ANY UDP ." -j DROP
-A PREROUTING -p udp -m udp --dport 53 -m string --hex-string "|0000ff|" --algo bm --to 65535 -m hashlimit --hashlimit-above 1/min --hashlimit-burst 3 --hashlimit-mode srcip --hashlimit-name DNS_LIMIT_ANY_UDP3 --hashlimit-htable-expire 240000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT UDP ANY ." -j DROP
-A PREROUTING -p tcp -m tcp --dport 53 -m string --hex-string "|00000000000103697363036f726700|" --algo bm --to 65535 -m hashlimit --hashlimit-above 6/sec --hashlimit-burst 30 --hashlimit-mode srcip --hashlimit-name DNS_LIMIT_ISC_TCP --hashlimit-htable-expire 10000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT ISC TCP ." -j DROP
-A PREROUTING -p tcp -m tcp --dport 53 -m string --hex-string "|00000000000103697363036f726700|" --algo bm --to 65535 -m hashlimit --hashlimit-above 1/sec --hashlimit-burst 60 --hashlimit-mode srcip --hashlimit-name DNS_LIMIT_ISC_TCP2 --hashlimit-htable-expire 120000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT ISC TCP ." -j DROP
-A PREROUTING -p tcp -m tcp --dport 53 -m string --hex-string "|00000000000103697363036f726700|" --algo bm --to 65535 -m hashlimit --hashlimit-above 24/min --hashlimit-burst 90 --hashlimit-mode srcip --hashlimit-name DNS_LIMIT_ISC_TCP3 --hashlimit-htable-expire 240000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT ISC TCP ." -j DROP
-A PREROUTING -p udp -m udp --dport 53 -m string --hex-string "|00000000000103697363036f726700|" --algo bm --to 65535 -m hashlimit --hashlimit-above 1/sec --hashlimit-burst 3 --hashlimit-mode srcip --hashlimit-name DNS_LIMIT_ISC_UDP --hashlimit-htable-expire 10000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT ISC UDP ." -j DROP
-A PREROUTING -p udp -m udp --dport 53 -m string --hex-string "|00000000000103697363036f726700|" --algo bm --to 65535 -m hashlimit --hashlimit-above 5/min --hashlimit-burst 15 --hashlimit-mode srcip --hashlimit-name DNS_LIMIT_ISC_UDP2 --hashlimit-htable-expire 120000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT ISC UDP ." -j DROP
-A PREROUTING -p udp -m udp --dport 53 -m string --hex-string "|00000000000103697363036f726700|" --algo bm --to 65535 -m hashlimit --hashlimit-above 1/min --hashlimit-burst 3 --hashlimit-mode srcip --hashlimit-name DNS_LIMIT_ISC_UDP3 --hashlimit-htable-expire 240000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT UDP ISC ." -j DROP
-A PREROUTING -p tcp -m tcp --dport 53 -m string --hex-string "|0000000000010472697065036e6574|" --algo bm --to 65535 -m hashlimit --hashlimit-above 6/sec --hashlimit-burst 30 --hashlimit-mode srcip --hashlimit-name DNS_LIMIT_RIPE_TCP --hashlimit-htable-expire 10000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT RIPE TCP ." -j DROP
-A PREROUTING -p tcp -m tcp --dport 53 -m string --hex-string "|0000000000010472697065036e6574|" --algo bm --to 65535 -m hashlimit --hashlimit-above 1/sec --hashlimit-burst 60 --hashlimit-mode srcip --hashlimit-name DNS_LIMIT_RIPE_TCP2 --hashlimit-htable-expire 120000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT RIPE TCP ." -j DROP
-A PREROUTING -p tcp -m tcp --dport 53 -m string --hex-string "|0000000000010472697065036e6574|" --algo bm --to 65535 -m hashlimit --hashlimit-above 24/min --hashlimit-burst 90 --hashlimit-mode srcip --hashlimit-name DNS_LIMIT_RIPE_TCP3 --hashlimit-htable-expire 240000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT RIPE TCP ." -j DROP
-A PREROUTING -p udp -m udp --dport 53 -m string --hex-string "|0000000000010472697065036e6574|" --algo bm --to 65535 -m hashlimit --hashlimit-above 1/sec --hashlimit-burst 3 --hashlimit-mode srcip --hashlimit-name DNS_LIMIT_RIPE_UDP --hashlimit-htable-expire 10000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT RIPE UDP ." -j DROP
-A PREROUTING -p udp -m udp --dport 53 -m string --hex-string "|0000000000010472697065036e6574|" --algo bm --to 65535 -m hashlimit --hashlimit-above 5/min --hashlimit-burst 15 --hashlimit-mode srcip --hashlimit-name DNS_LIMIT_RIPE_UDP2 --hashlimit-htable-expire 120000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT RIPE UDP ." -j DROP
-A PREROUTING -p udp -m udp --dport 53 -m string --hex-string "|0000000000010472697065036e6574|" --algo bm --to 65535 -m hashlimit --hashlimit-above 1/min --hashlimit-burst 3 --hashlimit-mode srcip --hashlimit-name DNS_LIMIT_RIPE_UDP3 --hashlimit-htable-expire 240000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT UDP RIPE ." -j DROP
-A PREROUTING -p udp -m udp --dport 53 -m string --hex-string "|096d6963726f736f667403636f6d000010|" --algo bm --to 65535 -m hashlimit --hashlimit-above 4/hour --hashlimit-burst 6 --hashlimit-mode srcip --hashlimit-name RATE_LIMIT_DNS_MSTXT_UDP3 --hashlimit-htable-expire 360000 --hashlimit-srcmask 24 -m comment --comment "RATE-LIMIT UDP TXT MICROSOFT ." -j DROP
COMMIT

Those rough rules will help. Please also consider forcing your resolver to answer queries via TCP. Please also consider setting up fail2ban which is far beyond this guide. It could be a thing I write in the future as the software is quite obtuse.

A good place to start is here:

https://www.teaparty.net/technotes/dns-fail2ban.html

DoH + DoQ

We are going to use the DoH-Proxy software written in rust. Its well supported, fast, mature and secure. I built it from source because I do not like cargo as a solution. You can use the rust compiler to do this. It should all compile nicely on most systems. Or you can use their preferred instructions

chmod +x the executable file wherever you compiled it to make sure it is executable and made a systemd process to start it. Lets make this service run as a user service. There is no reason for it to realistically have root and your systemd process shoud look like:

[Unit]
Description=Rust based DoH-Proxy userland service which starts at boot
After=network.target

[Service]
Type=simple
ExecStart=bash -xc '/home/(username)/<path>/doh-proxy -O -c 20000000 -C 20000000 -l 127.0.0.1:36443 -u 127.0.0.1:13753'
Restart=always
RuntimeMaxSec=5m

[Install]
WantedBy=default.target

Its man pages:

USAGE:
    doh-proxy [FLAGS] [OPTIONS]

FLAGS:
    -O, --allow-odoh-post      Allow POST queries over ODoH even if they have been disabed for DoH
    -K, --disable-keepalive    Disable keepalive
    -P, --disable-post         Disable POST queries
    -h, --help                 Prints help information
    -V, --version              Prints version information

OPTIONS:
    -E, --err-ttl <err_ttl>                          TTL for errors, in seconds [default: 2]
    -H, --hostname <hostname>                        Host name (not IP address) DoH clients will use to connect
    -l, --listen-address <listen_address>            Address to listen to [default: 127.0.0.1:3000]
    -b, --local-bind-address <local_bind_address>    Address to connect from
    -c, --max-clients <max_clients>                  Maximum number of simultaneous clients [default: 512]
    -C, --max-concurrent <max_concurrent>            Maximum number of concurrent requests per client [default: 16]
    -X, --max-ttl <max_ttl>                          Maximum TTL, in seconds [default: 604800]
    -T, --min-ttl <min_ttl>                          Minimum TTL, in seconds [default: 10]
    -p, --path <path>                                URI path [default: /dns-query]
    -g, --public-address <public_address>            External IP address DoH clients will connect to
    -j, --public-port <public_port>                  External port DoH clients will connect to, if not 443
    -u, --server-address <server_address>            Address to connect to [default: 9.9.9.9:53]
    -t, --timeout <timeout>                          Timeout, in seconds [default: 10]
    -I, --tls-cert-key-path <tls_cert_key_path>
            Path to the PEM-encoded secret keys (only required for built-in TLS)

    -i, --tls-cert-path <tls_cert_path>
            Path to the PEM/PKCS#8-encoded certificates (only required for built-in TLS)

Bingo enable it, start it, check its status. It should be running. I opted it to restart every 5 minutes because for some reason at a random point between 7 and 23 minutes it decides to stop answering queries. This might be due to running as a user process so I refresh the process and socket every 5 minutes. It doesnt realistically effect anything if you hav eturned logging off

I created a configuration file called 98-dnsDoH.conf in the conf.d folder (sometimes called http.d)

# DoH Port 80
server {
        include /etc/nginx/ErrorPages/error_pages.conf; # Error Pages GConf
        if ( $request_method !~ ^(GET|POST|HEAD)$ ) {
                return 405;
        }
        listen 80;
        listen [::]:80;
        server_name dns.<my-tld>.net;
        return 308 https://$server_name$request_uri;
}

# DoH Port 443
server {
        listen 443 ssl http2;
        listen [::]:443 ssl http2;
        include /etc/nginx/headers.d/98-dnsDoHHeaders.conf; # Security headers   
        include /etc/nginx/ErrorPages/error_pages.conf; # Error Pages GConf
        server_name dns.<my-tld>.net;
        if ( $request_method !~ ^(GET|POST|HEAD)$ ) {
                return 405;
        }
        location / {
                return 400;
        }
        location /dns-query {
                error_page 500 502 503 504;
                proxy_set_header Host $http_host;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_redirect off;
                proxy_buffering off;
                proxy_pass http://127.0.0.1:36443/dns-query;
        }
}

We want to return an error for any method that isnt explicitly after DNS to prevent anything useful from being extracted. Given that my server is listening on ipv6 primarily it is permissable to have the flag ipv6 only. You dont need my level of logging. I like to have it as insurance and helpful dianostics. so you can exclude error_log entries.

That pretty much covers it. Then just point all DoH resolvers towards the URL you defined it as. In my case
https://dns.< MY-TLD >.net/dns-query

Which by the way if you visit in a browers it should error 40x out :wink:

Im prepared for DoQ but fedora’s nginx version doesnt support it yet. When they upgrade to 1.19 I will add to any HTTP2 a HTTP/3 optional svc block as you can see is commented out. Eventually though I will and it will be sweet to have QUIC support.

I like being ready to deply as soon as possible thats my philosophy its OKAY to not agree with it and be different.

Enabling it on devices

Ive decided to support these questions on a case by case basis. There are far too many configurations to account for. I apologize for the inconvenience.

Conclusion to what we have

A DNS Server that has implemented the following:
Standard port 53 lookups anywhere via a publicly addressable server
A hardened DoT Server for port 853 DNS over TLS lookups
A hardened DoH Server to use DNS in the application layer to avoid blocking and censorship
DoQ optionally when its arrives for DNS over HTTPS via NGINX implementation

Now you have a server accessible via any method you want. This is currently what I have

Disclaimer:

Dont use my server because I wont whitelist a site for you. Make your own or expect breakage :wink:

Links to Infrastructure Series and Other Resources

Blog: Phaselockedloopable- PLL’s continued exploration of networking, self-hosting and decoupling from big tech

Phaselockedloopable- PLL’s continued exploration of networking, self-hosting and decoupling from big tech

Series 1: Native Dual Stack IP4+IP6

Infrastructure Series – Native Dual Stack IP4+IP6

Series 2: Wireguard Site to Site Tunnel

Infrastructure Series – Wireguard Site to Site Tunnel

Series 3: Recursive DNS and Adblocking DNS over TLS w/NGINX

Infrastructure Series – Recursive DNS and Adblocking DNS over TLS w/NGINX

Series 4: NGINX Reverse Proxy and Hardening SSL

Infrastructure Series – NGINX Reverse Proxy and Hardening SSL

Series 5: Taking DNS One Step Further - Full DNS Server infrastructure

Infrastructure Series – Taking DNS One Step Further - Full DNS Server infrastructure

Series 6: HTTP(S) Security Headers! You should use them!

Infrastructure Series – HTTP(S) Security Headers! You should use them! [NGINX]

Series 7: Use NGINX to inject CSS themes

Infrastructure Series – Use NGINX to inject CSS themes

ONE KEY TO RULE THEM ALL

Setting up a YubiKey Properly – One Key to rule them ALL!

Series 9: Infrastructure Series: BIND9 Authoritative DNS Guide “Please See Me Edition”

Infrastructure Series: BIND9 Authoritative DNS Guide “Please See Me Edition”

Buy me a crypto-beer

If you found this guide helpful you can donate Monero or Bitcoin to me at the following address in my User Card Profile

3 Likes

Reserved

UPDATE:

Rewritten and simplified on September 10th, 2022