So, a family member uses AT&T Fiber, and I had some issues previously with Systemd-resolved and my Home Lab’s VPN tunnel on Pop!_OS earlier. While troubleshooting this, I discovered some interesting behaviors with DNS on the AT&T network, and I believe that the evidence shows that AT&T is actively intercepting DNS port 53 queries and proxying them to use their DNS servers even if you configure clients to use another DNS provider. The AT&T gateway is intercepting port 53 queries mid-flight, including encrypted queries before the TLS handshake or SSL encryption, then spoofing these packets to make it look like it came from the DNS provider you set.
P.S. I discovered this because of https://dnsleaktest.com. When using the AT&T Fiber connection without my Wireguard Tunnel active (but with custom DNS servers defined), the DNS query would fail to resolve (and the F12 Network Tools showed NS_ERROR_UNKNOWN). If I went into Firefox and enabled MaxProtection it resolved.
These shenanigans have been allegedly known or seriously suspected for a long time, hence why Firefox and Chrome have native encrypted dns support, in one of the various ways it can be.
Also afaik it’s enabled by default in Chrome.
see the docs for firefox and a tutorial for chrome
Also Thunderbird mail client supports dns over https since last year
So if these are normie family members and 99% of their traffic goes through a browser or a mail client, it might be simpler to just configure those and accept the win updates and some other less important OS stuff uses an inferior DNS
One useful check is to separate “browser DoH fixed it” from “port 53 is being rewritten”. If you can, run a packet capture on the client while querying a public resolver over plain DNS and compare it with the same test over encrypted DNS. If plain DNS gets an answer but the encrypted path shows a different resolver path, that is strong evidence. For a practical workaround I would not rely on per-browser settings only; put DNSCrypt or DoT/DoH on the router or Pi-hole, then block plain outbound DNS for the LAN so devices fail closed instead of silently leaking around it.
I wonder how this affects usage of Wireguard on UDP port 53. Would be hilarious to waste their resources trying to inspect non-DNS traffic but I imagine that it would result in a bandwidth bottleneck…
After I discovered this, it is one of the reasons I took on the process to bypass the AT&T gateway.
To this day, I have a gigabit mini PC with two NICs and it does a security bypass to the AT&T network. I still pay for service, it’s not a magical get free internet but they don’t hijack my DNS.
Another reason is they leave a backdoor open for their internal systems to manage basic functions of the gateway, which means any bad actor who also gets this access…
Oh interesting. Might be useful for my own home setup. Currently, I have like this wierd convoluted DNS system that makes Pi-Hole forward WAN queries to pfSense since pfSense’s Unbound package natively supports DoH/DoT.
I HIGHLY suggest self hosting adguard home over pihole
Its simply better
I wrote the guide with pihole but overall it did not fit the use case as well as I hoped. With a toxic developer environment in terms of their insular behavior and religiousness around rules that don’t make sense and from the bugginess pihole exhibits as anything but a local internal dns server, its not the best software for the task
Adguard is explicitly designed to have mitigation techniques beyond simply rate limiting. You have the option to force upgrade DNS to TCP like bind9 in order to get rid of amplification attacks. See the amplification attack requires UDP because you dont need a confirmed handshake and you dont need to go through the kernel or netfilter and stuff. It also features built in DoT and DoH with your own certs from the get go
Its much more safe, secure and reliable to choose it over pihole if you will expose your DNS server to the greater internet like myself
Pi-Hole doesn’t support DoH/DoT natively, and I didn’t really look very deep when I figured that out because my brain went into instant problem solving mode. Unbound does support DoH/DoT, and it is, in fact, configured to use Cloudflare’s servers when forwarding queries to the WAN. So Pi-Hole isn’t using DoT/DoH at all when forwarding queries to pfSense; but a typical route goes to Pi-Hole » pfSense » Cloudlfare DoT.
You were able to get bypass the AT&T cryptographic lock between their SFP ONT and the AT&T gateway? AT&T seem to be the only major fiber provider in the country to force customers into using their anemic gateway. It’s quite awful compared to my GFiber setup at home.
Honestly, AT&T fiber is basically cable with [almost] symmetrical upload and download. That gateway is such a huge bottleneck that completely kneecaps the rest of the infrastructure, even though AT&T uses GPON rather than XGS-PON.
Does their gateway still suck even in IP passthrough mode?
That should disable firewall and make the thing just act as a modem and mirror packets to your actual router. It’s kind of like the old “bridge mode” for DSL modems. Even a very weak embedded CPU can handle a ton of plain software bridged traffic without NAT/firewall.
I don’t have the hardware but I can see some docs from them indicating it should
IP Passthrough on the AT&T gateway device isn’t true passthrough. It still has a state table with a limitation of 8192 states. If you’re on AT&T fiber, get a WAS-110 PON-on-a-stick, flash the 8311 community firmware, and ditch the gateway.
Would you happen to know if this also works with Quantum fiber? AT&T transferred my service to Quantum since they own them now, I guess. Their technician was very kind but conveniently just knowledgeable enough to dodge my questions when I asked about modifying the settings on their ONT. Tbh, he probably just didn’t know and that’s fine.