A primer on DNS over HTTPS
You are probably familiar with the term, but do you understand the technology and how it works? Let’s have a chat. This discussion thread’s purpose is not to scare you. The intent is that we all become more well-informed. Sunshine is the best disinfectant. You’re probably wondering what made me decide it was time for a forum post. Tim Holus and I were having a discussion. Here is a link to where we left off. DOH and it's effectiveness against ISPs - #20 by TimHolus . @TimHolus I felt my reply required a dedicated post. He is correct in regards to DoH being here to stay. However, I’m afraid I have to disagree with his take on it being an essential technology for the future. Many people reading this will probably not have been alive during the earlier days of the Internet. That is a history lesson for another day. Please do watch the youtube videos in the sources area at the bottom if you want the same point delivered by infinitely smarter people than I am.
The forum will attract younger people at some point. We must educate them as we got educated in technology while younger. DoH presents some problems for anyone who runs a managed private network. What DoH has done is encapsulate all of your DNS traffic into HTTPS on port 443, effectively encrypting it and removing it from your view to manage and deal with. Years ago, in 2013, a series of disclosures made the general public aware of how the world works. On path actors have not been good stewards. People understandably could have reacted better to what nation-states were doing; however, I digress. Thus, the motivation to protect the individual or user was born. Openness has been one of the defining characteristics of the Internet for as long as it has existed. It is paramount we continue to preserve this. Most requests for HTML pages and associated content are in plain text. The responses are returned similarly, even though HTTPS has existed since the 90s. Still, sometimes there’s a need for security and privacy. While internet traffic encryption has become more widespread for online banking and shopping, the privacy-preserving aspect of many internet protocols hasn’t kept pace.
In particular, when you look up a website’s IP address by hostname, the DNS request is almost always transmitted in plain text, allowing all the computers and ISPs along the way to determine what website you were browsing, even if you use HTTPS once the connection is made. The idea of encrypting DNS requests isn’t exactly new, with the first attempts starting in the early 00s, in the form of DNSCrypt, DNS over TLS (DoT), and others. Mozilla, Google, and a few other large internet companies are pushing a new method to encrypt DNS requests: DNS over HTTPS (DoH). Cloudflare, in particular, is quite insidious. The former companies do not have address agility on their DoH resolvers, but Cloudflare does. They make precisely ZERO promises to allow their service to be blocked. DoH encrypts the DNS request and serves it to a “normal” web server rather than a DNS server, making the DNS request traffic essentially indistinguishable from normal HTTPS. DoH is unblockable by design, as shortly after DoT came into existence, the IETF decided it wasn’t enough. ALL user traffic needed to be unblockable. Encrypted SNI is only going to make this worse. ECH will cause this to become ungodly difficult to block using traditional endpoint security methods. This is a double-edged sword. While it protects the DNS request itself, just as DNSCrypt or DoT do, it also makes it impossible for the folks in charge of security at large firms or any operator responsible for the management and use of their network under law to monitor DNS spoofing, and its moves the responsibility for a critical networking function from the operating system into an application. Worse yet, they want to avoid TCP, and the designers have a legitimate argument. TCP congestion control is atrociously outdated and unmaintained in OS kernels today. UDP allows them to do their own. People don’t understand or willfully ignore that UDP doesn’t need to ask the kernel to make a connection. An application can open a connection of its own accord. The application does not need to provide a valid destination. The reality is that an effective way to block this behavior is to ban UDP traffic on your network. I hate to agree with China here, but you must tell me your destination before I open the gates. If I do not want you going to this destination, you will not go. This has a lot of implications, but it’s a lot to unpack, and I will if asked. All these application platforms, including IoT, are the new way for companies to print money at their behest. DoH also doesn’t do anything to hide the website’s IP address that you just looked up. You still visit that page, after all, and with some traffic analysis, you can know that the prior request was DNS. What is sad is that some parents want to filter what their kids see on the world wide web. All of these techniques make parental controls incredibly easy to bypass. In my view, this violates parental consent and network consent simultaneously. This is not a “think of the children” argument. A legitimate case exists for a parent controlling what their kid may or may not visit. That is a parental responsibility, and in the same way, it extends the argument of the network operator’s responsibility to maintain the safety of his network and users.
In comparison to DoT, DoH centralizes information about your browsing in a few companies: at the moment, Cloudflare, who says they will throw your data away within 24 hours, and Google, who seems intent on retaining and monetizing every detail about everything you’ve ever thought about doing. We can get around this by hosting our own, but it is not a solid solution. We can get to that later.
Here is a primer on it from Cloudflare. At least they are honest enough to recognize the issues it poses. DNS Encryption Explained
DNS as an RFC
The first DNS server (Berkeley Internet Name Domain Server, or BIND) was written in 1984 by a group of UC Berkeley students based on RFC 882 and RFC 883. By 1987 the DNS standard had been revised several times, resulting in RFC 1034 and RFC 1035, which have remained unchanged. Here is a LINK to my bind thread if you want to know more.
Before we get around to encrypting DNS requests, it’s essential to be sure that the DNS server we’re talking to can be trusted. The need for this became apparent during the 1990s, culminating in the first workable DNS Security Extensions (DNSSEC) standard (RFC 2353) and the revised RFC 4033 (DNSSEC-bis). DNSSEC works by signing the DNS lookup records with public-key cryptography. The authenticity of a DNS record can thus be verified by the public keys for the DNS root zone, which is the trusted third party in this scenario. Domain owners generate their keys, which are signed by the zone operator and added to the DNS. While DNSSEC allows one to be relatively confident that the responses one gets from the DNS resolver are genuine, it does require DNSSEC to be enabled in one’s OS. Unfortunately, few OSes implement a DNS service that is more than just ‘DNSSEC-aware,’ meaning they do not validate the DNS responses. This means that today one cannot be sure that the DNS responses one receives are genuine. This is a fundamental problem, as DNSSEC is a better solution to the trust issue than any other current solution.
Consequences to Network Operators
These network operators are possibly corporate environments that rely on plaintext DNS inspection to enforce policies, or they are yourself running your network at home. You may care about privacy and network consent and do not want devices talking to specific destinations. DoH through this all out the window in the name of “protecting the user.” They designed something for the user without actually understanding who the user was. If devices fall back to plaintext DNS if DoH/DoT is unavailable, the network administrators could block port 853 with little risk because it is only used by DoT.
On the other hand, if they block port 443, then all HTTPS websites will become unavailable. Similarly, if they see an influx of DoT traffic, it could indicate an anomaly. Suppose some similar traffic spikes occur with DoH. In that case, it might not be possible to distinguish HTTPS from DoH traffic directly. DoH and DoT are comparable on a protocol level; DNS messages are encrypted in both cases. See also my Cloudflare blog post explaining DNS encryption, where I describe the technical protocol details, deployment choices, and different expectations from individuals and organizations.
The operating system has historically accepted whatever the local network advertised DNS resolver via DHCP. The corporate network administrator, the ISP, or your own recursive resolver typically configures this. They expect to be able to provide services such as malware blocking, parental filtering, blocking illegal content, and in some cases, query data logging.
DoH and DoT are great in protecting the privacy and integrity of DNS queries in untrusted environments such as airport Wi-Fi or even snooping/interference from the local government in some ways. Hiding from a nation-state is hard; DoH will not help you here. A nation-state has unlimited time and resources. You do not so stop being naive about your approach to combatting surveillance. However, since it was an emerging technology, not all existing DNS resolvers have support for it.
That put early adopters such as Mozilla in a difficult position. Should they abandon the idea of improving privacy or select a DNS resolver who supports DoH with a firm privacy policy? They ended up with the latter, which meant that the default DNS resolver provided by the operating system was initially ignored. This is probably the reason for the negative pushback against DoH from ISPs and governments. This is also my reason for pushback. No software, program, or application should have the right to bypass what I have defined as the legal boundaries of my network. The current threat model is that any device can have a DoH resolver, particularly with oDoH, to bypass your systems entirely. You aren’t going to block port 443 generically, are you? If the DoH resolver doesn’t go to a URL but rather an IP directly, then blocking it becomes much more difficult as most nefarious IPs have address agility.
Hiding these addresses in a simple TXT record provided by Cloudflare that you entered into the DNS system maliciously is relatively easy. It’s trivial, and that’s how C2 servers work. So now we have a non-respecting, global, and potentially viral internet. Organizations need to start evaluating the risk associated with the DoH protocol because attackers have already begun using DoH to look up command-and-control (C2) servers. The best-known example of DoH as a C2 mechanism came in April 2019 with the Godlua backdoor (360 Netlab, 2019). A newer variant of the Godlua backdoor runs on Linux and Windows. It uses a DoH request to grab a part of its C2 information.
Another way an attacker could use DoH in an attack is to trigger a redirected webpage as part of a spam campaign. Researchers at MyOnlineSecurity (2019) found a sample where an email attachment had a Base64 encoded string that would query Google DoH for a TXT record. The TXT record would have a JavaScript redirect to a spam webpage whose address is often changed.
Numerous DoH C2 proofs-of-concept are publicly available, meaning that the threat of malicious actors using DoH will likely increase rapidly. It gets worse.
The Internet will become a Single Point of Failure
Let us ensure we are on the same page about the definition of a SPoF and how to extend that definition to what we see. A single point of failure (SPOF) is a potential risk posed by a circuit or system’s design, implementation, or configuration flaw. SPOF refers to one fault or malfunction that can cause an entire system to stop operating. A SPOF in a data center or other IT environment can compromise the availability of workloads or the complete data center, depending on the location and interdependencies involved in the failure.
Mozilla’s answer to the IP tracking problem is that there is no problem because of the Cloud. As more and more websites and content distribution networks (CDNs) get lumped onto a handful of services (Cloudflare, Azure, AWS, etc.), the meaning of that single IP becomes less and less meaningful; you have to trust whichever Cloud service you pick not to steal your data, or go down for a day. The grim reality is that you are now the product again.
There was a massive downtime event on June 24th, 2019, when a configuration mistake at Verizon led to Cloudflare, Amazon, Linode, and many others being unavailable for much of the day. Then on July 2nd of that year, Cloudflare went down for about half an hour, taking down many websites that rely on its services. Coincidentally Microsoft’s Cloud-hosted Office365 also had a multi-hour outage that same day, leaving many of its users stranded and unable to use the service. Meanwhile, on US Labor Day weekend, a power outage at AWS’ US-East-1 data center led to 1 TB of customer data vanishing as the hardware it was stored on went FUBAR. Before adopting this model, some issues must be ironed out with this ‘centralizing the internet is good’ message. I’m afraid I have to disagree with this model entirely. We have other significant examples, but you get the point. Additionally, I talked about threat vectors and models earlier. Guess what? If an attacker can gain control of AWS infrastructure, they could do significant damage. That’s likely much harder than penetrating individual companies because AWS is very good at running a secure shop. Still, of course, it’s not impossible. That possibility should fundamentally not sit well with you. There are real but imponderable risks to a couple of companies controlling so much of the Internet.
This is a FLAWED design. It doesn’t work. We can continue to band-aid it or attempt to go in a different direction.
We don’t need DoH. We have VPNs
It’s in many ways astounding that in this whole discussion about privacy and tracking, there’s no mention of Virtual Private Networks (VPN). These solve the issues of encrypting your data and DNS queries, hiding your IP address, and so much more by simply moving the point where your PC or other internet-enabled devices ‘exists’ on the Internet. Most people will probably agree that the basic tenets of a VPN are a good thing. I fundamentally believe that internet privacy is more than just a good thing. It’s paramount to the success of the online world. Many people delay getting a VPN, considering it inessential or, worse, unnecessary. They shouldn’t. They make complex solutions like DoH but don’t consider the OG method of protecting your traffic. A good way of illustrating the necessity of a VPN is to show how exposed you are when your internet connection is not encrypted. Dissidents in authoritarian regimes have very commonly used VPNs for decades to get around internet censorship and, along with specialized forms such as the Tor network, are a crucial element in online freedom. You might think that HTTPS does the job, but it’s ok. For the uninitiated, HTTPS secures information communicated between a person’s web browser and a website. The lock in the browser address bar indicates it. At the same time, this provides added security while web browsing. Your data will still be vulnerable, mainly if you use public Wi-Fi. It’s a bit like closing your front door but failing to lock it. Sure, it’s better than keeping your front door open, but security could be tighter.
Whether you’re connected to the Internet in public or at home, without a VPN, you are exposed to a myriad of vulnerabilities. When browsing at home, your ISP can see everything you do and is probably logging it. Places with public Wi-Fi hotspots, such as coffee spots and airports, are vulnerable to hackers who can easily set up fake but convincing hotspots.
On the other hand, when you use a VPN, your data is not exposed. The origin of your data will be your VPN server. By using a VPN, your online actions will not be tracked and logged by ISPs and malicious on-path actors, nor will sensitive information be taken. Even if data is intercepted, it is encrypted, so it looks nonsense to anyone without a decryption key.
If you can trust a giant corp to run your DoH server, why can’t you reasonably find a trustworthy VPN provider? Why is this lost on people? It is an entirely valid statement that DoH honors its acronym by poorly doing what DoT already does. More focus should be on implementing DNSSEC everywhere, along with DoT (name server to name server) and QNAME minimization. Suppose true privacy by dodging tracking is your goal. In that case, you should look at VPNs, especially if you’re a dissident trapped in some authoritarian regime. The naive view of internet activists is that they believe if they drive the cost and means to be authoritarian that the nation-state will stop doing so. My lizard brain says no. The nation-states have more resources than the collective might of the first-world open-source community could dream of. You are merely contributing to them becoming more authoritarian. Stop and think about what you are doing. Mailing lists move so fast on RFCs that it becomes inevitable that your voice is left out. You have to declare email bankruptcy. Sadly what makes this worse is your opinion only matters before, not afterward.
Mitigating the new risks posed by DOH
(Reserved for an update later). I want to update this thread at some point with how to start to combat the dangers DoH poses to a network operator’s security. Still, I do not know enough about mitigating it yet save using an SSL proxy. It disgusts me that I need to decrypt SSL to do this. Let’s talk about free and open-source solutions below on how to do that. Let’s make some guides to combat DoH. DoH is acceptable for people, but knowing what it can do is also essential. You are responsible for the safety and integrity of those on your network. Ultimately we may have to return to the honor system in some form. You respect the rules of my network, and you may use it. You break those rules, and you are excommunicated, so to speak.
Conclusion
You are in a fight, but you don’t realize it. Before we were born, aircraft were made and bypassed the traditional laws of land and sea. There was a race to control the air. We are now in the information age, and there is a race to control the most information and encryption of that information. Large corporations have recognized that having information superiority even over that of nation states is a significant advantage in this paradigm, and they will continue to seek it out. Their IOT platforms will gather data on you and make money off of it. Their AI models will map your mind over time. You are merely a money printer to them, and if you do not want this to happen, we should do something about it. It is not going to be easy. It may not even be simple. To paraphrase and rip off JFK, We do these things not because they are easy but because they are hard; because that goal will serve to organize and measure the best of our energies and skills because that challenge is one that we are willing to accept, one we are unwilling to postpone, and one we intend to win, and the others, too. The first and primary way we do that is by questioning how our inventions can be misused. It would help if people were more open to a decentralized open internet. An internet and tech world where we can milk the good for the good. A web of trust of sorts where companies can innovate but we can out the braked on technological misuse. It seems the sentiment is dying. Perhaps I will become a relic like it will be if we continue down these paths. DNS over HTTPS is fundamentally flawed in that it was designed to protect the user without real regard for who the user was. We must learn to live with that now. Do with this as you may. Discussion below is welcome.
P.S Paul Vixie’s views align quite closely with my own. Dude’s an actual hero
Sources:
DNS over HTTPS - Wikipedia
What is DNS over HTTPS? Definition, Implementation, Benefits, and More
DNS over HTTPS · Cloudflare 1.1.1.1 docs
What DNS over HTTPS Is and How to enable It in Windows 10
https://www.cloudflare.com/learning/dns/dns-over-tls/
DNS-over-TLS | Public DNS | Google for Developers
DNS over HTTPS Vs DNS over TLS. DNS queries are sent from the client… | by R. Gupta | Geek Culture | Medium
DNSFilter: DoH Isn’t Better, It’s What Google Likes: DNS-over-TLS
https://www.cloudflare.com/learning/dns/what-is-dns/
What is DNS? – Introduction to DNS - AWS
What Is Domain Name System (DNS)? | Fortinet
What is DNS and how does it work? | Network World
