Hacker News new | past | comments | ask | show | jobs | submit login
Mozilla rolls out DoH to Canadian users of Firefox (blog.mozilla.org)
231 points by tristor 6 months ago | hide | past | favorite | 333 comments



There's a lot of lament that this bypasses the operating system level config, but I feel like we wouldn't be here if the operating system had stepped up their DNS any time in the last several years.

This is a workaround for the fact that many networks are hostile to DNS lookups (injecting ads, tracking, captive portals), and operating systems continue to just default to the ISPs DNS provider, which is often hostile.

OS's could have switched to running their own local caching resolver ala unbound years ago, could have integrated DNSCrypt or DoT years ago. But instead the defaults are still horribly unsecure for the average user.

For the techie user who's running PiHole or some custom local DNS, yeah this will be annoying, but for the average user who's just delegating to their ISP that is then tracking their DNS lookups and injecting ads if they mistype a domain, this is a win.


So the fact that some ISPs are apparently untrustworthy is enough to blow up the whole thing which has worked pretty well for the past 30 years?

There are plenty of valid use cases for captive portals. But yes, injecting ads and tracking are obviously messed up, but DNS is only a tiny part of that problem, and I don't see how using DoH will help with them. If you want to get rid those, then get rid of shitty ISPs. This is throwing out the baby with the bathwater.

This will simply lead to ISPs and network operators blocking DoH. And instead of a wonderfully decentralized DNS, we'll end up with centralized DoH in the hands of Cloudflare and their ilk. Who have been shown repeatedly to be user hostile; especially towards privacy conscious users.


> If you want to get rid those, then get rid of shitty ISPs.

Good luck, when 30M+ only have DSL as an option, and >70% only have one Cable provider, and one DSL provider.

FCC Data: https://broadbandmap.fcc.gov/#/area-comparison?version=jun20...

More Info: https://www.theverge.com/22177154/us-internet-speed-maps-com...


So the fact that some ISPs are apparently untrustworthy is enough to blow up the whole thing which has worked pretty well for the past 30 years?

Right. In place of that, we get one central point of DNS censorship. Where any unapproved site can be made to disappear.

"CIRA Canadian Shield" "Truly free". Right. "Freedom is Slavery" is more like it. In the US, Mozilla is apparently putting users under the thumb of Cloudflare.[1]

I'd rather have a mode where regular DNS and centrally controlled DNS are compared, and you get a popup if they differ. That warns you of censorship attempts. But that doesn't seem to be an option.

[1] https://forum.netgate.com/topic/133679/heads-up-be-aware-of-...


> In place of that, we get one central point of DNS censorship.

Just don't use Cloudflare as your DoH resolver. What is preventing DNS from becoming centralized, and why is DoH different? You can still use any resolver you want, and network-wide settings will come out once network hardware and operating systems catch up to the ecosystem and start implementing their own support.

It is easier to block DNS queries and censor specific sites over unencrypted DNS than it is to block specific sites when you can encrypt your queries to any DoH provider, even ones that are running locally on your own network.

In fact, one of the primary criticisms of DoH from UK ISPs is that it will make it harder for them to block pirate sites and fulfill court orders to block websites[0]. When you listen to the actual censors, they all want DNS to stick around. That should tell you something about whether DoH really is a tool for censorship.

> I'd rather have a mode where regular DNS and centrally controlled DNS are compared

Why would this be harder to do with DoH than it would be to do with DNS?

[0]: https://indico.uknof.org.uk/event/46/contributions/668/attac...


> I'd rather have a mode where regular DNS and centrally controlled DNS are compared, and you get a popup if they differ. That warns you of censorship attempts. But that doesn't seem to be an option.

This is difficult with CDNs because it's commonly intended that you get a different RR-set response depending on where your query originated from. So, somewhat awkwardly, there's no unique correct answer to a particular DNS query in many cases today.

I just looked up IN A www.google.com from two different networks, and got two different answers, yet I think both of them are correct and are the answers that Google intended for me to get!

(These discrepancies are themselves one reason that CDNs have been wary of large numbers of users not using ISP-provided recursive resolvers. There is a sort of compromise workaround called ECS, https://en.wikipedia.org/wiki/EDNS_Client_Subnet, in which the recursive resolver gives the authoritative server a hint about where the query came from, to facilitate giving a response that may be most relevant to that specific client, typically a nearby CDN instance.)


Would a networking expert with some historical perspective then not say that we're misusing DNS for routing or load balancing?


I'm sure there was a huge debate about this when CDNs first started doing it. (I think Akamai was already doing it 20 years ago or so, so it wasn't just invented now.)

You could, of course, try to handle it at a higher protocol layer. For example, you could try to dynamically generate HTML like

<img src="https://dallastx.mycdn.com/mysite/someimage.png">

for one visitor, whereas a different visitor would get

<img src="https://seoul.mycdn.com/mysite/someimage.png">

This would mean that latency for an initial HTTPS page load would be higher, but then for subresources and maybe other pages on the site it would be lower. But there are other difficulties, like requiring the web server to generate page content (at least slightly) dynamically instead of statically.

I'm sure that there are many people who felt that there was something seriously improper about intentionally serving different RRs within the same zone at the same point in time. From what I've heard, this practice is so common that most people working on related Internet standards just assume that this practice has to be accommodated because CDNs and their customers will definitely refuse to give it up. (I know this issue came up a ton in ESNI because in the first version there was a risk that you could cache one version of an A record and another version of an ESNI key record, so the two could be out of sync and therefore inconsistent, if they pointed to different CDNs or different CDN instances that had different public keys. A "nice" solution might be to say "well, don't create deliberately inconsistent sets of DNS records in your DNS zone, then" but apparently nobody found that solution at all acceptable for performance, so they created a more complicated workaround where both pieces of information are merged into a single rrtype.)


A useful check is to then compare the certificates presented by the sites the DNS wants you to talk to. If both paths present the same cert, you're good. If they present different certs, there's a problem.

Cloudflare's creation of fake TLS certs for sites they front-end may mess that up, though. Do you see the same TLS cert from all Cloudflare MITM nodes?


> In the US, Mozilla is apparently putting users under the thumb of Cloudflare.

Even if you don't think Cloudflare is perfect, don't you trust them way more than, e.g., Comcast?

> I'd rather have a mode where regular DNS and centrally controlled DNS are compared, and you get a popup if they differ. That warns you of censorship attempts. But that doesn't seem to be an option.

This would only fix the censorship problem, not the surveillance problem.


> Even if you don't think Cloudflare is perfect, don't you trust them way more than, e.g., Comcast?

Why would I trust a company I have no binding contract with, that doesn’t gain anything from my using their services, that doesn’t stand to lose anything if they lose me and others as a user of this particular service, etc over a company I entered into a legal agreement with, pay a monthly sum to provide a named service, and can sue in a court of law with some modicum of standing if things go sour?


You can't sue Comcast; you signed away your right to do so, individually and as a class, when you signed up for their service[1]. Most ISPs have class action waivers and binding arbitration, these days. In fact, you explicitly agreed to let them aggregate your data and sell it to others for their profit[2], assist third-parties in targeting you for advertisements[3], and combine data they have on file with you from other companies databases[4].

So, no, if I were in your shoes, I wouldn't trust Comcast at all.

[1]: Comcast Agreement for Residential Service, Section 13 and 14. https://www.xfinity.com/Corporate/Customers/Policies/Subscri...

[2]: Xfinity Privacy Policy. https://www.xfinity.com/privacy/policy "Create measurement and analytics reports for us and others"

[3]: Ibid. "We use the information we collect to... deliver personalized marketing and advertising for our own and others’ products and services"

[4]: Ibid. "We may combine information across our systems, platforms, and databases. This includes combining information we receive from third parties and information about your use of our Services."


> you signed away your right to do so, individually and as a class

Has this ever been tested in court? Specifically in a situation of a service almost everyone needs and they don't have a choice in the company? The contract can say lots of things, but none of them have to be legally binding.


Because Comcast has a repeated, long-term pattern of manipulating the contract to be one-sided against you, of taking every bad action that the contract might even possibly allow, and arguably of violating the contract, whereas Cloudflare does not have a comparable pattern of such actions?

Because there's about a 99 percent chance that the contract is either completely silent about the matters of immediate concern, or has some language that implies they are actively allowed to act badly in those ways; the contract therefore offers you absolutely zero protection; and there is absolutely no way you can renegotiate the contract?

Lawsuits are incredibly impractical, and that's assuming you even ever detect the behavior you expect to be suing over.


"I'd rather have a mode where regular DNS and centrally controlled DNS are compared, and you get a popup if they differ."

Not sure about "mode" and "popup" as I am not a fan of browsers and GUIs but I have been comparing results from about 50 or so DoH providers. I use DoH without a browser to gather DNS data in bulk.

We all have a clear idea about what "censorship" means but as we go further down the rabbit hole of myriad third party, non-ISP DNS providers how do we distinguish between "filtering" and "censorship". Projects like OpenDNS made a business out of DNS filtering. Now we are seeing others trying to make a business out of filtering ads.

It is well-known from studies using different DNS vantage points (looking glass) that DNS-based censorship is active in some countries and also that the number of countries doing this is generally rising. On a personal level, I do see differences in results from different DoH providers. What really bothered me recently was one provider returning "0.0.0.0" for certain domains, I am guessing because they were websites that report on the online ad industry. I am OK if they want to exclude certain domains, but returning the network equivalent of "any address" is, IMO, not an acceptable way to block access.

Personally, I think relying wholly on third party DNS is a bad idea. It is a useful resource, it's good to have many sources of DNS data, but third party operated DNS caches will never be authoritative for any domain. These third parties are usually just more internet middleman who want to make money from being in the middle of users network traffic. We always need access to the authoritative servers for each domain. That's our "source of truth".

DoH can be useful because I can download DNS data in bulk over TCP using HTTP pipelining. I load bulk data into haproxy's memory via a map file and I never need to do a single remote lookup. I run an authoritative DNS server that just answers all queries with the address of the proxy. Even more, I can add or delete entries from the map without restarting the proxy. The map is just a simple two column text file. This makes monitoring changes in DNS easy.

While I do not need "modes" and "popups" or other GUI-style conveniences, I can build a "no DNS" system into an OS that boots from USB stick into memory, never touches "disk" (can pull out the USB stick) and runs on a wide variety of hardware.


"Who have shown repeatedly to be user hostile; especially towards privacy conscious users."

What examples were you thinking of.

They are still the only CDN offering TLS1.3 with ESNI. Problem is they do not necessarily encrypt backend connections. TLS version and SNI are relevant IMO because even with DoH, without TLS1.3 plus ESNI, the ISP can still see domain names in the clear on the wire in client hello packets sent and TLS certificates returned from remote IPs. All those AWS-hosted sites you visit all require SNI, the domain name sent in the clear over the wire for your ISP to track. For many, you can use the cloudfront CNAME as the SNI but it would be easy for an ISP to keep lookup table for all AWS CNAMES. HN's hosting provider will not even support TLS1.3, it's TLS1.2 at best.

SNI is hostile toward privacy conscious users and makes censorship easier.


But what’s the alternative to SNI for multiple domains on one IP? And don’t say IPv6, because my ISP doesn’t even offer that yet…


ESNI (now called ECH): https://defo.ie

I use ESNI-enabled openssl from github.com/sftcd, 30 Aug 2019 release, for all HTTP requests to Cloudflare-hosted sites and it works well. A noticeable downside is one has to keep fetching a new key in a DNS TXT RR every hour. Another use for DoH; the key can re retrieved via DoH (no SNI required).


ESNI with TLSv1.3.

(The "E" means "encrypted".)


That does sound nice, but apparently it’s already been succeeded by TLS ECH, and neither of them are widely enabled by default, so essentially there is currently no SNI alternative.


> This will simply lead to ISPs and network operators blocking DoH.

With eSNI/ECH, browsers would just have to hardcode a bootstrap IP, and then ISPs won't be able to block DoH without also blocking all of every major CDN.


Might still work. All DoH requests go to the same IP from the "major" proviers.


yup, will have to block eSNI (disable all hosts on firewall unless looked up on local DNS, per application)


Maybe this will bring back a form of domain fronting. Let the user put in a list of hosts that your tyrannical dictatorship allows through its firewall, and then do a fake DNS lookup through your local DNS server for them, then use the resulting IP to access other hosts on the same CDN.


The "some ISPs" is really important here.

This is very much a US thing, I don't know as much about the situation in Canada, might be as bad. I really hope that this is not the begging of a roll out to the rest of the world.


Yeah agree, ISP choice here in Australia (at least in the metro areas) is plentiful and in my experience all my ISPs have behaved pretty ethically. I'm on Aussie Broadband now and their service has been fantastic, they even let you run your own line tests and connection resets from their customer portal.

I spent a number of years in the US at the mercy of Comcast and Later TWC in new york city and it was a very different experience. Also had a weird experience in Chicago where my building provided internet to all the Apartments with ads injected on 404 pages and the like. Really odd and no option to change.


All ISP dns servers in Australia are malicious. By law they must log all of your requests tied to your ID and they must block copyright infringement sites.

The first one alone is reason to have DoH enabled by default for all users in Australia.


> This is very much a US thing

DNS censorship is absolutely a thing in Germany and the UK for example. While here in Germany we might not get injected ads and other stuff, there are still more than enough problems to not make this a "US thing"


When I lived there a few years ago my ISP was frustratingly adversarial.

https://news.ycombinator.com/item?id=6993831


> which has worked pretty well for the past 30 years?

But DNS hasn't worked pretty well for the past 30 years. It's been abused for a pretty long time now, both for hijacking requests and for violating privacy.

I don't think that DoH advocates looked at a perfectly good system and just decided to replace it. They looked at a system with significant flaws, saw that this system was holding back consumer privacy and security, and proposed an alternative.

> but DNS is only a tiny part of that problem [...] If you want to get rid those, then get rid of shitty ISPs. This is throwing out the baby with the bathwater.

DNS is a huge part of that problem, because without encrypted DNS most people can't realistically keep their DNS queries private. What we're seeing is that we can do all kinds of fancy stuff around encrypting connections, but the DNS is always going to be a hole, and it's always going to make the other protections much less useful.

We want to eventually plug other holes that make websites identifiable. During that process, there are going to be periods where some of the holes are plugged, and some aren't. But that doesn't mean all of the efforts are useless, it means we're in the process of plugging security holes. The fact that we haven't widely deployed SNI encryption or TLS 1.3 does not mean that encrypted DNS is useless, it means that multiple technologies are required to work together to make progress towards meaningfully masking domain names.

We still have very hard problems to solve, like IP addresses themselves and statistical analysis of response sizes to determine which pages are being visited. But it's not much use trying to mitigate those problems if we can't even secure DNS resolution.

In regards to ISPs, in general (where possible) we like to avoid the model of "everyone is insecure but there are no bad actors." Imagine if we had rejected HTTPS because "this is only really a problem if your router is compromised or the ISPs are snooping." We have a solution now that helps reduce the amount of trust we need for ISPs. That's good, we should deploy it. Even if ISPs weren't a problem, it would be worth deploying.

> This will simply lead to ISPs and network operators blocking DoH. And instead of a wonderfully decentralized DNS, we'll end up with centralized DoH in the hands of Cloudflare and their ilk.

I don't see a lot of reason to believe this. First, there was nothing preventing ISPs from blocking DNS and forcing you to use their DNS resolvers in the past, so I don't see what DoH changes about that situation -- other than that it's a meaningful privacy increase, which might make bad actors more aggressive about blocking it. But surely we're not claiming that we should abandon meaningful privacy improvements because we're scared of what ISPs might do.

Second, DoH is decentralized, you can still use other resolvers other than Cloudflare, and you can still chain resolvers together. For the moment, I happen to recommend Cloudflare because I think it's offering more believable privacy promises than other providers, but I understand the concerns about Cloudflare's centralization. It's valid to use something else.

But seriously, what is centralized about DoH that isn't centralized about DNS? The fact that requests can't be hijacked? Did the Internet become centralized specifically because you can't MITM HTTPS requests now? I'm seeing people claim that DoH means every request will go to Cloudflare, and that's just flat out not true: anyone can set up a DoH resolver, even locally on their own network.

Okay, granted, not everyone on your network might respect your DoH settings you put on your Piihole or router, but that's not really different from today. Google already doesn't respect network settings for its DNS resolution on Chromecasts (or at least didn't last I checked). My browser already allows me to override my network settings for DNS. DoH changes nothing about that other than that OSes and routers are lagging on implementing network-wide DoH settings. If you think that staying with DNS would mean that IOT devices will perpetually respect your network settings for domain resolution, then I have bad news for you, because they're definitely going to stop doing that in the future regardless of what technology exists, and some of them have already stopped doing that today.


> Second, DoH is decentralized, you can still use other resolvers other than Cloudflare, and you can still chain resolvers together.

... but in practice almost all Firefox users will end up using one of the very few designated Mozilla TRRs. That's an increase in centralization, as measured on the ground. Social and behavioral effects count.

You can't claim one minute that "we're doing this to take care of naive users who can't protect themselves", and the next minute that "users can set up any arcane configuration they want".

In the end, you screw the naive users by increasing their practical centralization, and you screw the sophisticated users by adding complexity, breaking whatever protections they already have in place, and making it impossible for them to secure all of the applications on their systems at the same time.

Oh, and on edit, I guess I should address "first" as well as "second":

Nothing was stopping ISPs from blocking alternative DNS before... but very few people were using alternative DNS before, so it didn't have a lot of impact on ISPs' priorities. If you suddenly take away the ISPs' visibility into many people's DNS, then the ISPs get a reason to care. All kinds of things skate under the radar if they're not deployed at scale, but provoke responses when they get big enough to be threatening.


> ... but in practice almost all Firefox users will end up using one of the very few designated Mozilla TRRs. That's an increase in centralization, as measured on the ground. Social and behavioral effects count.

If we're going to go down this road and talk about defaults as they exist on the ground, then the reality is that most nontechnical users should be using Cloudflare's DoH resolver at the moment. Are there concerns about centralization? Yeah, of course. I'm as worried as anyone else about Cloudflare controlling the Internet, they are fundamentally a dangerous company. Being big, on its own, makes Cloudflare untrustworthy. But I'm also looking at Cloudflare's actions in the real world today, and they still have a better privacy record and are currently more trustworthy than any ISP in the US. Choosing them as a default in the US was the correct choice for Mozilla to make if you care about people's actual privacy in the real world today.

Mozilla's defaults here are also not set in stone. As network operators catch up with the technologies they've been ignoring, we'll get better OS and network integration for DoH settings. And as more providers start offering DoH, the defaults can change. Cloudflare is the default right now not because of a conspiracy but because regardless of how nervous they make us, and regardless of whether it's a good idea in theory in the future to have so much traffic routed through them, currently on the ground right now they are the best choice for most nontechnical users.

Additionally, currently on the ground there are few companies that are doing more than Cloudflare to decentralize DoH and improve its technical privacy guarantees instead of just asking people to trust them. Oblivious DoH[0] is an interesting proposal in that direction, there's also a decent amount of work (which Cloudflare is supportive of) going into research about by-default splitting DoH requests across multiple entities, so no single provider would get all of your queries. There are some obvious issues there to overcome regarding privacy, but it's still a somewhat promising proposal.

From a technical perspective, there is nothing about DoH that is centralized, and no part of the technology would force it to be centralized. There's no consumer lock-in, and there's no browser lock-in. Configuring DoH is not an "arcane configuration", it's just as easy as any DNS setup in your browser, even easier because there's less network interference and it's easier to debug.

But from a practical perspective right now on the ground, while I share concerns about Cloudflare overall and while I share concerns that Cloudflare consolidating power is problematic, I don't think it's accurate or reasonable to claim that DoH is a power grab. By default, Cloudflare isn't the provider in Canada. In the US, they have temporarily been chosen by the default by Mozilla because they are currently offering the best service. But anyone else could step into that position if they care to, there's no blocking or gatekeeping going on.

And frankly, if the only reason that DNS is decentralized is because most people's devices don't use a consistent resolver and just naively trust whatever network is were on, if the only reason why DNS is decentralized is because everyone's ISP sets it in the background without their knowledge, then in my opinion that's just a ridiculous way to guarantee provider diversity, and we should absolutely overhaul that system. Any system where my DNS provider changes behind my back when I connect to the wifi at a coffee shop is broken, and that is the experience that most nontechnical users have with DNS today. It is frighteningly insecure.

> You can't claim one minute that "we're doing this to take care of naive users who can't protect themselves", and the next minute that "users can set up any arcane configuration they want".

I don't think I do claim this. What I claim is that Cloudflare is an obvious default for nontechnical users, and I claim that DNS is susceptible to all of the same centralization concerns as DoH, and I cliam that changing a DoH provider in Firefox for most end users is no more difficult than it would be to change a DNS provider.

Most users won't change their DNS provider, which is why it makes sense to use a secure default rather than to let every random network in the wild decide. But if you are the type of person who wasn't just trusting network DNS, then DoH is not going to be a problem for you. It especially won't be a problem for you once OS manufacturers start allowing system-wide configs.

> If you suddenly take away the ISPs' visibility into many people's DNS, then the ISPs get a reason to care.

I buy this theory, I think you're very likely correct. But I don't think it's a compelling argument. If what you're saying is correct, then literally any privacy initiative that threatened ISPs in any way would be blocked. What's the takeaway from that, we should abandon efforts that threaten ISPs?

We are seeing a large amount of pushback from ISPs, from network operators, and from governments themselves over DoH. This does not suggest that the proposal is dead on arrival, or that it's just going to be globally blocked. If DoH didn't have teeth, we would not be seeing this reaction.

So what your argument is saying is that DNS skated under the radar largely because it wasn't helping anyone, and ISPs didn't feel threatened by it. But I'd rather throw my weight behind a technology that they do feel threatened by, rather than behind one that they don't care about. A world where we have to fight about DoH being blocked is still strictly better than a world where we don't have to fight because there's nothing to fight over and the majority of people are using a system that is trivially attackable.

[0]: https://blog.cloudflare.com/oblivious-dns/


For the record, I share your cautious confidence in Cloudflare. For an entity of its size, Cloudflare has been amazingly good both at being transparent and at actively finding ways that people don't have to extend any trust. Cloudflare even suggested real cryptographically blinded tokens as an alternative to CAPTCHAs (and saw it shot down by people who didn't understand the crypto, and were sure it was some kind of trap...).

Cloudflare could easily go bad in the future, but has been refreshingly good so far.

But most of the risks "native" to large commercial entities seem to have been using DNS queries to target advertising. That sucks, but it's not my top concern. I'm more worried about outside pressure to censor the Internet, which is more of a non-native risk. From my point of view, censorship is a bigger worry than targeted advertising. It can happen here, and it's already happening in a lot of "theres".

If some major government, or coalition of governments, or even major private pressure group, wants disappear something from the DNS, and there are only 15 or 20 TRRs, then it's a lot easier to lean on those TRRs than it would be to lean on hundreds of ISPs. In any given case, it may also be easier than leaning on the registrars.

Heck, since there's a centralized Mozilla policy for who gets to be a TRR, they can just lean on Mozilla. Maybe it won't matter because Chrome will have more market share or whatever, but it doesn't seem quite right for Mozilla to rely on that.

Even if Cloudflare happens to be much better about resisting filtering pressure than most ISPs, and stays that way, there's still only one Cloudflare that has to be subverted. And there's going to be pressure on Mozilla to include TRRs that might not even be as safe as Cloudflare.

You can already see the pressure to bake censorability into the TRR system. Look at the comments on the Mozilla consultation. You have a lot of people talking about how TRRs need to be able to (secretly!) respond to government blocking requests. You even have what looks like a coordinated push from UK commenters to protect secret non-government blocking at TRRs.

Yeah, we'll all disagree about whether X or Y should be censorable. Most people can name something that they think should be censored, and I think that a lot of people's lists are getting bigger rather than smaller. But once you start creating infrastructure for filtering, everybody comes out of the woodwork and starts trying to get their particular filtering priorities. It's hard for a centralized system to resist them.

The real answer is probably to switch to something a lot more decentralized and anarchic than the DNS can ever be. After all, in the end, there's only one ICANN. But that will take a very long time, if it ever happens at all. In the meantime, it seems really unsafe to concentrate power over the DNS in a limited number of pressure points.


At this point. The vast majority of ISP dns servers are hostile. Here in Australia, the ISP dns servers are hostile by law so 100% are hostile.

If you are in the minority who have a great DNS server, go to the firefox settings and make a change. Defaults should suit the majority.


I note Firefox is turning this on country by country. Perhaps they could turn it on for certain countries (possibly including yours, Australia) but leave it off for others (like New Zealand where all ISPs don't interfere with DNS).


Can they block DoH, with reasonable tradeoffs? How?

And will they care enough? I get the sense that they don't really gain that much from messing with DNS, relatively.


They couldn't necessarily block all DoH, but they could destroy its prevalence by just blocking all of the default resolvers and all of the resolvers that the browsers suggest to users. Those are public lists.

I suspect you're right that they don't get that much value from messing with DNS, though... not enough to deal with the user complaints they'd get. So they probably wouldn't bother unless pressured.

However, they might decide to "play chicken" with Mozilla and other sources of DoH apps by seeing who yielded to the user complaints first. There'd be a big temptation for Mozilla to fall back to "regular" DNS if DoH failed, and then the ISP would effectively win.


They'd have to block by domain name, right, and block all HTTPS traffic to those domains? I somehow suspect that browers can win that particular cat and mouse game just by making the domains too important to block (share the domain with something users care about). Maybe I'm missing something there though, it probably is more complicated than just that.


In the end, DoH has to either start with some kind of seed IP address, even if it's a cache-priming DNS A record, or use a (presumably attackable) non-DoH lookup to find its server. You can't start with a domain name for the server that you use to resolve all domain names.

But I don't think that's very important, because domain names always resolve to IP addresses anyhow, and you can't change the IP addresses that fast.

So you would have to share an IP address with something "too important to block". The political problem with that tends to be that the operators of things that might be too important to block tend to think that those things are too important to expose to any risk of being blocked, say by sharing them with something that somebody might want to block.

It's also not always administratively or technically trivial to share the domain name or IP address(es) of some giant public Web service with another fundamentally unrelated service like DoH.

I mean, sure, in the end, if enough large organizations tried hard enough, they could eventually come up with ways to make it pretty painful to block... but those large organizations tend to have overriding commercial concerns... and not to want to overtly taunt large governments...


> So you would have to share an IP address with something "too important to block". The political problem with that tends to be that the operators of things that might be too important to block tend to think that those things are too important to expose to any risk of being blocked, say by sharing them with something that somebody might want to block.

I think that's part of the idea of having Cloudflare run it. I'm far from a fan of them in general, but the one thing in their favour for this case is that an ISP would have to be pretty ballsy to block all Cloudflare's IPs.


They don't have to block all of Cloudflare, though; they just have to block 1.1.1.1, right?


Cloudflare could set it up so every IP in their address range was serving DoH and Mozilla could probably set up their clients to use random ones with retry or something like that.


I don't know why Cloudflare is inherently more trustworthy than your ISP.


> There's a lot of lament that this bypasses the operating system level config, but I feel like we wouldn't be here if the operating system had stepped up their DNS any time in the last several years.

OSes are relegated to support myriad environments with adhoc yet byzantine configuration. A browser's primary objective is to be a "user-agent" (as opposed to, say, subscribing to the whims of ISPs / Governments / anti-digital-rights-companies-masking-as-cyber-security-firms).

> For the techie user who's running PiHole or some custom local DNS, yeah this will be annoying, but for the average user who's just delegating to their ISP that is then tracking their DNS lookups and injecting ads if they mistype a domain, this is a win.

Internet censorship, which most of the population in the West isn't subject to or even aware of, is the primary reason DoH-in-a-browser is widely lauded. It just works, and thwarts all cheaper attempts at surveillance and censorship. Though, the likely effect would be, deep-packet-inspection becomes common place, but the imminent ESNI standard has that one covered, too.

As someone who builds and maintains an open-source DoH resolver, I have come to fully appreciate the control and power a resolver running over HTTP/S puts in the hands of the users, both in the client and on the resolver-side.


> the imminent ESNI standard has that one covered, too.

Too slow, I'm afraid. Word on the street is that any hello containing ESNI will be blocked from day one in too many places for it to ever get traction, including all of China. I believe draft ESNI is blocked by the GFOC today.

To even have a chance of making ESNI/ECH widespread, you'd have to make it mandatory for all TLS. That wouldn't fly politically in the working group, because the "enterprise" people have at least some influence there, and they want to do exactly the same kinds of DPI that North Korea would want to do. Even if it came out mandatory on the standards track, it would probably still fail at this point.

It might have worked if it'd been mandatory in TLS 1.2 or even TLS 1.3, but it seems too late now.

We'll see if they also disrupt DoH to the point of making it unusable. I bet they at least give it a try.


> That wouldn't fly politically in the working group, because the "enterprise" people have at least some influence there, and they want to do exactly the same kinds of DPI that North Korea would want to do.

If this was true, TLS 1.3 would have RSA kex. The "enterprise" people (specifically in fact EDCO, the Enterprise Data Centre Operators, Nalini Elkins' group) really wanted to keep RSA kex. It was removed in TLS 1.3 anyway because it's a bad idea.

In practice the intention is that ECH will be GREASEd, so yes, it will go from being nowhere to being apparently everywhere almost overnight. It's not specifically intended that GREASing ECH will defy the Great Firewall, but it's a possible outcome. China can decide its own policies without help from us.

For EDCO and similar outfits, ECH changes nothing.


> If this was true, TLS 1.3 would have RSA kex.

I hope you're right...

> In practice the intention is that ECH will be GREASEd

If even fake ECH extensions get rejected, then it's hard to deploy the whole GREASy burrito.

> China can decide its own policies without help from us.

... but China can make it uncomfortable to support ECH worldwide. It's not a free action for a service to risk being blocked in China, or for client software to accept being unreliable or unusable in China. That affects deployment in the rest of the world. We'll see who wins the war of wills, but I'm afraid that the "Western" entities that have the most control over what's widely deployed are not going to have will enough.


Any "service" that wants to support China needs to have serious presence there anyway.


The goal of ESNI is not to bypass China's intranet. The goal is to say to DPI countries such as the UK: either you implement a China-style great firewall or you implement a free internet. There is no middle ground where you can spy on people without it being obvious.


They actually are recommending for everyone to add fake ESNI/ECH data even if you don't intend to use ESNI/ECH.

See: https://datatracker.ietf.org/doc/html/draft-ietf-tls-esni#se...


If Microsoft, Apple, and Google all agreed to make ECH mandatory on the same future date, wouldn't that basically force the hands of every other enterprise?


I'm afraid browsers stopped being "user agents" a long time ago. It's one of my pet peeves. They're agents of whichever company rolls them out. Basically Google or Mozilla. And they defer to the whims of whoever built the website you're visiting for a good chunk of their (insane) behaviour.

If they were actual user agents, you would still have control over a lot of behaviour that's now been removed. Like actually remembering form fields you want to remember (instead, some websites tell the browser that it can't remember the user name, and the brower foolishly obeys). Or like copy/pasting text. Or like using the right mouse button (some websites disable it). Or like having control over keyboard shortcuts, which some websites override and which can't trivially be fixed.

In my book, a user agent acts in my interests and on my behest. Sure, most of these behaviours can be corrected using browser extensions, but requiring a boatload of extensions to have an actual "user agent" doesn't make me happy.


>There's a lot of lament that this bypasses the operating system level config

systemd-resolved has the clandestine ability to forward all your DNS traffic to google in many distributions of Linux. I would say the state of resolution in general is getting uncomfortably sketchy.

many wireless routers from Asus and others due to a buggy (or perhaps intentional) implementation of Dnsmasq will quietly intercept your DNS requests using their own built-in resolver. RT-ARCH13 is notorious for it even if NAT mode is disabled.

for my pihole implementation I block 53 outbound and do DoH resolution using cloudflared on the pi.

finally it bears remembering that the number one largest opposition to the use of DoH was ISP's who were (and still are) making a killing selling your search history. they went so far as to lobby congress to stop DoH. Now the very same ISP's we tried to protect users from are part of the DoH club anyway because, well, money talks.

https://arstechnica.com/tech-policy/2020/06/comcast-mozilla-...


I recently tried setting up microk8s, since Ubuntu was pushing it pretty hard and it seemed useful, but a lot of stuff wouldn't work at all.

Turns out that the microk8s config uses cloudflare DNS by default, and not your currently configured DNS or any kind of local DNS proxy. This was hugely frustrating because it's not the configuration that I would expect, it's not obvious why things are failing, and, in our case, it's sending internal DNS names to an external DNS server, unencrypted, over the public internet.

Surely it's convenient, but definitely not trustworthy.


"clandestine"? Please, don't be ridiculous. The choice of fallback DNS server is a default that is only used in the absence of any other configured DNS resolver. If you are compiling systemd yourself then this is one of many defaults that you need to be aware of. Everyone else will get the defaults that their distribution has chosen.


If Mozilla cared about preventing DNS tampering, they could have done DNSSEC 10 years ago. That would have totally killed ads, ISP typosquatting, and captive portals. They, like all browsers, rejected that source of end-to-end security.

The fact is that they want to enable DNS tampering, because they want to use it as a way of enforcing their idea of "safety". So they're building this giant, highly centralized, and easily pressured, infrastructure of "trusted resolvers" that can mess with your DNS results to their heart's content. This will, of course, be a huge attractant for bad actors who want to do things even more harmful than injecting obnoxious advertizing.

As for privacy, the only way that can work, even with something like DNSCrypt, is if the individual user chooses the resolver from among a very large set of possibilities.. which Mozilla is not enabling. Either that, or relatively exotic anonymization stuff that's not actually worked out, and that Mozilla has shown zero interest in working out. "Trusted resolvers" just move the privacy problem to a different place, and, again, it's a highly centralized place that's easy to subvert.

And DNS-over-TCP is desirable for sure, but it does nothing about tampering or privacy. It's just irrelevant, except maybe as a preliminary step making it easier to deploy some of the stuff that might actually help.


One of the reasons Mozilla couldn't "do" DNSSEC 10 years ago over plaintext DNS is that for some (enough) clients this doesn't work. Their DNS is broken, it was always broken, but so long as you only ever ask something easy like "news.ycombinator.com A?" it worked well enough. Asking DNSSEC queries doesn't work. "Last Change Broke It" rules apply so Mozilla gets the blame.

With DoH of course this is much less relevant. The people operating Trusted Resolvers for DoH are obliged to actually implement the protocol correctly.

If you choose a DoH provider which does DNSSEC for you (such as Cloudflare's 1.1.1.1), and you trust that provider, your answers are DNSSEC validated, today, without any changes to Firefox. Millions of people benefit, zero work.

If you're concerned that Trusted Resolvers aren't enough privacy, Oblivious DoH eventually means that the TRs don't see your query any more, so the people answering the query don't know who you are, and people who know who you are don't know what your query is.


I'll bet a comparable number of clients won't be able to reach DoH servers for one reason or another. Especially because there seem to be people who are going to actively try to block them. Whenever you change anything, something will break.

And it would have been relatively easy to put up a notice that said "your DNS server seems to be dropping all EDNS0 data or whatever stupid thing it's doing; would you like to switch servers"? Which would have avoided centralizing everybody by default.

I know there were enterprise firewalls that broke DNSSEC on any packet that went through them... but frankly the sellers of those firewalls deserved to get heat, and any enterprise that would deploy that kind of broken crap isn't an organization I'd want to support as user.

And that still doesn't justify putting DNS on top of HTTP of all things. But that's another story.

As for Cloudflare doing DNSSEC checks, the fact that I shouldn't be trusting Cloudflare to do the validation for me is kind of the point. Validation (final validation, at least) belongs in the endpoint.

If "oblivious DoH" is what it sounds like, it sounds like maybe I have to retract "zero interest in exotic anonymity". Which I am very happy to have to do.


DNSSEC has never validated at the endpoints. By design, validation occurs --- has to occur --- at the recursive server. The channel between you and your recursor is "secured" with a single bit, which says "yeah, sure, I did DNSSEC validation". Don't blame the systems software for this; it's a terrible protocol.


I am typing this on an endpoint that does DNSSEC validation.


You can also run default-free full-feed BGP4 on your endpoint, if you're just trying to make a point.


I don't run it to make a point. I run it because it's relatively easy to set up even with the presently available software, and it improves my confidence in what I get from the DNS. It would be even easier to set up on an endpoint if DNSSEC were something that people providing OSes cared about. In fact, there's no reason that setting it up intrinsically requires any user interaction at all; it could easily be the default.

You claimed that it "by design, validation occurs -- has to occur -- at the recursive server". That is false, and it is false in a way that's important to the issues under discussion. Nothing forces validation to happen at servers other than laziness and inertia.


I'm not sure you understand what I'm saying. DNSSEC validation occurs step by step through the recursive resolution process. You can't simply request records from a recursive server and then DNSSEC-validate them yourself. I don't think DNSSEC works the way you think it does.

Basically the message here is "DNSSEC will work when customer DNS servers go away and everyone just runs their own". Ok, I'll add that to the (long) list of obstacles to its adoption.


I understand exactly how DNSSEC works. It's been a while since I was really close to it, so I doubt I could sketch the packet formats any more, but I'm crystal clear on the structure... and I still admin it for multiple zones.

Insofar as I can extract any meaning from what you say, what you say is false. DNSSEC validation does not take place "step by step through the recursive resolution process". You seem to be conflating the resolver tree with the zone tree.

Any entity that can get access to the necessary DS, KEY, and RRSIG records (plus the actual data records) can do a complete local DNSSEC validation from the root to the leaf. It can get those records from anywhere. It doesn't even have to get all of them from the same place. It doesn't have to rely on any validation from the upstream servers, and is in a position to independently verify any validation they do do.

In fact, that's a basic DNSSEC design feature. One of the design goals was that intermediate responders didn't have to know about DNSSEC at all. Otherwise it would have been impossible to ever deploy it.

The protocol provides for asking a responder not to return invalid records, or even to return them but flag them as suspect. Clients don't have to use that functionality. Clients can always ask for all of data and for themselves. And any responder that does set any validation-related bits is expected to have verified the entire signature chain for itself, not deferred to any validation the next upstream responder claims to have done.

There's nothing "step by step" about it.

It's true that DNSSEC validation at the application end is <em>typically</em> done by a stub resolver, by which I mean a specific kind of limited DNS server that does recursive lookups and caches results for some defined population of clients, but is not authoritative for any zone. Unfortunately the word "resolver" is confusing, because it's used to refer both to stub resolvers and to more or less any software that does DNS lookups at a client, including libraries linked into application programs.

Nonetheless, the validating entity does not have to be a stub resolver or any other kind of DNS responder. It can be a "resolver" in the sense of a library, or it can be something else. Applications don't have to use the DNS protocol to talk to it, and if they do, they can do so entirely locally on a single host.

But you can still do on-endpoint validation using a stub resolver, just by using a local one that serves only applications on a single host.

To head off your next load of FUD about "running your own DNS server", such a local stub resolver is a small thing that can be installed by default, because you can get it working with zero configuration or administration, beyond maybe a forwarder address that it can get from DHCP.

Running an on-node caching resolver is nothing like running an authoritative DNS server, or even a stub resolver serving a whole network, and I know very well that you know that. Don't try to compare them.

A local resolver process doesn't demand a lot of resources by post-1990s standards, and the end user doesn't even have to know it's there. In fact, operating systems typically have similar local server processes anyway, for performance reasons. That's exactly what nscd or systemd-resolved does. Indeed, systemd-resolved, even though it was introduced for unrelated reasons, can be told to do DNSSEC validation, removing any need at all for a separate DNS-based stub resolver process.

I have trouble believing you're confused about any of this. What you're writing here, and what you've written elsewhere, sounds too much like the output of somebody trying to generate FUD that will convince the half-informed. And, yes, I did read that enormous screed you wrote a few years ago, with all the other gross, and suspiciously convenient, misunderstandings and mischaracterizations.


This is a long description of how an endpoint can do much of the work a recursive cache does for itself. We all understand that it can. You've clarified, you understand how DNS servers do lookups; I won't suggest that you don't again. It was never my claim that every server in the DNS topology needed to be DNSSEC-aware (though, obviously, every authority from leaf to root needs to be).

You haven't made any headway in your argument about how everyone is going to run DNS software that does this stuff. It's "easy" to do this stuff, in the same sense that it's trivial to run a local djbdns instance on your laptop instead of relying on your ISP's server. My point is: that's not how DNS is normally deployed.


> And it would have been relatively easy to put up a notice that said "your DNS server seems to be dropping all EDNS0 data or whatever stupid thing it's doing; would you like to switch servers"? Which would have avoided centralizing everybody by default.

Would you really expect ordinary users to understand that and respond appropriately?


> If Mozilla cared about preventing DNS tampering, they could have done DNSSEC 10 years ago.

DNSSEC only protects integrity, not confidentiality or availability. It still lets evil ISPs build a profile of every domain you've ever resolved, and to block your access to certain domains by just dropping your DNS request packets for them. Basically the only thing it stops is them sending you to a fake page (e.g., a pretty block message rather than a generic browser error).


Also no one cares about integrity since it is already validated by TLS. And it seems that DNSSEC is not even widely supported by TLDs. My domain says DNSSEC is not available.

Seems like DNSSEC is another failed spec like certificate pinning.


We need DNSSEC for eSNI/ECH to be possible.


The eSNI drafts I read say explicitly that DNSSEC is not required (there's a whole section about it), which is good, because if it were, eSNI would be dead in the water.


I looked again and you're correct. For other readers, the justification is basically that if an attacker can MITM DNS, then they can find out what domains you're going to even without compromising eSNI/ECH.


I double checked the RFCs, but I'm not kidding that you can generally work this stuff out axiomatically (for a change) --- if an important new standard requires DNSSEC, it's not an important new standard, because it can't happen. DNSSEC is moribund.


How could Mozilla have "done" DNSSEC 10 years ago? Isn't that pretty much just not in their control?


If TLDs are signed (which they have been for a long time), and if the browser checks DNSSEC, you can't typosquat, because you can't spoof the TLD server's signature on the typo. That requires no cooperation from anybody but the browser and the TLD.

WIth DNSSEC enabled, somebody can still impersonate an unsigned domain, but the whole reason that there are so many unsigned zones to begin with is that nothing checks the signatures, so there's no incentive to sign. By actually doing the checks, browsers, including Firefox, could have created an incentive for widespread DNSSEC signing.

So they could have both unilaterally gotten a lot of benefit for their users, and encouraged wider DNSSEC deployment that would have produced further benefit.

Add in DANE, and it would also have given a much more satisfying fix for a bunch of CA issues that they've chosen to band-aid with stuff that only half works. You could probably even have gotten to a world with zero-cost TLS certs well before enough pressure built up to make Lets Encrypt come along.


You're saying it's one way, but it's the opposite way. Huge portions of the Internet's user base resolves through servers that will validate DNSSEC if it's available. Still, nobody --- most especially not organizations with serious security teams --- signs their domains. That's not because they don't care about security. It's because DNSSEC makes sites unreliable, and the failure mode for DNSSEC is just about the worst possible mode: your site simply vanishes from the Internet, as if it hadn't existed at all.

None of this is hypothetical. It happened, for instance, at the launch of HBO NOW, when a screwed up DNSSEC record made the site vanish for everyone resolving through Comcast, which pointlessly verifies DNSSEC.


DNS without DNSSEC will also "make your site vanish" if you administer it incompetently. And that's not hypothetical either; it happens all the time.

As for the "servers that will validate DNSSEC if it's available", the operators of the servers that "huge portions of the Internet's user base" use are exactly the people who are potential threat actors in most DNS tampering scenarios. That's not useful or effective validation. It makes no sense to put the choice of whether to validate in their hands, and obviously nobody can trust their validation.


DNSSEC does not protect privacy which is more important than integrity since TLS already does that.

Even worse, making DNSSEC well used might have pushed privacy protocols out of priority as people claim DNS is good enough already.


Interesting, thank you. That does sound more satisfying as a fix (though also sounds harder and possibly riskier. If they're the only ones checking, will some sites be broken only in Firefox?)


> This is a workaround for the fact that many networks are hostile to DNS lookups (injecting ads, tracking, captive portals), and operating systems continue to just default to the ISPs DNS provider, which is often hostile.

This workaround is hostile to my own network (home or work) being able to track requests. I control my router / DCHP server, and I should be able to tell devices where to send DNS requests.

Anything that does not do what I tell it to do is malware IMHO.


You can’t track all requests anyway, as they can be trivially tunneled through encrypted streams with bundled certs.

This is the tradeoff of running arbitrary Turing-complete programs. If you don’t like what a program is doing, or that it doesn’t use libc or some standard way to do DNS resolution, your only options are to either change the program to do what you want, or don’t run it.


> You can’t track all requests anyway, as they can be trivially tunneled through encrypted streams with bundled certs.

Previously I can track devices (a) making DNS requests, and then (b) make TCP and/or UDP requests. Anything that made a data request without a DNS request was probably using a hard-coded IP, which would be suspicious.

With DoH, it's all-HTTPS all-the-time:

> DoH is an over the top bypass of enterprise and other private networks. But DNS is part of the control plane, and network operators must be able to monitor and filter it. Use DoT, never DoH.

* https://twitter.com/paulvixie/status/1053886628832382977

> or don’t run it.

So Firefox further circles the drain with regards to market share? Is this part of some grand 4D chess plan by Mozilla that is perhaps beyond my intellect?


The cat is out of the bag regardless of what Firefox does. Even without firefox any bit of malware could bundle their own DoH library.


> and I should be able to tell devices where to send DNS requests

But you can't. Even if DoH went away today, even if Mozilla and everyone else dropped the technology, you still don't actually have the ability to do that. Devices will still be able to ignore you.

It's not DoH's fault if devices on your network don't use your network-wide DNS resolver. IOT devices were going to move in that direction regardless. Increasingly, your smart TVs are going to ignore your network settings. They might not even make DNS requests over the correct port. They might hide them in normal Internet traffic. They might cache them locally. They might make a request for a domain that looks safe, and then use a different IP address that they have cached locally. They might even ship with their own embedded 5G chip and use it for specific exfiltration requests and updates. They might join your local network, read all of your unencrypted traffic, and then use a 5G chip to separately send data out.

If you think that blocking DoH will solve these kinds of problems, I feel like you are putting way, way too much faith into your existing DNS setup.

> Anything that does not do what I tell it to do is malware IMHO

Okay, that seems completely reasonable. However, my perspective is that any network that will MITM my requests or monitor what websites I visit without my permission is malicious IMHO. It sounds like the answer here is to block access to your network for devices that you don't trust, or to try and quarantine them -- not to demand that my personal device stop respecting my own entered DNS settings whenever it joins a new network.

To argue that my installation of Firefox or that my OS shouldn't be able to ignore network settings anywhere and that I shouldn't be able to choose how my requests are encrypted, just because it makes it easier for you specifically to enforce your network's security policy without explicit opt-in or consent -- I don't know, it just seems pretty backwards to me from how this kind of security stuff is normally supposed to work.

I know that this stuff will make it harder for some network operators to do their jobs but... so did HTTPS. I heard literally the exact same complaints during the early push to get every website encrypted. But we live with it, and now if you want to monitor my unencrypted network traffic, you need to get my permission to install a certificate. Control and consent slowly moves away from networks and onto devices over time, closer to the users themselves.


> Devices will still be able to ignore you.

Yes. But previously by logging requests to port 53 (and 853) I can hunt down those devices quite easy.

> However, my perspective is that any network that will MITM my requests or monitor what websites I visit without my permission is malicious IMHO.

If you don't like how I've set up my network, blocking your DNS requests, feel free not to use my network.

> To argue that my installation of Firefox or that my OS shouldn't be able to ignore network settings anywhere and that I shouldn't be able to choose how my requests are encrypted, just because it makes it easier for you specifically to enforce your network's security policy without explicit opt-in or consent

You can set up your devices however you want. But if they're on my network, they play by my rules.

> I know that this stuff will make it harder for some network operators to do their jobs but... so did HTTPS.

And network owners are free to block port 443 as well. Or any other port. People can choose to use those networks or not.


> But previously by logging requests to port 53 (and 853) I can hunt down those devices quite easy.

And IOT devices are required to use port 53 or 853? They don't have the ability to proxy their DNS requests through another service? DNS even as it exists today is not a mandate, this stuff can be circumvented.

> You can set up your devices however you want. But if they're on my network, they play by my rules.

I think that's fine. Block DoH on your own network. But don't complain about my browser turning it on. You can block 443 if you want, but I'm not going to be sympathetic if you argue that LetsEncrypt shouldn't exist.

There's a big difference between saying that you require every device on your network to plain-text its DNS queries to you for monitoring, vs arguing that it's harmful for Mozilla to make Firefox's DNS queries more secure for everyone else.

The only way DoH would be hostile to your network is if needing consent to monitor people's DNS queries when they join your network (via a settings change or confirmation that they're not using DoH) is a problem for you.


DHCP is a mechanism to automatically configure clients with useful defaults, like a DNS server. It's not a system for enforcing network policies. There was never supposed to be any requirement to observe what DHCP tells you.


I didn't realize that Fedora dropped that proposal https://pagure.io/fesco/issue/1511

May have been related to IPv6? Not sure.

I remember trying it out at the time (https://fedoraproject.org/wiki/Changes/Default_Local_DNS_Res...) and not running in to issues


Having had the joy of writing code to manage system wide DNS config for all 3 of the major desktop operating systems, it's very clear that nobody cares about DNS - the users just want it to work and the systems developers also just want it to work. On macOS, if you don't have an IPv6 enabled primary adapter, even if you have a working IPv6 VPN connection, you just won't be able to get AAAA responses regardless of your configuration. Watching for config changes is also a race ridden mess, applying DNS on Linux on major distros is also a crapshoot of which specific NM (NetworkManager) version the user is using (this can differ between a fresh Ubuntu 16.04 install and one which has been upgraded to 16.04 from 14.04), and which specific DNS mode NM is configured for. There's also systemd-resolved, with multiple versions and capabilities, and there's also resolvconf. In some cases, there's a notion of a global config, in others, the config is always bound to a specific interface. Windows is somewhat better in this regard. But at least the developers of NM and systemd-resolved are receptive and incredibly helpful.


How OS is going to circumvent local DNS? Hardcode 1.1.1.1? It's very easy for ISP to block this address.


Release (not beta) macOS already lets you configure your system to use DNS-over-HTTPS, if you know the magic incantation to chant at it. Apple hasn't added UI for it yet, but it's available and works when enabled.


> ISPs DNS provider, which is often hostile

Exactly which Canadian ISP does this? I don't think that's an issue in Canada, where this feature is being enabled.


On Linux there is systemd-resolved which is enabled installed and enabled on the most popular distros. It is a caching resolver that supports DoT (although it isn't enabled by default).

Having the pre-select a DNS resolver some of the same problems as having the application selecting the resolver. Specifically, it doesn't work as expected in networks with a local DNS server.


This is where iOS wins I think. According to Apple documentation, no app can override a systemwide DNS setting in iOS14. So if you were to apply a dns profile, it is guaranteed that apps cannot use their own dns resolver.


Which documentation? I searched the App Store Review Guidelines for "DNS" and didn't find anything.


They have, will Firefox respect the settings made there?

ie Windows 10 has it, macOS, Linux with systemd-resolver has it, Android too.


Mozilla wants to be the gatekeeper to their users. Every company and it's dog wants to do this. "If you are not paying for the product, you are the product" should become "if you are a customer/user, you are also a product".


I'm not sure how I feel about DoH. It seems like there a ton of unnecessary overhead added by placing the super lightweight DNS protocol on top of HTTP. My initial reaction is also that I might lose some control over my network and which domains hosts query on my network. Are things like DNS sinkholes and domain blacklisting still possible with DoH?


You don't get to control what my devices do just because you control a network they're transiting through. Even if DoH had never been standardised, clearly my devices can do this anyway.

On the other hand, if you actually control your devices DoH shouldn't be a problem, you can have them talk to whichever servers you prefer regardless of the protocol.

As to the lightweight thing. We get a few benefits from running DNS on top of HTTPS rather than say TLS (which also exists, as DoT)

1. HTTPS allows us to put parameters in the URL which are thus confidential (HTTPS will encrypt them) and have those control the DNS lookup. So dns-server.example can offer all its users their own URL, even though they all talk to the exact same server (dns-server.example.com) and a snoop can't tell that one device used the rank0 account while another device used the tialaramex account, those are encrypted in the URL, yet the actual server can see which is used and maybe my devices get "pure" answers while yours have ad-blocking or whatever.

2. DNS doesn't have a rich error vocabulary. For your ad-blocked domains, a traditional DNS server has to either lie (bogus answer) or claim no answer exists (NXDOMAIN or OK 0 answers) but HTTP has a rich variety of error codes. Maybe 403 Forbidden is appropriate for the Xian parent who doesn't want their teenager looking at pornhub.com and 451 Unavailable For Legal Reasons is appropriate for a DNS service which excludes Pirate Bay due to lawsuits.


> You don't get to control what my devices do just because you control a network they're transiting through.

And if I run a guest wifi in one of the many countries where I have legal obligations to take steps to block illegal traffic because I am ultimately responsible for the emissions of my network, what would you suggest I do?

> DNS doesn't have a rich error vocabulary. For your ad-blocked domains, a traditional DNS server has to either lie (bogus answer) or claim no answer exists (NXDOMAIN or OK 0 answers) but HTTP has a rich variety of error codes.

It's a good thing that DNS has a limited vocabulary of error codes. I want the garbage website to think it couldn't load the ad because of a mysterious DNS issue. Having a reason like "DNSHTTP 486: Blocked by Client" would just make it that much easier for the site to deny you access/track you/detect your blocking activity.


As with steps someone might have a legal obligation to take to prevent people from thinking negative things about the King or to prevent six from being an even number, you should do whatever you feel is appropriate. It might even make a difference. Perhaps you could put up a notice which says "I run guest WiFi here and I have legal obligations to block illegal traffic so, don't do anything illegal".

It's certainly weird to insist it's everybody's else's problem that you're subject to legal obligations that may be impossible to fulfil.

If the purpose of this "obligation" is to ensure nobody is technically compliant and so they can arrest, imprison or even execute whoever they want, it doesn't seem like that's my fault, or Mozilla's fault, or any of the authors and acknowledged contributors for RFC 8484. Any more than it's Giuseppe Peano's fault that six is even.

And if (as seems most likely) they're just covering their backside, the notice is probably sufficient.


> And if I run a guest wifi in one of the many countries where I have legal obligations to take steps to block illegal traffic because I am ultimately responsible for the emissions of my network, what would you suggest I do?

This is an oft overlooked part of these discussions. I manage a public network at a library (which is still the only access point for many people, unfortunately). DNS blacklisting certainly isn't the only tool in the toolkit, but it is one that DoH takes out. I'm genuinely conflicted on DoH because on one hand, I am a privacy advocate, but on the other I run a public network and that comes with certain obligations, both legal (we need to block filesharing) and to customers (we do traffic shaping because the budget it limited and bandwidth isn't cheap).


You do what every company does. Deploy industry standard methods and record your processes. And then when someone comes asking, you show them you did everything right and that is usually the end of it.


That's not the case in the US. Our ISP will absolutely be within their rights to cancel our service if someone on our network is violating their TOS. And we don't really have the money or the standing to fight it.


A world where it’s the local public network operator’s problem to filter client traffic came and went with TLS. You already live in a world where someone who wants to bypass your restrictions can do so trivially. So if you’re not in legal hot water today you won’t be tomorrow with DoH. You deploy whatever non-functional product your country requires for filtering and wash your hands of the responsibility.


> On the other hand, if you actually control your devices DoH shouldn't be a problem, you can have them talk to whichever servers you prefer regardless of the protocol.

I already do tell the devices I control which DNS servers to use via DCHP and IPv6 RA. Firefox is choosing to ignore that by default, and now I have to go through extra steps to solve a problem that Mozilla created by not simply re-using the previous solution (gethostbyname(3)).

At the very least if they had simply used the already existing DoT, instead of inventing a new protocol (DoH), I could have monitored port 953 traffic and see which devices were broken. Now I have to figure out devices trying to sneak through my policies.


It's good that they didn't use DoT, because then it would be too easy for adversaries who want to do censorship or surveillance to just block port 853.


Do you think that DoH prevents adversaries from doing censorship and/or surveillance?

> DoH encrypts precisely zero data that is not already present in unencrypted form. As it stands, using DoH only provides additional leaks of data. SNI, IP addresses, OCSP and remaining HTTP connections still provide the rest. It is fake privacy in 2019.

* https://twitter.com/PowerDNS_Bert/status/1175744071673028608

In the resulting HTTPS web request you have the hostname anyway. Think that ESNI will save you? Well that's being blocked:

* https://www.theregister.com/2020/08/11/china_blocking_tls_1_...

* https://www.zdnet.com/article/dns-over-https-causes-more-pro...


> DoH encrypts precisely zero data that is not already present in unencrypted form.

What's the point in locking my front door if I leave my window open? And what's the point in closing my window while my front door is unlocked?

This argument is circular. If you want to secure something that is widely insecure, then you have to start somewhere. It's not like people are ignoring SNI and IP addresses. They're just being handled in separate efforts than DNS is.

> Think that ESNI will save you? Well that's being blocked

If you're working with the assumption that any serious attempt to secure hostnames will eventually be blocked, then what's the point of anything at all? Should we just completely give up on security/privacy because the state won't allow it?

We can address ESNI blocks as a separate effort from DNS. We don't have to do literally everything at the same time, it's OK for us to gradually move towards better security and address each problem one at a time.

> Do you think that DoH prevents adversaries from doing censorship and/or surveillance?

Given the amount of complaining I'm seeing from multiple network operators on this very article, yes, there's a pretty strong likelyhood that it helps. Because if it didn't, then network operators wouldn't be complaining about it.

How do you square "DoH is useless" with these kinds of comments in your own linked articles?

> In a paper published last month, the SANS Institute, one of the world's largest cyber-security training organizations, said that "the unmitigated usage of encrypted DNS, particularly DNS over HTTPS, could allow attackers and insiders to bypass organizational controls."

> "The trend is unmistakable: DNS monitoring will get harder," the Dutch agency said.

> The Internet Watch Foundation (IWF), a British watchdog group with a declared mission to minimize the availability of online child sexual abuse content, also criticized both Google and Mozilla, claiming the browser makers were ruining years of work in protecting the British public from abusive content by providing a new method for accessing illegal content.

Does DoH make blocking/monitoring content harder or not? Is the UK lying when it says that DoH will make it harder for them to block content at the ISP level?


Sounds like eSNI is a good move if China is blocking it.


Sounds like eSNI is useless waste of effort since its intended audience can't use it.

Meanwhile eSNI/ECH breaks network visibility into the networks I'm responsible for managing (at home and work). If I'm supposed to be a good netizen and make sure that I quickly hunt down malware that's gotten on my network(s), blocking tools and techniques to do just that seems… silly.


> Meanwhile eSNI/ECH breaks network visibility into the networks I'm responsible for managing (at home and work).

Well, don't worry about it, since apparently no one can use it and it'll get blocked, right?

I don't understand the simultaneous argument of "this won't be usable because governments will all block it" and "this is going to make it impossible for me to monitor my network." Both of those arguments can't be correct at the same time; if your government won't block eSNI, then it'll be a privacy boost in your country and it's a realistic path for privacy advocates to pursue. If your government does block eSNI, then why are your worried about your personal network?


I'll do it too, for my own network. But adding it at firewall would be a good/acceptable idea... Currently this probably means running HTTP/S proxy.


_You_ are not in control of _your_ devices. Your television will use this technology to report what shows you're watching to the company that manufactured it; the web browser running on your computer will use this technology to report your activity and retrieve advertisements even if you've tried to block them elsewhere.


It amazes me how people think the worlds largest advertising company is pushing DoH out of alturism. That pihole you're running? That 'steals' money from google. DOH gets rid of that.


You can still use ad blocking with DoH. Just pick a DoH server that does so (or run your own) instead of using the default one. Or just use a browser extension like uBlock Origin instead. Nobody's trying to kill your Pi-hole with DoH. They're trying to kill your Pi-hole by serving ads from first-party domains.


You missed the point. OP is talking about app and IoT devs hard coding DoH servers into their products so that they can bypass any tracking, advertising, and privacy restrictions (aka piHole) you put in place.


Devices like that could just hardcode the IP addresses of their ad servers instead, so DoH isn't really the culprit there.


Yeah there’s nothing about DoH that a Google or any IoT device maker couldn’t have done 10 years ago if they really wanted to bypass local DNS servers.


The companies willing to do this already made their products fail to load when their ad domains could not be reached. I tried pi hole for a week and found every ad infested service would just throw an error.

This also means nothing for firefox. Firefox can not use DoH and iot products can just implement it themselves.


How many people would be interested in an open DNS service, with the publicly configurable rules e.g. blocklists/custom dns configurations, or rule-lists like ublock origin? I am thinking of starting one.


If your television or browser are running proprietary software while having network access, you have already lost.

Anything that you can do as a network admin to control "your" devices, an ISP can do to control its users. The sane way to resolve this conundrum is to fix your network endpoints to run only Free software, as the alternative would be facilitating centralized control on the large scale.


Your TV will have its own cellular chip soon to report back data regardless of what you do. Or it will automatically collect to Amazons mesh network.


You want to use my network you live by my rules. You don't have to connect to it.


So don't let untrusted devices on your network. Once they are on the network they are able to do anything you don't block.


> Are things like DNS sinkholes and domain blacklisting still possible with DoH?

they aren't, which is part of the reason DoH is implemented in the first place.


Yep. The entire point is to make it impossible to block ads at the network level. And since ad companies control the application level too, they then have a complete end-to-end ad delivery stack that you can't tamper with.


Yep. Everything's going to be locked from the bootloader to the screen and you WILL watch the ads.

What I think will really happen is the same thing as everything else. They'll use tech to take away features / abilities we have right now and then rent it back to us as a subscription.

"OpenDNS is now part of Cisco"

Add it up.


We are talking about Firefox, a web browser, that allows you the most control over your computer, has the best adblock technology, on which the author of Ublock origin has said his software runs the best, which comes with built-in anti tracking, which now comes with technology making it harder for public wifi to trick your computer into going to captive portals, often with ads.

And somehow you ad this up to making it impossible not to see ads and locked down computers.

How?


Firefox's primary sponsor is still Google. And whether they are pushing it because of malice or just incompetence, DoH was designed and built by Google to protect ad companies from network security. Implementing it by default is a hostile act, and one Mozilla should reconsider.


i have a pihole and have been worried that DoH would break it. i checked the network settings in firefox and the DoH setting is there but it is disabled.

I doubt that chrome will allow one to disable DoH but at least firefox does for now.


I believe Pihole automatically put in the NXDOMAIN entry needed to disable DoH on Firefox for you. But who knows how long Firefox will respect that, since there's nothing stopping ISPs from employing the same strategy to disable DoH.


Even if Firefox stops respecting that due to malicious ISPs using it, you'll always be able to turn off DoH in Firefox's preferences.


For now.

This is one step in the boiling-the-frog process.


>the entire point is to make it impossible to block ads

the entire point is to make it impossible to modify DNS requests at the network level. this has a lot more serious consequences than just blocking ads. especially for the parties involved with this, none of whom are advertising companies. phishing and data security are actual big issues with financial implications that companies want to prevent. just because blocking ads is the consequence that will most immediately impact you personally, doesn't make it the whole point.

this accusation is especially rich on an article about mozilla, one of the few companies fighting to make sure that advertisers don't control the application level.


> phishing and data security are actual big issues with financial implications that companies want to prevent

These are issues exacerbated by DoH, not fixed by it. DoH assists in the circumvention of security and monitoring. We block ads because they're security problems.


>DoH assists in the circumvention of ... monitoring

yes, that's the point. I understand you want to monitor traffic within your network, but forcing clients to use insecure protocols to enable network-wide monitoring means you're enabling network-wide monitoring. and that's the opposite of security.

remember that your browser traffic crosses multiple networks before it hits the website you're trying to connect to. forcing that traffic to be observable and modifiable in your own network means it will also be observable and modifiable by your ISP.


> forcing that traffic to be observable and modifiable in your own network means it will also be observable and modifiable by your ISP

This is factually false. And represents a key part of the problem in the rollout browser vendors have designed: Browsers should not be implementing their own network stacks. It's the wrong place to begin encrypting network requests, but for the "if you're a hammer" crowd, everything looks like a nail.

If the browser respected the OS stack, the OS could decide to encrypt requests, but it isn't given the choice. If the browser respected the network it's in, the network could decide to encrypt requests at the border, but it isn't given the choice.

Browser vendors decided the right choice was to build a product purpose-built to bypass nearly every good method of restricting malware, phishing, and, oh of course, ads. Google and their ilk would happily install ransomware on every PC on the planet if it would guarantee Google Ads couldn't be blocked, and that ISPs couldn't compete with their well-established surveillance tools.

The latter is why any suggestion DoH is to protect user privacy is silly... the user's privacy was compromised by the browser application pre-encryption. They just need you to believe the ISPs are somehow a bad actor for tracking you, so that only they can track you.


Control over DNS is neither necessary nor sufficient to block ads.


It's not sufficient in itself, but it bears most of the load of network-wide management of harmful traffic. It's the 99% effective method.

Endpoint-level control is no longer possible, since ad companies are skipping the OS network stack and bringing their own. Application-level control is less efficient, and more difficult, particularly when the applications are designed by the same ad companies trying to circumvent DNS control as well. (See Manifest V3.)


Any device you didn't have root on was free work around your DNS server already. This is weak security at best.

DHCP will catch up too. https://datatracker.ietf.org/doc/html/draft-peterson-doh-dhc...


Use MDM and configure the browser. Or block the special probe domain Mozilla has for this purpose.


Yes unfortunately this seems to be the reason. Earlier you could tell the resolver to look at files first and filter with your hosts file the ad and other sites. Now you have to do packet inspection. We are living on a planet that's revolving and evolving ...


Blocking ads at network level is very easy to circumvent. Just serve ads from your domain. It's impossible to block youtube video ads using DNS, for example, you must intercept their API queries and modify them on-the-fly which requires browser extensions or custom apps.


It's easy to circumvent if your ads are first party, as in YouTube's case. Most ads are not. Even most Google websites serve ads from ad-specific endpoints, as opposed to their own domain.


I think that's because those who care about blocking ads, will block them anywhere. And percentage of those who're using pihole or similar methods to block ads is negligible, so they don't really care about that.


And if your blocking ads your probably not the desired audience of them anyways.


All you need is to point DoH at your own DNS server


You can't do that with a Chromecast. The goal here is to sell devices which can ensure they always talk to their own DoH endpoint with no ability to secure it.


Chromecast was already bypassing local DNS for years before DoH was even conceived.

Blocking people from using DoH isn't going to solve the problem you are describing, it will just weaken users' privacy. Big vendors like Google will just create their own workarounds.


> Chromecast was already bypassing local DNS for years before DoH was even conceived.

Before DoH though, my understanding is a good gateway appliance could just overwrite those DNS requests as it sees them. Now they'll be encrypted, and you'll have no way to tell the Chromecast where to connect (or even to where it is connecting).


If the possibility of having the DNS packets rewritten was a concern for them, any junior developer could have easily implemented a proprietary REST web service to get the IPs over HTTPS just like DoH. Standardization of DoH in browsers isn't really furthering that issue.


How would they get the first IP for the REST client to contact?


That problem isn't solved by DoH, you still need to hardcode the DoH server IP there too if you intend to bypass the network level settings. So a proprietary system would be no different.

All the major appliance vendors (Google, Amazon) already have huge fixed IP ranges to devote to this purpose, which are effectively unblockable because they might be shared with important cloud services.


> The entire point is to make it impossible to block ads at the network level.

The entire point is to make DNS secure. For every 1 person who uses DNS to block ads, there are many thousands who just use their ISP's DNS servers, and thus remain subject to surveillance, ads, redirects, and other malice. You can't possibly believe in good faith that the entire point or even the primary point of DoH is to hurt the small fraction of people who use DNS to block ads, rather than to protect the much larger set of people who are subject to the whims and financial interests of their ISP.


ISPs do not inject malware and phishing scams, Google Ads does. The benefit of blocking ISP tampering is vastly outweighed by protecting the bad behavior of a far more malevolent party.


Google ads (like most ads) are hosted by websites that intentionally put them there; install an adblocker. Google doesn't MITM sites; ISPs and malicious networks do. ISPs are also known to surveil traffic on their networks.

I'm not disputing that Google ads and tracking, like all other ads and tracking, should be blocked. Run a browser you trust, and run an adblocker. But the widespread use of unencrypted DNS is a problem that needs fixing. And DoH provides the most viable solution for that problem, by running DNS over an ordinary HTTPS connection.


> Google doesn't MITM sites

AMP.


There's legitimate complaints about AMP, but it's not a MITM.


Ye I know. I was half sincere, half joking.


> ISPs do not inject malware

Not taking a side on this thread, but just want to point out that Comcast literally has a patent describing just such a mechanism. [0]

[0]: https://patents.google.com/patent/US20120224572


From said patent:

> including weather, emergency broadcast, and police stations

These seem like reasonable uses of this technology. My ISP has tried to inject a copyright violation notice before. (This was hilarious, actually, because it went to a guest in my house's browser, not one of mine, and I almost didn't hear about it at all, because they only sent it that way once... and I had to ask them to send me an actual letter about it with the details, which would've been the straightforward way to tell me in the first place...) Irritations with copyright holders aside, that was a notification of a mark against my account, which is arguably information that needed to be delivered to me.

Meanwhile, Google is preserving their ability to send phishing sites unimpeded.


Not a Verizon customer I take it?


Pardon my ignorance, but can’t you simply host your own DoH server where you choose the domains you want to block, which in turn points to a DNS / DoH server you trust? I guess it wouldn’t be more difficult than hosting your own filtered DNS server like Pi-Hole.


Yes you (or we) can. But you need to manually alter the settings everytime a new ff version is out. And other people can not. Brave new world. BTW how is Brave at this chapter ?


Why would you need to manually change your settings every time a new Firefox version is out? Firefox has settings UI to point to the DoH server of your choice and that setting does not get reset by new Firefox versions:

https://support.mozilla.org/en-US/kb/firefox-dns-over-https#...


Why every time? Isn't this a user config that overrides whatever default there is?


Here's a blog post from someone who uses nginx 'doh-to-dns' to connect their DoH clients to Pi-hole.

So, yes you can blacklist domains, as long as you can configure the DoH servers that the client uses.


> as long as you can configure the DoH servers that the client uses.

Which now involves not just getting every machine, but every application on every machine


Devs will just hard cord DoH servers into their products so that you can't block their tracking and ads.


I think you left the link out.




Yes they are but you'd have to run your own DoH server or follow the instructions to disable DoH like an ISP would (the possibility of which makes one wonder what the whole point is.)


Or just use one of the many public DoH servers that already do this.


The won't block my specific list of websites (reddit.com for example.)


Don't NextDNS and OpenDNS both let you do that?


Even if they do, you're now sending all of your DNS queries to some external service. That's a pretty serious privacy violation, even worse than 8.8.8.8.


Isn't this what basically everyone does anyway pre-DoH? Most people don't run their own recursive resolvers.


No, pre-DoH people are sending their queries to their ISP, not to a third party.


Do you not count your ISP's DNS server as "some external service" Where did you get "third party" from?


Still possible. The control moves from your home network, corporate network or local ISP to the default DoH resolvers defined in the browser. This can be changed, but we know most people outside of a corporation will not change this. So in this case, CIRA will be logging DNS requests and sink-holing undesirable domains presumably by Canadian standards.

People can still use browser add-ons to block domains, URL's, objects on sites, etc... uBlock is my favorite for this.


They aren't possible for the network to unilaterally impose on unwilling users. They're still possible if the user actually wants it, by just setting their client's DoH server to one that does them.


A network suggests a DNS server

You can choose to use that server or not

If a network intercepts your DNS traffic, you can encrypt it yourself. If you don't trust the network you should be encrypting everything anyway


> You can choose to use that server or not

If you're not using DoH, a malicious network will just redirect your requests to legitimate DNS servers to instead go to its own.


Can you expand on this a little more? I do not understand the reasoning here


The tools to administer a home network, such as DNS blocking and DNS sinkholes, are the same tools nations use to administer their populaces. If the default in Firefox is to enable blocking and sinkholes for the home administrator's benefit, it will similarly be to the benefit of oppressive administrations.

Home administrators who control their endpoints can turn DoH off with relative ease. By comparison, most governments cannot.


It would be nice if it would check the resolver config files (like the hosts file) with it enabled though.

That was a really easy way to block harmful stuff or name things without requiring DNS. Having to choose between encryption and easy configuration is pretty poor UX.


Naively, that seems like something that could be a very useful configuration option to set! Though I can also see why you might default to having it off if you don't trust the host resolver very much in the country-administering-people case.


You have to trust the local OS config. No amount of clever application programming will ever let you get around that.


The vast majority of people don't run their own DNS server, or have any control over the DNS server they use.

Most people use their ISP's or organization's DNS server, and have no control over it; instead, it's something used against them. In many cases, that DNS server may do some combination of tracking, redirection to ads, or other things that make it undesirable to use.

DoH eliminates one of the last major unencrypted protocols on the Internet, and instead uses an encrypted protocol to talk exclusively to the intended server without the possibility of interception.

If you want to talk to your ISP's DNS server, you're free to do so. But that should be your choice, not your ISP's.


DoH makes widespread censorship and surveillance via intercepting plaintext DNS queries much more difficult.

Of course, this is just one weapon in the arms race. It doesn't completely protect you, but every bit helps.


I think it's an extremely inelegant and overkill solution to a real problem. So, the web in a nutshell.


> DoH. It seems like there a ton of unnecessary overhead added

This. Why the hell everyone ignores DoT, much simpler one? Yes, China, censorship, but Canada? And I’m still unsure if there are http cookies in DoH, or not, or possibly maybe…


> Are things like DNS sinkholes and domain blacklisting still possible with DoH?

Surely they are... but the guy might not be your ISP but Cloudflare.

EDIT: In Canada that would be that so-called CIRA (by default). But you always have control who can pollute your DNS, right?


Oh, as an addition, the one that really protects you from fake DNS records is DNSSEC (given you have trust on ICANN or so).


Nice to see they'll be making it easy to opt out. I use a Pihole on my network and this would be an enormous PITA if it was hard (or worse, impossible) to disable...

Edit: I forgot, as another commenter pointed out, they also have a canary domain (use-application-dns.net) that you can blackhole to disable this at the network level (unless the user manually enables DoH).


PSA for network admins: Note that use-application-dns.net must return NXDOMAIN, not (successfully) resolve to 0.0.0.0.

I see plenty of people just adding it to their hosts file blacklists / DNS overrides like they do for other ad and malware domains, such that it resolves to a bad IP like 0.0.0.0. That's not what you should be doing in this case.


You can also just set the policy to disable this in registry or using policy.json on Linux. Mac has a similar setup, i'm just not familiar with it.


Yeah, I ended up going the canary domain route, and then setting up my pihole itself to use DoH using cloudflared outbound. Best of both worlds!


Yes, canary domain is fine - at least while it still works. I don't want to have to configure DNS for 50 different applications on 20 different machines. My OS does DNS, and my DHCP server doles out the DNS server I want my machines to use.


The Mozilla page on the topic (https://support.mozilla.org/en-US/kb/canary-domain-use-appli...) notes that:

> The use of this domain is specified by Mozilla, as a limited-time measure until a method for signaling the presence of DNS-based content filtering is defined and adopted by an Internet standards body.

Presumably that'd be the IETF, and I have no idea which committee would be working on this, if there's a draft RFC in the works, etc. But it's clear the expectation is some sort of standard mechanism will eventually replace the canary domain that yields the same functionality but in a less hacky way.


There was [1] which would've allowed websites to specify custom DoH resolvers for their subdomains, but it expired some time ago. The same authors are now working on [2] instead, which AFAICT is basically for the same thing.

There's also [3] which is a way for the DHCP server to also provide DoT and DoH servers to the LAN (in addition to the usual DNS servers).

[1]: https://datatracker.ietf.org/doc/draft-pauly-add-resolver-di...

[2]: https://datatracker.ietf.org/doc/draft-ietf-add-ddr/

[3]: https://datatracker.ietf.org/doc/draft-ietf-add-dnr/


As a sysadmin for an organization with our own internal DNS servers that clients need to hit, how will this impact us? Do I need to deploy a GPO to disable DoH in order to force FF to keep using our DNS servers?


There's a windows GPO key[0] and a canary domain[1] you can NXDOMAIN (use-application-dns.net). Both of these only control the default (it won't override a user personally choosing the setting).

[0]: https://github.com/mozilla/policy-templates/blob/master/READ...

[1]: https://support.mozilla.org/en-US/kb/canary-domain-use-appli...


Ah, so users can choose to override the local network administration? I didn't realise that. No way to prevent my family browsing to undesirable and unsafe sites without personally admin-ing each device and/or directly monitoring use?

I realise this is currently all leaky, my attitude was to put controls in place (DNS based) and if they learnt to hack around it then they're probably old enough to handle what they find ... doesn't work for malware, etc., however.

Presumably enabling DoH makes malware, advertising, etc., impossible to block?

When is MS Windows going to use DoH to force access for monitoring, currently I block a few domains.


> When is MS Windows going to use DoH to force access for monitoring, currently I block a few domains.

Microsoft doesn’t need DoH to circumvent DNS-based blocking. They control the OS and can make it do anything.


Have you seen this, MS running network traffic tunneled secretly over other people's networks?

Seems like it would be a CMA/CFAA infringement?


Microsoft has over 125 billion cash on hand[0]. I think they could spare some Azure compute to simply VPN all Windows DNS requests if they wanted (regardless of rfc8484).

0: https://www.microsoft.com/en-us/Investor/earnings/FY-21-Q3/p... (ctrl f "total cash")


I’m not saying they do. They wouldn’t even need to - they control the resolver. They could hardcode IP addresses, use domain fronting, use a CDN to proxy, load a list of IP addresses on every update check, … There are so many ways to circumvent DNS based blocking for someone that controls the resolver and network stack that it’s not even funny any more.

The fact that no one has shown that they do any of this tells you that they’re not trying very hard to circumvent blocks. Why would that change with DoH?


> No way to prevent my family browsing to undesirable and unsafe sites without personally admin-ing each device

No there's not, for good reason. From a technical perspective, this is indistinguishable from an abusive spouse or oppressive government trying to block sites.


That argument is pretty lame, buying a knife to chop carrots is indistinguishable from buying a knife to murder people.

Oppressive governments just block the providers of DoH that won't fold to them, meanwhile I lose the ability to control traffic on my own network.

I don't think DoH is going to make any difference to spousal abuse.


It's making the comparison to illustrate that the OS/User has no way to differentiate between a trusted router doing the blocking and the ISP blocking certain domains/injecting ads.

If you want to block domains on your devices, doing it on your devices themselves will be more reliable and implicitly requires you actually own & have admin rights on the devices you want to block domains on.


> Ah, so users can choose to override the local network administration? ... Presumably enabling DoH makes malware, advertising, etc., impossible to block?

It was always trivially possible to bypass DNS-based access control. Even before the establishment of DoH as a standard, you could have just served a text file containing the IP over HTTPS.


I hope by family you mean your children? Because otherwise this feature is very much designed to you and what you do.



Ah, thank you.


Do you use split-horizon DNS, or do all of your internal domains fail to resolve externally? If the former, then add your organization's domains to network.trr.excluded-domains. If the latter, everything will still work without you needing to do anything.


I recommend using Firefox's GPOs anyways: Not just to ensure users can't override disabling DoH, but also to turn off arbitrary extension install (a major malware vector) and other protocols designed to circumvent security inspection like QUIC.

If you operate any network security appliances of any kind, browsers are basically built-by-default now to circumvent them, so you need to be configuring them heavily via group policy.

Firefox, Chrome, and Edge all offer ADMX templates to add support for their policies pretty much drop-in to Active Directory.


You will want to employ DPI anyway, to make sure no other apps/devices on your network subvert your internal security. DoH is just another signature to detect and block.


It will automatically fall back to the internal DNS for names that don't resolve externally, so you might not need to do anything


This is how it was for me. Our internal names used the network dns and the real domains used DoH


It would be nice if Firefox supported some way to automatically rotate which backend provider it's using for DoH. This would keep any one entity from having a continuous feed of every DNS lookup.


Distributing DNS queries across several resolvers is a good idea to make sure there's no single resolver would be able to learn the entire browsing history of a user. However, this distribution algorithm should be designed with care because distributing queries in a random or round-robin manner would lead to even worse privacy since every resolver will eventually learn the entire browsing history of the user.

We have conducted some preliminary work on this idea and shared it with the team at Firefox, but I guess the current number of Trusted Recursive Resolvers is not large enough. However, I do believe that once Firefox have more than a dozen TRRs, (or if Firefox allows users to input more than just one customized DoH resolver), our proposed K-resolver resolution mechanism would help to increase the privacy benefit of DoH even further.

"K-resolver: Towards Decentralizing Encrypted DNS Resolution"

https://www.ndss-symposium.org/wp-content/uploads/2020/02/23...


That's an interesting idea. I wonder how you'd feel about this in relation to the trade-off against multiple resolver providers having information about your Internet traffic? By using a single provider, you limit your risk exposure, but also increase your reliance on that provider. So it's not clear-cut that this would necessarily be beneficial.


It's actually a very good point, and Tor is explicitely using a single entry guard because of that: it was analyzed that keeping a single guard for a long time is "on average" better than regularly changing guards.


That's a fair point. I think I would still be happier if no one entity had everything. But I do recognize the downside.


Rotating between a handful of providers could be a net negative for privacy:

- every FQDNs you resolve is leaked to at least one provider

- the FQDNs you resolve often (e.g. every day) are leaked to at all the backend providers over time

If you want to protect your privacy you need DNS resolution over Tor.


While I agree that rotating might be a net negative, that particular issue could be worked around: hash the FQDN and use the hash to determine which provider you use, so that a given user always resolves a given FQDN with the same provider. (Salt the hash, so that different users use different providers for that FQDN.)

It's still not clear if it's a win to rotate, but if you were going to rotate, that would help.


That's not rotation, by definition.

(and hardly beneficial for privacy: you just divided the FQDNs you leak across 3-4 providers)


The only advantage would be in reducing the ability to correlate a user's sites.

But nonetheless, I agree.


Great idea. Would a local load balancing proxy do the trick?


I use my PiHole to do balancing like this. I've set it up with DoT and DoH so that DNS requests are spread across several servers.

Then, I expose my PiHole as a DoT + DoH server (DoT for Linux and Android, DoH for Firefox) so that I can always use my ad block lists and secured DNS traffic.


Doesn't this just result in more organizations having your data? Presumably most DNS entries have some finite TTL, such as a few minutes / hours / days. Let's say one of your favorite websites has a TTL of 10 hours, and you visit this website once per day. Then eventually (soon!) you will have asked most or all of your servers for that domain, so now they all have your data. Or do you have your PiHole configured that it always asks the same server for the same domain?


You're right that this is the case. I set it up so that I can use the double encryption system with a DoH proxy when I eventually resolve this issue. In practice, 60% is my traffic ends up at cloudflare already, so I might as well use their DNS until then.

It's also an availability thing. There's that one archive website on a vendetta against cloudflare for not forwarding enough data in DNS requests (eDNS or something?) that I can access by using alternative servers as well. In the rare case of a cloudflare outage, this also ensures uptime.

DNS cannot be made fully anonymous, though that double encrypted DoH thing seems promising. Picking a few trusted providers isn't a problem for me personally.


You could share the namespaces using some consistent hashing so any given domain would hit the same server.

Maybe something exists for that already?


I don't understand.

I have a PiHole, and have the cloudflare DoH client installed on it.

Requests from my network go to the PiHole, and the PiHole is set to query localhost:5053, which then hits a DoH provider.

Why do you have separate setups for mobile vs. desktop web browser, why not have the OS do native DNS to your PiHole?


When I'm on my laptop, in the train, working on Windows, I can't use my home DNS without exposing it to the world and becoming part of every DDoS attack there is. So, for those situations, I've got Firefox set up to do DoH. Windows can do some DNS encryption through a complex setup of local DNS servers but it's nothing as easy as the standard Linux setup.

Similarly, I use DoT on my phone but if I want to connect to any hotspot, I have to disable DoT because the hotspot login detection fails with DoT configured. I think it's a Xiaomi thing. Regardless, I need to disable it to use. In one particular case (scanning groceries in the supermarket for the automated payout) I can't fall back to LTE like usual. Just in case I forget to turn DoT back on, I've also configured Firefox on my phone to use DoH because the mechanism is there and I might as well.

There's small edge cases where I just like to fall back to browser DoH. DoT and I'm network DNS still works fine, of course.


Not with a TCP proxy. To establish a HTTPS tunnel, the client needs to know the identity of the remote end for certificate validation. Randomizing remote ends with a naive proxy would lead to TLS errors.

So you'd likely configure your browser to do DoH destined for that proxy, and have the proxy generate DoH requests to the remote ends. The downside is you'd have to somewhat manually set up certificate trust between your client and your proxy - either by trusting the proxy's certificate, or installing a root CA that issues the proxy's certificate in your browser's trust store.


In the current designs, plaintext domain names are mainly exposed through two channels: the SNI extension in TLS, and traditional DNS name resolutions—the deployment of DoH/DoT is thus a prerequisite for ESNI. Equivalently, the use of DoH/DoT will not provide any meaningful privacy if domain names are still exposed through the (unencrypted) SNI extension in TLS handshake traffic.

Recently, there is a push for the deployment of DoH/DoT, with major organizations already supporting it (e.g., Google, Cloudflare, Firefox), though this has not been followed by an equivalent effort for the deployment of ESNI, which is now being reworked in an Internet draft called Encrypted Client Hello (ECH: https://www.ietf.org/archive/id/draft-ietf-tls-esni-12.txt). Unless the confidentiality of domain names is preserved on both channels (SNI and DNS), neither technology can provide any actual privacy benefit if deployed individually.

Also many websites hosted on IP addresses that do not host any other domains, the IP information can be used to infer which domain was visited. We have carried out some studies to investigate the extent to which domain name inference can be done through IP address information. For more technical details pls have a look at these papers:

"Assessing the Privacy Benefits of Domain Name Encryption" https://arxiv.org/pdf/1911.00563.pdf

"Domain name encryption is not enough: privacy leakage via IP-based website fingerprinting" https://petsymposium.org/2021/files/papers/issue4/popets-202...


Firefox has indeed been making efforts to enable ESNI/ECH alongside their efforts to enable DoH. As you mention, they are mostly only useful together.


yea, they have also disabled/removed ESNI since version 85 https://www.ghacks.net/2021/02/24/the-case-of-the-missing-es...

Hopefully ECH will make it this time


Chris Siebenmann has written extensively and early on what this means for organizations:

https://utcc.utoronto.ca/~cks/space/blog/web/FirefoxDNSOverH...


I wonder what's the attraction of split horizon dns especially at an academic institution. It seems it will obviously break stuff since all kinds of things rely on dns working as it was designed to, providing a global namespace.


I personally use split horizon DNS for my home lab. I have several things that I put behind Cloudflare, but I don't want my local access to go me > cloudflare > me, so I use split horizon DNS. If you're not behind something like Cloudflare you can use NAT reflection (hairpin NAT), but those are a bit of a hack.

Canada also has some large, private networks used by provincial governments that use split horizon DNS and DoH is going to be a huge pain for them. One specific example would be libraries that are on a private network and have access to resources handled differently depending on how you connect; via the private network = no auth, but via the internet = auth required.

Split horizon DNS is a simple, practical way of affecting how traffic is routed in a lot of cases.


> you can use NAT reflection (hairpin NAT), but those are a bit of a hack.

You think that hairpin NAT is a hack, but that split-horizon DNS isn't?


There are definitely large organizations who run split horizon DNS because back in the mists of time they made poor choices and assigned IP addresses that were in the public address namespace (ie not a 10. or 192.168.1. etc) that they didn't own randomly to a large number of legacy subnets within their network. It's now too painful to migrate them, so they are a wart that has to be lived with until they eventually retire.

In the meantime they have to use split dns so they are able to keep things kinda working.

I have personally worked at least a couple of large orgs that have this setup (ie 100s or 1000s of ancient devices on internal networks squatting on what are actually public IPs).


> Fortunately we already have a split-horizon DNS setup, because we long ago made the decision to have a private top level domain for all of our sandbox networks, so we use our existing DNS infrastructure to give BINAT'd machines different IP addresses in the internal and external views.

https://utcc.utoronto.ca/~cks/space/blog/sysadmin/BinatAndSp...


Thanks. It's interesting that after this account the characterisation "fortunately" is still used. But the motivations still give good perspective though i would have done things differently.


They force DoH in the name of freedom. What will be the DoH resolver ? Cloudfare ? Google ? One must use GP to disable this on Win or a DNS server on linux Everyday the reasons to keep firefox are converging to 0.


> They force TLS in the name of freedom. Who will be the certificate provider? VeriSign? Comodo? One must use certificate manager or their own certificate server on linux. Everyday the reasons to keep firefox are converging to 0.


> They force package managers in the name of freedom. Who will be the server? Canonical? RedHat? One must use wget, tar, and gcc on linux. Everyday the reasons to keep GNU/Linux are converging to 0.


How hard would it be to run a private DoH server? (This is a serious question, not a snarky comment)


Easy: https://dnsprivacy.org/wiki/display/DP/DNS+Privacy+Clients#D...

I want to add, that you can't run your own public DNS server nowadays, because it can be misused for DOS and hosting providers scan for open DNS servers.

So if you want your own public DNS resolver, DOH is much easier.


> because it can be misused for DOS and hosting providers scan for open DNS servers

Can you elaborate? I am thinking of doing exactly that, in case of need.



Very, very interesting… I forgot the fact that ip addresses can be spoofed in the UDP packets, essentially making DNS resolvers to carry out the attack for botnets.


DNS servers can be used in an DOS amplification attack by sending requests with spoofed ip addresses. So if you don’t take measures to prevent this it’s likely your server will be used in DOS attacks.

https://us-cert.cisa.gov/ncas/alerts/TA13-088A


One reason is that DNS is UDP based, on which source IP spoofing is possible.


You could try to run an IPv6 only DNS server, harder to find.


But it's not possible to configure via DHCP/network level autoconfiguration, is it?


No and it probably shouldn’t be going forward, by default the local network administrator is an untrusted assumed-malicious entity.

Can you imagine if your network administrator could just tell your computer to send all your unencrypted HTTP traffic to a proxy they control?


I run it on my pihole with Dnscryptproxy.

https://github.com/DNSCrypt/dnscrypt-proxy/wiki/Local-DoH

It isn't too hard.


DoH is configurable with GP or just a policies.json file if you want: https://github.com/mozilla/policy-templates/blob/master/READ...


GP is a bit obscure, I was hoping that we can simply use Firefox preferences to point at another provider.


You can set custom provider in the settings UI: https://support.mozilla.org/en-US/kb/firefox-dns-over-https#...


You can: Settings > General > Network Settings > Settings... > Enable DNS over HTTPS > Use Provider > Custom


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: