Hacker News new | past | comments | ask | show | jobs | submit login
DoH: Anti-Competitive and Network Neutrality Aspects (powerdns.com)
57 points by stock_toaster 50 days ago | hide | past | web | favorite | 50 comments

The EU might be doing a better job here-- but is this suggesting that I trust my US ISP's DNS servers? The ISP that injects HTML into my HTTP requests? The same ISP known to hijack DNS requests?

Don't get me wrong, the issues raised by OP are valid and need to be addressed-- already we see the strangely privacy-antithetical behavior of using DoH to get around CNAME tracking in adblockers (using Cloudflare no less) --but I don't think legislation and regulation have been effective so far. Expecting that to change in the near future seems like a pipe dream.

> The EU might be doing a better job here-- but is this suggesting that I trust my US ISP's DNS servers?

That's the false dichotomy. You shouldn't use either one.

The issue is that if you've already made your choice, e.g. by having your DHCP server give out the DNS you actually want to use or configuring it in your operating system, then having Mozilla default to ignoring your choice and overwriting it with Cloudflare is wrong.

And it would be one thing if it was something like people selling internet gateway devices that do this by default -- that's a great idea because it gets you away from the ISP's DNS by default but still provides a single place to set it the way you want it, and still allows the owner of an individual device to override it device-wide if desired.

Putting it in individual applications is the opposite. It means you have a full time job just changing it back to the way you want it in every application on every device, and even then you're liable to miss some.

> It means you have a full time job just changing it back to the way you want it in every application on every device, and even then you're liable to miss some.

Hopefully not. Firefox has some techniques for disabling DoH that rely on computer or network-wide changes. If other applications also adopt these or similar techniques, you'd only have to make the change in one place to turn it off for every application running on any computer on a network.

Specifically, as I understand it, Firefox disables DoH if

1) Windows Group Policy is used to manage a computer

2) A canary domain is blocked


(Disclosure: I work for Mozilla, but not on the Firefox browser)

> 1) Windows Group Policy is used to manage a computer

This is solving the problem only where it least matters. If you're using Group Policy then you can configure DNS using Group Policy anyway, so any default there is already easy to override. Where the default really matters is on the devices where the only central management is DHCP, because those are the devices that require manual interaction to fix an application that defaults to ignoring the DHCP configuration.

> 2) A canary domain is blocked

That could work, but what happens when the ISPs start blocking the canary domain? The FAQ says they'll monitor it but that doesn't tell you how they'll address it when it happens.

> but what happens when the ISPs start blocking the canary domain?

Put your own DNS server in front of it that resolves the canary domain, I guess.

The absurdity of rolling a DNS server solely to unblock a domain blocked by an upstream DNS server specifically so that an application can resolve that domain and consequently not actually use your DNS server is not lost on me.

Technically correct -- the best kind of correct.

Though obviously if you've got your own DNS server in front of the ISP's then you could just configure that to resolve names however you like. Which would then typically cause it to accurately resolve the canary domain by default and thereby cause the application to not use it. Hmm.

> Note that some countries have an underdeveloped ISP market, with large fractions of the population having no choice of broadband service provider. Regulation is then of the utmost importance to keep everyone honest, but in some of these countries the regulator has been captured by industry and is no longer very effective. This mostly goes for the US.

I think this is the key motivator for a move to DoH. Large US ISPs can’t be trusted.

> Large US ISPs can’t be trusted.

It's certainly unfortunate that we've gotten to this point. The net result is that nobody can be trusted: Not ISPs, not Cloudflare. There is a clear conflict of interest in Cloudflare's strategy, and their "aww shucks, we're just being the good guys" attitude is dishonest. At this point, one or many not-for-profit DoH providers (with easy selection of service in your browser or OS) would be the only true solution to this issue.

So don't trust Cloud Flare. DoH in no way depends on them. You can simply run your own DoH resolver.

>Large US ISPs can’t be trusted.

But why trust Cloudflare?

A clear privacy policy and an third-party audit.

All ISPs have privacy policies. 99% of all privacy policies are either copy/pasted from elsewhere or written in legalese so that company can do whatever they want. 'Internal business use' is a nice fodder for 'Selling data to increase this general ledger account for later business use.'

An audit is almost always under bad pretenses. If I spend $$$ on an audit company, the company itself has bias.

The difference between third-party audits and government issued audits (think food safety) is that the government has nothing to gain by a positive, or negative result.

IANAL, just cynical of people.

ISPs have those too (ISO 27001 is common). Again, why trust Cloudflare?

ISO 27001 is a standard for how to do policy. So from ISO 27001's point of view this is fine:

"Our policy is we'll do anything to make a buck".

The ISO standard sets out a correct way to develop this policy, write it down, ensure employees know about it, measure whether they're implementing it, and then for auditors to reassure management that all this is being done as described. But it doesn't say there's anything wrong with that policy I mentioned, beyond that maybe your ISO 27001 consultants will struggle to charge their usual fees for such a short and on-the-nose policy document.

It also matters who the auditors work for, and who they report to. Have you /read/ the audit reports for your ISP? No? Because they're confidential, only the ISP sees them. So, what use are they to you? For all you know the auditors found that the ISP doesn't comply with its own policies and shows no interest in doing so.

If you later find out the ISP isn't complying but the audit reports said everything was fine, you'll never know about that, and you can't fire the audit firm and get a better one, all of this is totally opaque to you.

In contrast Mozilla gets to insist upon reading the audit reports for the policy they agreed with Cloudflare and can insist upon a different audit firm if it decides the auditors aren't up to scratch. This has happened with CA root trust, the Hong Kong auditors for WoSign were disqualified in this way and the franchise owner's office in London informed of the problem.

Now, maybe you don't trust Mozilla either, but if you're running their web browser you're in a real pickle if you don't trust them. And if you don't run their browser who cares which TRR is configured in the browser you don't run?

I mentioned ISO 27001 because you mentioned audits, that it doesn't hold to much is clear.

Where can I read the Cloudflare audit that Mozilla got? If I can't then there is really no difference between them and the audit from an ISP.

CA audits are somewhat different, so I don't know why they should matter here.

That still doesn't answer the question why Cloudflare should be more trustworthy then an ISP.

Provided in return for a competitive advantage and a reduction in network neutrality.

I don't seem to understand the whole thing. To me the premises are:

1) we can't trust ISP with unencrypted DNS query

2) we can trust a third party with DNS query because those are encrypted.

My questions are:

- Why is 1 true? What are ISP doing with unencrypted DNS queries apart from knowing browsing habits?

- With DoH do they not know your browsing habit anymore?

- If there is a possible mitm attack with unencrypted, would it be fine if the ISP themselves ran a DoH server?

- For the 3rd party: why would a 3rd party want to run a dns server for free? ISP run DNS server but customers pay a bill to them precisely to do things like this.

in other words, in the long term, wouldn't it be a money sink for Cloudflare to run a DoH server?

- Instead of just one DoH endpoint, could browsers take a list of DoH servers, and picking the best one in terms of p95 latency, with some round-robbin tries occasionally to verify if anything changes?

This would solve the issue of one of the DoH service "having a bad day". Some amount of centralization would remain but based on an objective criteria (latency).

(The article answers some of those questions and/or points out the same concerns)

Why can't I, as an end-user, have the choice of who I want to trust?

My ISP, Cloudflare, Google, somebody else? Give me a dialog box in which I can choose whether to use DoH and if so which provider to use, and then I can make up my own mind over which of those parties I trust the most.

dnscrypt-proxy can now provide a local DoH server that Firefox can be configured to use. So that you can keep using whatever resolvers you want and have your filters, logging, etc. working as before.

I would argue that the original way to do this--where applications used the system resolver--gave you that option, as the system resolver could trivially be modified to do whatever you wanted (well, unless you are on iOS; on iOS you are effectively just screwed, because Apple wants to control the entire software experience... more on this in a later paragraph <- edit: and also, I realized you can also do this on iOS!). glibc, for example, allows you to just give it arbitrary software plugins to implement the resolver, and so you--the person who configures your computer--have complete control over not just "who" you want to trust, but exactly "how" you trust them: you can decide to talk to Google, Cloudflare, or your ISP, and you can use DNS over HTTPS, DNS over TLS, unencrypted DNS, or something insane like Active Directory for all anyone cared.

However, at some point, web browsers decided that they didn't like end users being able to have this control, as, in practice, no one ever took advantage of it and they always allowed the operating system to select, by default, "use unencrypted DNS to the current ISP", which was slow, insecure, and often just wrong (as ISPs--for at least four reasons I can think of, all but maybe one of which I will assert are "bad"--would sometimes return incorrect answers on purpose); and so, now, the new reality is that software is coming with embedded DNS resolvers that do exactly and only what the software developer wanted (which, in a world of not just codesigning but notarization, increasingly means the user has no ability to have any say in the matter what-so-ever), because "developers know better than everyone else" (originally, this was just for performance reasons or to avoid strange behavior from ISPs, reimplementing the logic, but not the protocol, though now they are using that control to take it further).

For clarity, I will restate "the premise": if software uses the system resolver (the one that people erroneously like to claim is somehow synonymous with "unencrypted DNS to your ISP"), anyone who wants to use DNS over HTTPS can, AFAIK on any non-iOS operating system (if nothing else, by simply setting their DNS server to, if for some reason you can't just plug in your own resolver backend; edit: oh, actually, even on iOS, by using a local Network Extension, which is how Cloudflare's app works), do that for not just their web browser but for all software on their entire system at once; and, when some new protocol comes out that is even better than DNS over HTTPS (maybe by solving some latency issue, or by using a decentralized resolver, or adding cryptographic verification), it would be equally easy to change your system resolver, and by extension all software on your entire computer, at once, to use that protocol.

And now, again, for clarity, I will restate "what happened": that this was deemed not something that either users or, interestingly, operating systems should be in control over (though it would be hilarious to see Apple try to fight this one out from their fortress of iOS, claiming you don't get to be in the App Store unless you use the system resolver, a similar battle to their issues with people doing HTTPS requests that don't use their App Transport Security library); and so, instead, software (possibly ever single piece of software you use: not just your web browser, but random desktop chat applications, e-mail clients, games... maybe even network-oriented command line tools such as curl!) will now embed their own DNS implementations, making it "your guess is as good as mine" as to what protocol or root of trust any specific piece of software is relying on :/.

Discovered this Fun fact last night; the systemd resolver since V242, used cloudflare as a fallback DNS. [0] The fallback has been Google and quad9.

In theory (haven't tested it,) this means that even if I give 0 DNS servers on a DHCP request, and I ignore any DNS requests to the gateway (if I wanted a machine totally unable to resolve) I can't do that because a distro choice to use the systemd resolver.

[0] https://github.com/systemd/systemd/blob/master/NEWS#L751

This is going to create a lot of noise (firewall drops) in networks where Linux is expected to be isolated. Many companies do this and many of those companies will start seeing alerts in their security operations centers. I am curious how most of them will respond.

It's trivial to disable the systemd resolver or change the fallback DNS servers to something other than Cloudflare.

Agreed, but people tend to not customize settings proactively. I usually see reactive changes. More often than not, people don't even know these changes to the base OS are occurring.

I get why you'd want to avoid Google, but why Cloudflare over Quad9?

While I disagree with Mozilla's decision to switch users to centralised DoH, I believe they are right in their assesment that the current DNS infrastructure is not very private.

And something like DoH sounds pretty cool.

Without encryption there are a few nefarious things ISPs can do quite easily:

- transparent DNS proxies

- DNS query tracking

These things are pretty bad if you're affected, but I believe this shouldn't be addressed by centralisation of DNS.

I think Dns-over-TLS (DoT) is a much better solution. It seems super weird to me that Mozilla isn't pushing for that, and is instead pushing for DoH+cloudflare.

> We have to keep in mind that if a DNS lookup is slow, the entire internet feels sluggish. Slow DNS = Slow internet.

Ridiculously low DNS TTL is an idiotic curse of the internet, but fortunately (and currently) one that can be (mostly) fixed locally.

This is a disinformation campaign. DoH doesn't centralize DNS or DNS privacy. Anyone can run a DoH resolver server, and it'll do just as good a job as Cloud Flare's.

The people crusading against DoH favor a different centralized DNS standard, DoT. DoT is DoH with essentially one difference, which is that it runs on its own port, so that network operators can block it. There's a sprawling cottage industry of security and analytics products that rely on passive DNS analysis to function, and people who sell those products or operate networks that use them have a strong disincentive to block DoT and require users to speak standard, plaintext DNS to servers they can monitor.

Note however that once it's up and running, DoT is just as centralized as DoH is; it has to be, because it provides the exact same service model. Centralization and competition has nothing to do with their real concerns, or they'd be lobbying against DoT as well.

I can't say I'm especially fond of Firefox's decision to default to Cloud Flare for its DoH resolver, because I loathe Cloud Flare. Google Chrome does something more sensible: it won't change your DNS server at all, but will upgrade you to DoH if your nameservers support DoH. You should use nameservers that do, because DoH is valuable: if you're in the US, your ISP is almost certainly passively monitoring and recording your DNS queries, and you shouldn't let them. Personal feelings about Cloud Flare aside, you are better off with them than with your US ISP's DNS servers.

There's a strain of Unix sysadmin criticism against DoH that says it's wrong for browsers to co-opt DNS resolution at all, and that they should use the system's resolver. This is nonsense. First, almost no matter which browser you use, you should trust their DNS resolution code more than your operating system's; it will be more security-conscious and probably more performance-conscious code --- DNS resolution maintenance is the job you get on Microsoft or Apple's OS teams to punish you when you break the build. More importantly, as anyone who's ever had to mess around with libares will tell you, browsers have always needed better DNS resolution than libc offers, because they need fast async lookups.

There's another strain of criticism against DoH that Unix nerds won't see much of but which is probably even more important to be aware of: there have been hearings in Congress about how DoH threatens law enforcement by depriving them of intelligence for investigations. Ultimately, this is of a kind with the DoT vs. DoH argument: the people lobbying against DoH simply don't believe DNS lookups should be private. Another "tell" that you're reading one of these people is when they start talking about how DNS privacy doesn't provide complete or perfect privacy --- that you shouldn't care if your DNS lookups are being monitored, because you're leaking so much information elsewhere. This is bamboozlement. DoH privacy is extraordinarily low-effort, actually modernizes the DNS protocol in some ways, and addresses a threat millions of users are directly facing right now.

You should be wary of any outlet that promotes the meme that DoH is somehow shady.

> crusading against DoH favor a different centralized DNS standard, DoT

I think that there are 2 camps against DoH. 1. Default centralization to specific points. EG firefox using cloudflare first and foremost, nobody else. 2. DoH adding complexity. DoT was (practically) superseeded with DoH anyway, if for nothing more than adoption.

> criticism against DoH that says it's wrong for browsers to co-opt DNS resolution at all, and that they should use the system's resolver. This is nonsense.

So... your argument is that I can't trust the OS to 'do the right thing' but I should trust the browser, because they know best? If you can't trust the OS, then how could I possibly trust a browser running on the said, untrusted OS?

Honestly asking how you came to that conclusion because the train of trust is broken on the OS level, so anything above is moot.

> DoH simply don't believe DNS lookups should be private

Problem with DoH is that it is private up to the endpoint. Nobody can listen on the request, but nobody is preventing the endpoint from telling everyone else that 'Joe Smith visited Youtube at {timestamp} Once the endpoint has the request, they have your info and can just as easily sell that to telcos.

End-to-End encryption and privacy is only as good as the people on the other side. Can't trust Alice with your message, then don't send it.

> modernizes the DNS protocol

If by modernize you mean convolutes. If I have a resolver on my network serving qwer.localnet. Browser asks cloudflare, returns nxdomain, then my system resolver asks for it; that is far more latency and is far from the modern proper solution for speed and privacy. Cloudflare now knows that I have a local domain, qwer.localnet.

With respect to your OS versus your browser, you're using "trust" in a different way than I am. Obviously, you have to "trust" your OS in a strict engineering sense; if the kernel is compromised, nothing the browser can do will meaningfully mitigate that. That's not what I'm talking about. I'm saying I "trust" my browser developer to care more about DNS security than I "trust" my OS vendor, just like I "trust" Chrome's X.509 handling more than I "trust" the certificate validation code that ships on the OS. It's not that I think the OS developers are malicious; quite the opposite. I just believe, with some evidence, that they're not incentivized to adopt modern security mechanisms for those features.

My operating system ships with IPSEC VPN support. I'm not going to use it, even though I think highly of the poor souls tasked with maintaining it for Apple; I use WireGuard instead, because I trust Jason and, more importantly, Jason's incentives, more than I trust Apple. I certainly don't feel like I need permission from my OS (in the form of them formally adopting WireGuard) to do so.

With respect to endpoint security – I assume really what you mean is the security of the resolver server that you use, which can indeed monitor your DNS requests – yes! You do have to trust the DoH server with your requests. You should choose one that you do trust. But because your US ISP is already violating that trust flagrantly, you're almost better off with any off-network DoH server you can find. Really, if you're a stickler, you'll just run your own DoH server. You can't do worse than the status quo ante.

I don't care about the "convolution" argument.

> There's a strain of Unix sysadmin criticism against DoH that says it's wrong for browsers to co-opt DNS resolution at all, and that they should use the system's resolver.

It's perfectly reasonable. I don't want each app maintaining its own config and having to gain in-depth understand of its custom solution when using another network. Just had that fun with gradle proxy settings in a global configuration file making downloads appear forever with no visible error message. Could have spend that quite a bit of time a lot more productive / enjoyable.

These kind of standards exist for a reason. The problem is Mozilla not having a way to distinguish an unaware user using the good old way from someone specifically preferring it. OS vendors could help, but relying on them doesn't seem like a good plan either.

It is not perfectly reasonable. Different applications have different requirements, and the applications running on my OS should not all be confined to the lowest common denominator of what the developer who broke the build and is stuck maintaining the DNS code decides to implement.

When the operating system provides performant DoH lookups for all applications, then sure, lobby your browser to use that code instead of their own.

of what the developer who broke the build

This is vicious slander. Would the developer who merely broke the build get to write an entirely new system resolver implementation in C++ from scratch, get it into a major OS release and thus break stuff for zillions of users until the whole thing has to be rolled back, never to be seen again? I think not.


The article claims that there is some push towards centralized DNS. I do not see the evidence to support this - the only reason Cloudflare is the default in Firefox is that DoH implementations are still rare. PowerDNS is free to implement DoH themselves if they want their customers to be competitive, and so is every ISP.

CF was chosen because they signed a contract with Mozilla to reduce the amount of data they collect to the bare minimum.

Can (US) National Security Letters override such contracts?

One nice thing about 'plain' DNS(-over-54; Do53) is that it is distributed in nature, so to get any kind of massive surveillance you have to visit multiple ISPs.

> Can (US) National Security Letters override such contracts?

Yes, and normal, everyday courts can compel companies to siphon their customers' data and keep it secret that they're doing so, despite any claims that the companies make in their privacy policies.

That's not in most people's threat model. If it is in your threat model, you can change the DoH provider to any such service you want. CF with the additional privacy policy is an excellent default choice.

The point presented by the post is plain wrong, people were using non-ISP recursors for a long time already and this was not problem due to the fact that EDNS Client Subnet has been a thing for a while now.

The problem mentioned in the post is specific to Cloudflare, as they do not support EDNS Client Subnet anyway, even over plain UDP53

more reading: https://www.samknows.com/blog/dns-over-https-performance

Mozilla did their own testing of DoH and its effect on performance in conjunction with Akamai:

Test announcement: https://blog.mozilla.org/futurereleases/2018/11/27/next-step... "To do that, we’re working with Akamai to help us understand more about the performance impact."

First results: https://blog.mozilla.org/futurereleases/2019/04/02/dns-over-... "The results were strong! Like our previous studies, DoH had minimal impact or clearly improved the total time it takes to get a response from the resolver and fetch a web page."

Final post: https://blog.mozilla.org/futurereleases/2019/07/31/dns-over-... "Furthermore, we found that web pages that are hosted by Akamai–a content distribution network, or “CDN”–have similar performance when DoH is enabled."

And nowhere is "TCP" mentioned. I suppose the connection just builds up itself in 0ms time.

Sure if you already have <5ms latency to your DNS, the additional RTT from TCP and TLS won't matter much. For the rest: they will.

That's valid, but it's also why HTTP/3 is being pushed for 1-RTT and 0-RTT connections (0-RTT is most likely going to be enabled for their DoH if it isn't already)


And for that you need TLS tickets/cookies, don't you? (So you can resume your crypto state.) Isn't that a good way to track users?

Instead of a query coming from an IP (probably NATed with IPv4), you can identify individual browsers AFAICT.

The DoH server gets to track users if they use a resumption method, yes. As I understand it Firefox puts extra work in to ensure that it doesn't share the backend cache between your HTTPS service and the DNS resolver so there's no risk that if the DNS service and the HTTPS site you're visiting are owned by the same people they can correlate that, but they would be able to see that it's the same browser asking for say, camelporn.example and visit-arabia.example

If you disabled resumption you could get rid of this issue at a cost of one extra round trip per DNS lookup.

Nobody else gets to track users this way (at least in TLS 1.3 and I'd assume DoH implementers will use TLS 1.3 since they are of similar vintage) because the keys aren't re-used so the identifier for the key will just appear to be random noise each time.

That is resumption. So you still have the overhead for the first connection. Assuming that you don't DNS query every few seconds you are gonna suffer.

On a side note, last I checked Quad9 and Cloudflare (need to recheck that) almost immediateley close the connection once you finished your query. Tested with DoT but the same TLS handshake is required for that.

I think you're confused. You're correct that the first connection is 1-RTT but the resumption is a new connection and is 0-RTT using a resumption PSK.

Let's say your browser was left open when you went to work, you get home, click the bookmark for your preferred porn site, and now the browser needs to resolve the FQDN for that site. How fast is that?

So if you have TCP fast open and 0-RTT TLS 1.3 that's exactly one round trip for your answer, just like traditional unencrypted DNS.

The DNS query gets wrapped as an HTTP request, which gets wrapped as a TLS plaintext, which gets sent as TLS 1.3 early data, which all goes in the Fast Open TCP packet.

The server checks the Fast Open cookie, opens the TCP connection, matches the PSK, unwraps the early data, unwraps the HTTP request, unwraps the DNS query

The server finishes the TLS handshake on the now open TCP connection, and using the now encrypted channel it sends back the HTTP response, which wraps a DNS answer and that goes back as the first TCP packet the other way.

Same number of round trips and packets as for DNS. It's very ugly and QUIC in particular will tidy this up considerably, but it's secure and fast.

But the original post makes not mention of anything other than non-optimal answers.

Before comparing various methods of contacting a recursor you have to decide what your threat model is; does it include your ISP snooping on you? If so then an encrypted transport of your dns queries is mandatory. If it doesn’t, stick with DNS53.

DoH can use UDP, but most dont, Also most people's local DNS (Here in America) Provided by their ISP is pretty shit. In my experience working at a MSP, having a local DNS server is always best

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact