
Passive observations of a large DNS service - xuande
https://blog.apnic.net/2018/08/14/passive-observations-of-a-large-dns-service/
======
stingraycharles
"Due to the centralization that is caused by public DNS services, large
content delivery networks (CDNs), such as Akamai, are no longer able to rely
on the source IP of DNS queries to pinpoint their customers. Therefore, they
are also no longer able to provide geobased redirection appropriate for that
IP."

I always thought that the go-to way to solve geobased redirection was to make
all your PoPs listen to the same IP(s) (I think some BGP is involved here),
and let Dijkstra's algorithm let all clients find the shortest path to the
nearest PoP? Isn't this exactly what Google and CloudFlare are doing with
their own DNS IPs?

~~~
lathiat
That technique is called "Anycast", it's mentioned at the very end of the
linked article.

The problem with Anycast is it works pretty good for UDP (such as DNS) but
doing Anycast with your actual TCP connection (particularly long lived ones)
can be problematic because of shifting routes, multipath, etc.. meaning that
sometimes your traffic will change data-center mid-flow or possibly even with
alternating packets in the case of multipath somewhere etc. So suddenly your
HTTP connection data is arriving at the wrong data center and the connection
breaks.

What most providers currently do (as far as I understand anyway, I'm sure
there is a lot of hyper optimization) is based around using Anycast to receive
the DNS lookup, and use information from that DNS lookup to send back a
relevant POP locations IP.

That's not always based purely on which DC received the DNS request though,
that's the most naive implementation, but most of the big CDNs optimise the
returned server based on the source IP of the lookup among many other factors.

Hence why it matters for public DNS, since they are the source of the DNS
request.

This article however also discussed that Anycast doesn't always route traffic
where you would "expect", for various reasons that extend from commercial
agreements and cost all the way to bad configuration and "no idea" you may not
get routed to the nearest DC.

There is also a DNS extension, called "EDNS0" (also discussed in the article)
that sends the /24 of the IPv4 IP along with the query, to allow the DNS
resolver to customize their response particularly for these centralized DNS
services (e.g. Google DNS supports this but only if you apply to receive them
as an authoritative DNS server) - however APNIC/CloudFlare explicitly chose
not to use this I believe supposedly for privacy reasons with 1.1.1.1.

~~~
markonen
The problem of running TCP over Anycast addressing tends to be overblown.
Cloudflare, for example, does _all_ of their client-facing HTTP traffic over
Anycast and so you can consider that as existence proof for it being a viable
strategy. Routes simply don't seem to shift all that frequently in practice.

~~~
akvadrako
I used to operate an anycast DNS service and we monitored it from probes all
over the world. Sometimes a random ISP in USA/Europe would start sending
queries to Japan and that could last for seconds to hours. I'm not sure how
often it happens, but it will certainly be enough that it will cause broken
connections for a few users.

I'm not sure how Cloudflare deals with it - potentially they could share the
TCP connection data between datacenters when needed.

Or maybe it has something to do with their how edgy their setup is - if it's
mostly intra-ISP cache servers being anycasted, the routing should be more
stable.

~~~
Artemis2
AFAIK a lot of Google’s HTTPS traffic is also routed using anycast. They also
offer this to GCP customers under GCLB ([https://cloud.google.com/load-
balancing/](https://cloud.google.com/load-balancing/)).

I don’t know how they deal with changes in routes.

------
kijin
Are out-of-country DNS queries necessarily bad? If I were in upstate New York,
for example, I would much prefer to use a resolver in Toronto than one in San
Francisco. Measuring actual distance or network latency would be a much better
way to detect suboptimal geolocation than simply checking whether the server
is in the same country as the user.

~~~
lossolo
> Measuring actual distance or network latency would be a much better way to
> detect suboptimal geolocation

And how would you measure that in case of DNS servers? What you mean by
_distance_ ? Amount of hops? Physical distance? One hop can have the same
latency as route with 3 hops. Physically closer POP can have higher latency
than one that is further away. And we didn't even touch the throughput.

It's not so easy as it sounds.

~~~
deathanatos
> _What you mean by distance ?_

I believe the person you're responding to meant miles. Physical distance.

> _It 's not so easy as it sounds._

Their point was not that it isn't easy (and I think they realize that) but
that the article seems to imply that out-of-country _is_ suboptimal, and
they're questioning that assumption. (Hence the SF vs. Toronto from upstate NY
example.) DNS resolvers are already computing some form of location to
determine whether a query is in or out of country, and presumably the resolver
knows its own location, so it effectively is already solving (to a degree) the
problem of computing distance; the poster's point was just that "in country"
is not necessarily "closest", "fastest" or "best" (or even that those three
are the same).

~~~
lossolo
> DNS resolvers are already computing some form of location to determine
> whether a query is in or out of country

Most of it is based on publicly available IANA databases (RIPE, ARIN etc)
which are not always accurate, as IPs are traded and moved between AS's all
the time + dynamic IPs that are given to users from example both coasts of
USA.

> the poster's point was just that "in country" is not necessarily "closest",
> "fastest" or "best" (or even that those three are the same).

And I wrote it's not easy to solve because you do not get accurate GPS
positions to most of IP clasess, which means it's a lot easier to point to the
same country and that's why almost every provider does that. Some use geo
coordinates of IPs but only for certain classes that have accurate positioning
and are inside the same country.

PS. I am building globally distributed anycast GEO-DNS service/system at the
moment.

------
phicoh
I would naively assume that with the huge number of POPs that Google has and
the desire to provide a low latency service, Google would only need to have
the queries that originate from their resolvers have the right local
addresses.

What might be a problem is that for those addresses there is no geolocation
information. Which Google could solve by supplying that to maxmind, etc.

Hence the need for the privacy violating subnet option.

~~~
JdeBP
It already does. There's a list of exactly that, supplied by Google,
hyperlinked-to in the very article headlined here.

------
jedisct1
I don't feel comfortable with DNS services being "observed".

~~~
guipsp
Do something about it then.

