
Google Public DNS: 70 billion requests a day and counting - abraham
http://googleblog.blogspot.com/2012/02/google-public-dns-70-billion-requests.html
======
davidu
Just to add a couple data points:

    
    
      1) Tons of random indy web crawlers, mail cannons, etc. use Google Public DNS.
      2) We actively discourage large machine-based usage of our service.
    

Even with some aggressive handling of large machine-based users, we still grow
at a crazy clip.

Our growth: <http://i.imgur.com/znfu9.png>

~~~
brador
Any idea on the limit for "large" machine based usage?

For examples, I use it with a PHP script that hits it a few hundred times and
have not yet had a problem. Should I be expecting one?

~~~
davidu
We had a 500 cluster web crawler using us. Probably doing about a billion
queries a day. They had the option to pay us or switch to Google Public DNS.
Guess which they did? :-)

A few hundred, even if a few hundred per minute, is not a problem.

~~~
blibble
I have operated a rather large crawl cluster on EC2 (several hundred nodes),
and I don't see why anyone would point one at a third party DNS service.

we operated for about 6 months using the default resolvers than come with the
AMIs, then Amazon contacted us, telling us they wanted us to stop using their
recursive nameservers... apt-get install bind, point at 127.0.0.1, redeploy
AMI, done (in under 5 minutes).

it isn't as if a crawler cares about an extra 250ms from non-cached entries,
and the bandwidth from DNS is trivial* compared to that of downloading pages.

... so why would anyone ever offer to pay you/anyone else for a recursive DNS
service? it's a trivial problem...

(* say 32 bytes for the query, 64 bytes for the response... 1B lookups is
~64gb, or $3.50 with Amazon's very expensive bandwidth costs)

~~~
davidu
250ms matters. That's 1/4 of a second. It matters for a crawler tremendously.

DNS query sizes are wrong. I'd double each. But still, inexpensive from a
bandwidth standpoint. I get it.

~~~
blibble
why would it matter at all, given a crawler hits many pages on the same
hostname? the fact the initial request takes 250ms more is meaningless.

when you also take into effect that crawlers are either: bound on sleep() if
they're friendly, bound on cpu if doing processing and you have a lot of money
for bandwidth, or if you don't have much money, bound on bandwidth.

and given that crawlers tend to be massively parallelised, the DNS query could
take minutes and you really still wouldn't care...

(go and read up on Amdahl's law)

------
freehunter
Being in the information security world, I have to wonder what percentage of
these requests are based in malware? I know at least the latest version of
ZeroAccess/Max++/Sirefef (which we managed to get before the AV vendors
released definitions for it) uses it quite heavily. That's one of the symptoms
we used to diagnose computers from a strictly network-level standpoint. No one
on our network should be using Google DNS, so any computers who were making
requests to 8.8.8.8 were likely infected (confirmed using other signatures).

That amounted to about 100 requests every day per infected computer just from
us, and ZeroAccess isn't the only one doing it (and isn't a rare trojan).

------
seancron
For those thinking about changing their DNS servers, you might want to take a
look at <http://code.google.com/p/namebench/>

It benchmarks global (like Google Public DNS and OpenDNS) and regional DNS
providers to show which DNS servers would be fastest for you.

------
jasoncartwright
For comparison, OpenDNS does ~37bn <http://www.opendns.com/technology/traffic-
stats/>

------
51Cards
That's a little over 800,000 requests a second on average. That's some serious
traffic.

------
c1sc0
I'm wondering if there's a detectable peak in people switching to Google DNS
because of censorship in countries like .be & .nl

------
TomGullen
I just switched to Google DNS. Before I switched I got a 38ms response time
when I ping google.com, now I get 285ms? I'm in the UK. Is this why it's slow?

~~~
zacgarrett
DNS has nothing to do with ping response times. A ping requests the IP address
from the DNS server and then does the ping. On most systems the DNS is cached
locally, so multiple requests will use the same information.

~~~
icebraining
_DNS has nothing to do with ping response times._

Not true, the feature is called GeoDNS and gives you different IP addresses
for the same domain based on your DNS server. See Locke1689's reply.

------
quux
Glad they're doing something about the CDN issue. This is the main thing that
keeps me from switching to Google DNS.

------
nuttendorfer
I used to use this but switched off over security concerns (Mainly because I
didn't want Google everywhere)

I haven't noticed any difference in load times so I guess it didn't do any
harm.

~~~
re_todd
It would be funny if they suddenly dropped the service because it wasn't
generating a good "revenue stream". It would be chaos for awhile.

------
sellandb
I would be interested if statistical data gleaned from DNS makes it's way into
any other service areas. DNS would seem like a useful way to rank the
popularity of web sites, I am sure there are some interesting enhancements
that could be made using that data.

~~~
tonfa
I think their privacy policy disallows it:
<http://code.google.com/speed/public-dns/privacy.html>

> We don't correlate or combine your information from the temporary or
> permanent logs with any other data that Google might have about your use of
> other services, such as data from Web Search and data from advertising on
> the Google content network.

And they say that the logs are only used for debugging, DoS protection and
abuse.

~~~
aw3c2
The quote from the policy says nothing about using or not using the
information about what DNS queries are made for data mining or ranking.

~~~
tonfa
Data mining and ranking is neither debugging, nor DoS protection or abuse
protection, so I don't see how it could be allowed.

The FAQ makes it even more clear:

> Is information about my queries to Google Public DNS shared with other
> Google properties, such as Search, Gmail, ads networks, etc.?

> No.

~~~
aw3c2
I did not mean "my queries" being "shared" but the query collection/archive as
a whole. It would be another source to know what domains ___the people_
__visit.

> And they say that the logs are only used for debugging, DoS protection and
> abuse.

Source?

~~~
tonfa
Rereading it, I see it's less clear how it applies to aggregated data.

------
xiaoma
I for one am very thankful for this DNS. Ever since moving to China, 8.8.8.8
has useful for getting around various flaws of the internet experience here.
Even with a paid VPN, it's nice to have an always working DNS server.

------
cowmixtoo
It seems like Google's DNS servers are always MANY hops away from me, no
matter where I am. The ping times range from 8ms to 45ms.

How is that faster than using a local DNS server?

~~~
secure
Obviously it cannot be faster than a local DNS cache/server (for the RTT).
What can be significantly faster is forwarding queries to Google Public DNS
instead of your ISP. At least in my experience, ISPs often have slow and
overloaded servers which are not well maintained. Recursively resolving
completely on your own is often even slower than either of those two choices.

~~~
a1k0n
ISPs often also have abusive DNSes that redirect unresolved domains to ad-
ridden landing pages, which is why I switched.

------
Codhisattva
What an enormous data mine.

~~~
Codhisattva
Down votes for pointing out that Google DNS is an enormous data mine? Why?

