When I made a DNS switch at reddit, even with a 5 minute TTL, it still took an hour for 80% of the traffic to shift. After a week, only 95% had shifted. After two weeks we still had 1% of traffic going to the old IP.
And after a month there was still some traffic at the old endpoint. At some point I just shut off the old endpoint with active traffic (mostly scrapers with hard coded IPs at that point as far as I could tell).
One of my friends who ran an ISP in Alaska told me that they would ignore all TTLs and set them all to 7 days because they didn't have enough bandwidth to make all the DNS queries to the lower 48.
So yeah, set your TTL to 40 hours. It won't matter anyway. In an emergency, you'll need something other than DNS to rapidly shift your traffic (like a routed IP where you can change the router configs).
So for my own stuff if there is a controlled change I try to keep the original address operating as a relay to the new address for a chunk of time, and for a while after that have it host a message saying "your DNS resolution seems to be broken, you shouldn't have been sent here, please report this to your ISP or local SysAdmin, if you want me to try fix it for you here is a list of my consulting fees".
The lesson I took away from hosting operations is that the implementation of internet standards has a long tail of customization; it's part of the job to accommodate them graciously. :)
Turns out their routers somehow sets an insane TTL (like max int), and reports this to their clients, which in turn also get stuck with an insane TTL. You have to reboot or flush both the router and the clients to get it unstuck. I don't know if it's a "feature" or some sort of memory corruption / race condition.
The routers were almost always Asus routers, ex: Asus RT66. And the same customers were repeat offenders (not every time, but often enough).
In the end we had them set DNS on their computers to something like 220.127.116.11
TTLs aren't respected -> people try lowering them even more.
Lots of low TTLs -> people configure a minimum or ignore them.
As an infrastructure provider with limited connectivity, seeing lots of low DNS TTLs means I'll just configure a saner minimum (say 1h) and consider it not my problem.
This is exactly why people set TTLs lower; they get ignored.
If you’re hosting for customers the difference here is that this is your friends fault, not yours.
I'm not really sure what's up with that, as no-one has ever reported the site not being reachable in the Facebook app after an IP change. Either it does some very aggressive caching, or something is pretending to be the Facebook app.
The author seem to be missing one of the big reasons ridiculously low TTLs are used: it lets passive eavesdroppers discover a good approximation of your browsing history. Passive logging of HTTP has (fortunately) been hindered as most traffic moved to HTTPS, but DNS is still plaintext.
Low TTLs mean a new DNS request happens (apx) every time someone clicks a link. Seeing which domain names someone is interacting with every 60s (or less!) is enough to build a very detailed pattern-of-life. Remember, it's probably not just one domain name per click; the set of domain names that are requested to fetch the js/css/images/etc for each page can easily fingerprint specific activities within a domain.
Yes, TTLs need to have some kind of saner minimum. Even more important is moving to an encrypted protocol. Unfortunately DOH doesn't solve this problem; it just moves the passive eavesdropping problem to a different upstream server (e.g. Cloudflare). The real solution is an encrypted protocol that allows everyone to do the recursive resolution locally.
I operate DNS for hundreds of thousands of domains. I've tried to reassemble browsing history from DNS logs, and I can tell you it is damn near impossible. You have DNS caches in the browser, the OS, broadband routers, and ISPs/public resolvers to account for - and half of them don't respect TTLs anyways.
The reason people set low TTLs is they don't want to wait around for things to expire when they want to make a change. DNS operators encourage low TTLs because it appears broken to the user when they make a change and "it doesn't work" for anywhere from a few hours to a few days.
It would be naive to think none do of course.
So if some things are cached, you won't get a complete picture, but the picture you get might be enough.
I did find this paper:
The violations in that paper that are important are those that have increased the TTL. Reducing the TTL increases costs for the DNS provider, but isn't important here. The slowest update was about 2 hours (with the TTL set to 333).
Of those that violated the TTL, we don't know what portion of those would function correctly with a different TTL (increasing the TTL indicates they're already not following spec). So I wouldn't assume that increasing the TTL would get them to abide by your requested TTL. They're following their own rules, and those could by anything.
Considering how common low TTLs are... you're worrying about a DNS server that's already potentially causing errors for major well known websites.
From my own experience when changing records and seeing when the long tail of clients stops calling the old addresses (with the name), it is a really long tail. An extreme example that lasted almost six months was a web spider that just refused to update their DNS records and continued to request websites using the old addresses.
Is there a lot of custom written code that does their own DNS caching? Yes. One other example is internal DNS servers that shadow external DNS. There is a lot of very old DNS software running year after year. Occasionally at work we stumble onto servers which are very clearly handwritten by someone a few decades ago by people with only a vague idea of what the RFCs actually say. Those are not public resolvers of major ISPs, so the above study would not catch them.
Naturally if you have a public resolver where people are constantly accessing common sites with low TTLs then issues would crop up quickly and the support cost would get them to fix the resolver. If it's an internal resolver inside a company where non-work sites are blocked then you might not notice until the company moves to a new web hosting solution and suddenly all employees can't access the new site, an hour later they call the public DNS hosting provider, the provider diagnoses the issue to be internal of the customer's network, and then finally several hours later the faulty resolver gets fixed.
It may have been Java client that was not restarted. At least for older versions of Java the default was to cache result forever.
Could it be that it wasn't using DNS at all? Just hardcoded the ip address?
Note that you can still get the benefit of a low TTL during a planned switch to a new cloud provider, server, or network even if you run with a high TTL normally. You just have to lower it as you approach the switch.
For example, let's say you normally run with a TTL of 24 hours. 25 hours before you are going to throw the switch on the provider change, change the TTL to 1 hour. 61 minutes before the switch, change TTL to 1 minute.
I can still imagine a benefit to short TTLs in the sense that you can maybe roll out your canary in a more controlled way. But that's a lot more complicated than the issue of quick switching.
If your cloud provider does an oopsie (e.g. https://news.ycombinator.com/item?id=20064169) and takes down your entire infrastructure, or you have to move quickly for some other reason, or you're recovering from a misconfiguration, the long TTL can add 24 hours to your mitigation time.
If you're just playing around with your personal project/web site, you just added a giant round of whack-a-cache to your "let's finally clean up my personal server mess" evening.
Subsequently, my previous employer lowered the default TTL simply because it got rid of all the bad Trustpilot ratings about customers being "prevented from leaving", and started offering a "move my WordPress site for me" service to profit from all the panicking new-comers who had no idea about how to do trivial things like importing/exporting a database and transferring files.
99% of apps will gladly trade a 30ms increase in session start (assuming the browser's prefetcher hasn't already beaten them to it) to not have to worry about things taking an hour to change. Not all efficiency is about how technically slick something is.
I mean, my company does this for certain failure scenarios involving our CDNs. Can anyone tell me why we're idiots, or is this just hyperbole?
amazon.com, reddit.com, facebook.com, and others use low TTLs on their domains for this reason. Anyone who can't maintain an anycast infrastructure around the world and doesn't want to depend on Cloudflare will use this method.
The only other alternative for that scenario is using anycast addressing, and that has a colorful bag of limitations that are quite different from those of low-TTL DNS (including being out of reach for most).
I see nothing wrong with using low DNS TTLs for failover - really don't understand the author's objections here, and them claiming that "DNS is not used for failover any more" significantly discredits them, IMO.
I came here to say exactly that. Our company uses DNS entries with low TTL for failover and load balancing purposes as well -- it's a very common approach. Services like AWS Route 53 and CloudFlare make it very easy to setup and low cost. I was surprised that the author didn't give much acknowledgement to this type of usage.
Within a single network, sure, load balance all you want. That’s not the scenario low TTLs go after.
What people use in those sitations is Anycast.
Of course, DNS itself, or e-mail, don’t need this kind of redundancy, since the NS (or MX) records themselves provides a list of failover servers. The corresponding alternative for HTTP, SRV records, has been consistently stonewalled by standard writers for HTTP/2, QUIC, etc.
There is an interesting draft RFC which I am keeping an eye on, but I don’t want to get my hopes up:
This requires that you blow a publicly-advertisable prefix for every unique combination of services you would want to fail-over.
E.g., if you wanted to be able to have independent fail-over between your customer-facing self-service portal and your webmail interface (each relying on specific state that you can't replicate synchronously, and can't guarantee replicate consistently with each other), you would need to /24s, one dedicated to anycast for the webmail interface, one for the self-service portal, and separate from any services which are active-active.
Whereas using DNS, you could use your other existing public /24s that you are already using for your active-active services.
In the last days of IPv4, an extra 2 /24s just for this is quite an expense.
Outside of a region, they use DNS and sometimes a combination of WAN accelerators and VPN's, dark fiber.
Or is there a better way to quickly invalidate DNS caches in case of emergency?
Not sure about any others.
Everyone probably starts with higher TTLs, then the first time they mess up an update they switch to a short one.
The author seems to not appreciate how big a problem a misconfigured, long TTL DNS entry could be for someone.
Other reasons for short TTLs... maybe an IP gets blocked/flagged by a large network and they need to change fast... or a network path is slow and they need to move to a new location.
So whats the solution? We are using AWS ALB/ELB and it states in docs, that we should have low TTL, and it makes sense. Servers behind LB scale up and down. What is the option B?
> $ drill detectportal.firefox.com @18.104.22.168
Is captive portal detection not a valid use case for low TTL? The entire point is to detect DNS hijacking of a known domain, which takes longer when you cache the DNS results...
But sure, there are other implementation approaches (e.g. injecting HTTP redirects), which I imagine is one reason why Firefox doesn't literally inspect the DNS reply.
After I complained I think they upped the ttl.. to 60.
After about 8 years we were were restructuring our DNS infrastructure for performance and I rate limited those two to 10 or so queries per second each. In that time there must have been 300 billion or so requests from those two boxes alone.
Some process on the system wants to connect to itself, which then causes a dns lookup. Add a high transaction rate on top of that, and 1,200/second is easy.
The funniest one I remember seeing was thousands and thousands of lookups for HTACCESS. Turned out apache was running on top of a web root stored in AFS and not configured to stop looking for .htaccess files at the project root, so it would try to open
This would be done something like twice for every incoming web request.
$ echo min-cache-ttl=300 | \
sudo tee /etc/dnsmasq.d/99-min-cache-ttl.conf
Don't forget to run
$ sudo pihole restartdns
$ drill ping.ring.com @22.214.171.124
;; ->>HEADER<<- opcode: QUERY, rcode: NXDOMAIN, id: 36008
;; flags: qr rd ra ; QUERY: 1, ANSWER: 2, AUTHORITY: 1, ADDITIONAL: 0
;; QUESTION SECTION:
;; ping.ring.com. IN A
;; ANSWER SECTION:
ping.ring.com. 3 IN CNAME iperf.ring.com.
iperf.ring.com. 3 IN CNAME ap-southeast-2-iperf.ring.com.
;; AUTHORITY SECTION:
ring.com. 573 IN SOA ns-385.awsdns-48.com. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400
There are caching nameservers that allow you to override the minimum TTL but be aware the device is likely relying on this being immediately up to date and may not work during a change with an extended TTL set.
;; ANSWER SECTION:
amazon.com. 60 IN A 126.96.36.199
amazon.com. 60 IN A 188.8.131.52
amazon.com. 60 IN A 184.108.40.206
;; ANSWER SECTION:
glacier.us-east-1.amazonaws.com. 60 IN A 220.127.116.11
;; ANSWER SECTION:
s3.ap-northeast-1.amazonaws.com. 5 IN A 18.104.22.168
;; ANSWER SECTION:
dynamodb.us-east-1.amazonaws.com. 5 IN A 22.214.171.124
From time to time, you need to do something with customer facing infrastructure: Remove the DNS entry, watch the traffic drain over the next 5-10 minutes, and then do what you need to do on the device, test, and then add it back in the DNS again, from which you can watch traffic return to normal levels and verify everything is good.
I just checked, and I actually have TTL forced to 1 day in dnscrypt-proxy. My internet experience is fine. I guess I never noticed in the last 2 years or so.
Instead of a browser doing
1. Local DNS lookup (resulting in expired entry)
2. DNS query
3. DNS response
4. HTTP request
why not do
2.1. DNS query
2.2. HTTP request
4. If DNS response changed and HTTP request failed, HTTP request again
Maybe use two expiration lengths, one that results in flow 2 and a much longer one that results in flow 1.
Since the traffic is low, though, I can afford to check for an IP change every ~5min, and although I set a TTL of ~15min on most services, the main CNAME (ovh-provided dynamic dns service, TTL set by them) is set to 60s.
My IPv6 record was set to 1h, but I'll look into increasing it. It is true that my mobile phone often pings my server, so I imagine that it could reduce the battery usage.
A company I worked with a couple of years ago was using Dyn as their DNS provider, and one day we got a notifcation that we had passed the usage limits for our account. This seemed impossible considering our site was getting a couple of hundred unique visitors a day. A few things came out of the analytics.
1) A short TTL on an A record had been left on from a website migration project. The majority of the requests were coming from our internal website administrators. I moved it up to a couple of hours and this went a way.
2) We were getting a huge amount of AAAA record hits. I think most modern browsers/OS try quad A first??? We didn't have IPv6 configured, and therefore the negative resolution had a TTL set to the minimum on the SOA record, which was 1second! A change of this to 60 caused a huge reduction in requests. I suppose I should have set up ipv6, but I didn't.
3) When we sent out stuff to our mailing list the SPF (or rather TXT) records saw a peak that was off the chart. We had a pretty settled infrasructure, so I moved that TTL to a day (I think from memory) and it flattened the peak somewhat.
4) There was a large peak in MX request around 9am. I put this down to people opening their email when they got to work and replying to us. I had to set the TTL to a couple of days (of course) to smooth that one.
I like to think it was worthwhile and improved things for users. I at least had a nice warm glow that I had saved the internet from a bunch of junk requests, and it just felt tidier.
It would be interesting to (ab)use this mechanism in the way you suggest. A recursive DNS server could ask to be NOTIFY'ed of changes in the zone they are querying...it would, of course, add load to the server, and it would need strict limits to avoid DoS, but it seems an interesting idea.
It seems that a lot of sites are ok with slightly higher latencies if it means greater operational flexibility.
So there's no single latency to report.
I'm kind of suprised that I can't see any other comments talking about GTM's (I assume you mean F5 Global Traffic Managers)
Where I am at the moment GTM's are used everywhere, and everywhere the TTL is set to 30s.
The only part of this that really annoys me is that the global default configuration, rather than serving up a subset of the list of IP addresses, only a single IP address is returned when you resolve down to the A record.
When I've pressed the issue that _at least_ on our internal GTM's we should just return a bunch of IP addresses every time someone resolves the address, I've been told that it would break load balancing... which blows my mind because who on earth is relying on DNS to load balance traffic with a 30s TTL, I would have thought that the normal thing to do, if you actually wanted load to balance, would be to result a subset of IP addresses in a different order and with a different subset each time. That way other DNS servers which resolvers that will cache that record can at least be returning multiple addresses to all the clients it serves, as opposed to everyone using that resolver getting stuck to a single address for 30 seconds...
But all of that being said, it would make perfect sense to me to just return like 4 IP addresses publicly for every resolution and rotating setting the TTL to like 30s so that clients could spend 30s iterating through the A Records they have cached, then hit your resolver up again and get a different sites addresses back if your site had gone down...
And then I change my DNS again to the new IP.
You have to tune a bit to get the right IP in your logs but so far it works.
Unless you do not return an RR set and what you return is based on geolocation and data center health.
Basically, your request is coming from them and wherever their servers are (US, I guess, though they probably have several data centers) and they route it to the final user.
I think using DNS-based geolocation sounds like a really bad idea: what am I missing?
Of course, caching gets more complicated and less useful with this.
Obviously it would be possible to opt out for situations where you genuinely need a low TTL on a domain.
I've worked on managing thousands of (sub)domains and the administrative overhead of changing the TTLs for everything manually would be considerable. I'd certainly like an automated way to say "These records should gradually increase TTL up to <X> time over <Y> time" (e.g., gradually raise TTL to 2 days over 2 weeks if there are no changes).
There are downsides to high TTLs though: (1) you need to remember to preemptively lower them ahead of any planned changes (if you want those changes to take effect quickly), and (2) you can't change the records quickly in an emergency. But, fortunately, lots of record types are ones that you probably don't need to change in an emergency -- and for ones that you do, you can use a low TTL.
Anyway, I'd personally like to see automated TTL management as a feature in DNS software.
Of course, the reality is some traffic will continue to flow to the old destination for as long as you care to measure. There's plenty of absurdly broken DNS caching out there.
I bet this is one of the main reasons for low TTLs.
But it's a risky play for providers. It reduces their DNS load (which is pretty cheap to handle), but increases the risk that a customer will come yelling why they couldn't fix their outage quickly because the algorithm increased their TTL to something large.
They've collected data on DNS queries "for a few hours". By definition, clients who have DNS cached (iow, most clients, since browsers and resolv calls in operating systems will do that for you), will not issue DNS requests for any records that have a TTL that has not yet expired.
So, they've caught all (well, all that were re-requested) the TTLs shorter than whatever "a few hours" is, and only those longer ones that expired exactly during the experiment and were re-requested.
To run a proper experiment testing for "short" vs "regular" (let's say 1-3 days), you need to collect data for days (eg. at least 7 days, preferably at least 30), but even that would not report most TTLs longer than 7/30 days.
Articles like this are bad because they can easily confuse even the knowledgeable people like the HN crowd.
However they wouldn't be quite as bad if web pages didn't load useless crap from 60 different domains.
FWIW, I've learned in the past that while there are plenty of people who claim to want communication to be as succint as possible, majority are unable to understand when somebody is really terse (while still saying exactly enough). I've learned to follow up such a terse statement with examples and longer explanations for the majority that does not get it.
But maybe it's just that people don't expect the mathematics-level precision on the internet :)
Connections are cached and reused.
Also what are you talking about? If the connection is closed, how would it be used?
Connections should only be closed due to? Inactivity. If a connection is closed, don't you think you would probably want to do another DNS request?
Also if your doing proper layer 4 load balancing using BGP, DNS is a moot point...
Or if the user closes the browser of if the server/proxy restarts. But yes, mostly inactivity on the order of a couple of minutes.
> If a connection is closed, don't you think you would probably want to do another DNS request?
That's the whole point of the DNS TTL, to say how long to go before doing another lookup rather than doing it each time you reconnect.
> Also if your doing proper layer 4 load balancing using BGP, DNS is a moot point...
BGP load balancing operates on layer 3, and is irrelevant as you still need to DNS lookup an anycast address. EDNS client subnet is better anyways.
I hope you didn't pay for your education.
By that logic BGP is a layer 7 load balancer since it has an application layer. BGP only exchanges layer 3 reachability information to update route tables therefore you can only load balance layer 3 with it.
Personal attacks and other things in your comments are against the HN guidelines. The goal is to talk about DNS/TTLs and their impact on performance not insult each other. https://news.ycombinator.com/newsguidelines.html
Yes, it is not exactly great, but at least it works well enough for self-hosting services.