BGP attacks change the semantic meaning of IP addresses themselves. DNSSEC operates at a level above that. The one place this matters in a post-HTTPS-everywhere world is at the CAs, which are now all moving to multi-perspective validation.
As you should be aware, multi-perspective validation does not solve anything if your BGP hijack is accepted to be global. You will receive 100% of the traffic.
DNSSEC does greatly assist with this issue: It would have prevented the cited incident.
1. Hijack the HTTP/HTTPS server. For some IP ranges, this is completely infeasible. For example, hijacking a CloudFlare HTTP/HTTPS range would be almost impossible theoretically based on technical details that I won't go through listing.
2. Hijack the DNS server. Because there's a complete apathy towards DNS server security (as you are showing) this attack is very frequently overlooked. Which is exactly why in the cited incident attackers were capable of hijacking Amazon Route53 with ease. *DNSSEC solves this.*
If either 1 or 2 work, you have yourself a successful hijack of the site. Both need to be secure for you to prevent this.
In summation, you propose a forklift upgrade of the DNS requiring hundreds of millions of dollars of effort from operators around the world, introducing a system that routinely takes some of the most sophisticated platforms off the Internet entirely when its brittle configuration breaks, to address the problem of someone pulling off a global hijack of all the Route53 addresses.
At this point, you might as well just have the CABForum come up with a new blessed verification method based on RDAP. That might actually happen, unlike DNSSEC, which will not. DNSSEC has lost signed zones in North America over some recent intervals.
I do like that the threat model you propose is coherent only for sites behind Cloudflare, though.
"I do like that the threat model you propose is coherent only for sites behind Cloudflare, though."
The threat model I proposed is coherent for Cloudflare because they have done a lot of engineering to make it almost impossible to globally BGP hijack their IPs. This makes the multi-perspective validation actually help. Yes, other ISPs are much more vulnerable than Cloudflare, is there a point?
You are not saying DNSSEC doesn't serve a real purpose. You are saying it is annoying to implement and not widely deployed as such. That alone makes me believe your argument is a bit dishonest and I will abstain from additional discussion.
No, I'm saying it doesn't serve a real purpose. I've spent 30 years doing security work professionally and one of the basic things I've come to understand is that security is at bottom an economic problem. The job of the defender is to asymmetrically raise costs for attackers. Look at how DNS zones and certificates are hijacked today. You are proposing to drastically raise defender costs in a way that doesn't significantly alter attacker costs, because they aren't in the main using the exotic attack you're fixated on.
If we really wanted to address this particular attack vector in a decisive way, we'd move away, at the CA level, from relying on the DNS protocol browsers use to look up hostnames altogether, and replace it with direct attestation from registrars, which could be made _arbitrarily_ secure without the weird gesticulations DNSSEC makes to simultaneously serve mass lookups from browsers and this CA use case.
But this isn't about real threat models. It's about a tiny minority of technologists having a parasocial relationship with an obsolete protocol.
yeah, the same for the rest.
your fanboys are happy and the rest is just tired, because everyone who does not share your point of view has a invalid opinion.
>It’s full of rhetoric and bluster, appeals to authority and dismissal of arguments not from what he considers an authority, and when he runs out of arguments entirely, he stops responding.
Or his broken record commentary on how Signal absolutely needs to ask people for their mobile phone numbers in order to at all be able to provide a functional service, and how doing so does not at all provide Signal with a highly valuable social network map. Exact same story as soon as the arguments are dismantled.
Counterpoint: no it isn't, which is why virtually nobody uses it. Even the attack this thread centers on --- BGP hijacking of targeted DNSSEC servers to spoof CA signatures --- is a rounding error sidenote compared to the way DNS zones actually get hijacked in practice (ATO attacks against DNS providers).
If people were serious about this, they'd start by demanding that every DNS provider accept U2F and/or Passkeys, rather than the halfhearted TOTP many of them do right now. But it's not serious; it's just motivated reasoning in defense of DNSSEC, which some people have a weird stake in keeping alive.
> Counterpoint: no it isn't, which is why virtually nobody uses it. Even the attack this thread centers on --- BGP hijacking of targeted DNSSEC servers to spoof CA signatures
Wait, wait, wait. How can you hijack a DNSSEC server? Its keys are enrolled in the TLD, and you can't spoof the TLD server, because its keys in turn are enrolled in the root zone. And the root zone trust anchor is statically configured on all machines.
DNS is an industry that has a race to bottom in terms of quality and price, and will generally avoid any costs in order to keep as much of the small margin they get between the costs from the TLD and the end user price. For many registrars and domains the margin sits around $0-1 per domain and year, and the real profit comes from uppsells and dark patterns that hikes up the price until the customer manage to leave.
High-end registrars that focus on portfolios and brand management generally do have any kind of authentication that the customers want, and the price will reflect that. Costumers are not just buying the service of a middle man that acts as a go-between the TLD and the domain owner.
You are again ignoring the fact that DNSSEC would have prevented a $152,000 hack. Yes, we are aware organizations are not always serious about security. For those that are though, DNSSEC is a helpful tool.
No, it isn't. It attempts and mostly fails to address one ultra-exotic attack, at absolutely enormous expense, principally because the Internet standards community is so path-dependent they can't take a bad cryptosystem designed in the mid-1990s back to the drawing board. You can't just name call your way to getting this protocol adopted; people have been trying to do that for years, and the net result is that North American adoption fell.
The companies you're deriding as unserious about security in general spend drastically more on security than the companies that have adopted it. No part of your argument holds up.
Citation? A BGP hijack can be done for less than $100.
"You can't just name call your way to getting this protocol adopted"
I do not care if you adopt this protocol. I care that you accurately inform others of the documented risks of not adopting DNSSEC. There are organizations that can tolerate the risk. There are also organizations that are unaware because they are not accurately informed (due to individuals like yourself), and it is not covered by their security audits. That is unfortunate.
Slack's house literally did burn down for 24 hours because of DNSSEC back into 2021.
When you frame the risk as "marginal benefit against one specific threat" Vs "removes us from the internet for 24 hours", the big players pass and move on. This is the sort of event the phrase "sev 1" gets applied to.
Some fun companies have a reg requirement to provide service on a minimum SLA, otherwise their license to operate is withdrawn. Those guys run the other way screaming when they hear things like "DNSSEC" (ask me how I know).
What percentage of the fortune 500 is served over DNSSEC?
Oh. I thought it burned down because of their engineers not having fully acquainted themselves with the tool before applying it. It's misguided to hold DNSSEC culpable for Slack's engineers' goof-up. Like advising people against ever going near scissors because they might run with one in their hands.
That is an extremely uncharitable take for an outage that involved two of the most poorly defined and difficult to (correctly) implement DNS features (wildcards and NSEC records) that the three major DNS operators mentioned in the Slack post-mortem (AWS, Cloudflare, and Google) all implemented differently.
IIRC, Slack backed out of their DNSSEC deployment because of a bug with wildcards and NSEC records (in Route53, not Slack), but the problem Slack subsequently experienced was not caused by that bug, but was instead caused by the boneheaded way in which Slack tried to back out of DNSSEC. I.e. Slack’s problem was entirely their own doing, and completely avoidable if they had had any idea of what they were doing.
Having read the post mortem, I disagree. Slack engineers did something dumb, under pressure during an outage. Even if they hadn't, they still would have been in a degraded state until they could properly remove their DNSSEC records and/or get the Route53 bug they hit fixed. In other words, they still would have had a 24+ hour outage, albeit with a smaller blast radius.
The design of DNSSEC is simply not fit for purpose for zone operators. It is far too easy to screw your zone up for far too marginal a benefit, to say nothing of the huge increase in CPU resource required to authenticate DNSSEC record chains.
The story for implementers is just as bad - the specifications are baroque, filled with lousy crypto and poorly thought-out options.
To give just one example, consider the NSEC3 iteration limit field. NSEC3 itself was designed mostly[0] to prevent zone enumeration when validating negative responses (which is trivial to perform with NSEC.) The iteration count was designed to give zone operators the ability to increase the cost to an attacker of generating a dictionary of nsec3 names[1]. Of course, a high iteration count also raises the cost to a non-malicious resolver of validating a negative response for a nsec3-enabled zone.
In good old DNSSEC fashion, the iterations field is a single number that is subject to a... wide variety of potential limits:
* 16 bits by the wire protocol
* 150 for 1,024 bit keys (RFC 5155 10.3[2])
* 500 for 2,048 bit keys (RFC 5155 10.3[2])
* 2,500 for 4,096 bit keys (RFC 5155 10.3[2])
* 0 (RFC 9276 3.2)
Why 0? It was noted -- after publishing the NSEC3 spec -- that high iterations just don't provide that much benefit, and come with a high cost to throughput. Appendix B of RFC 9276 shows a roughly 50% performance degradation with an iteration count of 100. So, RFC 9276 3.2 says:
Validating resolvers MAY also return a SERVFAIL response when processing NSEC3 records with iterations larger than 0.
Of course, their guidance to implementers is to set the limits a bit higher, returning insecure responses at 100 iterations and SERVFAIL at 500. That said, if you want to be maximally interoperable, as a zone operator, you should pretend like the iteration count field doesn't exist: it is standards compliant for a validating resolver to refuse an nsec3 response with more than a single hash round.
As I said, this is one example, but I'm not cherry picking here. The whole of the DNSSEC spec corpus is filled with incomprehensible verbiage and opportunities for conflicting interpretations, far beyond what you see in most protocol specs.
0 - also to reduce the size of signed top-level zones
1 - all NSEC and NSEC3 records, while responsive to queries about names that don't exist, consist of obfuscated names that do exist.
2 - According to the letter of the standard, the limits applied to the iterations field should be 149, 499, and 2,499. Implementations are inconsistent about this.
IIUC, if Slack had done the correct thing, only wildcard DNS records (if any) would have been affected. They would certainly not have had a complete DNS blackout. I would classify that as significant.
> The story for implementers is just as bad - the specifications are baroque, filled with lousy crypto and poorly thought-out options.
I don’t care. So is almost every other standard, but until something better comes along, DNSSEC is what we have. Arguing that a working and implemented solution should not be used since it is worse than a non-existing theoretical perfect solution is both:
1. True
2. Completely and utterly useless, except as a way to waste everyone’s time and drain their energy.
> IIUC, if Slack had done the correct thing, only wildcard DNS records (if any) would have been affected
There's the problem - you DON'T understand. Straight from the post-portem that you clearly have not read:
One microsecond later, app.slack.com fails to resolve with a ‘ERR_NAME_NOT_RESOLVED’ error:
[screenshot of error ]
This indicated there was likely a problem with the ‘*.slack.com’ wildcard record since we didn’t have a wildcard record in any of the other domains where we had rolled out DNSSEC on. Yes, it was an oversight that we did not test a domain with a wildcard record before attempting slack.com — learn from our mistakes!
> I don’t care. So is almost every other standard,
Cool story. I do care. I'd like to see greater protection of the DNS infrastructure. DNSSEC adoption is hovering around 4%. TLS for HTTP is around 90%. At least part of that discrepancy is due to how broken DNSSEC is.
They could have done a quick fix by adding an explicit app.slack.com record. But instead they removed the DNSSEC signing from the whole domain, thereby invalidating all records, not just the wildcard ones.
> I do care.
I will care once something else comes around with any promise of being implemented and rolled out. Until then, I see no need to discourage the adoption of DNSSEC, or disparage its design, except when designing its newer version or replacement.
> I'd like to see greater protection of the DNS infrastructure. DNSSEC adoption is hovering around 4%.
I work at a registrar and DNS hosting provider for more than 10.000 domains. More than 70% of them have DNSSEC.
> They could have done a quick fix by adding an explicit app.slack.com record. But instead they removed the DNSSEC signing from the whole domain, thereby invalidating all records, not just the wildcard ones.
1) That would do nothing to fix resolvers that had already cached NSEC responses lacking type maps.
2) That presumes the wildcard record was superfluous and could have been replaced with a simple A record for a single or small number of records. Would love to see a citation supporting that.
3) That presumes the Slack team could have quickly identified that the problem they were having was caused by the fact that app.slack.com (and whatever other hosts resolve from that wildcard) was caused by the fact the record was configured as a wildcard and would have been resolved by eliminating the wildcard record. If you read the postmortem, it is clear they zeroed in on the wildcard record as being suspect, but had to work with AWS to figure out the exact cause. I doubt that was an instantaneous process.
Any way you slice it, there was no quick way to fully recover from this bug once they hit it, and my argument is that the design of DNSSEC makes these issues a) likely to happen and b) difficult to model ahead of time, while providing fairly marginal security benefit.
At this point, I really don't care if you agree or disagree.
> I will care once something else comes around with any promise of being implemented and rolled out.
Yeah. DNSSEC is going to be widely deployed any day now. The year after the year of Linux on the desktop.
> I work at a registrar and DNS hosting provider for more than 10.000 domains. More than 70% of them have DNSSEC.
Cool. There are, what, 750 million domains registered worldwide? We are at nowhere near 10% adoption worldwide, let alone 70%. Of the top 100 domains -- the operators you would assume would be the most concerned about DNS response poisoning -- *six* have turned DNSSEC on.
> 1) That would do nothing to fix resolvers that had already cached NSEC responses lacking type maps.
The TTL for NSEC records are presumably way lower than the TTL for the DS records.
> 2) That presumes the wildcard record was superfluous and could have been replaced with a simple A record for a single or small number of records. Would love to see a citation supporting that.
It’s theoretically possible that it would not have worked for all cases, but that is, in my experience, very unlikely.
> Any way you slice it, there was no quick way to fully recover from this bug once they hit it
The bug seems to me to have been reasonably easy to mitigate, but their problem was that Slack did not know what they were doing. Thu bug itself was minor, but Slack tried to fix it by stopping to serve DNSSEC-signed DNS data, while long-TTL DS records were still being unexpired in the world. This is the worst possible thing you could do.
> Of the top 100 domains -- the operators you would assume would be the most concerned about DNS response poisoning -- *six* have turned DNSSEC on.
1. That number used to be zero, as tptacek liked to point out.
2. The huge operators often have fundamentally different security priorities than regular companies and users.
3. People said the same about IPv6 and SSL, which were also very slow to adopt. But they are all climbing.
> The TTL for NSEC records are presumably way lower than the TTL for the DS records.
Possibly. It was still an outage they had to wait out the TTL for, due to the design of DNSSEC.
> It’s theoretically possible that it would not have worked for all cases, but that is, in my experience, very unlikely.
This is completely unsubstantiated speculation on your part.
> The bug seems to me to have been reasonably easy to mitigate, but their problem was that Slack did not know what they were doing.
It is indeed reasonably easy to Monday-morning quarterback someone else's outage and blame operators for the sharp edges around poorly designed protocols.
> 1. That number used to be zero, as tptacek liked to point out.
Cool. so, at this rate, in another 100 years or so we should be at 50% adoption.
> 2. The huge operators often have fundamentally different security priorities than regular companies and users.
Priorities like uptime?
> 3. People said the same about IPv6 and SSL, which were also very slow to adopt. But they are all climbing
1) people started rolling IPv6 out once v4 addresses got scarce. There is no such compelling event to drive DNSSEC adoption. 2) SSL is easy to roll out and provides compelling security benefits. It is also exceedingly unlikely in practice to blow up in your face and result in run-out-the-clock outages -- unlike DNSSEC.
> This is completely unsubstantiated speculation on your part.
Do you have any support for your assumption that the wildcard record was vital and practically impossible to replace with regular records?
> It is indeed reasonably easy to Monday-morning quarterback someone else's outage and blame operators for the sharp edges around poorly designed protocols.
When Slack, being a large company presenting themselves as proficient in tech, make a tech mistake so bad that they lock themselves out if the internet for an entire day, a mistake even I know not to make, then I get to criticize them.
You just gave a link with a graph that shows a recent sharp drop in DNSSEC adoption as if it was a mic drop. The page I showed you barely even has text; it doesn't need any, the implication is obvious.
It used to show "a recent sharp drop", back when you originally gave the link. It quite soon started to climb again, and the climb has continued, as is now clearly visible. This was pointed out to you, but you acted, and are still acting, as if nothing has happened since that time you first looked at it.
Internally at slack the general consensus was that dnssec was a giant waste of time and money from a security perspective. We did it for compliance to sell into the Federal govt and federal contractors.
I'm not sure what this has to do with anything I've said on this thread, but we don't have to keep pressing these arguments; I'm pretty satisfied with the case I've made so far.
And at least Let's encrypt actually verifies DNSSEC before issuing certificates. IIRC it will become mandatory for all CA's soon. DNSSEC for a domain plus restrictive CAA rules should ensure that no reputable CA would issue a rogue cert.
"Most domains". Yes, it is possible that nobody bothers to DNS hijack your domains. Sadly I've worked for organizations where it did happen, and now they have DNSSEC.
I invite anybody who thinks this is a mic drop to pull down the Tranco research list of most popular/important domains on the Internet --- it's just a text file of zones, one per line --- and write the trivial bash `for` loop to `dig +short ds` each of those zones and count how many have DNSSEC.
For starters you could try `dig +short ds google.com`. It'll give you a flavor of what to expect.
And you still can't seem to make your mind up on whether this is because DNSSEC is still in its infancy or if it's because they all somehow already studied DNSSEC and ended up with the exact same opinion as you. I'm gonna go out on a limb and say that it's not the latter.
What do I have to make my mind up about? I worked on the same floor as the TIS Labs people at Network Associates back in the 1990s. They designed DNSSEC and set the service model: offline signers, authenticated denial. We then went through DNSSEC-bis (with the typecode roll that allowed for scalable signing, something that hadn't been worked out as late as the mid-1990s) and DNSSEC-ter (NSEC3, whitelies). From 1994 through 2025 the protocol has never seen double-digit percentage adoption in North America or in the top 1000 zones, and its adoption has declined in recent years.
You're not going to take my word for it, but you could take Geoff Huston's, who recently recorded a whole podcast about this.
