
DNS Flag Day on February 1, 2019: check your domains - el_duderino
https://dnsflagday.net
======
skywhopper
This page eventually gets around to the point, but the intro paragraphs don't
tell me what's actually happening, and are written in an extremely confusing
way. I almost just closed the tab since it's not even clear that any
explanation is coming.

"This change affects only sites which operate software which is not following
published standards. Are you affected?"

Affected _in what way_? Instead of linking to the Wikipedia page about what
DNS is (if your visitor doesn't know that, then the rest of the page is
pointless), there needs to be a sentence or three up top telling me what is
_actually happening_ and what the _actual impact_ could potentially be, and
for that matter, which standards you are even talking about. Then, tell me
what the forms you are asking me to type my domain names into are going to do.

As it is, this page is just going to leave a lot of people confused.

~~~
bigbugbag
this page explains things better: [https://www.isc.org/blogs/dns-flag-
day/](https://www.isc.org/blogs/dns-flag-day/)

A number of DNS software and service providers have announced that we will all
cease implementing DNS resolver workarounds to accommodate DNS authoritative
systems that don’t follow the EDNS protocol.

Domains served by DNS servers that are not compliant with the standard will
not function reliably after February 1, 2019, and may become unavailable.

If your company’s DNS zones are served by non-compliant servers, your online
presence will slowly degrade or disappear as ISPs and other organizations
update their resolvers. When you update your own internal DNS resolvers to
versions that don’t implement workarounds, some sites and email servers may
become unreachable.

Why make this change now?

Extension Mechanisms for DNS were specified in 1999, with a minor update in
2013, establishing the ‘rules of the road’ for responding to queries with EDNS
options or flags. Despite this, some implementations continue to violate the
rules. DNS software developers have tried to solve the problems with the
interoperability of the DNS protocol and especially its EDNS extension (RFC
6891 standard) by various workarounds for non-standard behaviors. This is not
unlike the way a driver with the right-of-way might hesitate at an
intersection before proceeding if there were another driver in the
intersection behaving erratically. These workarounds excessively complicate
DNS software and are now also negatively impacting the DNS as a whole.

The most obvious problems caused by these workarounds are slower responses to
DNS queries and the difficulty of deploying new DNS protocol features. Some of
these new features (e.g. DNS Cookies) would help reduce DDoS attacks based on
DNS protocol abuse.

------
kawsper
Testing my domains on Route53 gives me this:

> Minor problems detected!

> This domain is going to work after the 2019 DNS flag day BUT it does not
> support the latest DNS standards. As a consequence this domain cannot
> support the latest security features and might be an easier target for
> network attackers than necessary, and might face other issues later on. We
> recommend your domain administrator to fix issues listed in the following
> techical report: redacted.

~~~
bcaa7f3a8bbc
It means the authoritative servers of your domain simply does not support EDNS
(a 20-year-old protocol...), but it doesn't falsely advertising EDNS support,
which is what the Flag Day's targeting, so your situation is good.

~~~
colmmacc
Route53 absolutely does support EDNS0 and large answers without needing to
truncate.

The minor error reported is that Route 53 does not respond with a "BADVERS"
error in response to a query reporting EDNS version 1.

There is no EDNS version 1 standardized, and the specs say to respond with an
error when you see a version you don't support. Per the specs Route 53 is "in
the wrong". The thinking from the spec authors is that this kind of behavior
will make it easier for resolvers to negotiate future EDNS versions and figure
out what works. I'm skeptical that this will ever actually work in practice:
most protocols tend to have backwards-compatible version upgrades that don't
require round-trips.

The thinking on the Route 53 side, or at least my thinking - I wrote Route
53's EDNS support - is that we see incorrect versions occasionally on the
wire, from weird clients where people have clearly made some kind of mistake,
and that it is better to fail gracefully and give them /an/ answer than an
error that could lead to outages. I might be completely wrong about that, but
that's the thinking. We tend to always heavily lean in the direction that is
likely to increase availability and avoid any potential for outages.

~~~
JoshTriplett
The problem is that that behavior means people won't have to fix the software
that's emitting bad version numbers, which will then make it difficult to
upgrade to a real EDNS version 1 (or other version) in the future. A similar
problem often occurs with other Internet protocol upgrades, as well as with
things like Linux kernel syscalls that don't error out on unknown flags. If
you don't give an error on things you _don 't_ understand, you make it much
harder to build newer software.

If you see "incorrect versions occasionally on the wire from weird clients",
perhaps the people building those weird clients will know to fix them if they
get BADVERS errors.

~~~
colmmacc
Personally I have no appetite for breaking any customer. Sure they may be "in
the wrong" because they have an old dig recipe, or a load balancer health
check tool, or a latency measurers, or some kind of DNS canary, that
mistakenly used a wrong EDNS version, but if we change our behavior and break
them they will a) feel it viscerally and b) rightly, blame us for breaking
things. We're pretty serious about maintaining backwards compatibility always
and treating every API like a promise.

The other side of this is that being too tolerant can lead to network
ossification. This impacted TLS1.3's roll-out, which had to be made to look
similar enough to TLS1.2 session resumption that many network middle boxes
can't tell the difference and let the traffic through. This is less elegant
than a cleaner new design that isn't tainted by having to appear like the old
formats.

Personally, I worry about this a lot less and consider the extra effort to be
backwards compatible very worth it. It think it will still be very easy to
find totally unused never-seen-in-the-wild EDNS version numbers; there's no
shortage. On the TLS working group we've been through this a few times, having
to find unused magic numbers that hadn't been burned as part of the various
experiments. We've done with this protocol versions, cipher suites, etc.

On the DNS side; the proposed hypothetical EDNS version negotiation scheme
that might show up in the future requires round-trips and state. Resolvers
would have to first send an EDNSv1 query, observe it fail, store that state
somewhere and then try with EDNSv0. What if the authoritative server rolls
back their support? What if the auth service is mixed and only some of the
fleet supports the new version? These challenges tend to make that kind of
version negotiation impractical ... hence my skepticism that it will ever work
like that.

A small part of all this is that the DNS WG isn't really stewarded with these
practical issues front of mind. DNAME and DNSSEC have each had backwards-
incompatible implications that real-world operators had to fudge around and
ignore precisely what the specs say, to be able to satisfy customers. Due to
that history, IMO the DNS specs have lost a certain level of moral authority
that other specifications still enjoy.

~~~
JoshTriplett
> Personally I have no appetite for breaking any customer. Sure they may be
> "in the wrong" because they have an old dig recipe, or a load balancer
> health check tool, or a latency measurers, or some kind of DNS canary, that
> mistakenly used a wrong EDNS version, but if we change our behavior and
> break them they will a) feel it viscerally and b) rightly, blame us for
> breaking things. We're pretty serious about maintaining backwards
> compatibility always and treating every API like a promise.

That's _completely_ understandable.

What I'm wondering is whether you could accept it for now, but reach out to
customers you see it from and provide guidance, in addition to blogging about
the issue.

I'm not suggesting "turn it off tomorrow and break people", I'm suggesting
"work towards a long-term plan of turning it off".

~~~
colmmacc
That's probably what we will do. Rightly or wrongly, the new tool will
probably accelerate it because I'm sure we'll be getting support queries from
needlessly worried customers. Sometimes that kind of effort is good, but in
this case I have sad feelings about that result, because the impetus is
misguided. I totally predict that any future EDNS rev will _not_ use a trial-
and-fallback kind of negotiation, making all of this work pointless.

------
tzs
If you have N domains, all served from the same DNS server, you only need to
test one of them with their tester, right? This is about whether or not the
server handles the EDNS protocol, not about anything to do with the content of
any particular domain records, so it either works for them all or for none of
them?

~~~
detaro
yes

------
jaachan
I can't really find a simple explanation of what this means. The warnings in
the EDNS compliance tester aren't really helpful either. Is there a simple
explanation somewhere?

~~~
LeonM
_" Starting February 1st, 2019 there will be no attempt to disable EDNS as
reaction to a DNS query timeout. This effectivelly means that all DNS servers
which do not respond at all to EDNS queries are going to be treated as dead."_

So it basically means that authoritative DNS servers are now required to
support the EDNS protocol. If not, it is no longer guaranteed the domain will
resolve on DNS resolvers.

This is a performance improvement, because the EDNS fallback method requires a
timeout.

Edit: My answer isn't correct, see the comments below.

~~~
Twirrim
What is EDNS, and why would people want it?

~~~
JdeBP
The DNS protocol can be layered over UDP or over TCP. In its original form
DNS/UDP has some quite draconian packet size limits that are reached quite
quickly in the modern world. Originally, this mandated falling back to
DNS/TCP. But TCP is significantly more expensive as a transport protocol,
especially as the client has already had to try to perform the transaction
once over DNS/UDP before falling back to it, and trickier for servers to
implement than DNS/UDP.

EDNS0 ameliorated this greatly, allowing clients and servers to keep talking
DNS/UDP without falling back to DNS/TCP, up to much larger packet sizes.
_That_ is primarily why one would want it, even if one did not want any of the
other things that it incorporates.

~~~
bluejekyll
While packet size is one thing you can do with EDNS, it’s really a mechanism
to allow the DNS protocols to add new features, as there’s no version in the
DNS header.

Also, a lot of DNS hosts do not allow for large packet sizes over UDP to
attempt to reduce the effect of reflection attacks.

~~~
Dylan16807
> Also, a lot of DNS hosts do not allow for large packet sizes over UDP to
> attempt to reduce the effect of reflection attacks.

Do any allow a large response if you send a large (padded) request?

~~~
bluejekyll
Hopefully someone else can answer this, but I think Google will limit the
response UDP packet size to either 512 bytes or the size of the request.

So, yes, I think padding it could work. (EDNS does have a padding type)

~~~
Dylan16807
> (EDNS does have a padding type)

"The use of the EDNS(0) padding only provides a benefit when DNS packets are
not transported in cleartext. Further, it is possible that EDNS(0) padding may
make DNS amplification attacks easier. Therefore, implementations MUST NOT use
this option if the DNS transport is not encrypted."

Apparently it does have a padding proposal, but it wasn't thought through very
well. They only had the use case of confidentiality in mind, and decided to
deal with amplification by _forbidding cleartext use_ , no matter what the
response:request size ratio is.

~~~
bluejekyll
Even in the intended use scenario, I’ve personally found it difficult to
reason about how to append it to the end of a packet.

It basically needs to be the final record, which conflicts with things like
SIG0 that also want to be the final record.

------
runjake
This is going to flop.

I am very familiar with DNS. I run the tests and get minor failures. The test
results barely make any sense and don't really offer good pointers on how to
resolve them.

The documentation is obtuse in typical ISC fashion.

I could probably figure it out with more effort. But the average DNS Joe
won't, they'll ignore this and move on.

------
Kpourdeilami
Testing my domains on CloudFlare gives me:

> Serious problem detected!

> This domain will face issues after the 2019 DNS flag day. It will work in
> practice, BUT clients will experience delays when accessing this domain. We
> recommend you request a fix from your domain administrator!

Weird since they are listed as one of the supporters

~~~
bo0tzz
I had the same result, but after trying again I got an OK.

~~~
bcaa7f3a8bbc
I think the test works by testing querying an authoritative server with EDNS
on, and see if the request times out. Naturally, there will be false positives
due to random packet loss on the Internet.

------
pasta
Wow, EDNS is a DNS extension mechanism first published in 1999.

So I can imagine after 20 year it is time to move on.

When I first read it I thought it was some kind of 'new' tech.

~~~
tialaramex
More over, this isn't about "you have to implement EDNS" (you should, but you
don't have to). This is about "Either implement EDNS or when asked for EDNS,
just say you don't offer that".

This isn't so much "You must offer a vegetarian option" as "Don't offer a
vegetarian option but then when anybody orders it you force a steak down their
throat while screaming 'Choke on it you hippie!'".

------
peterwwillis
Woohoo!!!

The industry's complacency in getting broad EDNS support has been a huge
embarrassment imnsho. This is proof that we don't need to put up with "but the
middle boxes, argh!!?" excuses if the industry works together to force people
to be compatible with standards.

Thanks to all those involved with making this project a reality. It really
helps everyone on the internet, whether they know it or not.

------
eikenberry
Presumably they are not going to remove the workarounds until the software has
actually had a chance to disseminate? For example the resolvers they list are
all new releases and aren't in any current Linux distro. Which means they are
saying that in a couple weeks all Linux resolvers could stop resolving some
domains unless they adopt an unsupported release of the resolver?

They don't list it, but presumably the same applies to the Authoritative
servers. I run a pretty vanilla NSD install and it fails several tests.
NLNETLABS (the authors/maintainers of nsd/unbound) is listed as a supporter,
but they don't mention anything about this on their site nor does their
documentation mention anything about settings to control these features.

If this will actually have the consequences they describe it was terribly
mismanaged.

~~~
eikenberry
Just to clear the record for NSD users... I misread the results and my NSD
setup (4.1.0) does pass the test. It was my secondary at buddyns.com that is
having issues. I'll report the issue to them, but I'm still concerned about
the supported recursive resolvers not being compliant.

------
uniformlyrandom
So, this is just a front-end for
[https://ednscomp.isc.org/ednscomp](https://ednscomp.isc.org/ednscomp), which
is a front-end for [https://gitlab.isc.org/isc-projects/DNS-Compliance-
Testing](https://gitlab.isc.org/isc-projects/DNS-Compliance-Testing).

I kinda get the need for a web frontend for a C program (portability is an
issue), I miss the point of the second front-end. Why?

~~~
codetrotter
Because the OP link is spreading awareness. Your link states nothing about
what it does and probably the output of the one you linked is much less
understandable to the population at large as well.

~~~
uniformlyrandom
Well, I guess 'DNS flag day' is a catchy name, and the text is intentionally
FUD-y, so it has better chance of viral distribution. I just like the concise
summary on ISC page better, needs less scrolling.

------
tony_g
I have conflicting thoughts on this. I'm having a problem with the credibility
of this whole thing. Very few people know about it. Twitter @dnsflagday has 94
followers, telling me it's just an individual account with no public
validation yet. The ISC.org site also doesn't seem to be authoritative.

I understand the technical problem but I'm not convinced this is as big a deal
as it seems. There are some big names associated with the flag day event,
lending credibility. But there are also big names like Slack.com, Twitter.com,
GitHub.com, BitBucket.com, and web hosting companies, that all fail the
EDNS/dig test.

So - the stated threat is that sites will go dark on 1 Feb, but do we really
expect that to happen? Might these companies simply be threatening to reject
invalid EDNS responses? Will people blame Them for rejecting queries rather
than the services being resolved and their DNS?

I can't decide if this is something to scream loudly about, or to just let it
happen. I'm not asking for opinions on that. I'm asking for a credible source
of information to confirm what is truly expected.

Thanks.

~~~
roryokane
The site doesn’t claim that slack.com, twitter.com, etc. will go down when the
flag is flipped. Just try entering those domains into the testing widget
within the “Domain owners” section, and you can see that their status is not
so dire:

> Minor problems detected!

> This domain is going to work after the 2019 DNS flag day BUT it does not
> support the latest DNS standards. As a consequence this domain cannot
> support the latest security features and might be an easier target for
> network attackers than necessary, and might face other issues later on. We
> recommend your domain administrator to fix issues listed in the following
> technical report [link to report]

As other comments have said, the new rule does not require that all domains
support EDNS, merely that all domains respond with “EDNS not supported” when
applicable instead of pretending that they support it. Those sites you linked
follow that rule.

------
breischl
This is a bit trivial, but why is it called "Flag Day"? Because they're
flipping some configuration flag to change the behavior?

~~~
bcaa7f3a8bbc
Flag Day means a software change that is not backward-compatible, especially
in a network system where upgrade is scheduled early and everybody must deploy
upgrade at (before) the date, or the system will stop working. It came into
use when a change was made to the definition of the ASCII character set during
the development of Multics. The change was scheduled for Flag Day (a U.S.
holiday,
[https://en.wikipedia.org/wiki/Flag_Day_(United_States)](https://en.wikipedia.org/wiki/Flag_Day_\(United_States\))
), June 14, 1966.

See Jargon File.

[http://www.catb.org/~esr/jargon/html/F/flag-
day.html](http://www.catb.org/~esr/jargon/html/F/flag-day.html)

------
sandGorgon
which is the cheapest DNS service that allows one to buy "Registry Lock" (not
the simpler registrar lock). I'm talking out-of-band confirmation of any DNS
changes ?

The big ones like Mark Monitor, etc are super expensive. I thought Cloudflare
Business DNS would be cheaper, but they start at 1000-2000$ per year.

Any recommendations ? I have heard of Safenames..but not many others.

~~~
megous
It depends on the TLD. For example for my national domain, I can just send a
request form with my signature to the national TLD registry operator, and
that's it. My domain becomes untransferable, and I also have an option to
disable change of NS set.

It's free, except for the official signature verification, that can be done at
the post office for $1.5.

------
Tepix
How does tinydns deal with it? Does it need patches? Are they available? I
don't see it mentioned.

~~~
JdeBP
tinydns speaks the unextended protocol, but properly handles clients that try
to speak EDNS0 to it.

It does not exhibit any of the faulty behaviours that are common to bad EDNS0
implementations. It does not blindly echo various parts of the query back in
the response, for example. Indeed, the ISC test actually complains that
tinydns _does not_ blindly echo an OPT record back. (-:

But you cannot speak DNS/UDP to it with datagrams over 0.5KiB. Patches are
needed to make it do anything but wholly ignore EDNS (of any version) and
treat it as though it were the unextended protocol. But that is _not_ what
this flag day is about.

Indeed, given that tinydns does not send DNS/UDP responses greater than
0.5KiB, it does not trigger any EDNS0 problems in the network infrastructure
that a server is using, either.

Ironically, one of the people who tried to rewrite djbdns to have EDNS0
support discovered exactly the problem that this flag day _is_ about: The bad
content DNS servers for outlook.com in 2017-2018 gave bad responses to clients
that attempted to speak EDNS0.

* [http://www.nemostar.org/djbdns/changes.html](http://www.nemostar.org/djbdns/changes.html)

~~~
rconti
One pernicious problem I had with tinydns and EDNS support was in an
environment where one firewall, for whatever reason, did not permit tcp/53\.
So DNS replies worked, EDNS0 replies worked, but not from certain clients that
insisted on retrying the query over TCP.

Needless to say, after tons of time getting into the weeds with EDNS, and
STILL not understanding why certain clients weren't working, that was a fun
one to solve.

------
vertline3
My sites said all okay, I guess, what is this looking for exactly?

~~~
zzzcpan
[https://ednscomp.isc.org/compliance/summary.html](https://ednscomp.isc.org/compliance/summary.html)

~~~
vertline3
Thanks sir :)

------
wodenokoto
You might want to test your domain more than once. First I was told my domain
was in trouble, but after retesting I was told everything is a-ok ...

~~~
dttocs
This likely means that your DNS server IP is actually a load-balanced pool of
servers, with some running compliant software and others not. If you see any
errors when running the test repeatedly, I’d check with your DNS provider, and
share the test results.

------
LinuxBender
Minor issues for ycombinator.com [1] They use AWS Route53 so could be false
positive. I see this for anyone using Route53.

[1] -
[https://ednscomp.isc.org/ednscomp/39674a3069](https://ednscomp.isc.org/ednscomp/39674a3069)

------
aidenn0
Is there a simple command-line option I can run to check if the caches between
me and the internet are ready for this?

~~~
oriettaxx
[https://www.isc.org/blogs/dns-flag-day/](https://www.isc.org/blogs/dns-flag-
day/)

------
etbusch
Network Solutions is failing the test currently, anyone have info on when they
might become compliant?

~~~
tcombinator
I've got 2 clients using them and wish I could say this info surprised me, but
they are a terrible registrar & DNS provider. I've asked both of my clients to
reach out to them about the issue but don't have any information past that.

------
CedarHill
Can some ELI5 on this? Not entirely sure what's actually happening.

~~~
JdeBP
Read [http://jdebp.eu./FGA/dns-edns0-and-
firewalls.html](http://jdebp.eu./FGA/dns-edns0-and-firewalls.html) for an
explanation.

What is happening is that all of the stuff that I have been explaining in the
"Local fix" section for a decade and a half, alongside the practice that
clients had of timing out and then _retrying the transaction without the
extensions_ , is now being declared moot. If DNS lookup does not work because
of this problem, it is being declared _entirely the server-end 's problem_.
Clients are no longer going to be expected to put such local fixes in place,
or have timeout and retry bodges.

~~~
kstrauser
As someone who's been required to maintain bug-compatible software before,
this fills me with great joy. Congratulations on getting to deprecate the
brokenness!

------
jcoffland
Tested my Debian based server running bind, all good. Thanks Debian!

------
heyjudy
att.com is broken. Surprise, surprise.

------
rthomas6
Netlify domains get all green

------
Kye
Can I safely assume my Netlify-hosted domains will work?

~~~
LeonM
Depends, if you have hosted the DNS at netlify, then you can assume it is.

If you only run hosting at netlify (thus, having your DNS CNAME to Netlify)
you can still have problems.

Use the checker on the website to know for sure.

------
slackwill
ycombinator.com has some issues ;)

------
patrickg_zill
Hasn't dns been working for the past 25 years?

~~~
detaro
More than that: DNS servers made special exceptions for servers that do not
implement DNS properly and ignore some requests entirely. They're now stopping
that: servers either have to support a 20-year old extension spec to DNS, or
follow the original, 30+ year old specs and answer the query while ignoring
the extension.

