
IO domain DNS failure - yoshiokatsuneo
For 3 or 4 hours, IO domain DNS lookup often fails. Now, only 2 out of 7 IO domain nameservers respond, and other 5 DNS nameservers are dead.
======
xiaodown
AWS response:

Hello,

Some customers have reported intermittent resolution issues with .io domains.
We can confirm that Route 53 DNS services are operating normally at this time
and these issues seem to be related to the .io top-level domain provider.

We've investigated the issue and are primarily seeing incorrect responses from
two of the .io nameservers: ns-a4.io and ns-a2.io. These nameservers are
returning NXDOMAIN intermittently for domains that do exist. As a result, once
a resolver receives the erroneous response, it will cache the non-existence
for the negative TTL, which for .io is set to 3600 seconds (1 hour).

One suggestion we have is to increase the TTLs for your domains in Route 53 so
that the resolvers cache DNS answers for a longer duration.

Best regards,

(Customer support rep name redacted)

Amazon Web Services

We value your feedback. Please rate my response using the link below.

------
JonAtkinson
I recieved this reply from nic.io support a few minutes ago:
[http://imgur.com/a/wV2Kk](http://imgur.com/a/wV2Kk)

------
a10c
I have been seeing the same failures for many IO TLDs

[https://www.whatsmydns.net/#A/gitlab.io](https://www.whatsmydns.net/#A/gitlab.io)

Example: [https://i.imgur.com/SU4BAHK.png](https://i.imgur.com/SU4BAHK.png)

I have noticed the failures occurring moreso in Asia and Russia than in other
regions.

------
yoshiokatsuneo
So, "ns-a4.io" is the culprit! On ns-a4.io, xxx.io can be resolved, but
yyy.xxx.io cannot be resolved(delegated). Ref:
[http://qiita.com/MasahitoShinoda/items/40cd312fabc6db604b39](http://qiita.com/MasahitoShinoda/items/40cd312fabc6db604b39)

------
mic159
We are also having issues resolving some .io domains from AWS Sydney (ap-
southeast-2)

------
matt_oriordan
For those interested, we wrote a blog post about single points of failure with
DNS last Friday when the Dyn attack happened. Our strategy ensures our
customers are mostly unaffected by issues like this. See
[https://blog.ably.io/routing-around-single-point-of-
failure-...](https://blog.ably.io/routing-around-single-point-of-failure-dns-
issues-7c20a8757903)

------
yoshiokatsuneo
There are 7 IO nameservers (ns-d1.io, ns-l1.io ,ns-y1.io, ns-a1.io, ns-a2.io,
ns-a3.io). Only ns-a4.io and ns-a3.io respond.

------
yoshiokatsuneo
It looks, the the problematic nameserver "ns-a4.io" is down. So, the problem
should be disappeared...

At the same time, it looks, there is only one "ns-a3.io" nameserver working
for IO domain, at now...

~~~
xiaodown
Yep, rather than no such domain or a bad delegation, the "bad" servers seem to
just be down now.

------
jansh
For Planio ([https://plan.io/](https://plan.io/)), all accounts are available
via yourname.planio.net as an alternative while this outage lasts.

------
tkazec
We've been seeing this for exactly five hours now, all across East Asia.

Edit: Looks like .io is managed in London, and their customer service off
hours match the downtime.

------
alwaysanon
Two of their servers were sending out NXDOMAINs which are then cached by
downstream forwarders. It seems they have turned those off now.

------
matt_oriordan
This is still affecting us. Our customers on Heroku seem to be the worst
affected as Heroku is unable to resolve the DNS

------
asher_
We are experiencing the same from Vietnam

------
lifeeth_
.sh and .ac are also run by the same folks - looks like there are issues with
those tlds as well.

------
user9999
Having the same issue in Singapore. got nothing to do, helpless.

------
yjchang
Same here in South Korea. Still not fixed.

------
xiaodown
Getting significantly worse now.

