
.IO domain name reliability issues and how we’re working around them - sbierwagen
https://getstream.io/blog/stop-using-io-domain-names-for-production-traffic/
======
bryanlarsen
By that argument you should also stop using:

.org, .ngo, .lgbt, .asia, .aero, .info, .mobi, .pro, , .MN (Mongolia), .AG
(Antigua and Barbuda), .BM (Bermuda), .BZ (Belize), .GI (Gibraltar), .IN
(India), .ME (Montenegro), .SC (the Seychelles), and .VC (St. Vincent and the
Grenadines), .SG (Singapore) and .HN (Honduras) and more

All are provided or supported by Afilias, the company running .IO.

[https://en.wikipedia.org/wiki/Afilias](https://en.wikipedia.org/wiki/Afilias)

~~~
tbarbugli
Author here: the post is based on my direct experience with .io. It is not
very clear to me how Afilias is involved in the operational side of things,
AFAIK they might have just acquired the existing .io infrastructure and kept
it as it is (and same goes for the way more reliable .org TLD)

~~~
rmetzler
This is a blogpost by Planio about moving from .io to .com

[https://plan.io/blog/post/166816338743/moving-from-planio-
to...](https://plan.io/blog/post/166816338743/moving-from-planio-to-planiocom)

~~~
tbarbugli
> Using .io is unfair to the Chagossian People

> The reason closest to our hearts is the geopolitical background of .io
> domains which we only learned about recently.

Later in the article:

> Of course, we will make sure to redirect all traffic from *.plan.io
> appropriately so no existing links will break in the future.

Which to me sounds like they will keep paying for the .io domain for the
foreseeable future.

~~~
iamthirsty
"I don't like conflict diamonds, but since I have a bunch I might as well keep
them — I just won't wear them."

That's how it comes off to me.

~~~
an_account
If you already have a conflict diamond you already funded the mining effort.
Keeping or throwing it away has no effect.

------
foepys
I never understood why ccTLDs of some third-world countries became popular for
hosting production code. There is a real risk of somebody stealing your domain
due to vulnerable infrastructure and incompetence at the organizations that
sometimes employ less than 10 people. The notable takeover of the .io ccTLD by
Matthew Bryant [1] should have been a wakeup call for everybody.

1: [https://thehackerblog.com/the-io-error-taking-control-of-
all...](https://thehackerblog.com/the-io-error-taking-control-of-all-io-
domains-with-a-targeted-registration/index.html)

~~~
nkrisc
> I never understood why ccTLDs of some third-world countries became popular

Simple vanity.

~~~
tschellenbach
I think it's also about awareness. Most people don't know that IO or other
smaller TLDs are unreliable.

~~~
CamperBob2
They aren't unreliable. This article blows up a single minor incident into a
controversy of epic proportions.

~~~
sokoloff
The incident was far from minor IMO. I think .io has had 6+ hours of downtime
(full or partial/non-trivial) this year, long enough to push the annual
availability down to 99.93% from what I recall from memory of reading post-
mortems.

That was enough to spur us to begin planning to move production traffic off
our .io domain as well.

------
MistahKoala
There is also a good, ethical reason to stop using .io domains completely, as
the territory's population had no say in the governance of the TLD and are
deprived of any revenue from it.

~~~
jandrese
Is it expected that the population of a country would benefit directly from
TLD sales? I have not seen a dime of the proceeds from .us.

I guess the money probably goes into the treasury, but it can't be more than a
drop in the bucket.

I still think TLDs in general were a mistake. It always felt life a leftover
from the old Usenet dominated Internet. The Internet quickly outgrew the
categorization system and we ended up where we are today with most everything
being shoved in .com because it was the least strict.

~~~
pmyteh
The population of the British Indian Ocean Territory were all deported, to
clear the island of Diego Garcia for a large US airbase. It's one of the
grubbiest recent bits of British imperial manoevering.

~~~
Confiks
And "recent" here doesn't only mean "the 1970s". TIL: "In 2016, the British
government denied the right of the Chagossians to return to the islands after
a 45-year legal dispute". [1]

Also, in 2009 an attempt was made to prevent resettlement by declaring the
area a marine reserve. [2]

[1]
[https://en.wikipedia.org/wiki/Chagossians#Court_battle](https://en.wikipedia.org/wiki/Chagossians#Court_battle)

[2]
[https://en.wikipedia.org/wiki/Depopulation_of_Chagossians_fr...](https://en.wikipedia.org/wiki/Depopulation_of_Chagossians_from_the_Chagos_Archipelago#Diplomatic_cables_leaks)

~~~
namelost
Sounds like the UK has effectively given away the atoll to the US. The US is
not going to pack up their very important torture base just because the UK
asks them to. Denying the islanders permission to return is just a face-saving
move to avoid admitting that the UK no longer controls the territory.

------
advisedwang
I wonder if this is a factor in why large tech organizations like Amazon,
Facebook, Google, Microsoft etc have been so keen to register their own TLDs.
Now they only depend on root NS and their own infrastructure, and not _any_
ccTLD provider.

~~~
CydeWeys
Yes, this is one of the major reasons that we registered our own TLDs. It's
definitely not a secret.

Source: I'm the tech lead of Google Registry.

~~~
mmerlin
Any hint of a timeline for .app availability?

[https://www.registry.google/about/domains.html](https://www.registry.google/about/domains.html)

~~~
Veelox
You really expect to hear a product update via HN comment?

~~~
JdeBP
Given the likes of
[https://news.ycombinator.com/item?id=15353027](https://news.ycombinator.com/item?id=15353027)
and
[https://news.ycombinator.com/item?id=15269832](https://news.ycombinator.com/item?id=15269832)
, it is not beyond the bounds of possibility.

------
mschuster91
> In the first case, we would need to keep hundreds of DNS records in sync and
> double our SSL certificates;

The first thing is admittedly a PITA, but SSL certificates should not be a
problem. Either you use LetsEncrypt which automates the pain away anyway or
you do the sane thing and buy a wildcard cert - this has the added advantage
that no one can run a service discovery by simply grepping a CT log. Yes, I
know, security by obscurity, but scriptkiddies will go for the low hanging
fruit first, and having your domains show up in CT logs is ultra low hanging
fruit.

> secondly we would need to only change our infrastructure to not use any
> Route53 specific feature

You should not be locked in to Amazon (or for that matter any Cloud provider)
anyway, given how easy it is to get banned from them (hint: it's enough if
your Google account manages also the Youtube channel and videos on it get
striked too often).

> Using a widely used TLD like .com/.net/.org is the best and easiest way to
> ensure reliability.

Another caveat right here: .com and .net are operated by Verisign, while .org
is operated by PIR. You should always take care to choose a different operator
for the backup TLD!

~~~
tbarbugli
author here: About your second point. We spend quite some money on AWS every
month. As long as we keep paying our bills, I don't see why they would want to
cut us loose.

~~~
mschuster91
> As long as we keep paying our bills, I don't see why they would want to cut
> us loose.

I can imagine a competitor trying to boot you off by bombarding their abuse
team with bogus complaints and triggering automated actions. It's been done in
the past, the problem is that all major companies (not just hosting, but ANY
large company!) try to save on actual customer support and even more on stuff
they can't bill to the customer, like a properly staffed abuse team.

------
feelin_googley
.io domains are perhaps fashionable, but they are expensive.

For $0.88-$0.99/yr one can have a domain in the same registry as the Alphabet
Inc. website's domainname. For that price, it would not be a clever name, but
it could be an easy-to-memorize 6-digit number. What if it is only used for an
API endpoint?

A higher level of " _reliability_ " IMHO could be achieved by use and
publication of a stable IP address, perhaps anycasted if one can afford it. At
least it could be a backup for emergencies, such as DNS failures.

Consider that DNS itself e.g., disseminiation of root.zone, does not rely on
DNS. The IP address for ftp.internic.net is well known and rarely changes. As
I recall, when it does, they notify the public in advance. Some years ago if I
am not mistaken, there was a change from 208.77.188.26 to 192.0.32.9.

Another example is third party DNS providers. They too publish stable IP
addresses. Sometimes users might even memorize them, or store lists of these
addresses e.g. included within installed software.

As a user, I hold no bias against any company that publishes its stable IP
addresses. In fact, on the issue of reliablity I would hold them in higher
regard than those who rely 100% on DNS and third parties associated with DNS
service. DNS is reasonably reliable, but IMO not more so than a stable IP
address.

------
ca98am79
.io is managed by Afilias. They took over and migrated the registry to their
system in June. Afilias manages many other TLDs including:

.adult .ag .archi .bet .bio .black .blue .bz .global .green .in .info .kim
.lgbt .ltda .me .mn .mobi .ngo/.ong .org .pet .pink .poker .porn .promo .red
.sc .shiksha .vc .vegas .vote .voto

They are a very reliable registry operator. This was a bad screw up, and I
guess it had something to do with the migration.

~~~
finnn
>To our surprise, we found out that NIC.io could only be reached via phone
between 7 AM to 12 AM UTC Monday through Friday and did not expose any status
about the health of the service.

That doesn't sound very reliable to me...

~~~
ca98am79
Afilias tech support is 24/7

------
thisisit
> Back when we started in 2014 we decided that .io was great from a branding
> perspective. Stream is a technical product and our audience is mainly
> technical, so .io seemed like a great match. Using the same domain for the
> APIs was more of a consequence than a thoughtful decision.

Quite a lot of tech companies are using .io nowadays. If .io reliability is an
issue what is the next best alternative for tech domain names?

~~~
forgot-my-pw
Some newer TLDs that can be used in tech industry (not sure on their
reliability): .computer, .consulting, .design, .digital, .engineering,
.enterprises, .graphics, .guide, .network, .online, .plus, .productions,
.services, .site, .software, .systems, .tech, .zone

Note that some of these newer TLDs can be more expensive (up to $60/year).

~~~
jmelloy
The ICANN contract, which new TLDs have to sign and country codes do not, has
very strict penalties for having DNS go down for any reason. They're just as
reliable as com.

Disclaimer: I work for Donuts, owner of many new TLDs.

------
manigandham
.com/net/org are also faster, especially internationally, because of more
infrastructure. Algolia found the same during their testing [1].

Use the main TLDs for serving traffic and .io/* for corporate/marketing sites
or webapps if the occasional outage isn't a major problem.

1\. [https://stories.algolia.com/algolia-s-fury-road-to-a-
worldwi...](https://stories.algolia.com/algolia-s-fury-road-to-a-worldwide-
api-c1536c46f3a5)

~~~
dan15
Some ccTLDs have good infra though. Many of the smaller ones (particularly
ones ran by CoCCA, such as .cx) use Dyntld or Packet Clearing House which have
pretty good servers and connectivity.

------
stefanukchagos
Hello, I volunteer with UK Chagos Support Association, a voluntary Chagossian
support group that Plan.io and other .io firms and users have donated to since
this issue was raised in a tech magazine a year or two ago.

Someone Twitter tagged us and linked to this discussion and it's great to see
so much support for the community.

Couple of links on the background below.

[http://archive.chagossupport.org.uk/index.php/category/io/](http://archive.chagossupport.org.uk/index.php/category/io/)

[https://gigaom.com/2014/07/02/seats-io-donates-money-to-
chag...](https://gigaom.com/2014/07/02/seats-io-donates-money-to-chagos-
islands-charity-over-io-domain-use/)

A couple of .io start ups set up a site to encourage others to support us and
other Chagos-related support groups which allowed us to a lot more of our work
- campaigning, supporting community projects and issuing crisis grants - over
the past few years. Called 'The Dark Side of .Io,' it's actually offline now
but the owner has assured me it'll be back on shortly.

I've no expertise to comment on technical issues. On the moral side, I've not
met anyone in the community who has a serious issue with firms using the .io
domain name - most are impressed that so many firms have chose to back the
Chagossian community out of their own pocket. As someone mentioned below, as
recently as one year ago the UK government refused to allow Chagossians to
return to the Chagos Islands. The community will not see any money from the
renewal of the lease on their homeland for use as a military base.

These are the main issues, and while the .io domain name is somewhat symbolic
of the wider exploitation of the Chagossian people - others profiting from
their homeland with the community itself seeing no benefit - there is at least
a positive side to this, which is harder to find in the government's decision
to continue the exile.

If anyone is interested in finding out more please see our website (not .io -
but that's out of cost and ignorance factors more than ethics)

www.chagossupport.org.uk

We're actually planning a bit of a revamp of the site shortly and I'd be
remiss if when posting here I didn't ask for anyone interested in getting
involved in that - contact@chagossupport.org.uk

------
ca98am79
"Due to its decentralized nature..." should read "Due to its CENTRALIZED
nature..." If DNS were decentralized, like for example Bitcoin, so many
problems like this would never happen.

~~~
numbsafari
True... until "devops199" randomly gives you 150 million reasons otherwise.

------
lcurole
"In July 2017 a security engineer from Google was able to buy the domain of
one of the authoritative nameservers (ns-a1.io) and gain control of every .io
domain."

Ouch.

------
hodgesrm
Dumb question, but why not host DNS name serving somewhere other that NIC.io
so it does not go down? As I recall you can set the name server to live
somewhere else, though I don't have the account pages open.

The dnscheck.pingdom.com page shows that .io domains commonly route to
different name servers. Just try stream.io and slither.io and you should see
different authoritative name servers. (I picked those randomly for the test.)

~~~
mikepavone
DNS resolution is hierarchal. You start at the root DNS servers for a TLD to
find the authoritative servers for a given domain in that TLD. You can host
your authoritative servers wherever you want, but you have no control over the
root servers and it was the latter that had problems.

~~~
hodgesrm
Thanks, that clears things up.

------
henvic
Cross-posting a message I posted about one year ago because I am still mad
about what happened 7 years ago:

"In 2011 I paid for and registered o1.io (I really wanted 01.io, but back then
it was not available to register domains containing only numbers on .io). The
.io NIC web interface is really horrible and I ended up in an inconsistent
state after making the "horrible" mistake of clicking the back button. Even
though my Paypal account got charged, I received an email confirming I was the
owner of the domain, and so on, I couldn't access my account. Next step I took
was contacting them. I got ignored and after 7 days later they made a
transaction reversal on Paypal and I never heard a word back from them, even
though I have tried to contact. Some time later they made the o1.io domain a
reserved one and so if I wanted it now I would have to pay thousands. What a
shame."

[https://news.ycombinator.com/item?id=12980040](https://news.ycombinator.com/item?id=12980040)

------
timc3
For anyone recommending .com one of the simple factors that it isn't often a
good choice is people sitting on domain names for profit or other reasons.

So either you have to think up a completely ridiculous name for your new
service/product/company, pay stupid amounts or use something other than .com

~~~
wonderwonder
use .io or similar for any customer facing websites but a .com for any mission
critical api type calls. Customer never even has to know about the existence
of myUglyCompanyNameApiHandler.com but all api calls are handled by it.

~~~
OhHeyItsE
It's a good idea, but if your product is a website, cookies and authentication
in general are going to be a real challenge. Obviously you won't be able to
share cookies between the site and the api. But you have the additional
challenge of having your api domain treated as 3rd party and are thus subject
to all the security measures and sandboxing that come with that.

And what if your product is just an api for use by developers? Now the api is
part of your brand.

It's just a bad situation all around.

~~~
dan15
> Obviously you won't be able to share cookies between the site and the api.

I don't know of any API that requires cookies from the site. Usually you use
an access token or something similar for an API.

> But you have the additional challenge of having your api domain treated as
> 3rd party and are thus subject to all the security measures and sandboxing
> that come with that.

CORS solves most of the pain points. You just need to ensure the API is
serving the correct CORS headers.

------
elliotanderson
We migrated from serving production JS assets off our _.io_ domain to _.com_
after we found a number of corporates/schools in Ireland blocking the domain
at the TLD level.

Since then, the number of inexplicable error reports has dropped dramatically.

------
andyhmltn
It's hardly surprising sadly. Looking back through my submission history
there's two instances of incompetence by the .IO handlers:

1) Storing passwords in plain text: Although the post is now down, it pointed
to the fact that Nic.IO will email you your password in plain text. Something
which everyone that's spent more than a few weeks coding should know not to
do.

2) Another pretty major outage back in 2013 that rendered two of my sites
offline.

I personally stopped using them after my last IO domain lapsed but this should
serve as a stark warning to anyone thinking they can pick up a cute IO domain

------
gator-io
Remember this:
[https://www.icann.org/news/announcement-2017-09-27-en](https://www.icann.org/news/announcement-2017-09-27-en)

I think .io was part of the issue for the delay.

After the last outage, we at gator.io took it as a 'all hands on deck'
firedrill to get our api off .io. The problem is that many of our customers
have scripts on their sites with the old .io endpoint. Migrating them is very
difficult.

------
ryandrake
If these are just API endpoints (not typed in by users, and thus, no Marketing
value to the name) why not just register <random hash>.com?

~~~
tschellenbach
That's what we do :) It's ok for the marketing page to go down every now and
then. Not so nice for the API to become unavailable. (200 million end users)

------
zx2c4
WireGuard similarly moved from the .io to wireguard.com in July:
[https://lists.zx2c4.com/pipermail/wireguard/2017-July/001569...](https://lists.zx2c4.com/pipermail/wireguard/2017-July/001569.html)

Less trendy, but otherwise, zero regrets.

~~~
hoodoof
Nice, if you have the .com

------
andyonthewings
I guess that's why the domain name infrastructure should be decentralized.
Though I'm not familiar with the tech involved, does blockchain based things,
like namecoin, solve this problem?

------
45h34jh53k4j
.io represents the suppression of the Chagossian people.
[https://en.wikipedia.org/wiki/Chagossians](https://en.wikipedia.org/wiki/Chagossians)
English imperialism at its best.

Do not use this unethical TLD: [https://gigaom.com/2014/06/30/the-dark-side-
of-io-how-the-u-...](https://gigaom.com/2014/06/30/the-dark-side-of-io-how-
the-u-k-is-making-web-domain-profits-from-a-shady-cold-war-land-deal/)

~~~
gmiller123456
By those standards, can you provide a TLD that is not the result of some
atrocity or another?

~~~
45h34jh53k4j
.cx? (Christmas Island), which is a Australian territory? Maybe your right. I
think .io is especially egregious because of history of Chagossian people.

------
quirkot
It's not my area, but it seems like if your application is deeply dependent on
an API it might be a good idea to directly reference the IP address? Over
1000s of requests, wouldn't that save time?

~~~
mikeash
It would be slightly faster. However, if you ever had to change your server's
IP address (for example, because you had to move to a new hosting provider)
you'd be utterly screwed.

------
quocble
Said by the company that uses .io

~~~
andyhmltn
I mean if you read the article the reason they say not to use it is
specifically because they used it themselves and were burnt

------
rdiddly
TL;DR - company follows trend of misusing top-level domain of the British
Indian Ocean Territories, finds it unreliable and not-so-great, starts the
wheels in motion to switch away from it.

Addendum: Mind you I applaud their coming forward to report this. Hopefully
people get the message. It's not the first time fashion got the better of a
large number of people.

------
yeukhon
Sorry, but the title is click bait. Can people just write "Our IO domain
Resolution Failed, A Lesson Learned"?

In the "What Really Went Wrong" section, the author wrote:

> it does not take a lot of research to find out that the .io TLD team made
> several mistakes

and went on to cite two incidents, which are not exclusive to .IO domain. So
let's not blame ".io domain team" and only happen to .IO domain, try to
convince me (at least the way I read it based on the title) that using .io
domain is a bad idea. I should stop using .com then.

For me, I was looking for a "so we are going to switch away from .io domain in
the next year or so" at the end of the post, because .io is not good according
to the author. The plan instead is just add a backup domain, so it looks like
the author is eating his/her own words, even though the critical stuff are
going to run over .com. So let's just switch everything over, what's the big
deal for your user-facing website not over .com? Do people really think a big
deal now about .io vs .com when you have established a business?

Anyway, I do appreciate when a postmortem is available because I can learn
something new, but I do hope we write postmortem with an objective tone. So
no, if there is a downvote button on HN for the submission, I would downvote.

