

Amazon goes down; takes S3, Salesforce, Target, and others with them. - Sam_Odio
http://www.techcrunch.com/2009/12/23/amazon-down/

======
cperciva
And people told me I was crazy for hard-coding IP addresses for
s3.amazonaws.com and sdb.amazonaws.com into the Tarsnap code... :-)

~~~
akl
You _are_ crazy for doing that - one instance of downtime doesn't justify
ignoring all the advantages DNS brings.

~~~
derefr
Are there DNS servers that support versioning? The best solution I could
imagine would simply be to set Tarsnap to normally use the current DNS records
for S3, but be able to rollback to a valid zone record if they encounter an
update that makes the servers stop resolving.

~~~
cperciva
The Tarsnap client only talks to the Tarsnap server, but I do cache that
lookup in order to avoid problems with glitchy DNS resolution.

I could have the Tarsnap server cache DNS lookups if I was only worried about
working around DNS outages -- but as I said, that wasn't something I was
considering at all when I made the decision to eschew DNS.

------
justinsb
It is DNS. If you put the EC2/S3 address into /etc/hosts, the services work
fine. Affecting lots of other big websites as well apparently (target,
salesforce) because they all outsource to UltraDNS

~~~
aristus
I'm very surprised that they rely on a single vendor. But I guess DNS is one
of those things you don't think about until it fails.

~~~
metachor
The point of using UltraDNS is that it they provide fast "real-time" failover
of DNS routing. This is used, for example, for fault tolerance where the IP
address responding to a domain name might change due to failure scenarios or
load balancing (where a different server is now primary responder to the
domain name, maybe located in a different data center or country).
Infrastructure-wise, UltraDNS is kind of like the Akami of DNS, instead of
content distribution.

------
justinsb
I thought DNS was supposed to try backups servers automatically... any DNS
experts able to explain what's going on? Some of the ultradns servers are
returning (correct) values, others simply not responding.

~~~
lsc
yeah, uh, if you are smart, you have a secondary DNS provider. But that really
requires you managing it yourself. the problem was that many people outsource,
which usually means going with only one provider. (now, ultradns does have a
good setup, they probably aren't a bad choice for a provider, but having only
one is just plain stupid.)

------
boredguy8
Never let truth get in the way of a good headline.

~~~
artagnon
Haha.. true. What else can we expect from Techcrunch?

------
jseifer
It doesn't say in the TC article and I can't really tell from this thread. How
long was Amazon actually down?

------
ggrot
I wonder how much money amazon loses per minute 2 days before christmas. Ouch.

~~~
alain94040
At $20B annual, assuming Christmas is a big chunk, I'm guessing $2B (10%) in
the last two weeks, 2 minutes would then be worth $200,000?

I think my 10% number is too low, so maybe $500K for two minutes? But this is
not profit, just revenue.

~~~
wglb
Do you think it might catch up once it is available again? It is a pretty
serious shopping destination, and possibly people retry.

~~~
robryan
It really depends on downtime, if it was more than say an hour a lot of people
would probably buy elsewhere, small amount of downtime you wouldn't think
would effect sales though.

------
notmyname
surely they mean "Amazon goes down and takes the Internet with it"

~~~
slig
Only if TC was hosted there.

------
Sam_Odio
It looks like a DNS issue, since while <http://amazon.com> isn't working for
me <http://72.21.207.65> is (kind of).

EDIT: "No A records were found for amazon.com"
[http://www.zoneedit.com/lookup.html?host=amazon.com&type...](http://www.zoneedit.com/lookup.html?host=amazon.com&type=A&server=&forward=Look+it+up)

------
mattiss
When Rackspace goes down and takes TechCrunch with them, the whole INTERNET
goes down. When Amazon goes down, Salesforce, Target, and others...

------
tinio
It looks to be back up now.

------
newhouseb
Why wouldn't such a company run their own name servers? I understand it's "yet
another thing to maintain," but I've set up bind before... didn't seem that
bad.

~~~
rbranson
It's relatively trivial to outsource. It's one of those services that's easy
to measure, quantify, and manage (from an outsourced perspective). There's
also a bit more to it then that. The Anycast routing can be quite difficult to
setup and maintain. It's virtually useless outside of a very small set of
protocols (DNS being one of them), so it wouldn't make sense for Amazon to
bring that kind of talent in-house for something like DNS.

~~~
lsc
the mistake was outsourcing to only one provider. it's easy enough to setup a
BIND slave elsewhere that automatically transfers the zone from your primary
provider.

~~~
rbranson
I'm not sure that "setup a BIND slave" is quite that easy for someone with as
much traffic as Amazon.com.

~~~
lsc
I'm sure they could hire the muscle to do it.

I'm just saying, relying on just one company is usually a bad idea.

------
tybris
Internet can be so fragile.

