

Amazon EC2 and RDS in US-EAST zone down - akhkharu
http://status.aws.amazon.com/?
multi-az deployments affected too.
======
mikebo
Worst part of this outage: paying for a multi-az RDS instance and having
failover totally, completely, fail.

~~~
keithnoizu
I'm paying like 2,300 a month and even something basic like failover isnt
working. I'm not happy.

~~~
shiftpgdn
At $2300/month you could redundantly colo or lease VERY powerful servers in
3-4 data centers around the country.

~~~
dialtone
Except when you have to factor in all the plane flights to replace broken HDD.
And the risk of not making it in time for when it breaks.

~~~
shiftpgdn
Most colo facilities let you buy hands on time through their techs or include
a small amount per month for things like hard drive/ram swaps.

------
tolos
Every time (two out of two), by the time I click on "X is down" link, the
service/website is working again. Surely there is a better platform for
alerting about outages than ycombinator?

~~~
pjscott
Pingdom does a good job of it, if you point it at a public-facing web site you
particularly care about. I'm not affiliated with them; I've just been woken up
by them.

------
gregholmberg
Individual availability zones can be identified using the API.

    
    
       ec2-describe-reserved-instances-offerings --region
    

will tell you what the zone's identifier is.

After you list the permanent identifiers, you can match them up to find out if
your _us-east-1a_ matches my _-1d_.

This Alestic article shows how to label them all.

[0] "Matching EC2 Availability Zones Across AWS Accounts"
<http://alestic.com/2009/07/ec2-availability-zones>

------
dwhsix
Keep in mind AZs are different per account. My us-east-1b is not necc'ly your
us-east-1b (as someone reminded me on twitter just now).

------
pjscott
EC2 comes with a free Chaos Monkey service. It's called EC2.

I know, they're trying to make it reliable and they've got a bunch of very
hard problems to solve. That doesn't change the fact that sometimes some of my
servers just permanently stop responding to pings until you stop-start them,
or get crazy-slow I/O, or get hit by these once-in-a-while-and-always-at-night
outages.

It's great when you suddenly need a hundred more servers, though.

------
bad_user
I got notified by Pingdom that my domain was down before AWS had any info on
that status page of theirs. IMHO, they should improve on the latency of their
alerts.

~~~
NathanKP
Same here. In fact, the AWS dashboard was still showing 2/2 checks passed for
some 20 minutes after Pingdom told me my site was down.

Then the AWS dashboard finally updated and told me that 3 minutes ago my
instances became unreachable. That is pretty poor. AWS should be able to know
right away and email me themselves.

~~~
RegEx
I've learned to ignore the checks passed for quite a while, especially for
servers on load balance.

------
keithnoizu
I feel like you can't really say you're in the green when you still have
customers unable to use your service. My instance is still stuck in failover.

"9:39 AM PDT Networking connectivity has been restored to most of the affected
RDS Database Instances in the single Availability Zone in the US-EAST-1
region. New instance launches are completing normally. We are continuing to
work on restoring connectivity to the remaining affected RDS Database
Instances."

~~~
gooeyblob
Absolutely agree - that's just silly. Their status page is close to useless.

------
pearle
I'm running in us-east-1 and my EC2 instances and EBS volumes are still
responding ok for the moment...

Fingers crossed (just deployed to AWS less than 2 weeks ago).

------
mattwdelong
It's not entirely down as I can still access my instances. I'm in us-east-1b.

~~~
grourk
Your us-east-1b might be my us-east-1a.

------
KenCochrane
9:32 AM PDT Connectivity has been restored to the affected subset of EC2
instances and EBS volumes in the single Availability Zone in the US-EAST-1
region. New instance launches are completing normally. Some of the affected
EBS volumes are still re-mirroring causing increased IO latency for those
volumes.

~~~
KenCochrane
I'm still seeing issues, some instances that aren't starting, and others I'm
still not able to connect to. So I'm not sure what they are talking about.

------
pearle
Anyone have any details on why us-east-1 seems to be less reliable than the
other regions? Is it the oldest?

~~~
sausagefeet
I'm under the impression it's the most used.

~~~
NoPiece
It probably is the most used, being a cheaper alternative to us-west, but are
you suggesting it fails more because it is used more? It does seem that the
big AWS outages (in the us) have been concentrated in us-east. I have wondered
if it just because us-east is newer so they haven't had has much time to work
things out, or that the us-west team is a little better?

edit: btw, I am not dismissing "used more" as a valid theory. More use = more
hardware = more complexity which could lead to more failures.

~~~
rabbitfang
There are two different us-west regions. One in Oregon (priced the same as us-
east) and one in California.

------
rdl
I'm curious why no public paas is multiple AWS region.

~~~
malachismith
1) because AWS East is so much cheaper (and none of us like spending money) 2)
AppFog actually is multi region (and multi IaaS as well)

~~~
kanwisher
Oregon is same as AWS East, seems to have a smaller set of boxes, have gotten
errors in the past about not having any more servers to allocate.

~~~
malachismith
Same in that they are both AWS and sometimes generate errors - yes. Not the
same in that East has had four significant outages in the last 16 months and
West has not.

------
zedwill
Interesting enough not only the EBS is down, but ELB can not register
instances even if there are not EBS based and completely operational.

I have some live instances running without EBS disks that I can not place
behind the ELB as it is not working.

~~~
oasisbob
_I have some live instances running without EBS disks that I can not place
behind the ELB as it is not working._

ELBs are sometimes EBS backed.

------
DigitalSea
Issue #3298392 for EC2 this month. This is ridiculous, so many websites rely
on EC2 and it's proving to be extremely unreliable. Cloud computing is
definitely not the answer to everything it would seem.

------
stevefink
Cpu0 : 0.3%us, 0.0%sy, 0.0%ni, 0.0%id, 99.7%wa, 0.0%hi, 0.0%si, 0.0%st <\--
EBS subsystem is completely unreachable. I/O wait times are tanked across the
board for me (I'm in US-EAST-1).

~~~
nirvdrum
What zone? I really wish Amazon would provide that info, instead of saying
that it only affects one zone.

~~~
gabrtv
AFAIK zones are randomized. 1a for me is 1d for you.

~~~
mattwdelong
Do you know why this is?

~~~
blantonl
probably to prevent folks from all stacking up in a single AZ.

Just think if someone posted a blog post saying "I've noticed that EBS
performance is far better in 1d vs. 1a"

~~~
iharris
Yeah, for me, 1d experiences the lowest load of all zones. According to the
pricing history for spot instances, 1d experiences the fewest price spikes
compared to 1a and 1b. I'd be interested to see if other users have noticed
the same thing for their zones.

~~~
biot
I find that random(5) is the best performing. For okay but consistent
performance, random(5) is decent, but you should definitely avoid random(5)
due to high load.

------
rabble
Good time to consider Google's Compute Engine as an alternative? What will we
call it, GCE?

~~~
jfoutz
currently, it is a limited beta. Also, it looks to be more expensive.

~~~
malachismith
Actually, if you do the normalization to make it apples to apples (and adjust
for the difference in RAM) it looks price competitive. My numbers make it look
slightly more expensive than AWS EAST (teh suck) and slightly less expensive
than AWS WEST.

~~~
rabbitfang
us-west-2 (Oregon) has identical pricing to us-east-1 (Virginia).

------
mattbillenstein
I suggest until Amazon uses RDS For their database - that you don't either...

------
anuraj
Mine is okay

------
ahmedaly
dotcloud was down also but its now up. (they rely on ec2)

------
ahmedaly
My instances are not down too.. I will back it up now in case things go bad.

------
NathanKP
I am experiencing two out of four instances in us-east-1e unreachable.

------
misiti3780
my instances in us-east-1c are fine

------
malachismith
Goat rodeo.

------
cupcake_death
Yep - Forums are exploding

