Amazon S3 & SQS US East outages

[bgentry]

http://status.aws.amazon.com/

[bgentry]

What's the extent of the damage that everybody experienced from this outage?

[blasdel → bgentry]

I was having %90 of my requests time out.

Very little of the software I'm using handled the exceptions raised while this was happening. I checked the status page first thing, but it didn't get set to yellow for another 5 minutes after I started having problems.

[timf → blasdel]

> little of the software I'm using handled the exceptions raised while this was happening

Are saying there is a library you are using that failed to treat a timeout as an issue at all? (if this is for SQS, may I ask what that library is?)

[blasdel → timf]

The aws-s3 gem was throwing an exception correctly after a fairly long timeout ("We encountered an internal error. Please try again.")

But none of the software that was using the aws-s3 gem or causing it to be called caught the exception. I ended up with exceptions getting thrown all over the place, mostly in views from methods on data coming out of models (Rails was catching it at the top-level as ActionView::TemplateError)

[timf → blasdel]

Ah, thanks for explaining more.

[blantonl → bgentry]

I had an archive server go down because of queued up requests to send data to S3. Good lesson for us, we need to handle these types of issues better on our end.

[eclark]

Overall Amazon gets a good grade for their handling of the incident. They responded quickly and fixed everything in short order. ~30mins of downtime.

[seldo → eclark]

A blog post giving a little clarity on the source of the outage would not go amiss, but I guess they're still investigating.

[cperciva → seldo]

They usually don't do blog posts, but do put significant detail onto the status page.

[zacharypinter]

I guess this explains why Unfuddle was down earlier today.

[zzleeper]

In other words, my dropbox went down =/