
Ask HN: Is S3 down? - iamdeedubs
I&#x27;m getting<p>{
  &quot;errorCode&quot; : &quot;InternalError&quot;
}<p>When I attempt to use the AWS Console to view s3
======
boulos
Disclosure: I work on Google Cloud.

Apologies if you find this to be in poor taste, but GCS directly supports the
S3 XML API (including v4):

[https://cloud.google.com/storage/docs/interoperability](https://cloud.google.com/storage/docs/interoperability)

and has easy to use multi-regional support at a fraction of the cost of what
it would take on AWS. I directly point my NAS box at home to GCS instead of S3
(sadly having to modify the little PHP client code to point it to
storage.googleapis.com), and it works like a charm. Resumable uploads work
differently between us, but honestly since we let you do up to 5TB per object,
I haven't needed to bother yet.

Again, Disclosure: I work on Google Cloud (and we've had our own outages!).

~~~
JPKab
I've used both Google Cloud and AWS, and as of a year or so ago, I'm a Google
Cloud convert. (Before that, you guys didn't at all have your shit together
when it came to customer support)

It's not in bad taste, despite other comments saying otherwise. We need to
recognize that competition is good, and Amazon isn't the answer to everything.

~~~
vacri
My experience of support with Google Apps for Business makes me _very_ wary of
using anything Google for critical business infra. Google products are nice,
but as soon as you hit a problem or edge case, you're on your own in my
experience.

~~~
spuiszis
Wish I could +1 this more. Any time I get some error, I spend hours sifting
through old documentation and forum posts.

~~~
DerpyNirvash
Why not just open a support ticket?

~~~
foxylad
Exactly. Because they've usually only experienced support for Google's free
services, people assume all Google support is minimal - but it isn't. We pay
$150 a month for silver support, and in the extremely rare (several years
apart) case we need help, we get it.

~~~
vacri
Google Apps for Business is not free.

~~~
foxylad
True, it costs $5 a month - nearly free. It comes down to the eternal truth
that there is no such thing as a free lunch, and expecting a $100/hour support
person to be at your beck and call for $5 a month isn't realistic.

------
cperciva
S3 is currently (22:00 UTC) back up.

The timeline, as observed by Tarsnap:

    
    
        First InternalError response from S3: 17:37:29
        Last successful request: 17:37:32
        S3 switches from 100% InternalError responses to 503 responses: 17:37:56
        S3 switches from 503 responses back to InternalError responses: 20:34:36
        First successful request: 20:35:50
        Most GET requests succeeding: ~21:03
        Most PUT requests succeeding: ~21:52

~~~
kaishiro
Playing the role of the front-ender who pretends to be full-stack if the money
is right, can someone explain the switch from internal error to 503 and back?
Is that just them pulling s3 down while they investigate?

~~~
cperciva
My guess based on the behaviour I've seen is that internal nodes were failing,
and the 503 responses started because front-end nodes didn't have any back-end
nodes which were marked as "not failing and ready for more requests". When
Amazon fixed nodes, they would have marked the nodes as "not failed", at which
point the front ends would have reverted to "we have nodes we can send traffic
to" behaviour.

------
gamache
A piece of hard-earned advice: us-east-1 is the worst place to set up AWS
services. You're signing up for the oldest hardware and the most frequent
outages.

For legacy customers, it's hard to move regions, but in general, if you have
the chance to choose a region other than us-east-1, do that. I had the chance
to transition to us-west-2 about 18 months ago and in that time, there have
been at least three us-east-1 outages that haven't affected me, counting
today's S3 outage.

EDIT: ha, joke's on me. I'm starting to see S3 failures as they affect our
CDN. Lovely :/

~~~
xbryanx
I'm getting the same outage in us-west-2 right now.

~~~
WaxProlix
Same here, and it's 100% consistent, not 'increased error rates' but actually
just fully down. I'd just stop working but I have a demo this afternoon... the
downsides of serverless/cloud architectures, I guess.

~~~
pm90
Well what if you'd hosted it on your hard drive and it crashed? It seems like
the probability of either is similar nowadays.

~~~
jacobwg
The difference there is you can potentially do something about it, vs having
to wait on an upstream provider to fix an issue for everybody.

~~~
btgeekboy
"you can potentially do something about it" vs. "you have to do something
about it"

Perspective is everything.

------
alexleclair
Yup, same here. It has been a few minutes already. Wanna bet the green
checkmark[1] will stay green until the incident is resolved?

[1] [https://status.aws.amazon.com/](https://status.aws.amazon.com/)

~~~
matwood
I always joke that if one of those statuses ever went to red, it means the
zombie apocalypse has begun.

~~~
krylon
I've heard (on the Fnord new show on the most recent CCC congress, so take it
with a grain of salt and a bucket of humor) that Amazon's TOS are more or less
void when a Zombie Apocalypse breaks out.

They had some convoluted but fairly specific wording in their TOS, whoever
wrote must have had a lot of fun.

~~~
jfim
From [https://aws.amazon.com/service-terms/](https://aws.amazon.com/service-
terms/)

> 57.10 Acceptable Use; Safety-Critical Systems. Your use of the Lumberyard
> Materials must comply with the AWS Acceptable Use Policy. The Lumberyard
> Materials are not intended for use with life-critical or safety-critical
> systems, such as use in operation of medical equipment, automated
> transportation systems, autonomous vehicles, aircraft or air traffic
> control, nuclear facilities, manned spacecraft, or military use in
> connection with live combat. However, this restriction will not apply in the
> event of the occurrence (certified by the United States Centers for Disease
> Control or successor body) of a widespread viral infection transmitted via
> bites or contact with bodily fluids that causes human corpses to reanimate
> and seek to consume living human flesh, blood, brain or nerve tissue and is
> likely to result in the fall of organized civilization.

------
cyberferret
Well, at least our decision to split services has paid off. All of our web app
infrastructure is on AWS, which is currently down, but our status page [0] is
on Digital Ocean, so at least our customers can go see that we are down!

A pyrrhic victory... ;)

[0] - [http://status.hrpartner.io](http://status.hrpartner.io)

EDIT UPDATE: Well, I spoke too soon - even our status page is down now, but
not sure if that is linked to the AWS issues, or simply the HN "hug of death"
from this post! :)

EDIT UPDATE 2: Aaaaand, back up again. I think it just got a little hammered
from HN traffic.

~~~
AtheistOfFail
Could be worse, your entire infrastructure could be hosted on Heroku.

You don't use S3 but because they do, your entire infrastructure crumbles.

~~~
bananabill
I didn't realize Heroku used s3 until today, when my heroku app failed. Makes
me wonder why I'm using heroku instead of just aws.

~~~
ddollar
If you're looking for the simplicity of Heroku but want to run on raw AWS
check out Convox. It's open source and free to try.

[https://convox.com](https://convox.com)

Disclosure: I'm one of the cofounders

------
gmisra
FYI to S3 customers, per the SLA, most of us are eligible for a 10% credit for
this billing period. But the burden is on the customer to provide incident
logs and file a support ticket requesting said credit (it must be really
challenging to programmatically identify outage coverage across customers /s)

[https://aws.amazon.com/s3/sla/](https://aws.amazon.com/s3/sla/)

~~~
machbio
thats for below 99.9% - they are at 99.997% .. you are never getting that 10%
credit..

~~~
christop
0.1% of 28 days is 40 minutes, so it seems likely to happen.

~~~
machbio
I was calculating it for a year - maybe the availability applies to per
billing cycle - you may be correct..

------
geerlingguy
From Amazon:
[https://twitter.com/awscloud/status/836656664635846656](https://twitter.com/awscloud/status/836656664635846656)

    
    
        The dashboard not changing color is related to S3 issue.
        See the banner at the top of the dashboard for updates.
    

So it's not just a joke... S3 being down actually breaks its own status page!

~~~
etler
For this kind of page it might be best for them to use a data URI image to
remove as many external resources as possible.

~~~
Symbiote
Unicode characters would work fine, and be even smaller.

Warning sign, octagonal sign, no Entry (all filtered by HN).

There are plenty of possibilities.

------
jliptzin
Thank god I checked HN. I was driving myself crazy last half hour debugging a
change to S3 uploads that I JUST pushed to production. Reminds me of the time
my dad had an electrician come to work on something minor in his house.
Suddenly power went out to the whole house, electrician couldn't figure out
why for hours. Finally they realized this was the big east coast blackout!

~~~
Havoc
Disadvantage of being in the detail I guess. My thinking was Imgur seems
broken today >>> Something major on the intertubes must be fk'd.

~~~
TeMPOraL
Precisely how I discovered it. Imgur down. Imgur is almost like a piece of
critical Internet infrastructure. That + some other site misbehaving tipped me
off that something very wrong is happening...

------
ethanpil
Corporate language is entertaining while we all pull out our hair.

"We are investigating increased error rates for Amazon S3" translates to "We
are trying to figure out why our mission critical system for half the internet
is completely down for most (including some of our biggest) customers."

------
maxerickson
Coincidence?

[https://twitter.com/homakov/status/836649802842591232](https://twitter.com/homakov/status/836649802842591232)

 _I 've been fuzzing S3 parameters last couple hours...

And now it's down._

------
dang
All: I hate to ask this, but HN's poor little single-core server process is
getting hammered and steam is coming out its ears. If you don't plan to post
anything, would you mind logging out? Then we can serve you from cache. Cached
pages are updated frequently so you won't miss anything. And please do log
back in later.

(Yes it sucks and yes we're working on fixing it. We hate slow software too!)

------
greenhathacker
"I felt a great disturbance in the Force, as if millions of voices suddenly
cried out in terror, and were suddenly silenced. I fear something terrible has
happened."

------
chrisan
Down for us as well. We have cloudfront in front of some of our s3 buckets and
it is responding with

    
    
        CloudFront is currently experiencing problems with requesting objects from Amazon S3.
    
    

Can I also say I am constantly disappointed by AWS's status page:
[https://status.aws.amazon.com/](https://status.aws.amazon.com/) it seems
whenever there is an issue this takes a while to update. Sometimes all you see
is a green checkmark with a tiny icon saying a note about some issue. Why not
make it orange or something. Surely they must have some kind of external
monitor on these things that could be integrated here?

edit: Since posting my comment they added a banner of

"Increased Error Rates

We are investigating increased error rates for Amazon S3 requests in the US-
EAST-1 Region."

However S3 still shows green and "Service is operating normally"

~~~
gtsteve
To mitigate the effect of S3 going down I use cross-region replication to
replicate objects to another S3 region. If S3 went down in my primary region I
could update the app config to write back to the backup region and update the
CDN configuration to use the backup region as an origin.

I did that out of paranoia but it turns out this could happen to us. Does that
sound like a sensible approach?

Fortunately all my company's stuff is in eu-west-1 which still seems to be
fine.

~~~
kaishiro
I'm certainly not an expert here, but just to make you feel good (if nothing
else), this is exactly what we did this morning (Melbourne time). Woke up to a
bunch of flailing Lambda funcs on us-east-1. Luckily we're using Apex so cross
deploying them to Singapore took all of 30 seconds. We were concerned about
API Gateway since it was also sitting in us-east-1 but ended up not being an
issue. Realized that redeploying S3 and Lambda across regions can be done in
practically no time, but we would have been in trouble had we needed to
replicated our APIs in another region. Going to start exploring spec'ing up
our existing gateways in swagger to help with this.

------
johngalt
Sysadmin: I can forgive outages, but falsely reporting 'up' when you're
obviously down is a heinous transgression.

Somewhere a sysadmin is having to explain to a mildly technical manager that
AWS services are down and affecting business critical services. That manager
will be chewing out the tech because the status site shows everything is
green. Dishonest metrics are worse than bad metrics for this exact reason.

Any sysadmin who wasn't born yesterday knows that service metrics are gamed
relentlessly by providers. Bluntly there aren't many of us, and we talk.
Message to all providers: sysadmins losing confidence in your outage reporting
has a larger impact than you think. Because we will be the ones called to the
carpet to explain why <services> are down when <provider> is lying about being
up.

~~~
carbocation
People were joking about this but it turns out to be true: they host the
status icons on their service:
[https://twitter.com/awscloud/status/836656664635846656](https://twitter.com/awscloud/status/836656664635846656)

~~~
devy
Due to HN's flaky Cloudflare 503 Bad Gateway error, I noticed that Cloudflare
is also being affected by S3 being down in a similar but subtle way. See their
status page's broken logo on the upper left hand corner.[1] It was actually
directly linking to a S3 URL: [https://s3.amazonaws.com/statuspage-
production/pages-transac...](https://s3.amazonaws.com/statuspage-
production/pages-transactional_logos/retina/7809/JAVqjzc1Q32Y9zajzp4F)

[1]: [https://www.cloudflarestatus.com/](https://www.cloudflarestatus.com/)

~~~
qeternity
EDIT: Misread parent.

~~~
chinhodado
I guess they don't want to host their status page on their own CDN in case it
went down too.

~~~
devy
But only the logo image hosted on S3 is broken though. It seems to be
preventable if they host the logo image together with their status page.

------
jrs235
They don't show it on the status dashboard at
[https://status.aws.amazon.com/](https://status.aws.amazon.com/) (at least at
the time I originally posted this comment).

But if you go to your personal health dashboard
([https://phd.aws.amazon.com/phd/home#/dashboard/open-
issues](https://phd.aws.amazon.com/phd/home#/dashboard/open-issues)) they
report an S3 operational issue event there.

Edit: Mine is reporting region us-east-1

Edit 2: And now the event disappeared from my personal health dashboard too.
But we are still experiencing issues. WTH.

~~~
chickenfries
Also seeing the intermittent event on the personal dashboard. Wonderful.

~~~
jrs235
Apparently they need a special/separate status page for everyone's personal
health dashboard too. SMH.

------
atombender
It's interesting to note the cascading effects. For example, I was immediately
hit by three problems:

* Slack file sharing no longer works, hangs forever (no way to hide the permanently rolling progress bar except quitting)

* Github.com file uploads (e.g. dropping files into a Github issue) don't work.

* Imgur.com is _completely down_.

* Docker Hub seems to be unavailable. Can't pull/push images.

~~~
mcphilip
A doctor's office that's unable to process patients due to the outage:

[https://mobile.twitter.com/drjincali/status/8366578638879989...](https://mobile.twitter.com/drjincali/status/836657863887998976)

~~~
cookiecaper
I mean, that's not really AWS's problem, is it? Outages happen. If you have a
mission-critical service like health care, you really shouldn't write systems
with single points of failure like this, especially not systems that depend on
something consumer-grade like S3.

This appears to be a normal doctor's office where there are routine
appointments. Emergencies would be referred to the ER anyway. And while I
obviously don't know the details of how his office is run, you'd think that
you could get by on a pen-and-paper fallback to manage the office. Maybe
that's an advantage to keeping experienced office staff on board.

~~~
mcphilip
I work in the healthcare industry and there's a big push from AWS into
offering HIPAA compliant services for things like patient records. It's
becoming much more common to tie in third party services into electronic
healthcare software. Obviously no mission critical system should have a single
point of failure and doctor's offices should have fallback plans for handling
service outages, but most care providers don't have staff onsite with the
technical expertise to understand the extent of the coupling. I'm just closely
watching this space and found that tweet interesting in relation to the parent
comment's remark about realizing the scope of this S3 outage. There's no blame
unique to AWS here, but it is becoming an increasingly important piece of
plumbing in the industry.

~~~
inferiorhuman
Fine. But that's just buying into one of the very common misconceptions about
AWS (or any hosting provider), no? This idea that Amazon sells a fault
tolerant product. They don't. Amazon sells you tools that can make a fault
tolerant product, but making your own product resilient is entirely upon you.

------
fletom
what's truly incredible is that S3 has been offline for h̶a̶l̶f̶ ̶a̶n̶
̶h̶o̶u̶r̶ two hours now and Amazon still has the audacity to put five shiny
green checkmarks next to S3 on their service page.

they _just now_ put up a box at the top saying "We are investigating increased
error rates for Amazon S3 requests in the US-EAST-1 Region."

increased error rates? really?

Amazon, everything is on fire. you are not fooling anyone

edit: in the future, please subscribe to @MyFootballNow for timely AWS service
status updates
[https://pbs.twimg.com/media/C5xdm9_WMAAY7y_.jpg:large](https://pbs.twimg.com/media/C5xdm9_WMAAY7y_.jpg:large)

~~~
idlewords
@mikecb on Twitter explained it well. "The red icon is stored in S3 US East."

~~~
scarlac
While that may be true, that's not the reason you're seeing green. You should
have been seeing a broken image or a status page not finishing loading if that
was an issue.

~~~
general_failure
That was (obviously) sarcasm :)

~~~
tonyedgecombe
Humour is wasted on the internet.

------
STRML
It's not just us-east-1! They're being extremely dishonest with the green
checkmarks. We can't even load the s3 console for other regions. I would post
a screenshot, but Imgur is hosed by this too.

~~~
ceejayoz
IIRC the Console is mostly hosted out of US-EAST-1. Direct API calls to other
S3 regions are likely to work, but it's not surprising the Console's having
trouble.

------
rrggrr
Its unreal watching key web services fall like dominoes. Its too bad the
concept of "too big to fail" applies only to large banks and countries.

~~~
elastic_church
"To big [to allow] to fail" is what that term means.

This would extend to a service like amazon actually, where survival of the
service would be an extraordinary effort in case this problem lasted for a
long time.

The way you imagined it, as 100% uptime, is incorrect.

~~~
phil21
I didn't get that at all from the OP. His comment I'm fairly certain is of the
"why the heck are we centralizing the web to 2 or 3 infrastructure companies"
for what amounts to a minor amount of convenience.

We've seen this story play out in other industries and it never works out well
for average people. It's been astounding for me to watch the pace of this
centralizing and who is helping it along.

The tldr; point is that a single service provider should not have the amount
of control Amazon does over the Internet. At least that's my take.

I know my opinion on this wildly differs from the HN crowd and SV "decision
makers" these days - what is so curious to me is that this is a complete 180
from that same demographic even 10 years ago.

~~~
rrggrr
Yes: "why the heck are we centralizing the web to 2 or 3 infrastructure
companies". Its a systemic asset, unregulated, in a world where every systemic
asset is regulated (eg. utilities, transportation infrastructure, etc.).

~~~
witty_username
Regulations means you force your standards on other people who might not need
or want them.

If you want other infrastructure companies or decentralized internet, you are
free to do that yourself via voluntary means.

------
mabramo
Thanks for sharing. I overheard someone on my team say that a production user
is having problems with our service. The team checked AWS status, but only
took notice of the green checkmarks.

Through some dumb luck (and desire to procrastinate a bit), I opened HN and,
subsequently, the AWS status page and actually read the US-EAST-1
notification.

HN saves the day.

~~~
foxylion
On thing I learned here. When something seems horribly wrong, check HN first,
it may be "global" problem.

------
rnhmjoj
Wow, S3 is a much bigger single point of failure than I have imagined. Travis
CI, Trello, Docker Hub, ... I can't even install packages because the binary
cache of NixOS is down. Love living in the cloud.

~~~
stri8ed
Twilio call recordings appear to be missing as well.

------
benwilber0
Notice how Amazon.com itself is unaffected. They're a lot smarter than us.

~~~
yichi
I do recall reading somewhere that Amazon.com isn't actually hosted or fully
leveraging on the AWS platform, mostly due to the political struggle between
the AWS and the merchant department.

~~~
p0rkbelly
There are public talks on Youtube from Amazon.com titled "Drinking our own
Champagne" where they say the opposite.

~~~
krakensden
Yeah, that was the original pitch for AWS. Engineering presentations since the
initial launch have included "yeah... not quite" admissions though.

------
bandrami
And they've just broken four-9's uptime (53 minutes). They must be pretty
busy, since they still haven't bothered to acknowledge a problem publicly...

------
obeattie
Best thing about incidents like these: post-mortems for systems of this scale
are absolutely fascinating. Hopefully they publish one.

~~~
magic_beans
Have they even acknowledged the mortem..?

~~~
obeattie
Yes.

"We've identified the issue as high error rates with S3 in US-EAST-1, which is
also impacting applications and services dependent on S3. We are actively
working on remediating the issue."

~~~
noir_lord
> We are actively working on remediating the issue.

I do love corporate-speak.

It's a rapidly oxidising waste receptacle (rather than a dumpster fire).

------
AndyKelley
This seems like an appropriate time as any... Anyone want to list some
competitors to S3? Bonus if it also provides a way to host a static website.

~~~
ruchit47
If its static site, put it anywhere and just cache it with Cloudflare. (turn
on the always on feature)

~~~
dokument
You also get your ssl sessions backed up on global webcaches as an added
feature.

~~~
angry-hacker
What does it mean?

~~~
ArlenBales
He's referring to CloudFlare's newest feature, CloudBleed:
[https://en.wikipedia.org/wiki/Cloudbleed](https://en.wikipedia.org/wiki/Cloudbleed)

------
140am
[https://twitter.com/aws_shd/status/836635812020158464](https://twitter.com/aws_shd/status/836635812020158464)

"Increased API Error Rates - 9:52 AM PST We are investigating increased error
rates in the US-EAST-1" "S3 operational issue - us-east-1"

~~~
FLGMwt
Would love to know the threshold for "Increased" -_-

~~~
SteveNuts
"more than one"

------
mijustin
Started a list of "things to do when S3 is down."

[https://justinjackson.ca/s3/](https://justinjackson.ca/s3/)

What else should I add?

~~~
vinayan3
Yes. Go for a walk and interact with people in the community. You might meet
someone interesting and learn something as well!

~~~
contingencies
My suggestion would be to rediscover the clarity and focus of thinking about
systems and code on paper.

------
ethanpil
What kills me is that their status page still shows nothing is wrong.

[https://status.aws.amazon.com/](https://status.aws.amazon.com/)

~~~
jeffijoe
Their status page probably can't refresh because S3 is down.

~~~
amazon_throw
correct.

------
valine
Apple's iCloud is having issues too, probably stemming from AWS. Ironically
Apple's status page has been updated to reflect the issue while Amazon's page
still shows all green.
[https://www.apple.com/support/systemstatus/](https://www.apple.com/support/systemstatus/)

~~~
brandon272
I can't stream music from my iCloud library.

------
dfischer
Wow this is a fun one. I almost pooped my pants when I saw all of our elastic
beanstalk architecture disappear. It's so relieving to see it's not our fault
and the internet feels our pain. We're in this together boys!

I'm curious how much $ this will lose today for the economy. :)

------
homakov
Was just pentesting it, and have some minor result. If you are using S3
browser uploads, make sure parameters you supply to Presign do not contain \n
or it can lead to format injection [https://s3.amazonaws.com/doc/s3-developer-
guide/RESTAuthenti...](https://s3.amazonaws.com/doc/s3-developer-
guide/RESTAuthentication.html)

Many aws SDK libs don't remove \n for you.

(I hope it wasn't me who broke it lol)

~~~
buildbuildbuild
"Was just pentesting it" ... hopefully with their permission. Be careful.

~~~
homakov
It wasnt heavy pentesting, just some params jungling. No way it could cause
anything :) still funny coincidence

------
rawrmaan
Incredible how much stuff this affected for me. Opbeat is not loading and I
can't even deploy because CircleCI seems to depend on S3 for something and my
build is "Queued". This seems so dangerous...

~~~
piquadrat
Hi, Beni from Opbeat here. Our community site is indeed down unfortunately,
but the rest should be unaffected. Where do/did you experience issues?

~~~
rawrmaan
Hi Beni! My dashboard wasn't loading (the JS assets from cloudfront.net seemed
to be throwing an access-control-origin error) but it is now working again.
Very slow, but it works. Fortunately nothing critical is happening right now
so no worries. Love your service :)

~~~
piquadrat
Ah right, static assets are an issue, didn't notice it right away due to local
browser caching. Sorry about the trouble!

------
c4urself
It is, of course the checkmark will stay green throughout this as Amazon
doesn't care about actually letting its customers know they have a problem.

------
DenisM
Now might be a good time to ponder a lasting solution. Clearly, we cannot
trust AWS, or any other single provider, to stay up. What is the shortest,
quickest to implement, path to actual high availability?

You would have to host your own software which can also fail, but then at
least you could do something about it. For example, you could avoid changing
things during critical times of your own business (e.g. a tradeshow), which is
something no standard provider could do. You could also dial down consistency
for the sake of availability, e.g. keep a lot of copies around even if some of
them are often stale - more often than not this would work well enough for
images.

~~~
alexbilbie
High availability is improved by hosting in multiple AWS regions.

S3 offers alternative region replication functionality and you can use
Cloudfront of another CDN to load balance between buckets

~~~
DenisM
But do you always serve files from S3? Wasn't the entire S3 down today? Or was
it just some regions? I couldn't even connect to s3.amazonaws.com ...

~~~
bendbro
Only one region, us-east.

------
bandrami
That sound you hear is every legacy hosting company firing up its marketing
machine

~~~
yichi
I've always wondered why people dismiss dedicated hosting without a second
thought. It's actually cheaper than AWS if you factor in all of the
performance you get.

------
remx
Post about S3 not being a CDN hosted on an S3-powered blog:

[https://jdorfman.posthaven.com/medium-bitcoin-660x493-dot-
jp...](https://jdorfman.posthaven.com/medium-bitcoin-660x493-dot-jpg-cdn-
vs-s3)

The irony

------
devy
So S3's been down for at least 3 hours. Does AWS break this year's S3
durability & reliability promise of eleven 9s by now? [1][2]

[1]: [https://aws.amazon.com/s3/details/](https://aws.amazon.com/s3/details/)

[2]:
[https://en.wikipedia.org/wiki/High_availability#Percentage_c...](https://en.wikipedia.org/wiki/High_availability#Percentage_calculation)

~~~
dcosson
That's durability (data loss) not availability.

Here's [1] their official SLA. This outage so far brings them to less than 3
nines of uptime this month (43.8 minutes) but still more than 2 nines (7.2
hours) so it sounds like everyone gets 10% off their S3 bill.

Very curious if Amazon will apply this automatically or only if you complain.

Edit: from further down the same page, it looks like only if you write in to
support do you get these broken SLA credits. Kind of lame since everything
else about their billing is so precise and automatic.

[1] [https://aws.amazon.com/s3/sla/](https://aws.amazon.com/s3/sla/)

~~~
kseifried
I have gotten several credits from AWS for under a dollar due to incorrect
billing calculations on their end that they caught, notified me about and sent
a credit for. So I will assume yes, they do this automatically (if they don't
I'd be quite surprised).

------
caravel
But wait. Isn't S3 "the cloud". Everyone promised the cloud would never go
down, ever. It has infinite uptime and reliability.

Well good thing I have my backups on [some service that happens to also use S3
as a backend].

~~~
djhworld
I know your comment is in jest, but Amazon do say their API SLA for S3 is
99.99% available [1]

[1] [https://aws.amazon.com/s3/sla/](https://aws.amazon.com/s3/sla/)

~~~
hobofan
After more than an hour we are now at ~99.8%, so everybody affected should be
able to claim a 10% discount, right?

------
agotterer
Not sure if its related or not (I'll just assume it is), but dockerhub is down
as well. Haven't been able to push or pull for the last 15 minutes, some other
folks complaining of the same thing.

~~~
notheguyouthink
Yea my self hosted concourse is down. Feels quite bizarre to have our internal
CI so crippled by S3... though i'm still investigating, perhaps the workers
are just hung after all of the failed docker pulls.

------
robineyre
Hi all. I came across this forum on Google. I have the same error - and it's
all a bit beyond me. I'm not a techie or coder but set up Amazon S3 several
months ago to backup my websites and it generally works fine - and has saved
my bacon on a couple of occasions. (Also back up in Google Drive.)

As someone who's really only a yellow belt (assuming you're all black belts!),
just so I understand ('cos I'm cacking myself!) ...

I'm seeing the same issue. Does this mean there's a problem with Amazon? I
can't access either of my S3 accounts even if I change the region, and I'm
concerned it may be something I've done wrong, and deleted the whole lot. It
was working yesterday!!!

Would be massively grateful for a heads up. Thanks in advance.

~~~
z4chj
Yes this is an issue with Amazon and there is little you can do besides wait
until it has been resolved

------
flavor8
> Update at 11:35 AM PST: We have now repaired the ability to update the
> service health dashboard. The service updates are below. We continue to
> experience high error rates with S3 in US-EAST-1, which is impacting various
> AWS services. We are working hard at repairing S3, believe we understand
> root cause, and are working on implementing what we believe will remediate
> the issue.

"Believe" is not inspiring.

------
samaysharma
From [https://status.aws.amazon.com/](https://status.aws.amazon.com/): "Update
at 12:52 AM PST: We are seeing recovery for S3 object retrievals, listing and
deletions. We continue to work on recovery for adding new objects to S3 and
expect to start seeing improved error rates within the hour."

(I think the AM means PM)

~~~
boulos
And now:

> Update at 1:12 PM PST: S3 object retrieval, listing and deletion are fully
> recovered now. We are still working to recover normal operations for adding
> new objects to S3.

------
redm
It looks like the S3 outage is spreading to other systems or the root cause of
the S3 problem is affecting different services. There are at least 20 services
listed now. [1]

[1]: [http://status.aws.amazon.com/](http://status.aws.amazon.com/)

------
talawahdotnet
Yup it looks so. My console says I have zero buckets, my Lambdas are timing
out and [https://aws.amazon.com/](https://aws.amazon.com/) returns a big:

"500 The server encountered an error processing your request." message

~~~
alanning
Our lambda functions are also unavailable. Lucky for us we didn't move any of
our critical functionality to lambda yet although we are planning to once we
have an EC2 backup in place...

~~~
ec109685
Don't know if share911 is still your product (its about page is down), but you
could run your critical services on top of something like PagerDuty to give
you the reliability you need.

------
gaia
Sometimes refreshing the console gives this error instead of showing ZERO
buckets
[https://pbs.twimg.com/media/C5xZVGKUYAAXYGj.jpg:large](https://pbs.twimg.com/media/C5xZVGKUYAAXYGj.jpg:large)

------
vpeters25
Apologizes for the "me too" post:

It appears to be impacting gotomeeting, I get this error when trying to start
a 12pm meeting here:

CloudFront is currently experiencing problems with requesting objects from
Amazon S3.

Edit: ironically, my missed 12pm meeting was an Azure training session.

------
verelo
Years ago when we launched our product i decided to use the US-WEST-2 region
as our primary region and to build fail over to US-EAST-1 (Anyone here
remember the outage of 2011? Yeah, that was why).

There is something to be said about not being located in the region where
everything gets launched first, and where most the customers are not [imo all
the benefits of the product, processes and people, but less risk].

Good luck to everyone impacted by this...crappy day.

------
jedicoder107
Status Pages (Services & Products affected by S3 outage)

\- [https://status.heroku.com/](https://status.heroku.com/)

\- [https://status.aws.amazon.com/](https://status.aws.amazon.com/)

\- [https://medium.statuspage.io/](https://medium.statuspage.io/)

\- [https://status.slack.com/](https://status.slack.com/)

\- [http://status.filestack.com/](http://status.filestack.com/)

\- [http://www.trellostatus.com/](http://www.trellostatus.com/)

\- [https://health.autodesk.com/](https://health.autodesk.com/)

\- [http://status.ifttt.com/](http://status.ifttt.com/)

\- [http://status.imgur.com/](http://status.imgur.com/)

\- [http://status.docker.com/](http://status.docker.com/)

~~~
tyingq
Pretty good list of other affected sites/services in this article:
[http://venturebeat.com/2017/02/28/aws-is-
investigating-s3-is...](http://venturebeat.com/2017/02/28/aws-is-
investigating-s3-issues-affecting-quora-slack-trello/)

Some big names and services popular with HN mentioned there. Quora, AirBnb,
SendGrid, Downdetector(heh).

------
malchow
<% if(service.isUp || true) { renderGreenButton() } %>

------
Animats
Amazon outage just reported on NBC News.[1]

AMZN stock down $3.45 (0.41%).

[1] [http://www.nbcchicago.com/news/national-
international/Amazon...](http://www.nbcchicago.com/news/national-
international/Amazons-Web-Services-Down-Causes-Massive-Outages-
Online-415003823.html)

~~~
Animats
Just reported on USA Today.[1]

[1]
[http://www.usatoday.com/story/tech/news/2017/02/28/amazons-c...](http://www.usatoday.com/story/tech/news/2017/02/28/amazons-
cloud-service-goes-down-sites-scramble/98530914/)

~~~
Animats
Fox News now has the story.[1]

[1]
[http://www.fox5ny.com/news/238689310-story](http://www.fox5ny.com/news/238689310-story)

~~~
Animats
London Daily Express [1], CBS[2] now reporting the outage.

[1] [http://www.express.co.uk/life-style/science-
technology/77332...](http://www.express.co.uk/life-style/science-
technology/773320/amazon-down-web-hosting-services-not-working) [2]
[http://losangeles.cbslocal.com/2017/02/28/amazon-web-
service...](http://losangeles.cbslocal.com/2017/02/28/amazon-web-services-
disrupted-outages-reported-across-multiple-sites/)

~~~
Animats
Reuters, Associated Press, and The Hill now reporting the outage.

"[http://www.isitdownrightnow.com/"](http://www.isitdownrightnow.com/") and
DownDetector are down.

------
ianamartin
This is why it's important to write code that doesn't depend on only a single
service provider. S3 is great. But it's better to set up a Riak cluster on AWS
than to actually use S3, if you can.

The only services my team uses directly are EC2 and RDS, and I'm thinking of
moving RDS over to EC2 instances.

We are entirely portable. We can move my entire team's infrastructure to a
different cloud host really quickly. Our only dependency is a Debian box.

I flipped the switch today and cloned our prod environment, including VPN and
security rules, over to a commodity hosting provider.

Change the DNS entry for the services, and we were good to go. We didn't need
to do anything because everyone was freaking out about everything else being
down. But our internal services were close to unaffected.

At least for my team.

Obviously, we aren't Trello or some of the other big people affected. And we
don't have the same needs they do. But setting up the DevOps stuff for my team
in the way that I think was correct to begin with (no dependencies other than
a Debian box) really shined today. Having a clear and correct deployment
strategy on any available hardware platform really worked for us.

Or at least it would have if people weren't so upset about all our other
external services being down that they paid no attention to internal services.

Lock-in is bad, mmkay?

If your company is the right size, and it makes sense, do the extra work. It's
not that hard to write agnostic scripts that deploy your software, create your
database, and build your data from a backup. This can be a big deal when some
providers are flipping out.

All-your-junk-in-one-place is really overrated, in my opinion. Be able to
rebuild your code and your data at any given point in time. If you don't have
that, I don't really know what you have.

~~~
ec109685
Not knowing your situation exactly, but there could be a cost of running your
own infrastructure and not taking advantage of their services? For example,
are the chances of losing data higher in riak (take into account disaster and
operational bugs that could result in data loss or availability issues) than
in one of amazon's supported data stores.

I don't necessarily disagree with what you are saying but there is cost of
doing everything yourself.

You would have been equally protected if you had been in more than one region.

~~~
ianamartin
Yes. There's a cost to every decision you make. Sometimes it's the cost of s3
being down. Sometimes it's the cost of some developer time to make services
agnostic. The value of that isn't immediately obvious, perhaps.

But the developer cost here (my time) was worth it. Our shit wasn't down,
while everyone else's was.

I also want to point out that I spent minimal time setting this up. We can
deploy to GCE or commodity VPCs at a moment's notice, and that a project I did
over a couple of weekends piggybacking on the ansible playbooks I wrote for
AWS.

It's not that hard. You have to get your developers on board with being
provider agnostic, and you have to be agnostic yourself. But it is not
insurmountable.

It also help when you're the lead dev or your team and also have a good
relationship with the devops guy. :)

~~~
ec109685
You still didn't address the reliability aspect. S3's durability is likely
much higher than other solutions you might chose.

------
vegasje
We're in US-West-2 and our ELBs are dropping 5XXs like there's no tomorrow.
This is definitely cascading.

~~~
soccerdave
We're in US-West-2 and not seeing any issues. Are the instances behind your
ELB trying to access S3 in their application logic?

~~~
vegasje
Nope. No S3 logic behind the scenes. A few of the ELBs are fine, and a few are
not. Seems random.

The EC2 instances themselves are fine, but the affected ELBs are spitting out
500s.

------
huac
Canvas (the educational software platform) is down, and my friends/students
are in bad shape now. 'sso.canvaslms.com' returns 504, assume from this S3
outage.

------
etse
Anyone want to share their real experience with their reliability of Google
Cloud Storage.

------
scrollaway
Down in US-East-1 as of 17:40 GMT. Amazon SES also down in US-East-1 as of a
few minutes later.

Hearing reports of EBS down as well.

~~~
kureikain
EBS is down for 30% of my servers as well

------
oshoma
The status page shows a lot of yellow and red now.

From [http://status.aws.amazon.com/](http://status.aws.amazon.com/) Update at
11:35 AM PST: We have now repaired the ability to update the service health
dashboard. The service updates are below. We continue to experience high error
rates with S3 in US-EAST-1, which is impacting various AWS services. We are
working hard at repairing S3, believe we understand root cause, and are
working on implementing what we believe will remediate the issue.

------
FussBudget86
You think this is bad? Just look at what's happening in Sweden...

------
Fej
Okay, it's been a few hours and this is starting to get ridiculous. When was
the last time that we had a core infrastructure outage this major, that lasted
for this long?

------
kevindong
It really is amazing how many web services are dependent on S3. For instance,
the Heroku dashboard is currently down for me. Along with all of my services
that are on Heroku.

~~~
TravelTechGuy
Same here, but worse. Some of the apps I have hosted on Heroku (including
APIs) are showing "Application Error". Like you, tried logging into dashboard
and got a Heroku error page.

~~~
mcjiggerlog
Same here :(. Not sure why serving a connection to my dyno depends on S3 being
up...

~~~
kevindong
I'll bet that what Heroku does is precompile all the code when you push to
Heroku (or doesn't), saves it to a S3 bucket, and kills the running process
after a certain amount of inactivity. Once it detects a pending network
request, the code gets loaded from the S3 bucket into the EC2 instance, and
then your code spins up.

------
jpwgarrison
I am having trouble sending attachments in the Signal app - seems unlikely,
but could this be related?

[edit- looks like they do have a pretty heavy reliance on S3, per
[https://github.com/WhisperSystems/Signal-
Server/blob/master/...](https://github.com/WhisperSystems/Signal-
Server/blob/master/config/sample.yml) and various other sources.]

~~~
mayneack
[https://twitter.com/whispersystems/status/836651250842124288](https://twitter.com/whispersystems/status/836651250842124288)

------
ganesharul
Sendgrid, Twilio, Quora is also down. Is this related to S3. Entire world
depends on AWS

~~~
yodon
Sendgrid being down particularly hurts - we need it to notify our users of the
problem

~~~
jyriand
That's quite ironic.
[http://isitdownrightnow.com](http://isitdownrightnow.com) is also down.

------
booleandilemma
Thank you, HN, for giving me the answer the AWS Service Health Dashboard could
not.

------
BlackjackCF
Yes. Have heard confirmation from Amazon that this outage is affecting us-
east-1.

------
leesalminen
Now at the top of Drudge [http://drudgereport.com/](http://drudgereport.com/)

------
koolba
I bet the outage is related to the new color coded CloudWatch metrics:
[https://twitter.com/awscloud/status/836630468778864640](https://twitter.com/awscloud/status/836630468778864640)

As part of the release they wanted to make sure everybody gets a chance to see
"red" metrics.

------
mixedbit
'Increased Error Rates' is a bit harsh, couldn't they call it 'Sub-prime
Success Rates'?

------
Globz
My Atom keep crashing and the log says it can't resolve :

[https://atom-installer.github.com/](https://atom-installer.github.com/)

is there a part of this hosted on S3? I cannot open Atom anymore, it keep
crashing on the check for updates screen...

------
newsat13
Yup, same here. For a moment, I was worried that the UI showed 0 buckets. Gave
me a heart attack.

------
Animats
"Inc." is quoting comments from here.[1]

[1] [http://www.inc.com/sonya-mann/amazon-web-services-
outage.htm...](http://www.inc.com/sonya-mann/amazon-web-services-outage.html)

------
l0c0b0x
Google DNS 8.8.8.8 was (for the first time that I've noticed) spotty about 30
minutes ago. Something big is happening:
[http://map.norsecorp.com/#/](http://map.norsecorp.com/#/)

------
dangle
AWS is updating twitter here. No red icons on the status page IS an AWS issue:

[https://twitter.com/awscloud/status/836656664635846656](https://twitter.com/awscloud/status/836656664635846656)

------
linsomniac
We got timeouts to our bucket address from every location we tried starting at
10:37 Mountain time (GMT-7). Slack uploads started failing, imgur isn't
working, and the landing page for the AWS console is showing a 500 error in
the image flipper in the middle of the page. The Amazon status page has been
all green, but there is a forum post about people having problems at
[https://forums.aws.amazon.com/thread.jspa?threadID=250319&ts...](https://forums.aws.amazon.com/thread.jspa?threadID=250319&tstart=0)

In the last couple of minutes that forum post has gone from not existing to
175 views and 9 posts.

------
ayemeng
Funny, status page is incorrect because of S3

[https://twitter.com/awscloud/status/836656664635846656](https://twitter.com/awscloud/status/836656664635846656)

------
rabidonrails
Updated:

Amazon Elastic Compute Cloud (N. Virginia) Increased Error Rates less 11:38 AM
PST We can confirm increased error rates for the EC2 and EBS APIs and failures
for launches of new EC2 instances in the US-EAST-1 Region. We are also
experiencing degraded performance of some EBS Volumes in the Region.

Amazon Elastic Load Balancing (N. Virginia) Increased Error Rates more

Amazon Relational Database Service (N. Virginia) Increased Error Rates more

Amazon Simple Storage Service (US Standard) Increased Error Rates more

Auto Scaling (N. Virginia) Increased Error Rates more

AWS Lambda (N. Virginia) Increased Error Rates more

~~~
amasad
Where are you seeing these updates?

~~~
dlb_
[https://status.aws.amazon.com/](https://status.aws.amazon.com/)

------
joatmon-snoo
According to the personal health dashboard, they've root-caused the S3 outage
and are working to restore.

In the meantime, EC2, ELB, RDS, Lambda, and autoscaling have all been
confirmed to be experiencing issues.

------
nodesocket
Meanwhile engineers across the globe scramble to fix outages due to AWS s3,
$AMZN is unaffected on the stock market. Just shows the disconnect between
emotions and reality.

[https://www.google.com/finance?chdnp=0&chdd=0&chds=1&chdv=0&...](https://www.google.com/finance?chdnp=0&chdd=0&chds=1&chdv=0&chvs=Linear&chdeh=0&chfdeh=0&chdet=1488324845083&chddm=391&chls=IntervalBasedLine&q=NASDAQ:AMZN&ntsp=0&ei=6Ai2WOnrMsax2AaQ1rvwDg)

~~~
IAmGraydon
Why would this affect Amazon's stock? Amazon generates $136B per year. AWS
comprises $8B of that. Even if 5% of customers left because of this (and
that's never going to happen), it would barely create as much as a tiny ripple
in their ocean of revenue. Investors care not about Amazon's cloud services.

~~~
ec109685
AWS's profit helps fund the rest of their business.

------
bdcravens
I was listening to sessions from AWS Re:invent last night. What jumped out at
me was the claim of 11 9's for S3. How many of those 9's have they blown
through with this outage?

~~~
ta_wh
That's a durability target, not an availability SLA. Durability !=
Availability.

~~~
bdcravens
That makes more sense (since I was listening to it in the context of a
conference session, I'm not sure if I heard a distinctive term being used,
though I'm sure they're careful about the language they use)

------
trakl
[http://venturebeat.com/2017/02/28/aws-is-
investigating-s3-is...](http://venturebeat.com/2017/02/28/aws-is-
investigating-s3-issues-affecting-quora-slack-trello/)

[https://www.theregister.co.uk/2017/02/28/aws_is_awol_as_s3_g...](https://www.theregister.co.uk/2017/02/28/aws_is_awol_as_s3_goes_haywire/?mt=1488309539087)

------
dyeje
Experiencing issues with Elastic Beanstalk and Cloudfront as well.

~~~
seibelj
I cannot eb init or deploy to us-east-1

------
jasonl99
I can't download purchased MP3's from amazon's own site, I get "We’re
experiencing a problem with your music download. Please try downloading from
Your Orders or contact us."

When I go to my orders I get "There's a problem displaying some of your orders
right now. If you don't see the order you're looking for, try refreshing this
page, or click "View order details" for that order."

It seems that Amazon is eating its own dog food.

------
splatcollision
I just spent the last hour trying to figure out why in the hell I can't update
the function code on a lambda instance. Next time I will remember to check HN
first!

------
machinarium
Omg I wish I googled this earlier. Wasted hours debugging :(

~~~
SnowingXIV
Yep. I was wondering why my heroku deploys were hanging so I was looking into
every possible issue on my end.

And then I see the news.

------
spacecadets
There goes my Trello to do list. Now I'm lost. Oh well.

------
ryanmarr
My ELBS and EB related instances are also down. I can't even get to Elastic
Beanstalk or Load Balancers in the web console. Anyone else having this issue?

~~~
JBerryMedX
Yes, we're experiencing this issue as well. Are your ELBs also in us-east-1?

------
soheil
It doesn't look that bad, think about it S3 is such a critical part of almost
any web application, it is treated like a realtime micro-service. So looks
like most of the Internet in the U.S. is affected but nevertheless no one is
dead yet and the world has not ended. So even if hypothetically let's say
China attacked us using cyber-warfare it wouldn't be so bad after all... This
was kind of like a test.

------
khamoud
I think this explains why the docker registry is down as well.

[http://status.docker.com/](http://status.docker.com/)

------
sc30317
Still no RCA? I'd love to hear what the issue was for this. A couple of
coworkers and I are betting that it was a networking issue of some sort.

------
Animats
Negative comment on all this in Forbes.[1] Too much centralization. CEOs read
that.

[1]
[https://www.forbes.com/sites/ryanwhitwam/2017/02/28/amazon-s...](https://www.forbes.com/sites/ryanwhitwam/2017/02/28/amazon-s3-outage-
has-broken-a-large-chunk-of-the-internet/#4ec46414c467)

------
robxu9
New update:

"Update at 11:35 AM PST: We have now repaired the ability to update the
service health dashboard. The service updates are below. We continue to
experience high error rates with S3 in US-EAST-1, which is impacting various
AWS services. We are working hard at repairing S3, believe we understand root
cause, and are working on implementing what we believe will remediate the
issue."

------
artur_makly
[https://twitter.com/xiaodown/status/836656364965371904](https://twitter.com/xiaodown/status/836656364965371904)

[https://twitter.com/ArturMakly/status/836665379233628161](https://twitter.com/ArturMakly/status/836665379233628161)

------
fernandopj
Update[1]: AWS Status dashboard now showing icons other than green.
[https://status.aws.amazon.com/](https://status.aws.amazon.com/)

[1]
[https://twitter.com/awscloud/status/836662601090134017](https://twitter.com/awscloud/status/836662601090134017)

------
learc83
One of my heroku apps is down, and I cant' log into the heroku dashboard to
check it out. I'm guessing this is related.

~~~
robeastham
Me too. This couldn't have come at a worse time. Just launched a new site.

------
vinayan3
Yes it's down for me. I can't access files stored on S3. Also, the service I
run is hung trying to store files on S3.

------
JBerryMedX
My company's ELBs in us-east-1 are experiencing massive amounts of latency
causing the instances to be marked unhealthy.

------
leesalminen
FreshDesk makes extensive use of S3 and it's been unbearably slow to load for
the past hour or so. All on S3 requests.

------
rajangdavis
Hate to ask, but does anybody now of an alternative storage solution? Also,
anyone have any alternative to Heroku for now?

~~~
wsh91
Google Cloud Storage (as mentioned) for storage, Google App Engine for PaaS.
:)

(I work on Cloud, specifically Datastore.)

~~~
rajangdavis
Thanks for the recommendation; is there a way to do back ups?

For example, I have images and various assets stored on S3; would there be a
way to change the storage provider on the fly on a website?

The other case is could I have apps hosted on Heroku and set up a service to
duplicate the app code and database over to Google for redundancy? This isn't
super critical as the apps are not customer focused, but they generate content
that is customer focused.

------
janlukacs
We're down too with www.paymoapp.com - pretty frustrated that the status page
shows everything is up and running.

------
poofyleek
This is truly serverless computing at work.

------
xtus
After few requests timed out, started to dig a bit. The CNAME for a bucket
endpoint was pointing to s3-1-w.amazonaws.com with a TTL of at least an other
5600 secods. Doing a full trace was giving back a new s3-3-w.amazonaws.com The
IP related to s3-1-w was/is timing out, all cool instead for the s3-3-w.

------
knaik94
"We’re continuing to work to remediate the availability issues for Amazon S3
in US-EAST-1. AWS services and customer applications depending on S3 will
continue to experience high error rates as we are actively working to
remediate the errors in Amazon S3." Last Update 1:54pmEST

It shows up in the event log now too.

------
samgranieri
I'm running into timeouts trying to download elixir packages, and I'm willing
to bet this is the cause

------
melor
Only limited impact to Aiven services due to service migration capability
[http://help.aiven.io/announcements/aiven-customer-notice-
aws...](http://help.aiven.io/announcements/aiven-customer-notice-aws-us-
east-1-outage-on-2017-02-28)

------
mmansoor78
Per AWS :

For S3, we believe we understand root cause and are working hard at repairing.
Future updates across all services will be on dashboard.

[https://twitter.com/awscloud/status/836666548311859200](https://twitter.com/awscloud/status/836666548311859200)

------
axg
Amazon: The dashboard not changing color is related to S3 issue. See the
banner at the top of the dashboard for updates.

[https://twitter.com/awscloud/status/836656664635846656](https://twitter.com/awscloud/status/836656664635846656)

------
socialentp
Same here. I can log in to the new S3 console UI, but all of my
buckets/resources are missing. Same error as you in the old UI. Also unable to
connect through the AWS CLI (says, "An error occurred (AccessDenied) when
calling the ListBuckets operation: Access Denied"). Fun.

------
gopalakrishnans
Just in case if anybody needs :)
[https://www.microsoft.com/developerblog/real-life-
code/2016/...](https://www.microsoft.com/developerblog/real-life-
code/2016/05/23/S3-Proxy.html)

------
willcodeforfoo
Uh-oh. Same here... and tried taking a screenshot of pinging s3.amazonaws.com
and Slack upload hung.

------
devenrl
Sorry, my simplistic mind is only thinking this right now:

[http://alessandrobender.com.br/wp-
content/uploads/2015/07/fi...](http://alessandrobender.com.br/wp-
content/uploads/2015/07/fix.jpg)

------
eggie5
all of your jokes about the dashboard not turning red b/c the icon is hosted
on US EAST are true:

Amazon Web Services‏Verified account @awscloud 8m8 minutes ago More The
dashboard not changing color is related to S3 issue. See the banner at the top
of the dashboard for updates.

------
Animats
AWS is claiming that Simple Storage (US Standard) is starting to come back up
as of 12:54 PM PST.

------
adamveld12
Where is that "Show HN" that will let me check if a site is affected by an S3
outtage?

------
pfela
Their status page images are hosted on S3, so will be a while for the green
checkmarks to update

------
cdnsteve
Look like the dashboard has been updated to no longer use S3: AWS is having a
major meltdown right now

[http://status.aws.amazon.com/#ecr-us-
east-1_1488312155](http://status.aws.amazon.com/#ecr-us-east-1_1488312155)

------
tudorconstantin
"We have now repaired the ability to update the service health dashboard. " \-
full of yellow red icons now indeed
[https://status.aws.amazon.com/](https://status.aws.amazon.com/)

------
linsomniac
The AWS status page is still showing all green but how has a header saying
they are investigating increased error rates.
[https://status.aws.amazon.com/](https://status.aws.amazon.com/)

------
tzaman
It appears Docker Hub is hosted on S3 as well, none of the official images can
be pulled.

------
ruchit47
I have in the middle of thoughts of moving out of AWS and having a dedicated
provider as our billing has increased a lot with the scale. The only thing
which was holding me was the uptime confidence. Now I feel it's not a bad
idea.

------
jontro
I get this in my aws console.

Increased API Error Rates

09:52 AM PST We are investigating increased error rates in the US-EAST-1
Region.

Event data Event S3 operational issue Status Open Region/AZ us-east-1 Start
time February 28, 2017 at 6:51:57 PM UTC+1 End time \- Event category Issue

------
manmal
Our static site hosted on eu-central-1 is still up:
[http://www.creativepragmatics.com.s3-website.eu-
central-1.am...](http://www.creativepragmatics.com.s3-website.eu-
central-1.amazonaws.com)

------
metafunctor
Based on reports from the field, it looks like S3 was down for about three
hours for most of their customers.

S3 promises four nines of availability (11 nines of durability), so today we
got about 3-4 years worth of downtime in one fell swoop. Oops.

~~~
cowkingdeluxe
Where do you see four nines for S3 SLA?

[https://aws.amazon.com/s3/sla/](https://aws.amazon.com/s3/sla/) shows 99.9%

~~~
metafunctor
It was from memory. I distinctly remember it being advertised as "four nines".
Perhaps they've adjusted their marketing. The S3 FAQ still says [1]: "S3
Standard is designed for 99.99% availability".

[1]: [https://aws.amazon.com/s3/faqs/](https://aws.amazon.com/s3/faqs/)

------
linsomniac
We are starting to see recoveries, our SES emails have mostly gone out and our
data synchronization has updated 2 of our 3 feeds. Amazon has posted a message
that they expect "improved error rates" in the next 45 minutes.

------
nlightcho
[https://twitter.com/awscloud/status/836630468778864640](https://twitter.com/awscloud/status/836630468778864640)

At least now we can see all the network failures in full RGB.

------
tbeutel
I'm getting this using s3cmd:

$ s3cmd ls WARNING: Retrying failed request: / ([Errno 60] Operation timed
out) WARNING: Waiting 3 sec... WARNING: Retrying failed request: / ([Errno 60]
Operation timed out) WARNING: Waiting 6 sec...

------
francesco1975
Yes it is down

[https://status.aws.amazon.com/](https://status.aws.amazon.com/)

Half internet is down the data center in Virginia the one with the cloud is
totally dead apparently. Enjoy the cloud bullshit :)

------
phildougherty
I wrote a quick post discussing this outage. I figured I should share here
[https://blog.containership.io/aws-got-you-
down](https://blog.containership.io/aws-got-you-down)

------
jotaen
Does anyone have trouble with the Cloud Console? The JS assets for the
CloudFront dashboards seem broken, so unfortunately it’s not possible to
change the behaviours of the Distributions (e.g. to point them to another
bucket)

------
mmaunder
So much has broken thanks to this. Web apps, slack uploads, parts of Freshdesk
etc. I don't love you right now AWS.

[https://status.aws.amazon.com/](https://status.aws.amazon.com/)

------
Exuma
Great, all my billing services on Heroku are turned off. Why do they need S3
access for me to access my web dynos?

I'd rather my app load but appear broken so I can show my own status rather
than just shutting down every single app...

~~~
buildbuildbuild
Same here. S3 as a point of failure makes zero sense for dyno uptime, I'm very
frustrated with Heroku. (the dyno was already running, no need to download a
new slug in my opinion)

------
rrecuero
Is anybody else having trouble loading
[http://platform.twitter.com/widgets.js](http://platform.twitter.com/widgets.js)?
It is probably hosted on S3 I assume

~~~
ladybro
Fine for me.

~~~
rrecuero
Thanks. It's back up now :)

------
fjabre
I never understood why so many devs flocked to AWS. I actually find their
abstraction of services gets in the way and slows down my dev instead of
making it easier like so many devs claim it does. I prefer Linode.

~~~
Sanddancer
It lets them pretend that ops isn't a skillset you need people to specialize
in. Just throw more devs and servers at the problem instead of building a good
infrastructure.

------
samat
One of a really rare times when it's good to be in Europe (s3 works here).

------
tjpaudio
Interestingly, I placed an order on amazon.com and while the order appears
when I look at my account, none of the usual automated emails have come. I
wonder how deeply this is effecting their retail customers.

------
pmalynin
Down from the outside; The internal access (from within EC2) APIs still work.

------
awsoutage
[https://twitter.com/sadserver/status/818937064552951809](https://twitter.com/sadserver/status/818937064552951809)

Interesting tweet from last month.

------
jhaile
All of our S3 assets are unavailable. Cloudfront is accessible but returning a
504 status with the message: "CloudFront is currently experiencing problems
with requesting objects from Amazon S3."

------
ibmcloud
Our IBM cloud- Softlayer provides secure and stable cloud environment with
private network, for baremetal,dedicated, private and public cloud. Leave a
comment if you want to learn more. also HIPAA ready.

------
benevol
Here we go again:

Technology leads to technology (and wealth) monopolies, in other words: more
centralization. Which has always been bad.

Just like with Cloudflare leaking highly sensitive data all over the Internet,
a couple of days ago.

~~~
ta_wh
Wouldn't go that far. It's always been the case in the cloud that if you're
not region/provider replicated, you're susceptible to localized outages.

~~~
___start
but the sentiment is valid imo (albeit arguably benign)...

------
andrewfong
Yeah, we host on S3 (US-East-1 I think) with Cloudfront for caching / SSL.
Some of our requests get through but it's been intermittent. Lots of 504
Gateway Time-Outs when retrieving CSS, JS.

------
meddlepal
Totally fucked.

------
cdevs
I think there was some fontawesome loading issues related to this, I also
noticed a site trying to load twitter messages but couldn't Get the JavaScript
loaded during that time today.

------
mpetrovich
Update: AWS dashboard has been fixed and is now showing outages
[https://status.aws.amazon.com/](https://status.aws.amazon.com/)

------
ryanmarr
My EB instances and Load Balancers are also down. I can't even get to load
balancers in ec2 web console or to elastic beanstalk in web console. It's been
almost an hour now.

------
all_usernames
As of 4:30PM Pacific, we're still having trouble with EC2 autoscaling API
operations in US-East-1. Basically very long delays in launching new instances
or terminating old ones.

------
AtheistOfFail
We have a red error, finally!

Source: [https://status.aws.amazon.com/](https://status.aws.amazon.com/)

After two hours, they have finally updated their dashboard.

------
notheguyouthink
Same here, i mistakingly went to the dashboard first too. Silly me.

------
krlkv
S3 is down? Official Twitter feed is also "unaware"
[https://twitter.com/awscloud](https://twitter.com/awscloud)

------
netvisao
Looks like our dashboard is still sustaining it
[https://acedashboard.cbp.dhs.gov/](https://acedashboard.cbp.dhs.gov/)

------
ttttytjj
It's fixed... I mean the status page
[https://status.aws.amazon.com/](https://status.aws.amazon.com/)

------
garindra
DockerHub is down as well. DockerHub was down in Oct 2015 because S3 was down
in US-EAST. They should have known to cache images in multiple S3 regions
since then.

------
zedpm
Can anyone comment on mitigating issues like this with S3 Cross-region
replication? I'm reading up on it now while one of my services is dead in the
water.

~~~
thraway2016
The only appropriate comment is that this issue is affecting all of our
buckets, both in us-west and us-east. Replicating to another region would
yield no useful benefits in this specific failure scenario.

~~~
nolite
Can't agree with this. Buckets in eu-west-1 are fine

------
beeftime
[http://downforeveryoneorjustme.com/isitdownrightnow.com](http://downforeveryoneorjustme.com/isitdownrightnow.com)

RIP

------
rebornix
Alexa smart home component stopped working, if you try to reinstall the Alexa
app on your phone, you'll find that you can't even login anymore.

------
zitterbewegung
Can't access my website which is hosted on s3
([http://joshuajherman.com](http://joshuajherman.com)).

------
bkruse
This is one of the times that I am glad to be running my own distributed
object storage. I'm sure it's not as robust as Amazon, but......

~~~
jasonjayr
What are you using?

~~~
bkruse
Openstack Swift - easy to get setup and reliable. Takes a little expertise but
was well worth the investment in learning it!

------
shifted316
The status page is stored in s3. It can't be updated. The page you see is
cached in cloudfront. They are working on updating the status page.

------
contingencies
What a shame they took down MegaUpload! Clearly we need greater competition in
the wholly-owned-infrastructure, file-hosting-as-a-service space.

------
jefe_
Getting Issues with Citrix Sharefile api (which I've suspected to run in S3).
Seems to only be impacting writes in preliminary assessment.

------
LeonM
Just posted on their Twitter:

"The dashboard not changing color is related to S3 issue. See the banner at
the top of the dashboard for updates."

------
BrandonM
Additionally, Zendesk is apparently failing to process new tickets, so our
users can't report the errors they're encountering.

------
philliphaydon
I'm confused, just logged into work account, and site, and some contract stuff
I do. All use S3 / Cloudfront... no errors...

------
afshinmeh
Heroku as well: [https://status.heroku.com/](https://status.heroku.com/)

------
newman314
There is no cloud, there is only someone else's computer.

:(

------
bkanber
I'm having issues with CloudWatch and related monitoring services; eg auto-
scaling groups are unable to scale up or down.

------
vanpupi
Post your opinion on [http://wp.me/p7HKNy-5h](http://wp.me/p7HKNy-5h)

------
gcoguiec
> We have now repaired the ability to update the service health dashboard.

It seems their status page is hosted ... as a S3 static website.

------
c4urself
> We have now repaired the ability to update the service health dashboard \--
> AWS Status

Well that explains all the green checkmarks /s

------
stevefram
Yes, affecting elb in us-east-1 right now. web services are down and unable to
bring up the elb screen in the aws console.

------
alfg
Yeah, same here on US-WEST-2. Unable to use the S3 Console, but I can still
upload/get content via the API it seems.

------
4wmturner
Is cli working for anyone else? I can't use the console UI, but aws s3 ls and
get commands seem to be working fine.

------
travelton
I can't get to my Amazon Orders page. "There's a problem displaying some of
your orders right now."

~~~
palad1n
Yeah, it looks like that was part of the failure.

------
kyleblarson
Any specific regions? us-west-2 seems fine to me. [edit] now I can't see any
of my buckets in the web interface.

~~~
jrs235
Appears to be us-east-1

------
francesco1975
Yes it is down

[https://status.aws.amazon.com/](https://status.aws.amazon.com/)

------
Globz
Yes Trello is down and they are using S3 :(

------
cwe
Dropbox using this? Can't seem to sync

~~~
rckclmbr
They used to, but i think they are off it now.

------
bas
"Amazon CloudFront: Service is operating normally"

This is bullshit if you're using an S3 origin in your distribution.

------
ignaces
Heroku apps are also down because of this!

------
Svenskunganka
This can't be only the US-EAST-1 region. I'm a european resident and most
things are down for me too.

~~~
samat
I've checked eu-west-1 and it works fine for both reads and writes.

~~~
Svenskunganka
I'm not an AWS customer, but if it is comparable to Google Cloud's Multi-
regional Storage it should be geo-redundant. Doesn't S3 replicate the data
across regions, so in case a region goes offline it won't affect the service?

------
indytechcook
My EC2 Servers are also not provisioning.

~~~
amasad
Same here. I can't find any information about that though.

~~~
partisan
Unable to log into my servers. They are still up and taking traffic, but no
contact. Also unable to provision new servers at this late hour.

------
dgelks
Getting the same error on the GUI but the aws cli and sdk seem to be working
fine (our site is still up too)

------
maccard
My fire tv stick is totally unusable too. Seems I can't access any
applications (even Lodi or Netflix)

------
headcanon
Still down for us. S3 seems to be the only thing affected - our mobile apps
work fine (EC2 and RDS backend)

------
djb_hackernews
Anyone else seeing ELB/ALB issues?

~~~
redthrowaway
Yup. Some of our machines in us-east-1e dropped out of the load balancers.

------
mcheshier
If I listen closely, I think I can hear the pagers going off in South Lake
Union from Downtown Seattle.

------
kardashev
You'll remember me when the west wind moves

Upon the fields of barley

You'll forget the sun in his jealous sky

As we walk in fields of _green_

------
shiven
Status page is lit up like a Christmas tree! Looks like AWS finally found the
myriad non-green icons.

------
dageshi
Huh, I wonder if that's why Origin (EA's Steam competitor) cloud sync just
stopped working

------
skiril
they already admitted it:
[https://www.theregister.co.uk/2017/02/28/aws_is_awol_as_s3_g...](https://www.theregister.co.uk/2017/02/28/aws_is_awol_as_s3_goes_haywire/)

~~~
draw_down
"bit-barn bods" is some of the worst alliteration I've ever seen.

------
murphy52
TierPoint, a large hosting service, is reporting a massive DDOS attack on
their infrastructure.

~~~
ignaces
source?

------
k__
1\. Announce security vulnerability

2\. People push updates as fast as possible to fix security

3\. No tests, so everything blows up

------
balls187
Thanks for posting this. I've passed this information through my network.

Slack image uploads are hanging.

------
happyrock
Anyone doing a region failover? Any issues so far? We are making plans to flip
to us-west-1

~~~
whorleater
We did a failover to us-west-2 and it seemed to work.

------
myth_drannon
Looks like SoundCloud is hosting the tracks on S3 , can't program without my
music...

~~~
yuxt
this still works for me
[https://musicforprogramming.net](https://musicforprogramming.net)

------
JustinAiken
For our app, both S3 and SES have been _completely_ down in us-east-1 for
hours now.

------
thomassharoon
Is S3 down outside of Us-East too? I can't seem to create a bucket in US-West
or EU

~~~
officelineback
Other regions still work, but the web console relies on us-east-1 so you
should use the API to create new buckets until the issue is resolved.

------
KurtMueller
You can always check by going to www.isitdownrightnow.com/

Oh wait. The site sits on S3. Never mind.

------
zerotolerance
Do the engineering thing and build fault tolerant systems. Maybe adopt
features that have been around since 2015:

[https://aws.amazon.com/blogs/aws/new-cross-region-
replicatio...](https://aws.amazon.com/blogs/aws/new-cross-region-replication-
for-amazon-s3/)

------
nicpottier
SES seems to be down for us as well in Virginia. Of course nothing on the
status page.

------
mwambua
Not sure if it's related... but I'm having issues with Amazon Cloud-drive.

~~~
iamdeedubs
I would assume, I couldn't even share a screen shot of my evidence to my team
on slack!

~~~
mwambua
Lol... It's interesting how much depends on Amazon's infrastructure at the
moment.

~~~
vinayan3
Also, interesting everyone is posting that various sites which are down in
this thread. Feels like the internet is down!

------
jsperson
Just finished reading The Everything Store... I bet a "?" email went out.

------
magic_beans
Definitely experiencing non-loading for dependencies hosted on S3 at the
moment...

------
booleanbetrayal
S3 and Elastic Beanstalk (S3 dependencies) ... no issues with RDS at the
moment

------
mystcb
Been unreliability informed about 1 hour ETA for a fix. _fingers crossed_

------
ondrae
We're on AWS GovCloud and our S3 is all good. GovCloud is its own region.

~~~
neom
Shame we pay a bazillion dollars for it. Anyway, curious to know how you've
thought about mitigating something like this? I worry if it happens to us
govcloud users we obviously have much fewer options for redundancy.

------
amcrouch
It appears to be down. My website runs on S3 and my monitors are going nuts!

------
tech4all
Yes serious API problems started about 15 minutes ago. Around noon central.

------
edcoffin
If you've ever felt the AWS health dashboard was dubious before now...

------
oaktowner
Apparently app updates on iOS are failing right now, too. Could be related?

------
Exuma
Yep, currently have over 20,000 people on site seeing no images. Wonderful

------
DocK
Even Kindle books aren't able to be served; download attempts hang.

------
Rockastansky
Anybody else seeing 500 errors with AWS Cognito for us-east-1?

They are consistent for me.

------
murphy52
We host with TierPoint and they are reporting a massive DDOS attack

------
tomharrisonjr
We're seeing queries using Athena against S3 fail in us-east-1

------
reiichiroh
Any truth to this being a DOS by some kiddies named Phantom Squad?

------
framebit
Wow, amazing to watch stuff go down as this problem ripples out!

------
austinkurpuis
Same here. Also having trouble publishing to S3 via CLI and API.

------
Rockastansky
Is anyone else also seeing 500 errors for cognito on us-east-1?

------
kolemcrae
Yup. Every single image on my site is hosted there.... eek! :|

------
exodos
Yah getting the same error in multiple regions as of 1:12 EST

------
twiss
S3 Ireland (eu-west-1) seems to be doing fine at first sight.

------
j_shi
Is there a list of all apps/services that rely on S3?

~~~
mierenga
Would be easier to compile the inverse.

------
gtrubetskoy
they're down to 3 nines

edit: for the year, it only takes 52.57 minutes

------
simplehuman
It's still down. All morning! So much business lost.

------
hyperanthony
Experiencing issues with S3 and ELB for over an hour now.

------
jacobevelyn
We're getting errors indicative of an S3 outage too.

------
afshinmeh
yeah, looks like Travis CI is down, too:
[https://www.traviscistatus.com](https://www.traviscistatus.com)

~~~
freyr
And Trello

------
dorianm
Heroku API/Dashboard is down, Bugsnag is down, etc.

------
dorianm
Heroku API/Dashboard is down, Bugsnag is down, etc.

------
ajmarsh
via AWS twitter account "The dashboard not changing color is related to S3
issue. See the banner at the top of the dashboard for updates."

------
edgartaor
QUESTION. There could be data lost from this failure?

------
Rapzid
Upload failing for me from Sacramento --> us-east-1

------
the_arun
Quora is down too. Getting 504. Gateway Timeout.

Is it related to S3??

------
oneeyedpigeon
Finally! The status page admits something's up.

------
rbirkby
Can anyone get Alexa to play music? Is this related?

------
paulddraper
No, it's not [https://status.aws.amazon.com/rss/s3-us-
standard.rss](https://status.aws.amazon.com/rss/s3-us-standard.rss)

~~~
lancefisher
Looks like the RSS feed hasn't been updated for an hour and a half.

------
carimura
We're seeing recovery across our services now.

------
orn
I'm trying to reach S3 hosted website, no luck

------
ianopolous
I'm seeing the same error on eu-west as well.

------
sz4kerto
Dropbox is down as well. This is going to be gud.

------
manshoor
finally status are updated [https://goo.gl/wCINaC](https://goo.gl/wCINaC)

------
nomadic_09
I'm curious to see the postmortem for this.

------
aytekin
Never depend your business on a single provider.

------
oculusthrift
Yes. and Now my HN profile page is down as well.

~~~
grzm
I've been getting sporadic "Bad Gateway" errors on HN for the past half hour
or so, not apparently associated with any one feature (such as profile pages).
My suspicion is that it's unrelated to anything happening at AWS (HN doesn't
have any AWS dependencies, does it?), though maybe everyone checking HN to see
what the status is has increased load.

------
grimmdude
Making an already troublesome day worse. Yeehaw

------
chiph
I'm seeing problems with Kindle downloads.

------
nickstefan12
bets as to the cause? internal DDoS against their dynamo clusters backing s3?
DNS issues between amazon's services?

------
chx
hex.pm and docker hub are both failing, a lot of projects can't CI because of
these. The house of cards we built.

------
kopy
Looks like they store the statuses on S3

------
Trisell
US-West(Oregon) just went down as well.

~~~
deboflo
us-west-2 (Oregon) is still up for me from the CLI.

------
josephlord
Looks like it. Brief panic caused here.

------
vanpupi
Any opinions you can post on [http://wp.me/p7HKNy-5h](http://wp.me/p7HKNy-5h)
as well

~~~
Exuma
No

------
_callcc
SES also down.

------
outericky
StatusPage.io survived. Thanks gents.

------
jflowers45
trello and giphy both seemed affected

------
uranian
Heroku seems to suffer from this too

------
magic_beans
Dashboard has been updated, finally!

------
nicpottier
SES seems to be down for us as well.

------
nicpottier
SES seems to be downf or us as well.

------
nodefortytwo
Not seeing any errors from eu-west-1

------
rhelsing
Having issues as well.. big issues..

------
SubiculumCode
Mr. Robot live shoot? :)

slack file services down too

------
aabajian
Leap day bug?

~~~
kyled
And yet people think I'm crazy for wanting to wrap get time functions so code
can be tested...

------
soheil
wow even services like Intercom are affected, I can't see who is on my website
right now.

------
sweddle
yeah still all green in AWS status.... maybe their red and yellow icons are
kept on S3. :-)))

~~~
camperman
Many a true word is spoken in jest - that's exactly what the problem is :)

------
sk2code
Now what kind of business choose to remain down for 2 hours plus during the
peak business hours?

Seems cloud computing still has a lot to learn.

------
rhelsing
Is this only affecting US-EAST-1?

------
nvarsj
eu-west-1 is doing great. Obviously European ops are superior to their US
counterparts.

------
bseabra
Same here. We are seeing issues.

------
ARolek
Same here. US East (N. Virginia)

------
mvindahl
We all laughed at the notion om moon people dropping rocks at the earth.

Then they started dropped rocks on S3 and who is laughing now?

------
cryreduce
in the s3 web interface requests to S3 backend end with 503 Service
Unavailable

------
Beacon11
Works for me, in us-west-2.

~~~
joshuahaglund
you sure? it's not for me.

------
the_arun
Same here

------
thepumpkin1979
is it just us-east-1? could it be prevented by using a different region?

------
65827
Dead as a doornail for me

------
danielmorozoff
Yea seeing the same thing

------
SubiculumCode
news.ycombinator.com seems really slow right now. s3 dependencies?

------
Eyes
My website is not down.

~~~
thadjo
same

------
jgacook
Yup - dead in the water

------
baconomatic
Seeing it here as well.

------
qaq
Cmon but the cloud is magic and very reliable let's move everything to the
cloud

------
mrep
quite ironic that 'isitdown.com' is also down

~~~
Humdeee
you should always try 'isisitdown.com/down' first

------
AzzieElbab
Netflix is up. Enjoy

------
Raphmedia
Same here in US EAST

------
xvolter
Seeing the same here

------
julenx
Outage as a Service

------
ahmetcetin
The same here still

------
dbg31415
Yes, appears to be.

------
TheVip
Same problem bro...

------
prab97
Quora is down too.

------
sonnyhe2002
Is it down again?

------
jsanroman
It's down :(

------
0xCMP
SES is also down

------
mtdewulf
Yep, same here.

------
eggie5
is this affecting dockerhub for anyone?

~~~
sklarsa
Yep, can't push new images right now

------
jahrichie
same here, east us seems non-responsive

------
ahmetcetin
The same here

------
methurston
Down for me.

------
kangman
any one get more info from AWS?

------
kangman
what's the SLA for s3?

~~~
nomadicactivist
And my favourite part: "To receive a Service Credit, you must submit a claim
by opening a case in the AWS Support Center." \-- for a company that has built
itself on automation, surely you could automate some bill credits based on the
SLA.

~~~
bdcravens
Many companies don't give out refunds and you have to proactively request.
(alas my employer's value prop, in a completely different industry of course)

------
aarondf
Yes

------
davidsawyer
Same

------
GabeIsman
Yes.

------
dhairya
region-west2 is also down

------
thadjo
heroku API is down for me

~~~
smmnyc
They placed their API into maintenance mode:
[https://status.heroku.com/incidents/1059](https://status.heroku.com/incidents/1059)

------
kfkhalili
So why did the outage occur?

------
davidcollantes
Azure is also down. Related?

~~~
Theodores
What, to Yahoo Mail being down? Can't see the connection myself. Even with HN
I had the twitter status down page before I reloaded five seconds later, in
disbelief... Maybe it is not S3.

------
renzy
getting the same...

------
eggie5
yes, confirmed.

------
simook
yes it is.

------
b01t
yup

------
frik

      Increased Error Rates
    
      Update at 11:35 AM PST: We have now repaired the ability to 
      update the service health dashboard. The service updates 
      are below. We continue to experience high error rates with 
      S3 in US-EAST-1, which is impacting various AWS services. 
      We are working hard at repairing S3, believe we understand 
      root cause, and are working on implementing what we believe 
      will remediate the issue.
    

Amazon hosted their status page on their failing service, ouch. Now they fixed
the status page, after more than one hour.

    
    
      The dashboard not changing color is related to S3 issue. 
      See the banner at the top of the dashboard for updates.
    

[https://twitter.com/awscloud/status/836656664635846656](https://twitter.com/awscloud/status/836656664635846656)

------
vacri
So this is particularly weird - one of my instances was showing 0% CPU in
CloudWatch (dropped from 60% at the start of the event), but the logs were
saying 'load 500'. I ssh'd in... and the problem resolved itself. The only
thing I did was run htop to look at the load, and it dropped from 500
(reported in htop) to it's normal level. Just ssh'ing in fixed that issue.

------
thenewregiment2
soundcloud uses aws s3. it is still down.

------
fletom
are you openly admitting that the AWS service status page runs on AWS? because
that is far more embarrassing than this downtime ever could be

~~~
dang
That's piling on, and rudely at that. Please don't be rude on HN.

We detached this subthread from
[https://news.ycombinator.com/item?id=13756997](https://news.ycombinator.com/item?id=13756997)
and marked it off-topic.

------
soheil
I try not to put all my eggs in one basket, that's why for images I use imgur.
They have a great API and it's 100% free. There is a handy ruby gem [1] which
takes a user uploaded image and sticks it on imgur and returns its URL with
dimensions etc. On top of that you don't have to pay for traffic to those
assets.

[1] [https://github.com/soheil/imgur](https://github.com/soheil/imgur)

~~~
arrty88
imgur doesn't use S3 behind the scenes?

~~~
BoorishBears
It uses it, and it's completely down right now.

