
Ask HN: Can you get to AWS? - cornellwright
We&#x27;re having lots of trouble getting to many AWS&#x2F;Amazon resources in US-West2 and it seems like many other people are reporting problems with US-East. Currently we cannot even reach http:&#x2F;&#x2F;status.aws.amazon.com. What are other people seeing and does anyone have any pointers to what the problem could be? There seems to also be a bit of discussion on Twitter: https:&#x2F;&#x2F;twitter.com&#x2F;hashtag&#x2F;aws?src=hash&amp;vertical=default&amp;f=tweets#
======
colinbartlett
I often find out about these sorts of global internet issues early because of
a little project of mine:

[https://statusgator.io](https://statusgator.io)

It just monitors the status pages of lots of different services. When I get 30
notices from various unrelated services at once all with comments like
"Network connectivity issues", I know some kind of routing issue is plaguing
the 'net.

~~~
nthitz
What if statusgator.io or my routing to SG goes down? :)

~~~
vacri
Same solution as every monitoring system: run a second, separate one. Whether
it's the proverbial backhoe through the fibre link or a sysadmin fat-fingering
a conf or deploy, there's always a way for a single system to be brought down.

I saw an absolutely crazy way to do this with Nagios[1]. Nagios saves its
state in a file, and the suggested way to synchronise two separate systems was
to have them look to see if the other one is active. If it is, sync the
statefile across. If it isn't, start up nagios locally based on the latest
sync'd statefile. I mean, the solution works, but due to Nagios it's
inherently hacky. It's bizarre that Nagios doesn't (didn't?) have inbuilt
support for failover this way.

[1][https://allmybase.com/2010/10/04/setting-up-fully-
redundant-...](https://allmybase.com/2010/10/04/setting-up-fully-redundant-
failover-nagios-servers/)

------
mvelie
Looks to be because of:
[https://twitter.com/Axcelx/status/616058414746202113](https://twitter.com/Axcelx/status/616058414746202113)

------
bks
I run a spam filtering company in AWS-Oregon and we are under a DDoS that is
directed to some of our financial services clients...We know that our client
is the target because the Russian team claims responsibility opened up a
trouble ticket with us (not hosted in AWS) about 2 hours ago to let us know
that they would DDoS us and the network.

I am sure it's just a coincidence :-)

~~~
JohnHaugeland
It appears that it is. This appears to have been a BGP problem.

[https://twitter.com/Axcelx/status/616058414746202113](https://twitter.com/Axcelx/status/616058414746202113)

------
_jomo
There's another post on HN:

"EC2 us-east is down" \-
[https://news.ycombinator.com/item?id=9809304](https://news.ycombinator.com/item?id=9809304)

------
verelo
"Between 5:25 PM and 6:07 PM PDT we experienced an Internet connectivity issue
with a provider outside of our network. The issue has been resolved and the
service is operating normally. "

[http://status.aws.amazon.com/](http://status.aws.amazon.com/)

I really hate how AWS use PDT for their status page. Seriously, is there any
logical reason to use anything other than UTC? I find it means when there is a
report, I first need to convert the PTD time shown to UTC, then to local time
just so I can know if the issue is possibly the one I experienced. Unless
you're also in PDT, couldn't they just save everyone else one step by showing
all service issues in UTC?

~~~
jonas21
Whichever time zone you pick, everyone not in that time zone is going to have
to do a conversion. So why not pick the time zone where most of Amazon's
engineers, and quite possibly most of their customers' engineers, are located?

~~~
joshribakoff
PDT differs from PST. Conversion logic can get especially confusing if you've
specified a date time, and then the laws surrounding day light savings time
are changed to occur on a different day. Using UTC is like using a unix
timestamp, you don't have to worry about a lot of things like locales & local
laws, when the date time was recorded vs when its being viewed & what laws
might have changed in-between. For example the date/time that daylight savings
takes effect was changed in 2007. If you're trying to compare old logs to an
old AWS announcement, it could get confusing... whereas UTC should remain
immune to changes in laws.

~~~
verelo
This is exactly why I want UTC timestamps.

------
cgtyoder
Note at the top of
[http://status.aws.amazon.com/](http://status.aws.amazon.com/):

Internet connectivity issues

We are currently monitoring an external Internet provider issue that is
causing interrupted service connectivity to AWS services for some customers.
AWS services are not affected and continue to operate normally.

------
danieljurek
I can't get the status page to load:
[http://status.aws.amazon.com/](http://status.aws.amazon.com/)

Edit: Scratch that. Some stuff I'm hosting in AWS East is still working. Maybe
someone screwed up router configs on the west coast.

Edit: Rollback the scratch. It's oscillating between up and down right now.

------
NeutronBoy
Status page updated:

"We are currently monitoring an external Internet provider issue that is
causing interrupted service connectivity to AWS services for some customers.
AWS services are not affected and continue to operate normally."

[http://status.aws.amazon.com/](http://status.aws.amazon.com/)

------
aws_ls
I woke up to see two of my instances, running in AWS Singapore, at 100% and
unresponsive. Rebooting also did not help. They had to be killed. Thankfully
AWS had documented this on their support page:

"Additionally, some customers have reported continued connectivity issues for
some of their instances. We have seen with these reported issues that this has
been caused by a leap second bug within the instance operating system, which
results in 100% CPU utilization. We recommend rebooting the instance via the
EC2 Management Console or API, or resetting the operating system time to
resolve the issue. For further information see:

[https://access.redhat.com/articles/15145"](https://access.redhat.com/articles/15145")

------
ytjohn
[https://blog.thousandeyes.com/route-leak-causes-amazon-
and-a...](https://blog.thousandeyes.com/route-leak-causes-amazon-and-aws-
outage/?utm_source=news&utm_medium=outagereport&utm_campaign=blog)

------
richadams
A fiber cut in Oregon could be responsible
([https://puck.nether.net/pipermail/outages/2015-June/007906.h...](https://puck.nether.net/pipermail/outages/2015-June/007906.html)).
I've been seeing connectivity issues with us-west-2 for most of the day.

There was also a fiber cut in San Francisco area this morning
([http://www.usatoday.com/story/tech/2015/06/30/california-
int...](http://www.usatoday.com/story/tech/2015/06/30/california-internet-
outage/29521335/)).

------
jstratr
I can reach [http://status.aws.amazon.com/](http://status.aws.amazon.com/)
from my home Comcast connection in SF. The status page is showing green for
all services.

------
mondoshawan
Looks to be something strange with the network. Using a SOCKS proxy to a host
running on he.net, I can get to US-East machines, but not from my Cogent-based
connection in the office.

~~~
cornellwright
Yeah, that's what we're seeing. We can finally get to US-West2 again from our
office. Employees who were working from home didn't experience an outage at
all.

------
ra1n85
[https://twitter.com/Axcelx/status/616058414746202113](https://twitter.com/Axcelx/status/616058414746202113)

Yuck.

------
fuziontech
I'm able to tunnel into ec2 over our VPC and get to most things that are down
like slack and netflix. Amazon.com is still unreachable

------
Tekker
Leap second posts?

------
t3f
For what its worth, I usually start at [1].

[1] [http://internetpulse.keynote.com/](http://internetpulse.keynote.com/)

------
kirk21
Nope. A ton of websites don't work.(even amazon.com)

------
sergiotapia
I've been having issues all week reaching a lot of different AWS websites
including atom.io.

Switching my DNS to 8.8.8.8 fixed it for me. Something is up though.

~~~
halayli
faced similar dns issues using the default name server that ec2 points to. We
ended up running bind.

------
imroot
I just got an email from abovenet -- apparently they had a fiber cut and
they're having major routing issues affecting the san francisco area.

------
heavymark
Yes, it's been suffering from the massive load of launching Beats1. Beats1
from Apple went down, but looks like it's back up finally.

~~~
clebio
Citation?

~~~
mgingras
[https://www.apple.com/ca/support/systemstatus/](https://www.apple.com/ca/support/systemstatus/)

~~~
rgbrenner
and what makes you think that apple uses AWS? Considering they've invested
billions in building data centers for themselves in NC and CA.. and they've
building two new ones in EU, and one in OR.
[http://www.datacenterknowledge.com/the-apple-data-center-
faq...](http://www.datacenterknowledge.com/the-apple-data-center-faq/)
[https://www.apple.com/pr/library/2015/02/23Apple-to-
Invest-1...](https://www.apple.com/pr/library/2015/02/23Apple-to-
Invest-1-7-Billion-in-New-European-Data-Centres.html)

------
ikeboy
I'm connecting fine to us-east.

