
Heroku is down again - ardakara
https://status.heroku.com/
======
robbiet480
MASSIVE Storms in VA area where us-east-1 is. 326,000 customers without power
already, worst lightning I have seen in my 20 years of life. Sky is intense
blue/green/purple. This is most likely what the issue is

~~~
prezjordan
No matter how powerful we become as a species with our technology, we are
still at the mercy of the clouds. Pretty cool if you think about it.

~~~
oldstrangers
Or if we just built our power grid underground like rational people.

~~~
Ygg2
Underground cables have many problems like rats (and other vermin) or people
harvesting copper/metals.

~~~
Nitramp
Do they? Here in Germany the entire cabling within cities is underground, only
the high voltage long distance lines are above ground. I've never heard a
story about people stealing underground cables (they do steal e.g. train track
above ground cabling). That also wouldn't make sense, digging up those cables
is much more effort than taking them down from a post.

I've also never heard stories about issues with rats.

Power outages still happen, but they are quite rare - in 30 years I can only
remember twoish.

~~~
adamt
Rats are a common menace with all sorts of cabling. Large parts of scotland
recently lots broadband due to rats eating cables
([http://www.theregister.co.uk/2011/10/12/dirty_rat_downs_virg...](http://www.theregister.co.uk/2011/10/12/dirty_rat_downs_virgin/)).

Apparently it's the insulation on the wires that they like.

------
stbullard
EC2 status:

8:21 PM PDT We are investigating connectivity issues for a number of instances
in the US-EAST-1 Region.

8:31 PM PDT We are investigating elevated errors rates for APIs in the US-
EAST-1 (Northern Virginia) region, as well as connectivity issues to instances
in a single availability zone.

8:40 PM PDT We can confirm that a large number of instances in a single
Availability Zone have lost power due to electrical storms in the area. We are
actively working to restore power.

~~~
joelg87
8:49 PM PDT Power has been restored to the impacted Availability Zone and we
are working to bring impacted instances and volumes back online.

~~~
stbullard
9:20 PM PDT We are continuing to work to bring the instances and volumes back
online. In addition, EC2 and EBS APIs are currently experiencing elevated
error rates.

------
rgarcia
How many times does this have to happen before heroku spreads across multiple
regions?

~~~
fragsworth
You say this like they can just snap their fingers and provide regional
services.

~~~
inopinatus
Well, nothing of scale happens overnight.

But anyone deploying a critical application to AWS makes a point of cross-
region data replication. Heroku have long known that they lose potential
customers to, say, Engine Yard as a result of only hosting at US-East.

One can only conclude that this is a clear business decision on their part. I
can hardly believe that Heroku's engineers are incapable of it. Indeed I would
be very surprised to learn that they _haven't_ brought up an instance of their
platform at, say, US-West, for testing or proof-of-concept purposes.

Of course, productising that is a different matter. Extending the control
plane, front end, and pricing/billing systems might have considerable
associated project cost. Perhaps they have concluded that the costs outweigh
the additional revenue. Or, just haven't got around to it yet.

------
MicahWedemeyer
Can we update the title to something like "AWS US-east-1 is down" instead of
just Heroku?

~~~
sofuture
> informative titles get changed

> uninformative titles left intact

hn 2012

------
Apreche
Why does the AWS dashboard show all green when that is most definitely not the
case?

<http://status.aws.amazon.com/>

~~~
prezjordan
The red check marks are in VA and can't be displayed.

~~~
jc4p
If only they did multi-AZ hosting like they keep telling us to do when there's
outages :)

------
ultrasaurus
Yup, a lot of services served by AWS are having issues. We're seeing a huge
spike in incidents being triggered in PagerDuty.

(fyi: our customers are still being alerted)

~~~
scorpion032
What infrastructure do you use for alerting your customers?

------
devth
My app with 2 dynos is down.

Their status site is running fine altho it's not reporting errors:
<https://status.heroku.com/>

Their Helpdesk is down:
[https://api.heroku.com/helpdesk/login?timestamp=1341025835&#...</a><p>Devcenter
is down: <a href="http://devcenter.heroku.com"
rel="nofollow">http://devcenter.heroku.com</a><p>AWS isn't reporting any
errors: <a href="http://status.aws.amazon.com"
rel="nofollow">http://status.aws.amazon.com</a>

~~~
shuzchen
That's standard. You'd expect them to run everything they have on their own
service except for the status site.

------
ricardobeat
Please link to <https://status.heroku.com/>, pointing to a broken URL is
pointless.

------
darrenkopp
Think that netflix is down too. so much for chaos monkey?

~~~
jaequery
i think netflix ceo just signed up for a rackspace account

~~~
superuser2
Rackspace got hit by a truck last year and went down for a while too. Not
cloud != perfectly reliable.

~~~
davidandgoliath
Slightly different scenario however: the power was shut off by the fire
marshall if I recollect correctly.

Rackspace (and many, many other co's) tend to have functional UPS units &
generators. Amazon tends to choose the cheapest datacenter facility imaginable
and then these sort of failures occur.

Given their size they'll inevitably fix the power issues though -- they've got
the finances & they're capable to add a few levels of redundancy.

~~~
superuser2
I found the reports about the outage - it was 2007 (so obviously much more
than a year ago) but _very_ similar to one of Amazon's recent outages - the
truck took out a transformer, Rackspace fired up backup power, but cooling
failed to start so Rackspace had to shut it all down to avoid melting
everything.

Looks like Amazon wasn't the only one with inadequate testing of their
continuity plan. And I don't think Rackspace offered alternate Availability
Zones at that point.

------
sehugg
There are US-East problems, looks like:

    
    
      - One AZ is down
      - API commands are spotty and may return incorrect results
      - ELB looks screwed
      - IP reassignments don't seem to be working
      - Who knows what the fuck else is broken

------
KenCochrane
Don't think it was just heroku, lots of other sites were down as well,
netflix.com, etc. most likely another AWS issue.

------
reustle
I can't get to netflix.com. That is no good. Luckily all of my 75+ east
servers seem to be ok.

~~~
rorrr
netflix.com is down here as well

------
apawloski
Reddit also seems to be experiencing some difficulties. Are they still on AWS?

------
dragonstyle
We're in AWS East and definitely fighting some issues here, though we're
trying to understand what is happening.

------
drewvanstone
We have dozens of servers that are unavailable at the moment (US-East).
Obviously AWS is having major issues.

------
jc4p
It's been fifteen minutes since our site in US-East went down and AWS Status
hasn't said anything yet.

------
usaar333
Ugh, the EC2 administration console is done. Being in other availability zones
won't save you..

------
heliostatic
As of 10:16 CDT, I can't reach Netflix or Heroku, although AWS status
(<http://status.aws.amazon.com/>) is not yet reporting any current outages.

------
momoro
Aws East connectivity issues, 8:21pm:

<http://status.aws.amazon.com/rss/ec2-us-east-1.rss>

------
pud
This is the motivation I needed to spread my EC2 instances across multiple
availability zones. When the power comes back.

(fandalism is down)

~~~
sehugg
Don't assume that'll save you.

------
aquark
Just started up try using filepicker.io and it seems to be down too.

Beginning to feel pretty lucky though -- this is at least the 4th AWS-East
outage that has made enough of a splash to notice but missed my instances.
Upgrading to multiple availability zones was scheduled for Monday anyway.

------
manishm
Simple solution to this is to have a backup or failover to a non-AWS
Datacenter too, basically don't be just dependent on one Datacenter. E.g. MS
Azure/Google/Rackspace This not only spreads your risks but keeps your
customers happy.

~~~
digeridoo
There's nothing simple about that. :)

~~~
manishm
Yea, but it should be done by providers like Heroku who can talk to multiple
datacenters :)

------
bsaul
Google Appengine's just fine. Dont't know how many AZ i'm on and don't want to
know :)) The more i see about amazon failures the more i think VM are just not
high enough for me in the abstraction layer...

------
_nato_
I have 3 customers -- what the are they going to do, those poor souls!!

------
moskie
Comcast's login server looks like it's down too: <http://login.comcast.net>.
Prevents me from logging into HBO Go. No Netflix either. :(

------
sneaky_weasel
Everything still down for me. Would have expected some redundancy...

------
dnsco
Apparently it's AWS East.

------
coryshaw
Ahh I was just using crunchbase and its now down, must be related.

------
technotony
I just lost a potential hire because of this, was demoing my app to someone
and it wasn't working, she thought it was because of the product. Damn you
heroku!

------
knodi
Come on not another power issue, what happened to the generators... and the
back up generators that they fixed few weeks back.

~~~
excuse-me
The little red ribbon that you pull to get the AA batteries out is stcuk
underneath - they are looking for a pen to flick the battery but since
everyone switched over to Fire tablets there aren't any pens.

------
edouard1234567
idea for heroku : allow customers to host a "my app is down page for blah blah
reason" where they host their status page (rackspace I guess?). Who think this
would be useful? My users see a blank page right now when they go to ZeTrip,
I'd rather show them a static page saying : "our site is down due to amazon
lack of redundancy."

~~~
devicenull
Cloudflare lets you do this afaik. I'm not sure I'd trust a service to show a
proper 'this site is temporarily down' page when something very bad has
happened.

------
fosk
It's not Heroku that is down, AWS is down.

------
kevinprince
We use Heroku for our event tools, thankfully we have nothing live this
weekend or this would be a disaster.

------
RegEx
Loadbalancers are down for me: Getting 'Response contains invalid JSON' upon
attempted termination.

------
edouard1234567
"heroku status" command returns : All Systems Go: No known issues at this
time.

------
philip1209
Cloudflare Always-Up isn't showing on my page - is Cloudflare affected too?

------
jordanthoms
Heroku's uptime for June is going to be.. not so hot.

~~~
bohara
For shissle.

Had to move off Heroku for my latest app. That amount of downtime would put me
out of business.

To bad. I really like the Heroku platform

------
jaequery
did the datacenter get flooded or what? this is just "major" downage.

~~~
apawloski
It's likely related to the power outages across VA

------
mthreat
AWS should call them Unavailability Zones.

~~~
alanh
I had the same thought, but it’s not HN material.

------
_nato_
what a winner! And they charge!! FUX.

------
ahmedaly
I can't believe that guys at heroku are not ready for such situations!

They rely ONLY on virginia's instances because its the cheapest, without
caring about customers.. or thinking of replicating their services in multiple
locations for such issues!

