
Amazon's homepage was down - nbashaw
http://www.amazon.com/?down=yes
======
largehotcoffee
Unfortunately changing the URL from <http://www.amazon.com/?down=yes> to
<http://www.amazon.com/?down=no> does not appear to fix the problem.

~~~
up_and_up
This seems to be working for me: <http://www.amazon.com/?is_it_jeff=yes>

~~~
nbashaw
Looks like it's back up

~~~
amitlan
it is up.

~~~
HelloMcFly
Not for me (America).

~~~
shardling
Up for me (Kentucky)

------
evanjacobs
During my time as an engineer working on Amazon.com, we occasionally
experienced outages of various lengths. One of the surprising details about
these outages is that they really didn't result in any revenue loss. That is,
it appeared that customers would simply wait until the website was available
again to make their purchase. I would be surprised if that effect doesn't
still happen today especially with the availability of Amazon on a variety of
platforms (i.e. customers are comfortable ordering from their phones when they
couldn't get to the website from their desktop computers).

~~~
sneak
This is really insightful and sort of flies in the face of the research about
Google's page latencies affecting search volume (even subconsciously).

Thanks for sharing a non-obvious data point.

~~~
magicalist
It was actually someone from Amazon themselves, reporting on a/b testing they
did, that gave us the numbers of 100ms of latency cost them 1% in sales.
People might wait for Amazon to come back, but if it's slow to navigate back
and forth to compare brands, or slow to return search results, or slow to just
render the page, you can easily imagine losing some percentage of people or
just some percentage of what you could have sold to them.

<http://news.ycombinator.com/item?id=273900>

------
georgemcbay
"by nbashaw 28 minutes ago"

I thought for sure I'd have missed it and this would be one of those reports
where the service was back up before the story gained traction, but as of
12:07 PM Pacific/US time I cannot navigate to Amazon's home page..

The amazing thing about this for me is that it reminds me that it was only a
few years ago that even the biggest sites would have fairly frequent multi-
hour outages, but these days it is pretty rare for this sort of thing to
happen, particularly on a retail or otherwise direct-money generating site.

~~~
dos1
On the plus side, if they're willing to share, I bet this will be a very
interesting postmortem. Presumably Amazon.com is one of the more bulletproof
web properties in the world. Whatever could have occurred to take it down for
nearly an hour (at this point) can only be interesting!

~~~
nirvana
I can't compare to other web properties, but when I worked at Amazon, the
store going down was a regular event. Something broke almost daily, though it
was rare for the whole store to go down. (EG: You might not be able to search,
or checkout might be down, etc.)

The store went thru periods of relative stability, and relative lack of
stability, and in the periods where it was not doing so well, it (or a major
piece of functionality) would go down in some key area at least once a week,
sometimes multiple times a week during the holidays.

While it's been several years and I'm sure they've improved reliability, the
sheer mass of the store made it very slow to evolve. And as an ex-amazonian
sometimes I go and check for bugs that were issues back in the day- several of
them have come back over the years, which is not surprising given that the
entire group that was working on the parts I was working on disbanded because
so many people were driven off by bad management. (A one-two punch in that
case, a bad manager backed by another bad manager, neither of which had any
technical knowledge.)

At the time I worked there, large swaths of code in the store had no team who
was responsible because the team had been disbanded in one of the regular
shuffles of employees. Amazon had a tendency to get a team together to do a
feature, launch it, get the PR and the stock bump, then disband the team and
put them on other projects. Of course some of these things stuck around if
they were successful, but there was a lot of cruft from past efforts like:
Local restaurant menus, the movie times system, various "social shopping
features" (a perennial favorite to try again and again.) Hell, they used to
have catalogs for mail order merchants- scanned paper catalogs!

At the time, they were claiming that "AWS is what we built the amazon store
on!" (which was totally false, S3 was engineered completely separately from
the store, and to its credit, as obidos and gurupa were crap. The only thing
the store shared with AWS for at least the first several years was being
hosted in some of the same datacenters.)

At least at the time I worked there, I'd call it a mess held together by the
code equivalents of duct tape and bailing wire.

One of the things Amazon excels at is customer service, so when these problems
would impact the customer, their bacon was often saved by customer support
fixing the problem manually (eg: messed up orders, etc.)

Granted, operating at Amazon's scale is not trivial matter. But Amazon is a
retailer and stock marketing company (Eg: one of their primary products is
Amazon stock), more than an engineering company.

I'm kinda amazed that people perceive them as a "tech giant" along with
Google, Facebook and Amazon. Shows the power of a good (actually, _GREAT_ )
side business like AWS. They get the credit for building something good and
scalable with AWS, but of course it was a separate team lead by a senior
executive with enough political clout to shelter that team.

~~~
gokulk
'I'm kinda amazed that people perceive them as a "tech giant" along with
Google, Facebook and Amazon ' err.. we are talking about Amazon here

~~~
InclinedPlane
Amazon is a weird company, and it has lots of parts. Even at, say, Microsoft
there can be a huge amount of variation from division to division and team to
team on how things are run, the corporate culture micro-climate, etc. At
Amazon this is even more true, each team is substantially on their own, and
while there is a certain amount of global overarching corporate culture every
group is different and some groups buck against the trend successfully.

------
adrr
Maybe they should consider hosting in the cloud.

~~~
mixedbit
They should definitely try AWS. It is ridiculous that a simple online store
manages its own infrastructure.

~~~
hayksaakian
AWS was actually UP all this time, so perhaps that's not a bad suggestion.

~~~
thspimpolds
Wow. You really don't know why AWS exists. AWS is Amazon.com.

[http://www.zdnet.com/how-amazon-exposed-its-guts-the-
history...](http://www.zdnet.com/how-amazon-exposed-its-guts-the-history-of-
awss-ec2-3040155310/)

As for why, it's easy. Taxes. Much like Walmart rents its stores form an LLC
it owns to write off the taxes and bring down the liability of the largest
revenue sector, Amazon can write off their server costs since they can "rent"
them from AWS LLC. While AWS makes a good chunk of change, it has nothing on
amazon.com so by making AWS its own entity (and event better for them that its
publicly available) they get a gigantic tax write off and AWS makes capex
expenditures saving them taxes. All In all, the shell game must save amazon
millions just like it does for Walmart

~~~
_frog
I think you might have missed a joke there.

------
nbashaw
Sorry for the weird query string (?down=yes), but HN already had a amazon.com
submission

~~~
film42
You beat me to it by 30 seconds.

~~~
peterkelly
I was 400ms too late :(

~~~
mkopinsky
Or he was 400ms too early...

------
edanm
Interesting. Never thought I'd see that.

Does anyone have statistics for Amazon homepage uptime? I don't remember the
last time I heard about Amazon being down.

And an hour after I read Patrick's (patio11) article on the Rails
vulnerabilities. It's a scary day indeed.

~~~
dhosek
It's actually surprising it isn't down more often—internally, everyone has
write access to prod and the rule is that if you deploy something to prod you
need to be able to roll it back.* Apparently, though, someone has failed on
the second item.

* Or so I was told in a job interview with the big A a few years back.

~~~
georgemcbay
I find this extremely hard to believe. (Not calling you a liar, but I think
something must have gotten lost in translation).

The possibility for theft and fraud would be so massive if every dev at Amazon
had write access to production that I find it nearly impossible to believe
this is true.

~~~
dsl
Developers probably have access to most production systems. Credit card
processing and source of truth on orders that get shipped are most likely
segregated. (actually PCI dictates that physical and data access controls be
in place so only essential employees can access card data)

~~~
georgemcbay
Who cares if I can access the credit card processing system if I can insert
random code elsewhere in the system that redirects you to my phishing page
whenever you enter credit card information?

~~~
greghinch
Given that you would be an Amazon employee with a solid audit trail leading
back to you in that scenario, I'd say it's pretty likely you'd be caught and
prosecuted rather quickly.

~~~
stcredzero
Yes, I and my coworkers could've sold the realtime trades of a petroleum
multinational to the highest bidder, including ones that hadn't happened yet.
That would've been easy, and would've been worth 100's of millions to someone.
Not getting caught and having your life ruined -- that was scary and would've
been hard. Now, if I was working for a sovereign power, like China, and my
life was there anyhow, then pulling stuff like that in the US wouldn't be so
hard.

~~~
greghinch
When did amazon start selling petroleum futures?

~~~
stcredzero
Amazon is not a petroleum multinational. Guess again.

------
pjungwir
Interesting for all those people chasing "five nines": If 45 minutes today is
their only downtime for the year, their annual uptime for 2013 will be just

    
    
        1 - 45/(60*24*365)
    

or 0.99992.

~~~
wikwocket
"5 nines" works out to about 5 minutes of downtime a year: very challenging to
achieve. For reference, 4 nines is about an hour, and 6 nines is only ~30
seconds of downtime a year!

~~~
pestaa
Lucky I have no need to do this math! Our requirements is 100%!

------
danvideo
it looks like just homepage

all the internal links seem to be working fine <http://www.amazon.com/gp/site-
directory/>

edit: added less ugly link

~~~
nbashaw
Good catch - title updated accordingly

------
philwelch
Cue HN admin changing submission title to "Amazon.com: Online Shopping for
Electronics, Apparel, Computers, Books, DVDs & more" in 3...2...1...

(cf. <http://news.ycombinator.com/item?id=5142851>)

~~~
lucb1e
Less cynicism on HN might be a good thing, too.

~~~
stcredzero
I'm going read that as ironic. (Or, did I write that...?)

------
atlbeer
How many $/seconds do you think the homepage being down costs?

~~~
dangrossman
Since Amazon's retail operations are unprofitable, they're actually gaining
money.

~~~
dbaupp
Is this seriously true?? (If so, where do they make their money?)

~~~
ef4
It's true that their profit margins can be vanishingly thin, but that doesn't
mean they don't make money.

For some classes of items, they can sell at cost and still make money, because
their operations are allegedly so good that they can turn over the inventory
before their own payment to the supplier is due.

For example, say Amazon buys a book today and payment is due to the publisher
in 30 days. They sell the book tomorrow at cost. Now they get to sit on the
full price of the book for the rest of the month. In fact, take that money and
buy another book, and sell it right away too. Keep that up, and you have a
very big pool of money always sitting in your bank account. Money that can be
profitably invested in other activities.

Why would a publisher give them 30 days to pay? Because they're Amazon. It's
good to be big.

~~~
jeremyjh
Net 30 is very common, even if you are not Amazon.

~~~
stcredzero
And lots of big companies will take 2X or 3X longer to pay you if they can get
away with it.

------
noname123
<http://aws.amazon.com/console/> and aws is still up.

------
khangtoh
The really interesting thing that really happened was Techcrunch's reporting
of this news worthy event.

[http://techcrunch.com/2013/01/31/amazon-com-
down-503-service...](http://techcrunch.com/2013/01/31/amazon-com-
down-503-service-unavailable/)

It's sad that "tech" bloggers don't research and report on news worthy things
anymore, they just take what's on Hacker News and call it news.

~~~
zbowling
Traditional news does with the AP and Reuters news wires. Hacker News is a
newswire of the tech world.

------
rosser
It appears just to be the homepage, but all deep links are unauthenticated.
That is: if you were logged in before the site started misbehaving, and use a
deep link, you're not logged in on the page that loads.

------
gchucky
Per <http://gizmodo.com/5980618/amazon-is-down>, a hacker group named Nazi
Gods has claimed responsibility for the downtime.

~~~
dsl
They are just trolling. Wait for something to go down on its own then spin it.

These twitter gems demonstrate the cluelessness of the "hackers":
<https://twitter.com/NaziGods/status/297074050881183744>
<https://twitter.com/NaziGods/status/297070145141104641>

------
k2xl
Maybe they forgot to pay their EC2 bill

------
jpsim
Thankfully, it looks like all of AWS is still up.
<http://status.aws.amazon.com/>

------
dm8
It's up now! But it's strange that they were down. And don't they run on AWS
themselves?

~~~
ultimoo
I remember being in a talk by Dr. Vogels last year and he mentioning that
*most of the Amazon.com North America services moved over to AWS in September
2011, many other services outside of NA were yet to move.

------
dtwhitney
I've been having odd behavior with DynamoDB all day. I wonder if it's related.
The AWS Health Dashboard says things are fine, but I'm not so sure:
<http://status.aws.amazon.com/>

~~~
jeffbarr
Have you reported your issues in the Forum or via the "Report an Issue" page?

------
setheron
JeffB is for sure getting paged.

------
druiid
Been a while since I've seen the amazon homepage down. Wow.

I know from the e-commerce side, when walmart.com went down last year we saw a
traffic increase (enough to actually link to to the outage for walmart). I
wonder if it'll happen here.

------
notdonspaulding
What's the guess as to how much revenue Amazon loses for every second of
downtime?

~~~
runarb
Not all abandon sales are a loss for Amazon. Many customers will simply just
com back and buy another day because they don't know about the competition.

------
bowyakka
Good thing that hacker news is their first port of call for monitoring then !

------
gwf
I managed to clip the isitdownrightnow.com status while it was down:

[http://clipboard.com/clip/LQOyBd-
lpIMCZzlmvYHqted7GrrQ3zkJis...](http://clipboard.com/clip/LQOyBd-
lpIMCZzlmvYHqted7GrrQ3zkJisue)

------
kno
Today's deals works fine
<http://www.amazon.com/gp/goldbox/ref=cs_top_nav_gb27>

------
dfriedmn
Free month of Thinkful to the team that's supposed to be keeping it up. Could
have order something and had it delivered in the time it took to get back
up...

------
st3fan
OH NO I MISSED THIS IMPORTANT EVENT!?

Why does this post have 168 points?

------
Posibyte
Still down at 2:00 PM CST. I hope we see a good post-mortem out of this. For
their record amounts of uptime, I can only expect something extreme.

~~~
officemonkey
Still down at 1423 CST.

~~~
officemonkey
However, I just reloaded and it loaded back up nice and pretty.

------
reefoctopus
Well this explains some things I had to deal with this morning. Apparently
some people had issues with EC2 as well.

------
trebor
It's up on the east coast, but that was almost 50 minutes of downtime. (I
heard that they make $100,000/minute...)

------
ForFreedom
Apart from it being down all hits from hackers are bound to just create load
on the server. :)

~~~
lucb1e
My server with an Intel Atom, Windows 7, and 100KB/s upload connection didn't
go down from reaching the Hacker News homepage. It's laughable that any other
website does. For Amazon, traffic from sites like Hackernews must be
completely negligible.

------
creativityhurts
It's still down <http://cl.ly/MZNo>

------
cm2012
Hah, we're losing $3 a minute from lost sales while they're down - the
travesty!

------
runarb
Back up her in Norway also.

------
joetek
... 35 minutes and anonymous hasn't claimed responsibility yet?

~~~
alexakarpov
of course they did: <http://gizmodo.com/5980618/amazon-is-down>

bogus, I can assure you

------
davej
Back up @ 19:46 GMT.

~~~
davej
... and it's down again @ 19:47GMT.

------
k2xl
more detail: <http://www.isitdownrightnow.com/amazon.com.html>

------
wglb
No, it is not.

Don't post outages here.

------
gohwell
I wonder how many millions they've lost

------
blambeau
Is Amazon running Rails or Rubygems?

~~~
dsl
The blessed languages of Amazon are C++, Java, and Perl.

~~~
alexakarpov
Ruby is replacing Perl, and that's been going on for a few years now (and we
have a pretty active RoR community).

~~~
aioprisan
stats?

~~~
alexakarpov
not sure what kind of statistics would be good and not violate the NDA =) but
our internal rubyhackers mailing list is one of the busiest lists that I'm on.
I myself am not a Ruby fan, so don't know any details.

------
jerhewet
> ping ec2.amazonaws.com

------
lucb1e
Any news on what happened?

------
trimbo
But... I can't read my front page letter from Jeff?

------
gesman
The beginning of the end of the world!

------
thechut
Backup for US EAST at 14:58

------
ivanb
OH MY GOD

