
Amazon Web Services are down - yuvadam
http://status.aws.amazon.com/?a
======
timf
Some quotes regarding how Netflix handled this without interruptions:

"Netflix showed some increased latency, internal alarms went off but hasn't
had a service outage." [1]

"Netflix is deployed in three zones, sized to lose one and keep going. Cheaper
than cost of being down." [2]

[1] <https://twitter.com/adrianco/status/61075904847282177>

[2] <https://twitter.com/adrianco/status/61076362680745984>

~~~
campnic
"Cheaper than cost of being down." This is very insightful. Many of us look at
the cost of multi zone deployments and cringe, but its a mathematics exercise.
(.05 * hours in a year)*(cost of being down per hour) = (expected cost of
single zone availability). Now just compare to 2-3x your single zone
deployment cost. Don't forget the cost of being down per hour should include
lost customers as well.

~~~
dfranke
I'm actually surprised if incurring 50% extra hardware costs really is cheaper
than the cost of being down. If Netflix is down for a few hours, then it costs
them some goodwill, and maybe a few new signups, but is the immediate revenue
impact really that great? Most of Netflix's revenue comes from monthly
subscriptions, and it's not like their customers have an SLA.

~~~
imajes
Actually, they do. and Netflix proactively refund customers for downtime.
Usually it's pennies on the dollar, but i've had more than refund for sub 30
minute outages which have prohibited me from using the service.

Netflix are very very sensitive to this problem because it's much harder for
them to sell against their biggest competitor (local cable) since they rely on
the cable to deliver their service. If the service goes down, then the cable
company can jump in and say, "You'll never lose the signal on our network" --
blatantly untrue, but it doesn't matter.

When you're disrupting a market, remember that what seem trivial is in fact
hugely important when you're fighting huge well-established competition :)

------
asymptotic
Amazon's EC2 SLA is extremely clear - a given region has an availability of
99.95%. If you're running a website and you haven't deployed across across
more than one region then, by definition, your website will have 99.95%
availailbility. If you want a higher level of availability use more than one
region.

Amazon's EBS SLA is less clear, but they state that they expect an annual
failure rate of 0.1-0.5%, compared to commodity hard-drive failure rates of
4%. Hence, if you wanted a higher level of data availability you'd use more
than one EBS volume in different regions.

These outages are affecting North America, and not Europe and Asia Pacific.
That's it. Why is this even news? Were you expecting 100% availability?

~~~
hoop

        Amazon's EC2 SLA is extremely clear -
        a given region has an availability of 99.95%.
        If you're running a website and you haven't
        deployed across across more than one region then,
        by definition, your website will have 99.95%
        availailbility. If you want a higher level of
        availability use more than one region.
    

Good point.

let P(region fails) = 0.05% and let's assume (and hope) that the probability
of failure of one region is independent of the state of the other regions.

P(two regions fail) = P(one region fails and another region fails) = P(region
fails) * P(region fails) = 0.05% * 0.05% = 0.0025%

Making your availability = 100% - 0.0025% = 99.9975%

Ultimately it's more of a business decision if you want to pay for the extra
0.0475% of availability. I would think (or hope) that most engineers would
want it anyway.

The numbers at this size appear insignificant. How would one (say an engineer)
convince "the management" that the extra 0.0475% of availability is worth the
investment/expense?

~~~
jerf
"let's assume (and hope) the probability of failure of one region is
independent of the state of the other regions."

In practice, that's not true, and it's not true enough to ruin the entire rest
of your calculations. For Amazon regions to function independently, they'd
have to be actually, factually _independent_ ; there is no interaction between
them. The reaction to one node going down is never to increase the load on
other nodes as people migrate services, etc. There's fundamentally nothing you
can do about the fact that if enough of your capacity goes out then you will
experience demand in excess of supply.

If you want true redundancy you will at the very least need to go to another
entirely separate service that is not Amazon... and if enough people do that,
they'll break the effective independence of that arrangement, too.

(This is a special case of a more general rule, which is that computers are
generally so reliable that the ways in which their probabilities deviate from
Gaussian or independence tends to dominate your worst-case calculations.)

~~~
hoop
I agree with you 100% that they're not independent, but I don't know enough
about the data to model the probabilities of failure and availability in a HN
comment :-)

After today's event, it would certainly be interesting to see how resource
consumption changed in other availability zones and at other providers during
this outage.

I wonder if that could be measured passively? What I mean is, by monitoring
response times of various services that are known to be in specific regions
and seeing how that metric changes (as opposed to waiting on a party that has
little-to-no economic benefit to release that information.)

------
yuvadam
Current status: bad things are happening in the North Virginia datacenter.

EC2, EBS and RDS are all down on US-east-1.

Edit: Heroku, Foursquare, Quora and Reddit are all experiencing subsequent
issues.

~~~
wmoxam
Not all EC2 & EBS instances are down. I have several in US-east-1a and 1 is
down, while all of the others are working.

~~~
jedberg
Same with us. About 10% of our 700+ volumes are having problems right now.

It's hard to tell for sure since there isn't any load.

------
dsl
4/21/2011 is "Judgement Day" when Skynet becomes self aware and tries to kill
us all. <http://terminator.wikia.com/wiki/2011/04/21>

I am just a little freaked out right now.

~~~
jedberg
Don't worry. If skynet is in EC2, we'll be fine.

~~~
HelloBeautiful
You don't understand, Skynet is using all Amazon resources, hence the outages
;-)

Amazon have stated many times that amazon.com itself runs mostly on AWS
platform, but it works fine now ...

~~~
cicloid
AWS platform on a private cloud, it is not the same as AWS platform for us
commoners.

------
ig1
A couple of hours into the failure, and no sign of coverage on Techcrunch
(they're posting "business" stories though). It shows how detached Techcrunch
has become from the startup world.

Edit: I tweeted their European editor about it and he's posted a story up now.

~~~
dwc
Perhaps this isn't really news. These days it's normal.

~~~
dwc
It's ugly, but true enough. You don't have to like it to acknowledge it. It's
just another cloud outage bringing down one or more high profile sites. It's a
"dog bites man" story.

------
mcritz
This feels the same way as hearing that the whole Internet just got shut down.

------
kylec
I guess this is one Reddit outage that can't be blamed on poor scaling

~~~
jedberg
Thankfully, no. :)

But yeah, right now we're shutting everything down to try and avoid possible
data corruption. Once they restore service, hopefully we'll be able to come
back quickly.

~~~
thirtysixred
SHUT. DOWN. EVERYTHING.

Good plan though. :p

~~~
jedberg
My original comment was "We're going Madagascar on the servers." Then I
remembered I was on HN, not reddit. :)

~~~
JonnieCache
_> Then I remembered I was on HN, not reddit._

Right now we may as well be on reddit.

------
mtodd
Why is ELB not mentioned at all on the Service Health Dashboard?

We're experiencing problems with two of our ELBs, one indicating instance
health as out of service, reporting "a transient error occurred". Another, new
LB (what we hoped would replace the first problematic LB), reports: "instance
registration is still in progress".

A support issue with Amazon indicated that it was related to the ongoing
issues and to monitor the Service Health Dashboard. But, as I mentioned
before, ELB isn't mentioned at all.

~~~
jen_h
We've got a single non-responsive load balancer IP in one of our primary ELBs
(others have been fine for several hours now), so while everything else for us
is up & running, still have transient errors for folks that get shunted to
through that one system.

The interesting thing about the ELB in a situation like this is that I believe
it may, in many instances, be better to hobble along and deal with an elevated
error rate if at least some of your ELB hosts are working than to re-create
the entire ELB somewhere else, especially if you're a high-traffic site where
you may hit scaling issues going from 0 to 60 in milliseconds (OMMV, but we've
been spooked enough in the past not to try anything hasty until things get
back to normal).

~~~
mtodd
We have an identical load balancer to one that is causing problems so we're
lucky enough to reroute traffic through that one instead to get to the same
boxes. (The boxes serve two different APIs through two different DNS CNAMEs so
we split the ELBs for future and sanity). In this case, it's helped us out.
Alternatively, we would've just routed all traffic to our west coast ELBs.

------
potomak
Quora says: "We'd point fingers, but we wouldn't be where we are today without
EC2."

~~~
olegp
Nice way to point fingers while saying you're not.

------
helium
I just launched a site on Heroku yesterday and cranked up the dynos up in
anticipation of some "launch" traffic. Now, I can't log in to switch them off.
Thanks EC2, you owe me $$$s

~~~
mike-cardwell
Isn't that a design flaw in Heroku? Shouldn't you be able to log into Heroku
and change stuff like that even if the entire of Amazons cloud service is
down?

~~~
Maxious
> Nothing special-case here: we deploy with git push, just like any other
> Heroku user. Dogfooding is good for you.
> <http://blog.heroku.com/archives/2009/4/1/fork_our_docs/>

~~~
mike-cardwell
That's all well and good, but it's no use for their customers if/when Amazon
goes down.

------
mathrawka
I think this is a good example of how the "cloud" is not a silver bullet to
making your site always up. AWS provides a way to keep it up, but it is up to
each developer to ensure that they are using AWS in a way to make sure their
site can handle problems in one availability zone.

I think we will see more of a focus from big users of AWS about focusing on
how to create a redundant service using AWS. Or at least I hope we will!

~~~
jedberg
This outage is affecting all AZ's in the East. So even a multizone setup
wouldn't help for this one. Only a multiregion setup.

This outage is a lot like having your entire datacenter lose power.

~~~
6ren
If "multi-region" means North America, Europe and Asia Pacific, doing so would
also improve world-wide latency (e.g. here in Australia...).

Could you use this outage to justify switching to multi-region?

~~~
Maxious
A blog post last month touched on this: "Q: Why is reddit tied so tightly to
the affected availability zone?

A: When we started with Amazon, our code was written with the assumption that
there would be one data center. We have been working towards fixing this since
we moved two years ago. Unfortunately, progress has been slow in this area.
Luckily, we are currently in a hiring round which will increase the technical
staff by 200% :) These new programmers will help us address this issue."

Not sure if the costs of data transfer between regions (charged at full
internet price) would justify the added reliability/lower latency though.

------
alexpopescu
Instead of enumerating who's down, I'd be more interested to hear about those
that survived the AWS failure. We could learn something from them.

------
espeed
Quora is down, and evidently "They're not pointing fingers at EC2" --
<http://news.ycombinator.com/item?id=2470119> \-- I was going to post a screen
shot, but evidently my Dropbox is down too.

------
dmuth
Holy crap. An Amazon rep actually just _posted_ that SkyNet had nothing to do
with the outage:

[https://forums.aws.amazon.com/message.jspa?messageID=238872#...](https://forums.aws.amazon.com/message.jspa?messageID=238872#238872)

------
powdahound
I'm seeing 1 EBS server out of 9 having issues (5 in one availability zone, 4
in another). CPU wait time on the instance is stuck at 100% on all cores since
the disk isn't responding. Sounds like others are having much more trouble.

------
jws
Silver lining: Hopefully I can test my "aws is failing" fallback code. (my GAE
based site keeps a state log on S3 for the day when GAE falls in a hole.)

~~~
G_Morgan
This code should be well tested by now. Amazon is doing you a failure by being
rubbish.

------
smtm
AWS/S3 has become the new Windows - great SPOF to go for if you want to
attack. This space needs more competition.

~~~
dangrossman
Two years ago TechCrunch was publishing an article every time Rackspace went
down listing all the hot startups down along with it. AWS is no more a SPOF
than any other major hosting provider.

------
paraschopra
<http://venuetastic.com/> \- feel bad for these guys. They launched yesterday
and down today because of AWS. Murphy's law in practice.

~~~
hysterix
Wow. I can only imagine the intense frustration the site owner must be feeling
right about now. Makes you really stop and question the whole "cloud" based
service. Or at least should make you realize you need fall-backs other than
the cloud service itself.

------
smackfu
So when big sites deal use Amazon Web Services for major traffic, do they get
a serious customer relationship? Or is it just generic email/web support and a
status page?

~~~
mleonhard
There are subscriptions for various levels of support, from $50/mo for 12 hour
response time to $15k/mo for 15 minute response time.

<http://aws.amazon.com/premiumsupport/>

~~~
smackfu
Interesting pricing. The Platinum seems priced to have no one use it,
considering how much of a jump it is over Gold.

~~~
iclelland
I would say, rather, that it is priced to have very specific sorts of customer
using it.

The relationship between the pricing tiers changes fairly drastically,
depending on how much you are already spending on Amazon Web Services. Gold,
for instance, starts out at 4x the price of Silver support, but by the time
you're spending 80K/month on services, it's only a $900 premium (and stays
there no matter how much bigger your bill is). At the $150K/month level, it's
a 2x jump from Gold to Platinum, which may or may not be a huge jump,
considering the extra level of service you get.

------
frekw
It's a bit ironic that Amazon WS has become a SPoF for half the internet.

------
jedberg
Yes, they are. :(

~~~
athesyn
My four-day weekend is already off to a bad start(UK here).

~~~
taitems
It could be worse/better, you could have an Australian 5-day weekend.

~~~
endgame
To clarify, this Monday is ANZAC Day, a commemorative holiday for troops who
fought for Australia. Because that's also Easter Monday, the ANZAC Day holiday
is moved to Tuesday, despite commemorative services being held on Monday.

~~~
kristianp
Actually the official line is that Easter Monday got moved to Tuesday.

~~~
endgame
Interesting. Thanks. What did the Catholic Church have to say about that? Is
it that Easter Monday is still on Monday, but the holiday is on Tuesday?

~~~
spydez
Easter is the holiday, and it's on Sunday. I'm pretty sure the Pope doesn't
care much what people do the day after Easter (or the day after that).

~~~
endgame
Ah, thanks. I misread the wikipage, and didn't realise.

------
olegp
Assuming the problem is indeed with EBS, I would say this should be a warning
sign to anyone considering going with a PaaS provider, which Amazon is quickly
becoming, instead of an IaaS provider like Slicehost or Linode.

The increased complexity of their offering makes it more likely that things
will break, leaving you locked in.

I did a 15 minute talk on the subject, which you can check out here:
<http://iforum.com.ua/video-2011-tech-podsechin>

EDIT: here are the slides if you can't bother watching the video
<http://bit.ly/eqDNei>

~~~
ghshephard
Every time someone makes the claim that downtime should be a warning sign
about going with a PaaS provider (or, indeed, an IaaS provider, or in some
cases, people even make this claim about going with someone else's Data
Center) - I always respond: "And why do you believe that you would do any
better?"

Every environment I've been involved in as an operations professional for the
last 15 years has experienced downtime regardless of how much we invested in
staging, testing, change control procedures, redundancy and ITIL methodology.

While it is true that there are some environments in the world that experience
no, or close to no downtime (The NYSE, google.com, the Space Shuttle) - it's
usually not worth the investments those organizations make in ensuring 100%
uptime.

I'm more interested in seeing what the monitoring methodology is, what the
response protocols are, and what the reported downtime has been over the last
year has been, than I am about getting too concerned about the occasional
outage.

~~~
olegp
The problem with Amazon is that despite touting an open API, their
infrastructure internals and practices are a trade secret, so the likes of
Eucalyptus are having to play catchup. In other words, I cannot replicate
their infrastructure in my own data center, even I had the money to pay them.
I suspect this is the main reason that Heroku didn't move off Amazon, and not
the fact that Amazon was providing them great value for money.

There is definitely value in platforms, but those platforms should be built on
completely open standards with no vendor lock in or influence, as is the case
with OpenStack for example:
[http://www.theregister.co.uk/2011/02/10/rackspace_buys_opens...](http://www.theregister.co.uk/2011/02/10/rackspace_buys_openstack_partner/)

If you didn't watch my talk, I should point out that I'm working on Akshell
(<http://www.akshell.com>), which is itself a platform provider, so I am in
fact agreeing with you.

~~~
pzb
Compared to PaaS provides, Amazon is easy to migrate off of because they are
simply giving you virtual hardware.

The basics of their setup are well known: Xen and Linux. EBS is some sort of
block-based network storage. True, we do not know the backend storage setup,
but it does not really matter. The Xen VM simply sees a disk device. NetApp,
EMC, Dell, HP and many others all have products that offer similar
functionality, including snapshots.

The only part that is potentially hard to migrate off is the security groups,
and then only if you are using named groups. If you are just using IP based
rules, most any firewall would work.

------
vnorby
From EngineYard: "It looks like EBS IO in the us-east-1 region is not working
ideally at this point. That means all /data and /db Volumes which use EBS have
bad IO performance, which can cause your sites to go down."

------
oomkiller
They better start writing their explanation now. Multiple AZ's affected?

~~~
jedberg
They better be cutting me a check too.

~~~
ig1
As an advertiser on Reddit, you guys better be cutting me a check too :)

(although my app is also dead right now so not as if the advertising would do
me any good...)

~~~
jedberg
Send me the link to your ad (when we come back up) and I'll comp you a day.

~~~
reustle
Isn't it paid on impressions?

~~~
AgentConundrum
I don't know how the sidebar ads work, but the featured links at the top of
the page are run on a sort of auction system. Everybody that wants a piece
pays however much they want (minimum of $20) for the day, then all the ads are
totaled up and each ad gets a percentage of pageviews corresponding to the
their percentage paid on that days revenues.

If that wasn't clear (and I'm not sure it was), assume you and I were both the
only advertisers on reddit for a day. If I pay $20 and you pay $20, we would
both have our ads displayed on 50% of pageviews for that day. If instead I
paid $80 and you still only paid $20, then I would get 80% of pageviews to
your 20%, regardless of the total pageviews for the day.

------
antonioe
Had our blog go down. Didn't realize it was AWS wide..did a reboot. Now I am
in reboot limbo. Put an urgent ticket into Amazon. They just said they are
working urgently to fix the issues. Let's see how long this goes.

------
swedegeek
In case anyone is late to the party and missed the non-green lights on the AWS
status dashboard, here is the page as of about 9:30 EDT...

<http://screencast.com/t/p69xAoDJRSer>

------
ig1
Given that Heroku's parent company (Salesforce) owns a cloud platform, it
seems kinda inevitable now that Herkou will perhaps sooner-than-later switch
back-ends (or at least use both)

------
jjm
Everyone talks about SLAs but I believe it doesn't consider the fact that the
EBS vols are still up (not on fire, and available) and are phantom writing or
that the network is queued up the wazoo so writes don't even happen in a
timely manner as you'd expect.

~~~
elliottcarlson
I'm not sure that just because they are up, yet unusable, would negate an SLA.

You could have a dedicated server in a datacenter - if the network goes out,
your machine is still up and happily waiting to serve requests - but it's
still unusable and not actually in service.

------
mathrawka
So do we get some credit on our AWS accounts? I haven't really read their SLA
for EC2.

~~~
dpogg1
I did the math on it, with the downtime so far it's almost approaching the
magic 99.95% barrier where everyone gets a 10% bill credit. Wouldn't that be
something to light a fire under the collective arses of those in charge of
keeping EBS stable.

~~~
lylejohnson
> Wouldn't that be something to light a fire under the collective arses of
> those in charge of keeping EBS stable.

Yeah, the only possible explanation is that Amazon's put a bunch of lazy guys
in charge of EBS. How hard a job could it be?

~~~
dpogg1
I'm certain they're plenty talented and hard-working, but there's got to be
something in their way. Just like the guy said as he was leaving Reddit,
management seems to be pointing fingers in different directions as to why this
is going on and how it can be resolved. It's sad, though, because I don't want
to think of Amazon as a company with a potentially toxic corporate culture. I
enjoy my free Prime membership as a student :D

~~~
mleonhard
Amazon's corporate culture works pretty well. The problem is that these large-
scale multi-tenant services are new technology and very difficult to create
and run.

------
espeed
[http://www.information-age.com/channels/the-cloud-and-
virtua...](http://www.information-age.com/channels/the-cloud-and-
virtualization/news/1619753/amazon-web-services-struck-by-performance-
issues.thtml)

------
dmuth
Being unable to get much done here, my co-workers have found other things to
do in the office: <http://www.youtube.com/watch?v=u1-oGxDHQbI> :-P

~~~
jedberg
What did I just watch and why?

------
tybris
All hosting services go down occasionally. If you want to stay up you need to
build a fault-tolerant distributed system that spans multiple regions and
potentially multiple providers.

Also, Amazon should fix EBS.

------
xlevus
Ruh Roh. The service I'm using to acquire accommodation seems to be dependent
on AWS. Guess I'm going to be homeless tomorrow if it doesn't get fixed. :X

Note to self. Don't ever build a service reliant on AWS.

~~~
crashpadder
If its us (crashpadder.com) - sorry for the inconvenience. Email us hello [at]
crashpadder.com and we'll do all we can when we're back up.

We'd moved to AWS from an even worse host a few months ago, and until this
morning had been pretty impressed...

------
ck2
So what percentage of the top 1000 sites are now crippled by this?

------
singlow
It's definitely a limited outage. My three instances seem to have operated all
night with no problem. Two of them are EBS instances.

------
greaterscope
Wish were able to download our ebs snapshots, which are supposedly hosted on
S3. What does everyone else do?

~~~
dangrossman
I take the snapshots. I also have servers send backups to each other each
night. I also have a nightly cron job run and rotate backups of the most
critical databases to an external drive on my home network. A Tonido Plug does
that job (Ubuntu on a tiny ARM server in a plug that costs virtually nothing
to run).

Now, some of the databases are simply too large or under too much load to take
a live backup while the sites are running. Those I run on Amazon RDS with the
MultiAZ feature enabled. There should be two copies of the database running at
all times, both servers keeping a 3 day binlog for point-in-time backups, and
making a nightly snapshot to S3. I have to rely on Amazon for that.

But I still take daily home backups of the most valuable individual tables off
those servers, like user registrations and payment records. Even if I can't
have off-site backups of the whole database, I'll have off-site copies of the
part I'd need most in case of an Amazon-entirely-offline catastrophe.

------
hcentelles
It seems like availability zone us-east-1c it's working, i can launch a EBS
backed instance right now.

------
piramida
Today, April 21st 2011, according to the "Terminator", Skynet was launched...
No wonder AWS is down

------
antonioe
1:23EST and Reddit is back up. Quora/4SQ still down. My site still down.

------
kennethologist
Thankfully, my major clients are using the Asia EC2 Instances!

------
pextris
reddit.com is down, but luckily <http://radioreddit.com> is not.

------
jwr
It's really mostly EBS failures, so the title is overly dramatic. And EBS has
been known to have issues.

~~~
dogas
This is incorrect. EBS is part of the issue, as well as intermittent
connectivity failures for EC2 instances.

We're also seeing connectivity issues with our elastic load balancers.

------
hendi_
Yay for relying on the cloud \o/

~~~
muppetman
Yes! It's like those dumb people that rent out 5 floors of a 100 story
building, instead of building their own SkyScraper. Then, horror of horrors, 2
of the 8 lifts breaks down for 12 hours.

Such idiots for paying rent, even though building SkyScrapers isn't their core
business.

~~~
hendi_
Poor analogy; if the lifts break, just use the stairs.

Note that I've written "relying", as in "depending on the cloud without any
alternative backup". _Depending_ on the cloud means depending on something you
cannot influence.

~~~
muppetman
Ok, it's a bomb scare and the building's shutdown for 12 hours.

How I was supposed to get ' "relying", as in "depending on the cloud without
any alternative backup" ' from what your originally wrote, I don't know.

~~~
hendi_
I'm sorry if you didn't understand "relying" as I meant it. I'm not a native
speaker, so my assumption of the meaning of "relying" may have been wrong.

I'll try to elaborate: yes, there are of course certain things that you cannot
avoid. That's why I meant you should not "rely"/"depend on the cloud without
any alternative backup" on the cloud being available. Redundancy, if your
service is crucial!

If you're a small company, and cannot go to your building for a day, well then
I hope you're not that important so no big damage (financial or otherwise) is
caused. Just enjoy your free day ;-) But if there's a bomb threat in, for
example, a hospital, I surely hope they have some kind of backup plan!

Cloud as an alternative mechanism, or maybe even the primary resource for your
site -- great, why not! But please don't just throw everything into "the
cloud" trusting on the belief that the cloud's operator is impossible to fail.

~~~
muppetman
You said 6 words in your original post. And now you're trying to suggest I was
to get the above 166 words from the original 6?

You raise perfectly valid and interesting points. It's just a shame they're
not the parent comment because then they would have invoked some thoughtful
discussion.

~~~
182446
It's a shame your sarcastic reply derailed the conversation.

~~~
muppetman
Fair call.

------
marjanpanic
Amazon down - just one more reason to try out BO.LT and their amazing CDN and
page sharing services...

Just launched: [http://techcrunch.com/2011/04/21/page-sharing-service-bo-
lt-...](http://techcrunch.com/2011/04/21/page-sharing-service-bo-lt-lets-you-
copy-edit-and-share-almost-any-webpage/)

------
Jun8
This is like witnessing your parents having sex while a kid: you sort of knew
this is a possibility but it is a devastating blow to your belief system
nevertheless.

The amount of services I use that depends on Amazon is amazing. They have
really become the utility company of the Web.

