
Lessons We’ve Learned Using AWS (2010) - denzil_correa
http://techblog.netflix.com/2010/12/5-lessons-weve-learned-using-aws.html
======
mtdewcmu
I worked on a web store once that depended on a couple of 3rd party web
services and set up subscription billing. The project turned out to be like an
iceberg, where 10% of the work went into features, and 90% of the work went
into heading off all of the various possible failure modes. When the system
failed, people's credit cards got billed the wrong amount and either the store
didn't get paid or the customer got overcharged. So I can relate to all of the
effort they put into fail-safety.

------
SFJulie
[http://www.networkworld.com/article/3037428/cloud-
computing/...](http://www.networkworld.com/article/3037428/cloud-
computing/netflix-is-not-really-all-in-on-amazon-s-cloud.html) Well they also
learned to not put their most important asset on AWS.

~~~
amaks
Kind of makes sense given the recent outage. At least it looks like it's a
good idea to diversify the cloud technologies across several vendors.

------
venkasub
Architecting applications for the cloud is a non-trivial problem. Lot of
forethought needs to go in place while choosing a particular infrastructure
component. And the ideas of resiliency, robustness (what can be bucketed into
'graceful-degradation') etc along with the most important Monitoring needs to
be put in place right from the initial days of the app, so that rearchitecting
is does NOT end up being a costly affair.

~~~
Johnny555
Architecting for a physical data center is also a non-trivial problem, and not
all that different than architecting for the cloud.

~~~
bluejekyll
Couldn't agree more. The cloud just makes it clear where you didn't architect
for resiliency properly, in your own DC you work around this by doing things
that you can't do in the cloud.

But the biggest difference is automation. Because in the cloud you have a full
suite of APIs, you can literally automate ever piece of your infrastructure.
In your own DC you have to design this stuff to be automated from the ground
up, which just making a poor layer2 choice can make that task astronomically
more difficult.

------
skyisblue
What happens when multiple regions on AWS are down? Would it be a good idea to
also have a complete backup infrastructure on another cloud service?

~~~
Johnny555
At my company, we've decided that if multiple AWS regions are down, we're down
too.

We use so many AWS services that porting to a second cloud provider is cost-
prohibitive at this point. So we're willing to accept that if 2 USA regions
and one European region are down, we're down too.

~~~
npolet
> At my company, we've decided that if multiple AWS regions are down, we're
> down too.

We've come to the same conclusion as well. We cannot afford the engineering
hours to run duplicate infrastructure just in case the big boys go down.

If multiple AWS regions go down... we wait until they are not down. Our
developers sighed in relief when we decided that this was going to be our
course of action... but we're approaching the scale that requires us to really
mitigate against large scale outages like this. For now we bury our heads.

