
Open Guide to Amazon Web Services - forwidur
https://github.com/open-guides/og-aws.git
======
PaulRobinson
A 15-minute scan read of this - specifically the sections on the stuff I've
worked with the most - suggests this is a very, very good addition to the
official documentation.

I would as a minimum recommend anybody/everybody considering AWS to read and
think about the "When to use AWS" section. Whilst it is an excellent set of
tools that have completely changed the economics of deploying software, there
are times when you should use Google Cloud, times you should use bare metal,
times you should use Heroku. AWS is a complex beast. Heroku is simple, but has
limitations.

There are a bunch of apps I'm thinking about building at the moment where I
realise a hybrid approach is best: some of GCP's stack, some of AWS', and a
small amount of my own bare metal. Knowing when to choose which is not
intuitive and comes with time, but there are big, big clues that will help the
uninitiated in that section of this open guide.

Also, if you're looking to the future, the AWS Lambda and Google Functions
stuff is perhaps the most exciting stuff to start building knowledge up of now
if you're a developer, I think.

~~~
TheDong
> There are a bunch of apps I'm thinking about building at the moment where I
> realise a hybrid approach is best: some of GCP's stack, some of AWS', and a
> small amount of my own bare metal. Knowing when to choose which is not
> intuitive and comes with time, but there are big, big clues that will help
> the uninitiated in that section of this open guide.

unless you have a metric shitton of money to blow, there's never a good reason
to start with that.

The most expensive part of any of those cloud providers is networking. If you
need to transfer data from bare metal <-> aws, you'll need direct connect
which charges basically an arm and a leg. Transferring between aws <-> gce is
expensive for the same reason. Sure, if you're apple scale and need better
data redundancy maybe it's okay. _maybe_. But that's not an app you think
about building as an individual or small company.

I also don't think GCPs stack has anything whatsoever that AWS's doesn't have,
so it's odd to mention it in that phrase.

If you'd be so kind as to provide an example application you're thinking
about, and the reason each of those is needed for some part of it, I'd be
happy to hear it!

------
andrenotgiant
I recommend you also make the content available on a one-topic-per-page format
ASAP before someone else does and takes credit for it.

WHY: Google still doesn't handle anchor-links very well. You have 1000 amazing
articles on a single page. Each section (e.g.: "High Availability on AWS")
would be a great resource for someone searching on that topic in Google. But
when you put it all on one page Google infers "1/1000th of this page is about
high availability on AWS" and gives better rankings to a page that is 100%
about high availability on AWS.

I'm sure it would be pretty simple to write a script that breaks up topics
into individual pages. I love the style of having it all on one page but I
think it would be a waste of your hard work not to get all this great writing
in front of search.

~~~
zalzal
I understand the concern. We'll try doing something about that. That said,
single page on GitHub for the moment means (1) discoverability directly on
github.com, which helps everyone and (2) browser search on the whole guide
(which actually is more helpful than you might think!).

~~~
andrenotgiant
Completely agree, once I discover a guide like this, I bookmark it, come back
to it, and really value the ctrl-f-ability.

I was recommending the one-topic-per-page idea for others who haven't yet
found this nugget. I think a lot more people will discover it and benefit from
it if they are finding it from specific google searches.

I know HN can be a source of a lot of unfounded flyby critiques, I dont want
to contribute to that trend. I see you have a pretty good contributing guide,
maybe I'll try and submit a PR with a solution in the spirit of Hacktoberfest!

------
nzoschke
This is great. I've been working on AWS for close to 10 years now and an open
guide is something I both need and want to contribute to.

Many of us have simple goals on AWS. The official AWS docs are thorough, but
are too technical. There are blog posts about anything, they can be hard to
find or get out of date.

I hope this open guide helps us all get our jobs done faster and easier!

~~~
zalzal
Very glad to hear. Its this sentiment exactly that led us to get this started.
We all have 100s of valuable tricks and gotchas we learn over the years, but
99% of the time fail to write down and share them helpfully. Do join us on
Slack/GitHub and help us get your tips included, too.

~~~
nzoschke
Done and done. I was pleasantly surprised to see my ECS tips already linked. I
hope I can remix this knowledge for the guide!

[https://convox.com/blog/ecs-challenges/](https://convox.com/blog/ecs-
challenges/)

------
xbryanx
Wow, the link to
[http://www.ec2instances.info/](http://www.ec2instances.info/) alone is so
helpful. I wish I'd had this set of resources a year ago when I spent weeks
trying to understand AWS' own documentation.

~~~
forwidur
Right? I wish Amazon were just running a page like that themselves.

~~~
jeffbarr
Which aspect of it do you find the most useful?

* All of the instance types on one page? * All of the per-type facts in one row? * Sorting?

Let me know and I will share it with the team.

~~~
piinbinary
* Everything on one page

* Doesn't take 30+ seconds to load

* Sortable and filterable

------
zalzal
Remember, this isn't a blog, it's living GitHub project: If you see value in
info like this, consider contributing or giving feedback to improve it. :)

------
stormy
What I would consider one of the most important pieces of this guide is closer
to the bottom ([https://github.com/open-guides/og-aws#aws-data-transfer-
cost...](https://github.com/open-guides/og-aws#aws-data-transfer-costs)) where
it covers cost management strategies. The Data Transfer Costs diagram makes
the buried details of AWS networking costs stand out in a digestible way. I've
read the AWS docs on this many times and still missed out on some of the
nuggets exposed in the diagram.

------
robertleon
As a consultant that often recommends migration to AWS services for clients,
this is a treasure-trove of information when looking at each individual use
case and making a determination about how best to advise. It's often difficult
to know with certainty whether AWS vs Google Cloud vs bare-metal is the best
course of action, and the advice and information here goes a long way in
helping make those decisions easier.

------
imperialdrive
One of the biggest lessons I've learned is that you need occasional EBS-to-EBS
backups. Anyone that had to recover from snapshots knows the painful reason
why...

~~~
markwillis82
Why is it painful recovering from snapshots? (have just moved to AWS so have
not experienced this yet)

~~~
imperialdrive
I get a lot of shit for not giving straight answers... just spin up an
instance, put a gig of data on EBS drive, snapshot, create EBS from snapshot
as if you were recovering, and try pulling 100+ megs of data off it... you'll
never not keep EBS copies again. big clue: pre-warming

it will take you an hour to do, and you'll be years wiser

this is probably the number one reason people experience extra extra downtime
when suffering from rebuild from whatever issue... and EBS volumes in certain
regions can and will experience silent deaths

------
questionr
Someone want to start one for Google Cloud?

------
tptacek
The "use IAM roles for EC2" recommendation is a bit sketchy. The current
security zeitgeist, not just after Colin's post but also after DerbyCon and
Black Hat, is that EC2 roles are dangerous and, when under attack, not very
predictable.

~~~
dastbe
Do you have links to the DerbyCon and Black Hat talks? And could you clarify
what "when under attack, not very predictable" means?

~~~
tptacek
An attacker who compromises an EC2 instance can quietly grab the instance role
credential and use it even after losing access to the instance itself.

~~~
voganmother42
"Have the application retrieve a set of temporary credentials and use them."
"In the case of Amazon EC2, IAM dynamically provides temporary credentials to
the EC2 instance, and these credentials are automatically rotated for you."
Attacker should only have access until creds are expired no ?

~~~
gtsteve
That's right. Instance store credentials have an expiration time of a few
hours. However, if the instance policy is very open you could create yourself
a new IAM account or use STS to maintain persistence after the generated
credentials expire.

This is why it's important to lock down instance profiles to do only what the
application needs to do and no more. For example, you may give the permission
to s3:DeleteObject, and in the event that the box is compromised the attacker
would be able to delete files in your S3 bucket. However, if you don't give
access to s3:DeleteObjectVersion you can evict the attacker and restore the
deleted objects with relative ease.

This is why I would not recommend giving access to s3:* to an instance profile
(or indeed, any production credentials).

~~~
voganmother42
Thank you for the reply - that makes sense to me, least privilege seems to be
the primary defense in that case. Having explicit creds you rotate yourself I
could see having benefits as far as control, but also requires more work /
potential for implementation mistakes.

~~~
gtsteve
Well, the AWS credentials auto-rotate. It does however provide a familiar
place for an attacker to go to get the instance credentials, but that doesn't
really help. At some point, those credentials must exist in plain-text for you
to use them. If they're in a config file, they can be read out, if they're in
RAM they can be pulled out with a debugger. At least if your box is
temporarily owned due to a zero-day that you later patch, the credentials
aren't going to be valid for long - although that situation would be hardly
ideal!

You've also got to go to the trouble of getting the credentials on your box to
start with. With instance roles, you can launch an instance and have it
immediately capable of doing what your application needs. In the case of most
applications my company runs, the instance profile is enough and no further
security credentials are required. When database credentials are required,
they're retrieved via S3, authenticated by the instance profile.

~~~
voganmother42
we use iam roles and credstash(dynamodb and kms) for retrieving database
credentials. My comment was mostly in terms of the fact we cannot control the
rotation for roles, say in the event of a breach like where someone committed
keys to github and I can explicitly expire/rotate(assuming those keys were not
themselves temporary and have not already expired :))

~~~
gtsteve
I believe you can actually [0]. In a production setting it's a lot harder to
accidentally leak the credentials - my concern would be if someone compromised
the instance or if it was tricked into opening the instance store up to the
net, such as a badly configured nginx instance (how you'd do that accidentally
though I have no idea)

[0]
[http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use...](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_revoke-
sessions.html)

~~~
voganmother42
Good point! slightly less granular than per key but still very helpful,
thanks!

------
jmickey
I still don't get why Opsworks is not getting more love?

I guess people don't like Chef? Opsworks has enabled hassle free deployments
for us over the past three years or so at no additional cost. :)

~~~
melvinmt
Yep, not sure where this perception of "nobody's using it" comes from but I
have been using it in 2 different companies in the last 3 years as well with
nothing but love. In fact, if it were the case that "nobody's using it for
good reasons", maybe we should ought to know the reasons?

~~~
eric_h
Been using opsworks for about a year now and while it has very significantly
streamlined our provisioning/deployment tasks, "nothing but love" is not quite
how I'd describe it.

It does have some warts.

~~~
derwiki
Biggest wart we have is that it randomly picks a machine to run migrations on,
if it's a deploy with migrations.

~~~
roadrunnerfreak
You could code up something in the deploy hook to select the master node
(mostly the first instance in the layer) to run migrations and you could
disable the "Run Migrations" when you deploy. I do this for the Rails app in
my company.

------
chirau
Does anyone here use x1.32xlarge instances? If so what kind of stuff are you
doing with it? That thing looks beastly

~~~
forwidur
My guess is that there are companies with "legacy" applications, that can't
really be re-written into a distributed system, have a large footprint, but
still need to be run.

The special sub-category of those are huge RDBMS instances - a pretty common
choke point in growing companies with weaker engineering teams. Some of those
companies would pay basically any price to keep those DBs running.

------
gshakir
This is great. I created a pull request for my S3 Infrequent Access
calculator. [https://github.com/open-guides/og-
aws/pull/110](https://github.com/open-guides/og-aws/pull/110)

------
emmjay
> A single EBS volume allows 10k IOPS max. To get the maximum performance out
> of an EBS volume, it has to be of a maximum size and attached to an EBS-
> optimized EC2 instance.

Out of date; EBS volumes can be up to 20k IOPS per volume and what is "maximum
size"? To get the maximum performance out of a volume depends on workload, the
instance size you've attached it to (rather than EBS Optimization) and the
number of IOPS provisioned, and whether you've prewarmed it from a snapshot
restore or not.

> A standard block size for an EBS volume is 16kb.

A block can be 1kb -> 256kb in size. It depends on the application.

> EBS volumes have a volume type indicating the physical storage type. The
> types called “standard” (st1 or sc1) are actually old spinning-platter
> disks, which deliver only hundreds of IOPS — not what you want unless you’re
> really trying to cut costs. Modern SSD-based gp2 or io1 are typically the
> options you want.

The ST1/SC1 wording is misleading. You only need '100s' of IOPS when dealing
with big blocks for ST1, and SC1 isn't performance oriented at all.

~~~
trowawee
Might be more helpful to make a PR, rather than comment here. It'll take about
the same amount of time.

------
nodesocket
I converted to pdf using Typora[1].

[https://github.com/nodesocket/og-aws-pdf/raw/master/the-
open...](https://github.com/nodesocket/og-aws-pdf/raw/master/the-open-guide-
to-amazon-web-services.pdf)

[1] [https://www.typora.io/](https://www.typora.io/)

~~~
zalzal
Let us know if you find the PDF helpful? We could post one on the GitHub repo
too — though it's so full of links I'm not sure how useful it is.

~~~
bogomipz
I think the PDF is a useful option, use case being offline reading.

------
xarope
Great work! I started using AWS back when it was just simple websites, and the
plethora of services now (50!), and pricing (especially pricing!), is
overwhelming to track.

So overwhelming, in fact, that I decided it was easier to get some VPSs and
use common, work anywhere, tools to manage (e.g. saltstack), than have to
skill up on AWS specific stuff.

------
kregasaurusrex
Thanks a lot for posting this, I went to a linux conference over the weekend
and was talking with some friends about their datacenter jobs. I felt
hopelessly lost in trying to understand all its intricacies at routing,
storage, and backup levels where this guide gives a good bird's-eye view of
the stacks.

------
usdeveloper
The Azure team could learn a thing or two about putting together good
documentation for their products.

------
wslh
I would add as a VPC gotcha the use of the EIP_Disable_SrcDestCheck flag [1]
to enable layer 2 capabilities. This is a feature that is only present in AWS.
Neither Google Cloud Engine nor Microsoft Azure have it. So, if you craft an
Ethernet packet modifying the destination address but not the destination IP
in your local subnet, the packet will be sent to the computer by IP and not by
MAC address as you expect in an Ethernet network.

[1]
[https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_N...](https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_NAT_Instance.html#EIP_Disable_SrcDestCheck)

------
chapingt
As someone who is having to learn AWS very quickly for an urgent project, I
thank you.

------
ohstopitu
I have recently started out on AWS (I initially used AWS like I used to use
Digital Ocean, however after trying out Serverless, I'm of a different mind
and changing my ways to do it the AWS way), So this is pretty awesome!

~~~
buckbova
Are you thinking dynamo for a backend, RDS, or other?

~~~
ohstopitu
Dynamo for the backend and ElasticCache (Redis) for some cache

~~~
buckbova
Know of any good resources, links, etc for dynamo or going to figure it out as
you go?

~~~
ohstopitu
I had tried a lot of databases (postgres, mongo, couch and very recently
Rethink) before trying out Dynamo. So I just jumped in, and started something
basic, and read tutorials as I went along.

There's still a lot of stuff I don't fully know about (for about Read / Write
volumes that is set - I left it at a default of 5) but I guess, I'll learn as
I go along.

------
aidos
Great guide. I've been using AWS since there were only a handful of services
and it's become increasingly hard to keep up with all the additional ones that
have been added in the last few years.

EFS had completely passed me by. Does anyone have experience with it? I'm
wondering what it would be like to use for Whisper / Graphite (just on a
single machine). I'm less interested with concurrent access and more
interested in not having to resize drives as data grows / overprovision drives
all the time.

~~~
ljosa
The latency is higher than I had hoped. I wrote 10,000 files with 10 kb in
each. It took 23 ms per file on average. Then I read them back. That took 8 ms
per file on average.

That's way too much for the use case I was contemplating, so I didn't
investigate further.

~~~
aidos
It definitely _felt_ a bit slow rysncing to it last night. In the Whisper use-
case, there are a ton of small appends to do every minute - so that could be
an issue. I'm going to set up a machine with a linux 4 kernel today to try it
on (as that's what they recommend, along with async mode).

------
nodesocket
Wow, this is a treasure chest of information. Bookmarked for sure.

------
alexnewman
A good edition, but I wish there was a place for horror stories about this
tech. For instance, we can't launch or than 4 or 5 containers a second on our
ecs clusters.

~~~
dastbe
Are you measuring by placement, by container is running, or by container is
fully initialized?

------
simonw
This is so needed. I find Amazon's official documentation to be way too full
of buzzwords and marketing speak. I just want someone to tell me what the
thing does!

~~~
alexbilbie
[https://www.expeditedssl.com/aws-in-plain-
english](https://www.expeditedssl.com/aws-in-plain-english)

------
xorgar831
I think a better approach would be to use annotations on the current AWS docs
so that additional information is inline with the official documentation so
you have both in the same place. The Hypothesis project is working on such a
browser plugin that does this for example and is having success with academic
research already. [https://hypothes.is/](https://hypothes.is/)

------
vmarsy
Thanks, I like that Service Matrix[1] !

I've a few questions for AWS experts :

The only container orchestration that is open source seems to be Kubernetes.
Is it easy to run on AWS?

What's the equivalent of Azure "Service Fabric" in the AWS world? (and in the
Google Cloud?)

[1] [https://github.com/open-guides/og-aws#service-
matrix](https://github.com/open-guides/og-aws#service-matrix)

~~~
nzoschke
You probably are aware, but AWS has a container orchestration service built
into the platform with ECS. The container agent is open source
([https://github.com/aws/amazon-ecs-agent](https://github.com/aws/amazon-ecs-
agent)).

We're building an open-source platform at Convox that leverages ECS very
successfully. [https://github.com/convox/rack](https://github.com/convox/rack)

In my experience, ECS is easy to run, as it's a first class part of the
platform. Boot up the right "cattle" AMIs with the right ASG configuration and
you're good to go.

K8, Docker Swarm, Mesos and Nomad have plenty of documented success but you to
stand up and operate the orchestration layer yourself. This is booting up
"pet" AMIs and making sure they are monitored, etc. Then you boot up your
"cattle" AMIs to run your apps.

The Convox philosophy is that you get application portability by packaging
your app correctly with Docker. The orchestration layer should be invisible,
something that you shouldn't build or operate yourself.

------
sharmak1
Fantastic guide! The cost management part with spot and network usage is
extremely helpful and practical. Thanks for pushing this out!

------
dsmithatx
Awesome! As I read this I wonder exactly how many 100's of hours I could of
saved the past 7 years if I had this resource.

------
mxuribe
Although I'm familiar (high-level only) with numerous topics/services related
to AWS, I'm still doing things _the legacy way_ on providers like Digital
Ocean (which I'm 100% happy with), and by no means a guru of AWS...So this
guide looks awesome for someone like me!

Kudos to the authors and contributors!

------
bmoresbest55
Wow, this is exactly what I need right now. Thanks to the original author
(github.com/jlevy I believe).

------
user5994461
Question to readers:

\- What are your goals on AWS?

\- What topic do you need help with? What articles would you like to be
written?

------
alainchabat
[https://github.com/Netflix/ice](https://github.com/Netflix/ice) looks pretty
good. But you have to pay $780 for the highstock license to use it. Anyone has
a free alternative to this?

------
falcolas
Sadly, I could never get the company lawyers to approve contributions under a
CC-BY-SA.

Of course, I'm not 100% sure I could get them to approve contributions to any
external repo due to liability concerns, etc.

~~~
jedberg
Can you just contribute as you instead of as your company?

------
derricgilling
Really like the single page format. Much easier to search compared to
scattered documentation on AWS's own site. Definitly like the 1:1 mapping to
Google /Azure

------
nepotism2016
Nice work but seriously can you please avoid using acronyms?!? You have plenty
of space to write

Simple Storage Service

Rather than making me scroll down to find out what KMS stands for!

Cheers

------
SteB
Great job. How are you guys keeping up to date with AWS and all the new
updates/features they launch every day?

------
mhewett
Can you add one column to the table: a brief description of the service?
Thanks for collecting all the information.

------
alexmorenodev
Do you mind if I translate it to PT-BR?

~~~
forwidur
The license allows you to do that and many other things.

Also, you might want to wait a day or two before starting and let the dust
settle a bit. ;-)

------
machbio
Wish there was more information about Elastic beanstalk - it always confuses
me about how it works..

~~~
drewmassey
I've had the impression that elastic beanstalk (which I use) has suffered the
fate of a few other Aws offerings in has been seen as less trendy than
Docker/ECS. (See also: cloud search vs elasticsearch). But EB can do some
things very well and very painlessly.

~~~
abrookewood
EB tends to work very well when you're requirements fit within its framework -
and very badly when you try to do anything differently. We've moved to
CodeDeploy because: EB was slow to deploy; often left applications in an
'unknown' state after deployment; ties application configuration to
deployment; and generally felt fairly restrictive.

------
bogomipz
This is fantastic! Nice work.

------
pawanpe
Awesome, thanks!

------
Dustin82
Nice

------
swingbridge
Excellent!

------
gjolund
This is a goldmine.

I've been compiling a lot of tips and tricks personally that I use to help
train coworkers. I'm definitely going to cross reference and see if I can open
a few useful PR's.

------
andrewvijay
This is fantastic. I was thinking about it in the afternoon and I see it now!
Very useful for guys like me who are just booting up in the back end and
devops side!

