
Footgun Prevention with AWS VPC Subnetting and Addressing - vinnyglennon
https://ukcloud.pro/aws-vpc-subnetting-and-addressing-6dd627a0ef50
======
nodesocket
Google Compute Engine makes VPC networking so much easier than AWS. They
automatically create solid defaults with a /20 subnet per region (4096
addresses) which can all communicate with each other privately. No need to
setup VPN's for cross region private communication or deal with NAT instances
or internet gateways. Private traffic is always private. If you want to expose
a VM just attach an IP address (static or ephemeral).

Also a cool trick is GCE instance names automatically get mapped in your VPC
DNS. If you create an instance named web1, all other instances should be able
to ping web1 out of the box with no additional configuration.

I can't stress enough how much better GCE is than AWS in terms of sane
defaults, user experience, performance and cost.

~~~
skywhopper
GCE does have a huge benefit in having been built several years after AWS
organically grew into a set of services that now in retrospect seem to have
obvious defaults. From my very limited experience with GCE, it seems to have
its own oddities that make managing it from an organizational level weirdly
difficult, but I can't argue about the details in a well-informed manner. I
will say that while the lack of dead-stupid cross-region VPC peering is
annoying in AWS, having talked to some of their engineers about the issue, it
seems that their reluctance to make global features dead-simple stems from
hard experience of their own in the risks of global interdependencies and the
failure conditions that can result.

But to your specific details, I'm curious about the "private traffic stays
private" and "no NATs required". So does GCE provide implicit NAT, or does it
just work how AWS public subnets work (ie, give it an IP and it's on the
Internet, don't give it one and it isn't)?

~~~
user5994461
>>> having talked to some of their engineers about the issue, it seems that
their reluctance to make global features dead-simple stems from hard
experience of their own in the risks of global interdependencies and the
failure conditions that can result.

Poor excuses. "It's too hard".

It is very hard to make global services. Google was the first to beat the
technology long before every one else. Most of their cloud services are
worldwide. They proved that it's possible.

Having spent much time in AWS. It seems the technology that runs it is
exclusively mono region. All regions are segregated and independent, like a
full AWS clone. There are no service that span regions.

AWS is simply not global. IMO it hints that AWS does not have the technology.

~~~
count
1) Mono-region is a feature not a bug. Regulatory requirements, citizenship
requirements, the ability to not have a single global point of failure, etc.
are all huge benefits to the regional breakdown as it stands today. Nothing
prevents you from deploying multi-region.

2) IAM is effectively global. Route53 is effectively global. AWS 'has the
technology', they've just chosen a different engineering stance than Google,
and are very up front about that.

~~~
user5994461
1) Locality requirements should be fulfilled by attaching or limiting
workloads to regions. Not by having the entire infrastructure and everything
only ever exist in a single place (which breaks numerous legal requirements by
the way).

2) ELB/ALB are not. S3 is not. AMI only exist in a single region. EBS cannot
be moved, ever. Billing is not really.

AWS is heavily region centric. For instance, it is impossible to list all
instances attached to your account.

In command line tools and the web UI, it asks you a region at startup, and you
will only see instances in that region.

~~~
count
1) They'd have to figure out how to constrain their service teams who support.
It's not just customer locality that's important.

2) Yep, agreed there. Again, the choice that was made :)

------
sudhirj
With all the ceremony required to get private subnet instances talking to the
internet, why not just put all instances on the public subnet but cut off
incoming connections using a security group?

~~~
kiallmacinnes
Defence in depth? An extra layer between an attacker and your secret data is
rarely a bad thing.

The tradeoff against convienence is worth it for some environments. E.g. what
if AWS has an issue with applying security groups properly and it fails open?

Arguably, failing open is the right thing to do _in some cases_ as it avoids
causing an outage for your users. However it does expose them to a risk they
may not have been prepared for.

I know OpenStack had an issue a few years back where, when the component
responsible for managing the security group rule implementation was reset, it
failed open until all rules were rebuilt (at the time, this was implemented as
iptables rules on the hypervisor's). Another bug (well, poor implementation)
meant it could take several minutes or more to rebuild the rules.

It happens, and you should be aware of and understand the risks of things like
this happening when designing your infrastructure and choosing which tradeoffs
are worth it for your particular use case.

~~~
sudhirj
The defence in depth concept makes sense. The good news is that if I
understand correctly, security groups fail closed - so that makes things a
little safer.

Just spoke to an AWS Architect, and the points made were similar - more secure
default state, less chance of screwing up - ham fisted attempts at security
groups can open things up too much, private subnets are tough to expose
unintentionally, adding multiple security groups makes the rules less
restrictive, possibly in unintended ways, etc.

------
hoodoof
x

~~~
cthalupa
Amazon doesn't know how many resources you're going to spin up and how many
ENIs they will need, so it's hard to really tell you how big of a subnet to
create.

You can also use the AWS NAT Gateway which doesn't require you run your own
NAT instance

~~~
vacri
Annoyingly, the VPC NAT Gateway is more expensive than running your own nat
instance. If your traffic is low enough to be handled by a t2.small or lower,
it's cheaper to run your own. Most of my NATs are nanos or micros.

AWS also double-dips on the traffic charges for the VPC NAT - you're charged
for the traffic it transfers, but you're also charged for the same traffic in
the general bill, from what I've been able to glean. Given that traffic is
where AWS is not competitively priced, it's something to be cautious of.

~~~
Pirate-of-SV
NAT Gateway support bursts up to 10 Gbps[1] and t2.small something like
125-200 Mbps [2]. For production use cases this can make a huge difference.

[1] [http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-
na...](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-nat-
gateway.html)

[2] [https://stackoverflow.com/questions/18507405/ec2-instance-
ty...](https://stackoverflow.com/questions/18507405/ec2-instance-typess-exact-
network-performance)

~~~
vacri
Sure, but you also don't need a big NAT Gateway if you're doing the majority
of your external chatter via an ELB. Our setup is basically production load
goes through an ELB, and incidental traffic goes through the default route
(the NAT), so the NAT really only handles traffic like me ssh'ing in and
wanting to install a tool, or config management setting up a new server.

But yes, if you're going to be sending big traffic through the NAT, a t2.small
doesn't have the network performance.

