
Going Multi-Cloud with AWS and GCP: Lessons Learned at Scale - jbyers
https://metamarkets.com/2017/big-cloud-data-aws-and-gcp/
======
nodesocket
One of the biggest benefits of Google Cloud is networking. By default GCE
instances in VPC's can communicate with all instances across zones and
regions. This is a huge plus.

On AWS, multi region involves setting up VPN and NAT instances. Not rocket
science, but wasted brain cycles.

Generally, with GCP setting up clusters that span three regions should provide
ample high availability and most users don't need to deal with the multi cloud
headaches. KISS. You can even get pretty good latency between regions if you
setup North Carolina, South Carolina, and Iowa. Soon West Coast clusters will
be possible between Oregon and Los Angels (region coming soon).

~~~
sdrothrock
I may be completely off here, but isn't this due to their underlying
architecture decisions? That is, AWS from the start has kept all regions
completely separate, so that problems in one region do not influence another.
But GCP has has issues with failure across regions IIRC.

~~~
outworlder
Having a software defined networking spanning across regions and failure
cascades across regions are two different things. There's nothing preventing a
vendor from presenting to you a single network, while they are actually
distinct networks.

~~~
hossbeast
Having distinct networks in different regions encourages you to architect your
application in a fault tolerant way.

~~~
kakwa_
Or the contrary. In most cases there is something to synchronize between
regions, like a replica of the data.

With difficult interconnection of regions, it makes it somewhat harder to do,
and it can easily end-up with "meh, AZs are good enough".

------
ad_hominem
If any Google Cloud people are listening I wish you had an equivalent to AWS's
Certificate Manager. Provisioning a TLS certificate which automatically renews
for eternity (no out-of-band Let's Encrypt renewal process needed) and
attaching it to a load balancer is so nice compared to Google Cloud's manual
SslCertificate resource creation flow[1].

To a lesser extent, it's also nice registering domains within AWS and setting
them to auto renew. Since Google Domains already exists, it would be neat to
have this feature right inside Google Cloud.

[1]: [https://cloud.google.com/compute/docs/load-
balancing/http/ss...](https://cloud.google.com/compute/docs/load-
balancing/http/ssl-certificates)

~~~
rmhrisk
We hear you, while I can't speak to future products and features I can say we
understand there is room to improve the SSL provisioning and lifecycle
management story in our products and we are making investments in that area.

------
vira28
One thing that I liked with GCP is their recommendation for cost saving. I
spun up a compute engine for a hobby project and within minutes they gave
recommendations to reduce the instance size and how much i can save. I don't
think AWS offers something like that. Correct me if I am wrong.

~~~
obulpathi
Even better are Google managed services (PubSub / Dataflow / Datastore), which
scale up and down based on usage (cloud native products) and thus save money
automatically compared to their equivalents in AWS (Kinesis / Kinesis
Analytics / DynamoDB) which does not autoscale.

~~~
ranman
DDB recently added autoscaling -- [https://aws.amazon.com/blogs/aws/new-auto-
scaling-for-amazon...](https://aws.amazon.com/blogs/aws/new-auto-scaling-for-
amazon-dynamodb/)

~~~
CSDude
It does not work well, it gives late responses

~~~
ranman
Really? Feel free to email me about that randhunt@amazon

------
azurezyq
One extra point for tracking VM bills:

GCE bills are aggregated across instances. To get more detailed breakdown, you
can apply labels to them and the bills will have label information attached in
BQ.

Alternatively, you can leverage GCE usage exports here:

[https://cloud.google.com/compute/docs/usage-
export](https://cloud.google.com/compute/docs/usage-export)

Which has per-instance per-day per-item usage data for GCE.

Disclosure: I work for Google Cloud but not on GCE.

------
manigandham
When it comes to GCP:

\- They have _Role Based Support_ plans which offer flat prices per subscribed
user which is a much better model. [1]

\- Live migration for VMs mean host maintenance and failures are a minor
issue, even if all your apps are running on the same machine. It's pretty much
magical and when combined with persistent disks, effectively gives you a very
reliable "machine" in the cloud. [2]

1\. [https://cloud.google.com/support/role-
based/](https://cloud.google.com/support/role-based/)

2\. [https://cloud.google.com/compute/docs/instances/live-
migrati...](https://cloud.google.com/compute/docs/instances/live-migration)

------
user5994461
>>> on AWS you have the option of getting dedicated machines which you can use
to guarantee no two machines of yours run on the same underlying motherboard,
or you can just use the largest instance type of its class (ex: r3.8xlarge) to
probably have a whole motherboard to yourself.

Not at all. Major mistake here.

When you buy a dedicated instances on AWS, you reserve an entire server for
yourself. All the VMs you buy subsequently will go to that same physical
machine.

In effect, your VMs are on the same motherboard and will all die together if
the hardware experiences a failure. It's the exact opposite of what you wanted
to do!

~~~
ranman
I think two concepts are being conflated:

Dedicated Instances: [https://aws.amazon.com/ec2/purchasing-options/dedicated-
inst...](https://aws.amazon.com/ec2/purchasing-options/dedicated-instances/)

and

Dedicated Hosts: [https://aws.amazon.com/ec2/dedicated-
hosts/](https://aws.amazon.com/ec2/dedicated-hosts/)

~~~
stephengillie
At my current job, we're looking into DIs to reduce our SQL costs. With
standard Spot/RIs, we're paying per-core for SQL Server. But with a DI, we're
expecting to be able to license against the physical sockets instead.

> _You can use Dedicated Hosts and Dedicated instances to launch Amazon EC2
> instances on physical servers that are dedicated for your use. Dedicated
> Instances are Amazon EC2 instances that run in a VPC on hardware that 's
> dedicated to a single customer. You can also use Dedicated Hosts to launch
> Amazon EC2 instances on physical servers that are dedicated for your use._

> _Dedicated instances may share hardware with other instances from the same
> AWS account that are not Dedicated instances._

> _An important difference between a Dedicated Host and a Dedicated instance
> is that a Dedicated Host gives you additional visibility and control over
> how instances are placed on a physical server, and you can consistently
> deploy your instances to the same physical server over time._

It looks like you can launch DIs on your DHs, or on any arbitrary host; but
once you have a DI on an arbitrary host, only your VMs will run there; so a de
facto Affinity policy. And any instance you launch on your DH is automatically
a DI.

Is there a benefit to running DIs without having a DH? It sounds like having a
DI gives you 90% of a DH. The DH gives you is a few hardware details (which
might be essential for licensing), and like GP suggested would let you choose
Affinity (or Anti-Affinity) between them manually.

 _As a result, Dedicated Hosts enable you to use your existing server-bound
software licenses like Windows Server and address corporate compliance and
regulatory requirements._

This is the first I'm hearing about DHs, and it sounds like that might be what
we need, instead of the DIs we've been telling other teams about.

~~~
samstave
If you have hipaa reqs, your signing an agreement with Amazon will require you
to host pii/phi on a dh

~~~
otp124
This changed [https://aws.amazon.com/blogs/apn/aws-hipaa-program-update-
re...](https://aws.amazon.com/blogs/apn/aws-hipaa-program-update-removal-of-
dedicated-instance-requirement/)

~~~
samstave
Wow. Thanks!

------
dswalter
If AWS were to go to a per-minute billing cycle, they would be instantly more
price-competitive with Google's offering. Or, to put it the other way around,
those leftover minutes form a significant chunk of AWS's profit margin.

~~~
obulpathi
I don't think so. GCP's bill is usually about 50% of AWS's bill for same
application, if you run it full hour (from my personal experiences and from
several others as well: [https://thehftguy.com/2016/11/18/google-cloud-
is-50-cheaper-...](https://thehftguy.com/2016/11/18/google-cloud-
is-50-cheaper-than-aws/)). GCP has lot more cost saving features like seamless
scalability, custom shapes, sustained discounts and so on. If you workloads
span less than hour, GCP can offer more then 50% savings.

~~~
ranman
I refuted some of the networking claims in that article previously (I work for
AWS). Especially the bizarre claims that you have to get a C4.4xlarge for
1gpbs... The 220 mpbs network cap claim is just not true. Just run iperf3 on
any aws instance to a GCE instance and you can see greater than 220mpbs.

~~~
isatty
Honestly we all know that the small instances have terrible CPU that doesn't
let you use the advertized 1Gbps anyway. Other than that, even if AWS let
1Gbps traffic go on for a while, you get throttled pretty quickly from my
experience.

------
matt_wulfeck
> _As we investigated growth strategies outside of a single AZ, we realized a
> lot of the infrastructure changes we needed to make to accommodate multiple
> availability zones were the same changes we would need to make to
> accommodate multiple clouds._

Maybe he author means multiple _regions_? Multi az is so easy. Everything
works. Multi region is much harder.

------
whatsmyhandle
Very nice writeup! A nice, detailed read that was easy to understand.

It seems to focus more on raw infrastructure (EC2 vs GCE) instead of each
company's PaaS offerings. Obviously AWS has the front runner lead here, but
would be super curious in a comparison of RDS vs. Cloud Spanner for instance.
(pun unintentional, but then realized, and left in there)

~~~
outworlder
This should be RDS vs Cloud SQL

------
swozey
Great thorough comparison and falls very into line with my experience.
Definitely worth the read. Thanks!

------
throwaway0071
Off Topic: it's frustrating that these companies spend quite a lot of time and
money learning about the complexities of their infrastructure but when you're
interviewing at such companies, you're expected to have answers for everything
and a complete strategy for the cloud.

/rant

------
hobolord
Great post! How difficult is it to switch from an AWS EC2 instance to the GCP
version?

------
mrg3_2013
Nice post! I will be using it as a reference.

