
Moving The New York Times Games Platform to Google App Engine - spyspy
https://open.nytimes.com/moving-the-new-york-times-games-platform-to-google-app-engine-e9337f2c9444
======
obulpathi
Two things keep coming up while comparing GCP and AWS:

* This accomplishment would not have been possible for our three-person team of engineers with out Google Cloud (AWS is too low level, hard to work with and does not scale well).

* We’ve also managed to cut our infrastructure costs in half during this time period (Per minute billing, seamless autoscaling, performance, sustained usage discounts, ... )

~~~
vira28
This thread here came at the right time. Today whole day I attended the
DynamoDB training. Honestly, one thing that I understood is its cost based on
reads and writes per second. Irrespective of the amount of read data per
operation (whether its 1 bytes or 100 bytes), its always charged for 1KB. So,
as a work around what they suggested is using a Kinesis, a Lambda and an
another service to make the write operation as a batch, in such a way the
reads are near 1KB always. He pitched it like thats the perfect way to do. The
problem I see is too many moving pieces for a simple thing to achieve. If the
Dynamo team makes the reads cost based on the actual data we are all set.

~~~
obulpathi
Yep, same problems with many other services:

* Kinesis Streams: Writes limited to 1K/sec and 1 MB / shard, reads limited to 2K/shard. Want a different read/write ratio? Nop, not possible. Proposed solution: use more shards. Does not scale automatically. There is another service called Kinesis Streams that does not offer read access to streaming data.

* EFS: Cold start problems. If you have small amount of data in EFS, reads and writes are throttled. Ran into into some serious issues due to write throttling.

* ECS: Two containers can not use same port on same node. Anti pattern to containers.

AWS services have lots of strings attached and minimums for usage and billing.
Building such services (based on fixed quotas) is much easier than building
services which are billed purely pay per use. This complexity + cost
optimization pressures lead to complexity and require more human resources and
time as well. AWS got good lead in Cloud space, but they need to improve their
services without letting them rot.

~~~
oblio
> * ECS: Two containers can not use same port on same node. Anti pattern to
> containers.

Could you elaborate for this? I'm not sure I understand, are you saying that 2
containers cannot be mapped to the same host port? Because that would seem
normal, you can't bind to a port where there's already something listening.
But I guess I must be missing something.

~~~
alexeldeib
I'm not familiar with the details of AWS here, but maybe the OP means mapping
two different host ports to the same port on two different containers? That's
all I can imagine that would be a container antipattern in the way described.

~~~
hobofan
That is perfectly possible with ECS, so I don't know what OP was referring to.
The thing I remember though is that you have to jump through a lot of hoops
like making 4 APIs calls (or worse with pagination) for what should have been
a single call to make such a system work on ECS.

------
ciguy
As a DevOps consultant I've actually worked with clients migrating stacks to
and from GCE/AWS (Yeah, both ways, not the same client).

What I've found in aggregate is that GCE is a bit easier to use at first as
AWS has a LOT of features and terminology to learn. When it comes down to it
though, many GCE services felt really immature, particularly their CloudSQL
offering.

One client recently moved from GCE to AWS simply because their CloudSQL (Fully
replicated with fail-over setup according to GCE recommendations) kept
randomly dying for several minutes at a time. After a LOT of back and forth
Google finally admitted that they had updated the replica and the master at
the same time, so when it failed over the replica was also down.

There were other instances of unexplained downtime that were never adequately
explained, but overall that experience was enough for me (And the client) to
totally lose faith in the GCE teams competence. Even getting a serious
investigation into intermittent downtime and an explanation took over a month.
By that time our migration to AWS was in progress.

GCE never did explain why they would choose to apply updates to replica +
master SQL at the same time and as far as I know they are still doing this. I
asked if we could at least be notified of update events, was told that's not
possible.

There were other issues as well that taken together just made GCE seem
amateurish. I'm sure as they mature a bit things will get better, and it is
cheaper which is why I wouldn't necessarily recommend against them for
startups just getting going today. By the time you are really scaling it's
like they'll have more of the kinks worked out.

~~~
karmakaze
Why CloudSQL instead of Cloud Spanner? If for existing SQL workloads I can
understand but for new services I'm admiring Spanner over DynamoDB

~~~
vira28
Looks like Spanner is a relational database. DDB is just a key-value pair. So,
is it fair to compare against them or am i missing something?

~~~
ubernetes
Why wouldn't it be "fair"?

~~~
dullgiulio
Because a key-value store is a foundamentally simpler data structure (it's an
hash) than a relational database, which tracks the relations between different
data types. If you make an advanced use of the key-value store, you have a lot
of logic in the application (for example to key management, cascade operations
between related data...) which a relational database should do for you. It's
not fair because there is a development cost in using the key-value you are
ignoring.

~~~
ubernetes
Dynamo is not a k-v store.

------
neya
Hey community, let me share my experience with AppEngine. I work in a small
firm, where we've developed a massive Software Application comprising of 12
medium-sized apps. I went with Phoenix 1.3 w/ the new umbrella architecture.

With AppEngine, the beauty is that you can have many custom named
microservices under one AppEngine project and each microservices can have many
versions. You can even decide how much percentage of traffic should be split
between each of these microservices.

What's awesome is, in addition to the standard runtimes (Ruby, Python, Go,
Java, etc.) Google also provides something called custom VMs for AppEngine,
meaning you can push docker based setups into your AppEngine service, with
basically any stack you want. This alone is a HUGE incentive to move to
AppEngine because usually custom stack will require you to maintain the server
side of things, but with Docker + AppEngine, zero devops. Their network panel
is also very intuitive to add/delete rules to keep your app secured.

I've been using AppEngine for over 4 years now and every time I tried a
competitive offering (such as AWS Beanstalk, for example) I've only been
disappointed.

AppEngine is great for startups. For example, a lesser known feature within
AppEngine is their real-time image processing service API. This allows you to
scale/crop/resize images in real time and the service is offered free of
charge (except for storage).

Works really well for web applications with basic image manipulation
requirements.

[https://cloud.google.com/appengine/docs/standard/python/imag...](https://cloud.google.com/appengine/docs/standard/python/images/)

The best part is, you call your image with specific parameters that'll do
transformations on the fly. For example, <image url>/image.jpg?s=120 will
return a 120px image. Appending -c will give you a cropped version, etc.

I really hope to see AppEngine get more love from startups as it's a brilliant
platform, much more performant than it's competitors' offerings. For example,
I was previously a huge proponent of Heroku and upon comparing numbers, I
realized AppEngine is way more performant (in my use case). I'm so glad we
made the switch.

If you're looking/considering to move to AppEngine, let me know here and I'll
try my best to answer your questions.

~~~
smn1234
The way you've delivered this "experience" makes you sound like you either
work for Google or were asked to make a sponsored statement - for credits or
$.

~~~
sunsetMurk
or a passionate user who had a great experience?! I love finding solutions
which require less 'square peg round hole'. Unfortunately, rare these days
when piecing together a stack w/ the myriad of platforms/frameworks/etc.

~~~
BoorishBears
I'm usually not skeptical of comments but this comment definitely feels
"artificial".

I think Google has better things to do than to pay people to comment on HN,
but I do think either this person is trying too hard to sell us on Google
Cloud because they like it (which isn't a bad thing per say)

Edit: I thought about it and they probably aren't related to it, probably just
really enthusiastic about it (good thing) but they want to sell us on it (eh,
not sure how I feel about)

~~~
sunsetMurk
Yep - I guess I'm a bit empathic to the comment. I'm always trying to sell
what I'm using to others, to get more into that camp, to generate more
discussion and innovation. But, it's all just like ice cream [1] anyways.

1-
[https://twitter.com/adamlaz/status/900621343347146752](https://twitter.com/adamlaz/status/900621343347146752)

------
nrjames
I migrated a big data stack to GCP from AWS. Reasons: GCP has better
documentation, the AWS console and various services confuse the heck out of me
(I guess I'm getting too old), and the security integration between GCP
services saves a huge amount of time. It's super easy and very fast to used
the Google Compute Engine VMs. Given that the company I work for uses G Suite,
it's a piece of cake to implement SSO and other integration pieces. It's also
cheaper for us than AWS and more performant.

~~~
jsudhams
Wait until GCP becomes older with bigquery and other versions. Also GCP does
not have regions ready and it takes forever to get a new region. Waiting for
Mumbai region for more than 3 to 4 months now.

But one thing i like about GCP is that it allows to the limit setting in terms
of cost and ensure you wont cross it. In case of AWS it can give alerts but
for some reason say all you team in in one location and there is emergency
like flood etc and you dont check email then you are done. I stopped using AWS
after i learned that there is simply no way to set limit. Waiting for GCP to
open their Mumbai region. sigh.

Also AWS is very deceiving with free tier , there is simply no way to
understand which products get free and worst case is after free tier you will
get charged.

~~~
yeukhon
AWS doesn't add a new region every 3-4 months either. Adding a new region is
very complicated. Normally vendor does not actually build the DC themselves.
They would source from other data centers in the region whenever possible.
Building a new DC is not something can be taken lightly. Then finally local
laws.

------
vs2
"Due to the inelastic architecture of our AWS system, we needed to have the
systems scaled up to handle our peak traffic at 10PM when the daily puzzle is
published."

WT... I had to reread this to make sure I didnt misunderstand... why not work
on making the current arhictecture elastic?! #cloudPorn

~~~
tyingq
The "inelastic" might have been a shot at AWS. When pressed, the AWS people do
use phrases like "pre-warming", "over provisioning" and "advance notice"
around their ELB/ALB setup and ECS.

Google's cloud salespeople pitch that they don't require any of that.

~~~
g09980
Curious if the need for pre-warming ELB/ALB still applies. Last time this came
up, an AWS employee mentioned it is no longer necessary
([https://news.ycombinator.com/item?id=14052079](https://news.ycombinator.com/item?id=14052079)),
but would be nice if this was documented.

~~~
callalex
I don't want to dox myself but about a year ago when my employer forgot to
notify AWS about switching our production traffic (about 5K rps at that time)
from one ELB to another we failed requests for several minutes before we
decided to just switch back to the old ELB then ask them to do a prewarming
before we switched again.

------
pgrote
Interesting they are using Medium instead of in house publishing tools. It is
the first time I've noticed then open.nytimes.com articles.

~~~
devrandomguy
Didn't you hear? Medium pays per clap now, so the NYT authors are all jumping
ship /s

------
Jedi72
Getting pro GCP articles to the top of HN must no-doubt be a high priority for
the Google marketing team. This is the nature of modern advertising, sneakily
trying to subvert your thinking by masquerading as something else.

~~~
theDoug
We actually spent the entire day at the Giants game today. ¯\\(ツ)/¯

There's no incentive for high ranking HN posts, or any HN posts, actually. If
there were, you wouldn't see others continually submit our news here before we
do. This was a nice and unprompted post for everyone in GCP to read, as well.

(Disclosure: I work on GCP as a product marketer.)

~~~
ben_jones
A GCP product marketer responding to negative commentary on HN within ~20
minutes of it being posted strikes me as automated.

EDIT: It is a double standard though, HN readers want access and responses
from people on the GCP team but at the same time tinfoil hat subliminal
marketing etc.

~~~
sb8244
I would be reading this every 10 minutes for new comments if it was my product
up for discussion.

~~~
bastawhiz
Ten minutes even seems like a lot. If I saw my website on HN, I'd have alerts
set up for new comments.

------
NightlyDev
This doesn't really make much sense to me. How many peak users are there?
What's the number of requests per second?

I can't imagine that the load would be so high that it wouldn't be possible to
do it without GCP with three developers.

It would be way more interesting with performance details. :)

------
bsaul
Has anybody had successful experience deploying docker containers on appengine
? Last time i tried, i had such a bad experience in terms of deployment speed
( time to build the image, then upload it, then waiting for the stuff to
deploy) that i reverted to managing my own gce instance.

But maybe i had bad luck..

~~~
tarr11
I just tried to move a rails app to appengine. It uses the flexible
environment (eg, docker). It took like 10 or 15 minutes to deploy, each time.

Heroku does my deploy in about 5 mins.

[https://groups.google.com/forum/#!topic/google-
appengine/hZM...](https://groups.google.com/forum/#!topic/google-
appengine/hZMEkmmObDU)

GCloud is cheaper though.

Also, VMs spin up in GCloud amazingly fast. Like 5 seconds. Feels like
somebody at Gcloud just needs to go and fix this. No reason this is so bad...

~~~
brianwawok
Deploy is 15 seconds on GKE. Wonder why app engine is so bad.

~~~
lftl
With GKE you don't necessarily update your load balancer rules with a deploy
correct? The linked thread points the blame for app engine deploys on waiting
for GCLB to update.

~~~
brianwawok
Correct, you have software based load balancing under the covers.

My Google Load Balancers never move.. It is a single thing that points each
node (physical machine) in the cluster, and distributes traffic between then.

Each node knows how to route traffic to each app. So when I deploy that app,
the software load balancer at the node level will slowly move traffic over
from old app to new app. Entire thing is MAGICAL. And 0 downtime, very very
fast deploys.

Edit - But yes this explains iy. Changing the google load balancers is like a
5 minute ordeal. Total pain. Nice that with GKE you only need to touch them
when your node count changes, which can be very rare (~monthly for me)

------
foxylad
OT. How nice not see a single "avoid all Google services because reader"
comment. Maybe we are finally moving on.

~~~
CaveTech
Yet here you are.

------
merb
Uh thanks to this article I've seen that AppEngine know supports Java8, this
is really really cool.

------
revelation
I was sure this was about some multiplayer game thing, but no, it's a
crossword. Not entirely sure what they are even scaling here, I was expecting
an article about a CDN..

~~~
jprob
The game allows you to sync your game progress across multiple devices and
it's subscription based so a CDN wouldn't do much help there.

Also, realtime multiplayer crosswords are coming! I'll be speaking at GothamGo
this year about that exact topic.

------
kennethh
Anyone know how much it cost to add a custom domain and SSL to
AppEngine(Standard og Flexible)? I have been looking and not able to find out
how much it cost.

~~~
iamgopal
SSL is free. Custom domain also free.

~~~
kennethh
That explains it, would be nice if they could write it down somewhere as a
feature:)

~~~
rhodysurf
Its free for custom but only to use existing certs. You still have to set up
the cert and such with your domain as you normally would. You also get the
*.appengine.com domain with automatic SSL>

~~~
iamgopal
SSL cert provided by Google is free and is in private alpha stage. It's coming
soon with GA.

------
zitterbewegung
Nice advertisement that Google bought from the NyTimes.

------
mbesto
This reads eerily like a press release for GCP...

~~~
s17n
But check out OPs posting history.

~~~
lallysingh
4 posts about NYT crossword. So they're talking about their work, and their
work uses GCP.

~~~
spyspy
Quite

