
Node.js on Google App Engine Goes Beta - boulos
https://cloudplatform.googleblog.com/2016/03/Node.js-on-Google-App-Engine-goes-beta.html
======
davidjgraph
So many negative views of GAE, I have to add our, overwhelmingly positive,
experience.

We were paying $200/month for a load balanced EC2 setup in 2011, that's for
around 2k unique daily users. The load balancer kept on false triggering new
instances, so we went to GAE in 2012.

Since then our uptime has been horrifically good, heavily thanks for GAE,
trashing that of our competition, who mostly use AWS.

Last month we paid $150 for everything on GAE, traffic, CPU, etc. G-Analytics
shows a peak of 54k daily uniques last week. All the scaling is dealt with for
us, we can handle literally any traffic peak. i.e. x27 more traffic, lower
cost, better uptime, spending virtual zero time looking at the operations side
of things.

This is a mission critical app, although, we've never invoked support, so I
can't comment on that side.

~~~
stephenitis
did software change to account for efficiencies?

~~~
davidjgraph
No, the basic architecture hasn't changed, but the app does a lot, lot more
than it did in the AWS days.

------
mattbillenstein
Don't use AppEngine. I migrated Homejoy off of AppEngine in 2014 and I can't
recommend the platform for any reason whatsoever. It's slow, expensive, not
based on OSS, and not particularly stable. Bits of it (urlfetch, search, etc)
would go down for hours at a time and you're basically hosed until they fix
it.

I think Heroku is usable with a similar level of effort -- and you have a much
easier migration path to something else if you're not using Google's
proprietary datastore IMO.

~~~
justinblat
Apologies you had a bad experience. There's no doubt the original App Engine
had some... quirks.

You may want to take a look at Managed VMs. All of the new runtimes are based
on Docker images, are entirely extensible, and don't require the use of the
original App Engine APIs:

[https://cloud.google.com/appengine/docs/managed-
vms/](https://cloud.google.com/appengine/docs/managed-vms/)

You can use whatever Database or NPM modules you want.

~~~
hypertexthero
It's great to see App Engine getting better! I'm exploring it again after
trying the original app a few years back and am enjoying the improved
interface and abundant documentation and tutorials.

One suggestion: Make a cloud storage product option free up to a few megabytes
for hosting small personal sites. Maybe it exists already but I haven't found
it.

~~~
glaberficken
You can easily run a small personal site on the free tier of GoogleAppEngine.
Just google for "app engine static site"

There are a ton of options on what you can achieve inside the free tier. But
the simplest one for the use case you seem to require is to author a static
html site in a good WYSIWYG html editor and then deploying it as a static site
in GAE.

~~~
hypertexthero
Thank you — I will try this!

Edit: My use case is to move a couple of static sites to a host that supports
a secure HTTPS connection by default, which App Engine seems to do:
<[https://cloud.google.com/appengine/docs/python/console/using...](https://cloud.google.com/appengine/docs/python/console/using-
custom-domains-and-ssl>). I'll try moving one of my sites over and set up SSL
with with a certificate from
<[https://letsencrypt.org/>](https://letsencrypt.org/>).

------
Strom
This seems to be for _Managed VM_ [1] only, so it's a case of better
documentation & tooling, but no new functionality. You can already run
arbitrary code of any language on Managed VM instances.

[1] Managed VM is a sort of meet-half-way service from Google that is quite
freeform like Compute Engine, but also offers access to a good chunk of App
Engine's services. Still, it's very much not the standard App Engine.

~~~
plexicle
Oh, are you kidding me? Man I got so excited when I saw the title, I already
spammed the link to a couple buddies. I should have read it first.

I want real GAE. Not managed VMs. I want to be able to spin up a quick GAE
version on a deploy, and if it doesn't get the traffic to scale up, then it
just sits there and doesn't cost me anything.

Managed VM has 2+ instances running all month and is an incredible PITA to
spin down if I need to temporarily. (The current workaround is to deploy a
non-VM app and set that as default, which is just insane.)

GAE apps based on Managed VMs need a free quota as well. I have many apps but
some of them don't do much most of the time. A couple will spike hard
randomly. I don't mind paying when it does. I like the security that if/when
something really takes off, GAE has my back.

In the meantime, I'll stick to Go on GAE. I'd love to be able to use Node, but
things add up quickly if I have a couple hobby apps.

I'm glad that they are pushing Node and I'm a HUGE fan of the Google Cloud. I
just really want Node as a native GAE thing.

~~~
thesandlord
There is a easier way to spin them down now: gcloud preview app versions stop
main

~~~
plexicle
Well, that's good at least. Must have been very recent, I was talking to
support only a couple weeks ago.

~~~
justinblat
I think technically that command is still hidden lol. You can run it, it just
doesn't show up in the docs. We'll be officially releasing that one in a few
weeks.

~~~
dlor
Although "gcloud preview app modules stop <module> \--version=<version> is
documented, and does the same thing.

------
justinblat
Greetings folks, I'm the PM for Node.js on App Engine. If anyone has any
questions, feel free to ask me anything!

~~~
tomh-
I have one, how can European individuals use the paid version of any Google
cloud services? Last time I checked only commercial entities were allowed to
use the paid version.

~~~
boulos
Warning: IANAL. First, sorry for the pain! As an explanation, it's all about
VAT (determining it, collecting it, etc.). Outside of Ireland we dump it on
you:

[https://support.google.com/cloud/answer/6090602](https://support.google.com/cloud/answer/6090602)

But specifically:

> “Business” status means that you'd like to see a potential economic benefit
> from your development activities, for example: using the Google Cloud
> Platform to develop prototypes or applications with a view to generating
> revenue in the future. Most software developers -- including affiliates,
> sole traders, self-employed merchants, partnerships, students and others --
> use Google Cloud Platform for business purposes.

I can't help you decide what your VAT on the use of our services would be
(this is the realm of accounts and lawyers). I know AWS makes a different
decision than we do, and we're very aware of the friction (again, sorry!).

Disclaimer: I work on Compute Engine, but again IANAL.

~~~
tomh-
Thanks for your answer, although its worrying we need to have a lawyer around
to see how we can signup without accidentally violating my local country's VAT
regulations as a developer who just wants to pay and build stuff :(

------
shubb
I used google app engine for a project with very specific requirements - the
client wanted an application for internal use that didn't have any ongoing
costs (sign off) and would look after itself for a couple of years.

The platform has a lot of quirks, and I had to do a lot more learning than I
expected to get the fairly simple project done. But once it was up, it did
it's thing.

I'd definitely recommend for free hosting internal applications and personal
projects that are just for you to use. Just understand that you'll need to
shape your application around the wierdnesses of the platform.

------
esilverberg2
I wrote about my (unsuccessful) app engine experience many years
ago...hopefully they've improved since then.

[http://www-cs-students.stanford.edu/~silver/gae.html](http://www-cs-
students.stanford.edu/~silver/gae.html)

~~~
boulos
Good writeup! I think the _main_ change is that App Engine has really morphed
into a "You know, you don't have to use Datastore". Cloud SQL was added
precisely because of the issues you raise: being on App Engine meant being in
a totally foreign world, and not being able to get out easily.

If you were to try again today, you'd take stock Django 1.9 via pip install
Django -t lib and using vendoring:
[https://cloud.google.com/appengine/docs/python/tools/librari...](https://cloud.google.com/appengine/docs/python/tools/libraries27#vendoring)
. You point it at your Cloud SQL instance in your DATABASES line
([https://cloud.google.com/appengine/docs/python/cloud-
sql/dja...](https://cloud.google.com/appengine/docs/python/cloud-
sql/django#usage)) and you're done.

tl;dr: Yep! Things have improved in the last 5 years.

Disclosure: I work on Compute Engine.

~~~
sspross
It sounds like you know what you're talking about. Would you mind to take a
look at my issue/question on the appengine django skeleton[1]. The Problems
start after you've pointed your database to cloud sql und start using GAE
specific features like task queue etc.

[1] [https://github.com/GoogleCloudPlatform/appengine-django-
skel...](https://github.com/GoogleCloudPlatform/appengine-django-
skeleton/issues/14#issuecomment-196490921)

~~~
boulos
For your first question (playing with Cloud SQL) you've got two options.
Directly interact with Cloud SQL from your laptop or like you said have a
"development MySQL" separate from prod / staging.

Pointing at your Cloud SQL instance (maybe using a staging or development
database) can be done in settings.py. The switch between modes let's you
specify the IP address for your Cloud SQL instance when you're not on App
Engine (which uses a special UNIX domain socket). We also recommend you
connect over SSL when doing so outside (see more at the link I provided, or
search for cloud sql ssl).

Having a local environment for MySQL is also easy enough depending on your
box, but has the usual difficulty of needing to have a good test suite / being
populated with useful data. I think it's worthwhile if you can pull it off,
but I've seen companies big and small decide its not worth it (and instead use
a separate table or database on the same server, maybe as a different user as
another line of "don't screw up prod").

Finally, the interaction between Django's development server and the App
Engine one isn't great. My response was intended to remind people that if
you're not looking to use App Engine specific APIs you really can just get
going and almost not notice. _But_ you should be able to run your migrations
speaking to Cloud SQL remotely without needing to import say Task Queues. So
my workflow was to use manage.py for things like that, while using the
dev_appserver when I actually needed to emulate App Engine (as opposed to the
Django runserver command).

Does that help?

Edit: I forgot to add that we're working on decoupling the "emulation" of our
services so that this kind of interop works better. See
[https://cloud.google.com/sdk/gcloud/reference/beta/emulators...](https://cloud.google.com/sdk/gcloud/reference/beta/emulators/)
for Datastore and PubSub (and more on the way).

------
cdnsteve
What version of node is supported? Currently in all other languages they are
usually a major version behind (in everything).

~~~
justinblat
You can specify whatever version of Node.js you want to use in the engines
section of your package.json:

[https://cloud.google.com/nodejs/resources/runtime#engines](https://cloud.google.com/nodejs/resources/runtime#engines)

~~~
BinaryIdiot
Frustrating that it doesn't specify how it gets those versions. Is it always
direct from the node folks so it's going to always have all versions or do
they have their own cache?

~~~
diggan
Reading the source code of the docker container they use
([https://github.com/GoogleCloudPlatform/nodejs-
docker/blob/ma...](https://github.com/GoogleCloudPlatform/nodejs-
docker/blob/master/base/install_node#L34-L44)) it seems like it's reading the
available versions from an URL
([http://storage.googleapis.com/gae_node_packages/node_version...](http://storage.googleapis.com/gae_node_packages/node_versions))
which points to some Google hosted storage. So I'm guessing they have some
process to update this list after a new release and also caches the versions
themselves.

~~~
dlor
Yep, we update the available versions constantly and just serve them from GCS
to reduce download time.

I'll get this documented somewhere.

Disclosure: I work on GCP.

~~~
BinaryIdiot
Fantastic, thanks!

------
TheAceOfHearts
So, considering that it's beta, where should I expect rough edges? Is there
any list that shows what's missing to go from beta to full production? I'm
actually really interested in trying this out, but I want to know what
potential issues I should expect to deal with.

Something I found really awesome about Google's services is that they provide
fake local implementations that you can run from the CLI. This means that you
can run tests without having to hit the outside world!

Somewhat related to that, if you're using S3, check out s3rver [0]. It's a
node implementation of fake-s3 [1], which is a local implementation of S3 that
you can use during development and testing.

Looking through the examples, I'm confused about the deployment strategy for
SPAs. Looking at the webpack example [2], you build the frontend app every
time the server runs? That's REALLY slow for any reasonably sized SPA :(. It
means you generate a new build each time the server restarts, and for
something like node that's really common. (AFAIK, the suggested strategy is to
just let it crash and restart quickly.) But it also means that you might have
multiple servers running different version of your dependencies (since there's
no shrinkwrap). Even worse, if you're running multiple servers with slightly
different dependencies, it might also mean that you'll end up building
slightly different versions of your SPA in every server for any given release!
I'd love to see an example where you build and tag the SPA in a single place,
and you make sure all the servers are serving the same bundled code.

[0] [https://github.com/jamhall/s3rver](https://github.com/jamhall/s3rver)

[1] [https://github.com/jubos/fake-s3](https://github.com/jubos/fake-s3)

[2] [https://github.com/GoogleCloudPlatform/nodejs-docs-
samples/b...](https://github.com/GoogleCloudPlatform/nodejs-docs-
samples/blob/master/appengine/webpack/package.json)

~~~
eknkc
This is my understanding about Google Cloud services;

    
    
      - Alpha means sub par quality / rough edges / unresolved issues.
      - Beta means fairly stable but no SLA, no guarantees from Google.
    

So I'd guess it's fine to use now.

~~~
Xorlev
With Google it's either beta or deprecated.

------
mastazi
Wow, I thought that GAE was basically dead and now I find out that, besides
the latest addition of Node.js, it supports Python 3, Java 8 and even Docker-
based custom runtimes...

~~~
ekso
Why did you think GAE was basically dead?

------
smegel
Interesting things are happening to GAE these days. They are moving towards
custom runtimes using docker that the user can configure...even their
datastore API is being replaced by something resembling S3. GAE -> Google
Cloud Platform.

~~~
thesandlord
Cloud Datastore is more like DynamoDB, not S3. Cloud Storage is like S3. Yay
names!

Custom runtimes are trying to give people the "No Ops" of App Engine with the
flexibility of Compute Engine. Like always, there are tradeoffs, but more
options is usually better :)

(I work for Google Cloud Platform)

~~~
TheLogothete
Do you work on the database products? Please, PLEASE collaborate more with the
analytics team. We want export to BQ for regular accounts. It's not a big ask
and it will improve the lives and businesses of MANY people. I think it will
affect google cloud revenues quite positively too.

------
atonparker
We've tried node.js as a managed VM a few times before this. One of the
biggest pains is logging. Despite documentation implying that JSON logs are
supported (can't find this anymore, maybe it was removed), we still haven't
been able to get GCP to read JSON logs. You lose out on all of the nice
filtering and request grouping provided by the GCP log viewer.

The issue is here[0] and it would seem they still haven't addressed it a year
later.

[0]
[https://code.google.com/p/googleappengine/issues/detail?id=1...](https://code.google.com/p/googleappengine/issues/detail?id=11678#c5)

------
itaysk
I'm surprised they didn't have it until now...

~~~
joeblau
It's not really that surprising. They started with Python because the inventor
of Python worked there. Then they added Java because... well it's Java. Then
they added GoLang because they invented it. Then they added PHP because most
of the internet still runs on that. So Node is the next logical step... or
Ruby probably would have worked as well.

~~~
aleaxit
Actually, when we started (GAE) with Python, my friend Guido didn't YET work
here -- I was finally able to convince him to join, shortly after GAE's
launch, in part thanks to the prospect of working on GAE's Python (which is
what he ended up doing here, especially developing ndb, but also many other
little things). Python was at the heart of Google for long before that (read
Levy's "In the 'plex"...!-) -- or _I_ wouldn't have joined Google 11 years
ago, BTW.

As for Java, well -- guess who announced Java to the world at Moscone Center
in SF 21 years ago...? One Eric Schmidt, then Sun's VP for SW products (he
reminisced about that this morning at his Next keynote)... who just happened
to be Google's CEO when Java was announced as GAE's 2nd language!

------
vskarine
Has anyone tried running MeteorJS on GAE?

------
outside1234
Well that took forever. Congrats on getting to the right place though Google.

------
lambdacomplete
The only reason why I'm still on AWS: I don't own a credit card.

------
ungzd
Quirkiness of Javascript + extreme weirdness of App Engine (especially
datastore) = best of both worlds.

~~~
boulos
Aww, I like Datastore.

 _But_ , there's no requirement that you use Datastore. The point of App
Engine is that it's "you just give us code, we run it for you". With Managed
VMs (built on top of Compute Engine) we're able to offer that programming
model with a more flexible environment.

As someone mentioned elsewhere, there's even "just hand us a random docker
container". We've had people run _ffmpeg_ because you can just hook up App
Engine's request/response and autoscaling to an arbitrary thing, and you've
got an autoscaling web service.

tl;dr: Don't get hung up on the old "extreme weirdness of App Engine", this is
different.

------
exotiik
and make sure billing is enabled !

------
fibo
Great news!

------
ionwake
All I know about GAE is that when my python app started being used by HN users
I upgraded my account to a paid one to increase performance.

In the transition , without any warning, google took the app down, for a day.
Thus killing all chance of acquiring a userbase now that I had hit the
frontpage.

When I went onto the IRC GAE channel to ask someone for help to unlock or keep
the app running during this important moment the reply from a staff member was
' You should have planned better'. I was surprised to hear this response and
did not pursue the matter further. Its all in the IRC logs if any of this
sounds doubtful.

TL;DR; Surprisingly, I did not find the GAE platform to be a supportive
environment for startups.

~~~
Strom
Do you remember the day when this IRC event took place or what nick you used?
I have the logs, and would like to look into this. Also, when you say "staff
member", do you mean Google's staff?

The reason I'm asking is that I've been active in the GAE IRC channel for 6
years and I don't remember anything like this ever happening. I do remember a
few cases where people have come with similar problems complaining that their
free quota ran out and then regular people telling them that they need to
start paying to use the service further.

There really hasn't been an active Googler in the GAE IRC channel for many
years, so that fact makes all of this particularly fishy.

~~~
ionwake2
Hi,

Thanks for the reply.

I would like to retract the statement that it was a member of staff, you are
correct, this may have not been the case. It was a while ago and it may have
just been someone in the know, who didn't actually work for google.

I didnt expect anyone to even notice my comment.

I should have been more careful with my statements.

I will be removing the comment when HN allows me to log back in (apparently I
can log in again in about an hour).

~~~
avar

        > I will be removing the comment when HN allows me to log back in.
    

In that case, here's a copy of your initial comment, because removing top-
level comments that have ongoing follow-up discussion makes for very confusing
discussions:

    
    
        > All I know about GAE is that when my python app started being used
        > by HN users I upgraded my account to a paid one to increase
        > performance.
        >
        > In the transition , without any warning, google took the app down,
        > for a day. Thus killing all chance of acquiring a userbase now that
        > I had hit the frontpage.
        >
        > When I went onto the IRC GAE channel to ask someone for help to
        > unlock or keep the app running during this important moment the
        > reply from a staff member was ' You should have planned better'. I
        > was surprised to hear this response and did not pursue the matter
        > further. Its all in the IRC logs if any of this sounds doubtful.
        >
        > TL;DR; Surprisingly, I did not find the GAE platform to be a
        > supportive environment for startups.
        >
        > ionwake @ ~2016-03-21 22:xx UTC

~~~
ionwake
Well for some reason I can't seem to be able to delete my comment, so I guess
there isn't much I can do =(

~~~
pygy_
The edit window is open for one or two hours.

~~~
ionwake
Yes but my NOPROCRAST settings blocked my access for an hour so I wasn't able
to.

