> Anybody has a reasonable theory for why this took so long, despite users asking for it for years?
Well, initially, their investment in more runtimes was covering additional languages, not covering more versions of languages they already had. (IIRC, AE started with Python 2, added Java first, and then later Go and PHP.)
As that was happening, they gradually shifted focus from building out the old, fairly limited model of App Engine to building out other parts of the cloud platform, including (particularly relevant to future App Engine work) Compute Engine, to have a more robust cloud offering covering more use cases.
Then, they went back and returned to improving App Engine -- leveraging the now-broader cloud platform, and particularly Compute Engine -- shifting from the old model for new runtimes to the Flexible Environment (originally called "Managed VMs") model (built on Compute Engine, and which, in addition to including some additional Google-provided runtimes, means people can package up their own runtimes for App Engine).
Since doing so, they've released official Google runtimes for Python 3, Ruby (which people have been asking for longer than Python 3), and Node built on the Flexible Environment.
The architectural switch to the "Flexible Environment" is what I missed. I can appreciate why this broader new approach for language runtimes took some time.
But then again the Flexible Environment seems to be a very different product with different features/characteristics and pricing than what I once knew as "App Engine". I would then argue that Python 3 never came to "App Engine", only to their new product. Which is a valid strategy for Google of course, but now stifled my enthusiasm. Probably I only need to learn more about Flexible Environment style App Engine to become hyped again.
(It wasn't obvious to me why, given a first read of the docs; the fact that applications run in VMs in Flex seems more like an implementation detail. Am I missing something?)
We use both and try to stick with standard runtimes wherever possible, resorting to flexible runtimes for things doing heavier processing and/or with lots of instances.
The standard runtimes are superior in a lot of ways, two big wins being lightning fast deployments (seconds vs. many minutes) and batteries included for all the app engine services. Logging in the flexible environment can also be a bit lacking. When you have something running in standard runtimes you basically don't have to think about many of the things that can normally go wrong on a server.
That said, many things about the flexible environment are rapidly improving, they've certainly come a long way since we first started playing with them around 6 months ago.
I think this is my main problem with Google services. It takes them forever to listen to people. And I don't think other cloud services listen any better but at least it seems they give you what you want faster. So they have a better mechanism to know what works and making a product of it.
Does anyone have any numbers on the cost difference between standard vs flexible app engine usage?
Flexible environment supports any language/environment you want as long as you implement the HTTP endpoints it expects. I would be much more impressed if they brought Python3 to the standard environment which seems to be much more cost effective for many types of web apps. Flexible environment essentially requires you to run a VM all the time, which can get pricey, versus a much more managed sandboxed environment that can sleep and be awaken quickly as needed to serve requests. Also, Python 3.5 would be awesome to leverage asyncio, but that's probably still a bit too bleeding edge.
We use a combination of both at my company. We use the standard environment for the majority of our services, doing things like proving APIs, serving webpages, performing datastore interactions, etc. The flexible environment starts to make a lot of sense when you're doing heavier weight processing, text analytics in our case. App Engine instances are relatively cheap when you're using the smallest instance sizes, and have even more benefits if your application has idle time when they can spin down completely, but costs start to go up very quickly if you need to move to the bigger instances in the standard environment [1]. The flexible environment is current billed at the rate of the machines you're using, so is much cheaper for bigger instances [2].
The other downsides are the deploys are slower, for now at least, and it's still in Beta. Beta means it has SLOs, which are like reliability goals, but it doesn't have SLAs, which means you get refunds for downtime. App Engine Standard does have SLAs:
Beta products are also more willing to make backwards incompatible changes with shorter deprecation timelines. But overall it's certainly a lot more, ahem, flexible. Sneaky awesome feature is ability to install things like imagemagick , phantomjs, or ffmpeg, which to shamelessly self-promote I wrote about here:
That's using a custom runtime, which won't be as officially blessed, but you can of course just extend from the official runtimes, since they're on Google Container Registry and open-source:
Can't edit the parent anymore: Actually the flex price should go down compared to Compute Engine because it should be optimised for a more specific use-case?
waprin's comments are on the nose with what we've experienced.
There are pain points with the flexible environment that we don't have to worry about at all with standard runtimes. The two biggest ones for us are slower deployments and a lack of monitoring tools.
Just today we actually switched one of our App Engine modules to Flexible runtimes out of necessity as we we're hitting quotas for App Engine infrastructure, and in doing so we no longer have easy access to metrics on how many instances are running or memory usage for the service. The "Cloud Trace" tools are also impaired when you aren't using standard runtimes. In short, you lose visibility into your instances. It ends up being a tradeoff between developer time and cost.
With certain runtimes in the flexible environment, you also lose out on pieces of the App Engine infrastructure like memcache, which are invaluable in optimizing services.
Ahhh yes we've been hit with this. There are subtle differences in the GAE APIs when using flex. Trace as you mentioned, but also channels api doesn't work at all, cloud debugger only sometimes works, etc.
Most of these have gotten a lot better since managed vm's first rolled out (3 years ago!?!?!?!?!)
If you have very little traffic then what you are saying is true, flex GAE requires you to keep one VM running.
For our our app (Streak) we run 100's of instances, we moved some of our services from standard GAE to flex GAE and saw a HUGE reduction in costs. Depending on workload we saw 5x-10x reduction in cost. To see why this is possible, look at the price of a standard GAE instance and the price of a default flex instance. For roughly the same price you get a way more capable machine.
You just have to have enough traffic where you'd like to keep at least one instance running all the time. Enough traffic in this case is a really small number. If you're traffic is lower than that number than who cares about optimizing costs, its gonna be cheap no matter what.
On lowish traffic apps, it's way way more expensive as you have to pay for the VM's to be permanently running. AFAIK you also need to have 2 app servers running as they get restarted once w eek (at least thats how it is for Java apps I believe)
You can use Postgres and Redis. :) I'm working on writing new tutorials for both of those, but in the meantime:
You can host your own Postgres on Compute Engine [1] or use a hosted provider like ElephantSQL [2]. You can use whatever client library you want to connect to postgres in App Engine flexible.
Same story for redis - you can host it on Compute Engine [3] or use something like redislabs.
that's not the same as fully managed postgresql (which only aws has (at the time))
elpahntsql is not really cheap compared to aws master-slave.
Just comparing the smallest instances, elephantsql:
- db.t2.micro 1 year 20 USD / month (250 USD upfront) - 24.82 USD monthly (that's a multi-slave with auto failover)
compared (with 30gb (SSD) storage it's at 30 USD) to elephantsql only 2GB data on a shared database server 20 USD
- db.t2.small 1 year 41 USD / month (500 USD upfront) - 50 USD monthly (that's a multi-slave with auto failover) (with storage around ~60 USD) vs elephantsql Happy Hippo single dedicated database 99 USD 100gb data
you kidding right? I would need to pay twice the price.
Redis is provided by Click to Deploy, i.e. "Popular open stacks on Google Compute Engine packaged by Google", and is in beta (Estimated costs: $307.80/month).
Looks like Redis Labs has a trivial free tier, and the 1GB plan is cheaper on GCE than AWS ($57/mo vs. $71/mo). I'm pretty annoyed that I had to create a Redis Labs account to see that though.
How do you deal with securing the DB's in this setup? Is it possible to isolate the compute engine DB ports to only your appengine instances, or do they have to be open to the whole world?
App Engine Flexible runs on GCE instances, which all have internal IPs. So you can just firewall off external traffic but allow internal, which is the default anyway.
Global GCE instances within your project, yes. Not every GCE instance everywhere, that would not be very helpful :)
If you wanted to have some instances that couldn't talk to your database, easiest thing to do is probably throw them in a separate project or create a new subnet that you firewall off and create your non-App Engine instances in that subnet.
You can absolutely use Postgres or Redis. If you spin them up on GCE you can connect to it from GAE Flexible. The only support lacking is a managed Postgres or Redis service like Cloud SQL does for MySQL.
I wish there were a clearer delineation between developer downloads on PyPI and automated builds, since it seems there's been a bit of a cultural sea change over the past year or so in favor of Py3. I'd love to know which version humans are downloading.
Yes but if automated pip installs numbers are way larger than manual installs you end up with projects supporting both with roughly 50-50 numbers.
I agree with the parent post: now is the time. Open source projects should to stop supporting py2 in their new major release and cleanup their codebase from the hacks made to support both languages.
>I agree with the parent post: now is the time. Open source projects should to stop supporting py2 in their new major release and cleanup their codebase from the hacks made to support both languages.
The danger with this for Python is that it can give the impetus (is that the word?) and motivation for people with py2 legacy projects (which is the majority of people) to say "fuck this" and go try another language that doesn't pull this shit on them. Go, JS with Node, Ruby, whatever...
If they're not gonna get compatibility anyway, they might jump ship altogether. At least for green-field projects...
Really cool. Would be interesting if App Engine becomes a spot for larger framework driven web apps like Django, or if/when ruby is supported, ROR. It could be a very efficient way of doing so because you could have a single instance running with a gazillion gunicorn/unicorn worker processes to distribute the load across processes.
Ruby (and Node) are supported on the Flex Env the same way Python 3 is. JRuby may also be usable on the standard environment's Java runtime (ISTR seeing a how-to in the past, but I also seem to recall at one point seeing that there were some issues.)
Developer Relations team exists so that doesn't happen so sorry to hear. Part of the problem is some of our feedback mechanisms need work. If you go to:
cloud.google.com/python
You'll see we have a lot of tutorials that link to Github repos . If you file issues there you are more likely to get help. The caveat is we are not a support team and have no guarantees (just a likelihood), so if you need paid support look into our options for that instead.
I felt Google was much easier to get started in than AWS. And it's easy to forget that it's quite possibly the most complicated system most people who use it interact with. It probably has a few magnitudes more expressive power of a, say, a webpack+babel+package.config (yeah, hard to measure, I know). And that's a competition it easily wins.
Yes I have used AWS a lot and been happy with the UI, it gets the job done.
Google Cloud Platform UI (and speed) was just actively hostile towards me.
For example I went to the App Engine section to create a simple flask app. It keeps forcing you through a stupid tutorial trying to guide you around the site. Believe me being a developer myself I'm no stranger to web and its interfaces and can quickly navigate my way around this stuff but this had me hitting my head on the desk.
Anybody has a reasonable theory for why this took so long, despite users asking for it for years?