Nice writeup. We've been using Streak to manage our sales process and we're pretty happy with it - by putting the CRM in email (rather than the other way 'round) it is much, much closer to the "natural" way we work.
I'm consistently surprised that Google App Engine doesn't get more pickup among the HN community. No, it's not perfect, but it lets you build and scale a system with almost zero attention to ops. Lockin? Most startups fail because they never achieve product/market fit, not because they are unhappy with their hosting provider. Less ops means more time making features.
FWIW, App Engine went through quite long periods (as in, well over a few months at a stretch) of serious scalability problems: the latency was enormous, and even by Google's own reporting standards (they had websites you could use to monitor the environment) everything was coming up red; it got so bad once that they stopped charging for the service for a while (not a day, but a couple months) until they could get it remotely operational again.
I really do not get the impression that the people on the App Engine product mailing lists actually feel like they are able to spend less time on operations, and there are numerous alternatives that let you use more standardized stacks to get similar benefits, especially if you are serving less than a few tens of millions of users (which, honestly, it is not clear App Engine would handle in practice anyway, even though in theory it would work great).
Yes, I remember the latency problems with the old Master/Slave datastore. However, Google rolled out the High Replication Datastore almost two years ago and the M/S option has since been deprecated. The HRD has had an excellent track record; nobody complains about the datastore anymore. It's actually pretty remarkable how much appengine has evolved in the last few years, especially compared to other PaaS offerings.
When things go wrong (and they still occasionally do - on every platform), I'm one of the more vocal critics on the GAE mailing list. However, the zero-ops promise has held up pretty well. There's always something to complain about, but isn't it.
Also, that last statement is a little puzzling. GAE already handles tens of millions of users (probably considerably more) in shared infrastructure. The typical bottleneck (the database) isn't an issue because the database is already federated across every app in the system; you don't have to scale dedicated resources.
My final comment is per-user: it is clear that Google can handle tons of separated users, but their data store transaction model has some centralized bottlenecks that alternatives either don't or that can be worked around with some not-difficult work. Scalong tons of integrated users isn't a problem, honestly (and to be clear: they seem to be using less shared infrastructure over time, which is the explanation for their recent pricing changes).
The real issue is that people don understand that scaling challenges really aren't that bad until you have tons of users; in the few millions of clients range you can very easily handle the traffic with one or two centralized RDBMS systems (and in fact, this is how many sites actually work in practice, even large ones you use daily).
There are only two limits you have to watch out for when working with the datastore, and they're both fairly well documented:
* You must be careful of entity group boundaries. This ends up being fairly easy to do, especially with cross-group transactions (added last year). It's definitely a design consideration, but one you internalize very quickly and as long as you don't screw up your database design, isn't a scaling bottleneck.
* Tablet splitting. If you write monotonically increasing values to one index at high rates (1k/s is what I've heard, but I haven't hit it myself) you may have to work around this issue by federating the index. It's not pretty, but it's not a common problem.
I agree that traditional RDBMS architectures can handle quite a lot of traffic before they hit a wall. It's just a lot more ops work. What I like about GAE-land is that you just have to be a little more careful in your data modeling and you can stop worrying about all the usual problems like monitoring the performance of your database server. I've built rather large sites on RDBMSes in the past, and getting the ops right was always painful.
>... especially with cross-group transactions (added last year)...
Thanks for mentioning this: I didn't know they had added this feature; that definitely makes some of the data modeling problems easier. Without this, I found it often led to situations where you had to make horribly painful decisions that affected both functionality and scalability very early on in your project lest you end up with tons of data that you now have to programmatically and mostly manually migrate (which is why, to me, I keep being surprised that you keep mentioning ops challenges for normal databases: with the ability to do global schema and index changes concurrently and safely inside of transactions, I found the amount of time and thought I need to put into managing something like PostgreSQL a fraction of most alternatives that supposedly scale better).
Streak co-founder here - yes, we're very surprised more startups don't use AppEngine. We think its mostly because it isn't noted as sexy.
We've been on AppEngine for under a year and its been great. Its been much more stable since the HRD (high replication datastore) launch and the SLA was made available. We rarely think about how we'll scale some feature or our app - we've moved from megabytes of data to terabyte scale without many changes at all.
What has been your experience with the Prediction API?
It the one part of Google's offering I'm thinking of using at the moment to experiment with recommendation features without having to spend too much time implementing and scaling the algorithms ourselves. The post only briefly alludes to it but doesn't go in any details.
Even though App Engine's features seem nice on paper, and they have relatively decent documentation, I won't use them because I'm afraid that I'll have to contact customer service at some point.
My experience so far with Google is that if something is broken you can post to a newsgroup with the hopes that someone will read it and take action eventually, make enough stink that your story gets to the frontpage of HN or reddit, or know someone with pull at Google.
I'd much rather spend the time up front to build a solution that doesn't require good luck to get support than to gather the ops savings from going with GAE.
My biggest complaint with google appengine is that they don't send you an email when you start reaching your usage quotas. I hope someone from AppEngine sees this. Otherwise it's been great. ;)
App Engine Magic (https://github.com/gcv/appengine-magic) is a library that makes it easier to use GAE with Clojure, but it hasn't been updated since GAE SDK 1.6.3.1.
There is also gaeshi (https://github.com/slagyr/gaeshi), which is a newer GAE library for Clojure that's a little easier to work with, but it doesn't have much documentation or cover all the services.
Does anyone have experience on how using Clojure on App Engine plays out in practice and any advice on which path to take?
I'm consistently surprised that Google App Engine doesn't get more pickup among the HN community. No, it's not perfect, but it lets you build and scale a system with almost zero attention to ops. Lockin? Most startups fail because they never achieve product/market fit, not because they are unhappy with their hosting provider. Less ops means more time making features.