tl;dr - By doing a detailed analysis of his app, he was able to discover that a lot of what he was using App Engine for wasn't being used or could be made more efficient with relatively little effort.
Having worked on App Engine, I'm sure almost all (if not all) of the apps I've built ended up in this situation. Because it was "free" there wasn't a lot of reason to focus on being efficient (especially in terms of space). Getting an app to work and serve its purpose while maintaining a level of perceived performance was the top priority. Any work after that was usually not worth it because the apps tended to be small in scope and non-critical.
That said, it feels like if I were to build on App Engine again, it would require more development time to keep it free. With a couple of scripts to setup a free micro EC2 instance, that might make it easier to build those small apps there instead. Not sure if that's good or bad for App Engine, but I suspect bad.
Could someone please explain to me why this is different from hosting your own environment? In your own environment, $x / month gives you an environment that can handle a certain amount of load. You can either spend time (= money) on optimizizing your application so you get more bang for your buck from your hosting environment, or you can spend that money on your monthly hosting bill. And it's your job figuring out an optimal balance between those two things.
The advantages of appengine compared to an EC2 micro instance are that there is every little that can break over time. Develop once, deploy, forget about. Also: It's only free the first year, you still have to pay for bandwidth, it will cost more effort to scale and you can just have 1 free micro instance, but 10 free apps.
Great article. I haven't used App Engine before but plan to very soon. I was just at the Google Developer conference in South Africa and the reason they cited the price increase was to force more efficient implementations, implying that the cheap pricing allowed people to be lazy. At the time I thought that was a positive spin, but it seems to have worked in this case. Good work!
What helped me was pushing frontend tasks into the backend to utilize the full quota for those two. Adding some pre-checks to the tasks (e.g. asking questions like "do I really need to scrape this page again now?) were helpful as well.
Nevertheless, I feel that it has become a bit of a drag deploying a full-blown web-app on App Engine as these restrictions are running against fast iterations and quick deployments. I wish there were more tools to test the loads locally.
Or you could use a PaaS provider like Google App Engine and not have to deal with building, hosting and monitoring your own servers and stack of technologies.
Not to mention effortless scale-up when you need it, predictable pricing and built-in redundancy.
For anything but simple apps like this PaaS fall apart, first the pricing scheme is no inline with even a basic VPS. As you need to add features, if they don't support them your fairly screwed. App Engine took a year to add cron jobs. They still have a proprietary engine that I have to custom code for, and if you ever want to switch vendors you basically have to rewrite everything. TLDR : If you can rewrite all the code for your service in a few weekends then its fine to be on a PaaS
A very interesting thread. I haven't worked in this particular area, but isn't IaaS the happy medium?
If you chose a platform, for example Node, you could shop around for the best value between Heroku, Joyent, CloudFoundry etc. In theory, the only lock-in would be your initial choice of platform, and you would keep the flexibility to scale up instances to meet demand.
Are things quite different in reality? I'd be interested to know.
I'm guessing you've never spend half an afternoon on a holiday finding an Internet cafe, convincing them to let you run PuTTY and SSHing into your "complete control" server to fix a trivial problem that brought down your app?
Comparing a bus arrival service for a 1/4 million population city with Reddit is .. irrelevant, at best.
I'll take "Google screwed up, it'll be up when it's up" over "I screwed up, I'm dropping everything and scrambling to fix it" any day of the week.
Also just to scope the discussion: I'm not talking about big, important things - those with ops budgets and rotas - I'm talking about small/low-budget/side-project type thing, like the OPs project.
It would mean spending time configuring a server, which I have no interest in doing. I know nothing about Linux, and haven't dealt with any server configuration in over a decade.
The minimum number of things I have to deal with the happier I am - I want to get some functionality out there, and being able to just code it, upload it, and then let Google worry about databases, backups, upgrades, etc. makes life staggeringly simpler for me.
Ok, so he found that with some effort he can get the price down. But it's still the case that if you build on the App Engine that you're locked into it. Personally, if I had to rewrite my app to be more efficient anyway, I would be moving to a platform that wont lock me in. After all, I'm sure they'll also have altruistic reasons the next time the raise the price. And the next time. And the next time. Etcetera.
After using App Engine for two projects at the beginning of this year, I have to say that Heroku is better in almost every way (especially if your app is already open-source and using git for version control). The ability to use an actual production server (Gevent, for example) on a free dyno is awesome.
I am using both at the monent. And I must say that Heroku defenetly has it's advantages, but appengine has: lower latency (for me), more Apis, cdn built-in, ability to run different/test different versions at the same time, backend instances (Heroku has ec2 though), free webadmin with graphs and ability to see errors etc.
Having worked on App Engine, I'm sure almost all (if not all) of the apps I've built ended up in this situation. Because it was "free" there wasn't a lot of reason to focus on being efficient (especially in terms of space). Getting an app to work and serve its purpose while maintaining a level of perceived performance was the top priority. Any work after that was usually not worth it because the apps tended to be small in scope and non-critical.
That said, it feels like if I were to build on App Engine again, it would require more development time to keep it free. With a couple of scripts to setup a free micro EC2 instance, that might make it easier to build those small apps there instead. Not sure if that's good or bad for App Engine, but I suspect bad.