

Google App Engine: Datastore Performance Growing Pains - adora
http://googleappengine.blogspot.com/2010/06/datastore-performance-growing-pains.html

======
chasingsparks
I just started writing a GAE open-source community tool for rare disease orgs
two weeks ago. I thought it was slow, but wrote it off as a limitation
inherent to the Datastore. I moved a lot of things to the Memcache level,
mitigating (mostly) the slow queries. That being said, I am glad it was
actually a real problem.

I'm also impressed that they are not charging anyone until the problem is
solved.

~~~
richardw
GAE does have some performance surprises even during normal performance
periods, so it's good that you're using Memcache etc now.

Couple other ideas:

Try to minimize read effort as much as possible, so e.g. pre-calculate as much
as you can and rather optimize for performance than data cost. Data is cheap,
so de-normalise where you need speed. Calc offline, so add a task to calculate
something and return immediately to the user.

------
jasonkester
I like the way they're dealing with this. Acknowledge the problem (and
actually call it a "problem" instead of handwaving aroung it), and stop
charging for the service until they fix it. This is a good pattern to follow
if you run a service that ever runs into issues that affect your users.

On the performance issue itself, I actually used AppEngine for a project when
it first came out and was astounded at how poorly it performed in general. Any
page requiring more than a single "select where id={}" query would regularly
throw CPU warnings at you.

I couldn't imagine building a site even as straightforward as HN on top of it.
As soon as you had a page needing a dozen queries to construct, you'd run into
the wall.

------
TimothyFitz
If I'm reading the current status graphs correctly they're regularly seeing
1.5-2.5 second queries and 400ms direct fetches. That seems unbearably bad.
Can anyone running mission critical apps on the platform comment?

~~~
axiom
Yeah we've been freaking out about this. We have our app used in the
classroom, and have had several sessions disrupted because of this. Needless
to say our customers were pissed.

We're in the process of doing an emergency move over to EC2 because of these
issues. That being said, I must admit that this is a really decent response on
Google's part. Unfortunately the problems go deeper than just these latest
problems with datastore.

~~~
aarontcheung
Just wondering, are you porting to mySQL, Cassandra, or HBase? We're currently
moving off app engine also, and would love to hear more about your final
setup.

