

Google App Engine is now much faster, after latest maintenance - ordinaryman
http://code.google.com/status/appengine/detail/datastore/2010/11/06#ae-trust-detail-datastore-get-latency

======
mmastrac
Our app (<http://gri.pe>) feels a lot snappier after this latest bit of
datastore maintenance. We've been using as much memcache as possible to avoid
datastore access because prior to this maintenance, datastore access was spiky
and even the simplest of queries had a small chance of timing out.

I can say for sure that some of our datastore-heavy cron jobs were taking >
200ms, but are around 70ms now.

It's a shame that the stats for > 12hrs ago seem to have been wiped. I bet
we'd be seeing a marked improvement on our ms/request stat.

~~~
bad_user
Unrelated question: how are you doing those cron jobs.

I tried using the task queue, but it kept timing out on me, even when I broke
my tasks into smaller ones (couldn't figure out a threshold as the behavior of
the datastore was totally unpredictable).

~~~
madflo
Via Cron Jobs ?
[http://code.google.com/appengine/docs/python/config/cron.htm...](http://code.google.com/appengine/docs/python/config/cron.html)

~~~
bad_user
Quote from that article:

    
    
         A URL invoked by cron is subject to the same limits 
         and quotas as a normal HTTP request, including 
         the request time limit.
    

And I'm looking for some tips and tricks to do heavy processing (which is
really needed when working with non-relational datastores).

~~~
mshafrir
The key is to combine cron jobs with Tasks. Tasks are am easy to use and
powerful feature of Google Appengine that let you deal with the standard
limitations. For example, rather than doing some operation on 1000 entities in
one request, you spin off 1000 tasks, each handling the operation on one
entity. Read up about them at
<http://code.google.com/appengine/docs/python/taskqueue/>

You can also explore the new Mapper API at
<http://blog.notdot.net/2010/05/Exploring-the-new-mapper-API>

~~~
mmastrac
Agreed, the Mapper API is the best way to deal with processing large numbers
of entities from the datastore.

AppEngine requires careful thought of application design at this point.
Everything is an HTTP request and subject to the same restrictions.

I've heard rumours that they'll be launching longer-lived tasks at some point
in the future, but I have no idea when that will happen.

------
gfodor
As any ops guy will tell you, this is the finest form of graph porn there is.

~~~
pjscott
It would have been sexier on a semi-log-y scale. That said, as someone who's
always been annoyed at the high latency of App Engine's datastore, this is
_huge_ if they can deliver this level of latency with low variability.

------
endlessvoid94
This headline should say "GETs to the datastore are now much faster".

EDIT: I neglected to see the other datastore figures.

~~~
ordinaryman
You should also check out the latency for Put, Update, Delete and Query on
that same page.

~~~
endlessvoid94
Ah, you're right.

------
fharper1961
I just tried a high traffic site that runs on App Engine (appbrain.com). It
used to feel a bit slow, and now it really flies.

It definitely makes me reconsider using App Engine for new projects.

Disclaimer: I have no connection to Appbrain aside from meeting the developers
once.

------
k7d
Good news but I really hope Google figures out how to solve
DeadlineExceededError. They are popping up from time to time for my app
(<http://www.knowcase.com>) and I don't think it's datastore related.

~~~
ma2rten
This reply from a few days back should shed some light on the issue:
<http://news.ycombinator.com/item?id=1823362>

------
iampims
They even solved a bug with the datastore quota not being properly updated
after a massive delete of entities.

Kudos to the team, and I, too, hope that the latency will remain this low
_forever and ever_ …

------
wkornewald
It's still too early to tell. This would not be the first time that speed
improves after a maintenance and then it slowly gets worse every week until it
looks almost like before the maintenance.

~~~
benjaminfox
Is it possible that the degradation in performance is due to more developers
building more apps and driving more traffic to App Engine as time goes on?

~~~
wkornewald
The performance degrades too quickly for this. In two or three months we might
be on our old performance level, again. Well, I seriously hope that we'll stay
fast this time. Only time will tell.

------
tropin
This site, probably NSFW, makes use of the datastore (for the thumbnails) and
it's now much faster, it's a pity we can't compare speeds with the old
latency.

[http://wallres.saiswa.com/?tags=girls&pagina=23](http://wallres.saiswa.com/?tags=girls&pagina=23)

~~~
jws
My dinky app dropped from 60ms/request to 20ms/request, but over the next 3
hours crept up more or less linearly to about 30ms/request.

~~~
jspencer
I missed the word "app" when reading your comment and thought it was a clever
joke.

------
joshzayin
Looking at the error rates, I wonder what happened at around 12pm?
(Screenshots: <http://grab.by/7gJ4> <http://grab.by/7gJ5>
<http://grab.by/7gJh>)

~~~
forgotusername
From the maintenances mailing list:

The maintenance work has been completed. The datastore was in a read-only
period between 12:02-12:45 for a total of 43 minutes. Some users reported two
periods of 500 errors unrelated to datastore writes, but these issues should
no longer be occurring.

On Nov 6, 12:01 pm, App Engine Downtime Notify <i...@google.com>

------
cageface
The sense that the app store has been a neglected, second-tier priority is
deep enough now that it will take them a while to regain significant developer
trust.

AWS is very close to a de-facto standard now.

Edit: I mean app engine, the hosted web service. Not the app store. Thanks.

~~~
grandalf
I agree. However I hope they do regain that trust b/c app engine is a far more
pleasant platform to develop on.

~~~
c1sc0
Can't repeat this enough: the main reason I develop on appengine is that it
removes server admin headaches completely from the equation. Ok, you get other
headaches in return, but most of the time it's stuff I can deal with.

------
smoody
the app engine applications gallery is now gone, too. coincidence???

<http://appgallery.appspot.com/results?topapps=true>

~~~
realmojo
enterprisey.

------
gmosx
that 's great :) hopefully, now they can work on improving support for JDO/JPA
and release those great features announced in Google I/O.

------
grandalf
thank you google! wow, this is amazing.

------
tropin
I expected something more than a graphic.

~~~
tropin
Ah, thanks for the hate, ordinaryman.

~~~
ordinaryman
Not me. And, even if I wanted to, I can't.

Also, I guess Google folks would post about the performance boost later.

~~~
tropin
Ok, please excuse me.

~~~
YuriNiyazov
In either case - from the HN guidelines: "Resist complaining about being
downmodded. It never does any good, and it makes boring reading."

