

Some quick Django optimisation lessons - pajju
http://lukeplant.me.uk/blog/posts/some-quick-django-optimisation-lessons/

======
lucian1900
I've also run into O(n) and even O(n*m) # queries situations. They're entirely
avoidable by simply sticking closer to the relational model than the object
model. Sadly, this is almost always impossible with just Django's ORM. You
very quickly end up writing unmanaged models and raw SQL.

SQLAlchemy is so much better when it comes to this, since things like joins
are always explicit. However, it's rather hard to use it together with the
Django ORM bits, especially the admin. One method I want to explore is
replacing only the raw SQL bits with SQLAlchemy, and use its reflection to map
to the Django ORM models.

~~~
beagle3
Not django specific:

Personally, after years of trying to solve the Object/Relational mismatch, I
realized that it's better to drop the object side and stay with the relational
side if you need a database.

The price is a little bit of syntactic sugar. The return is the ability to
reason about your database and how it accesses data.

web2py eschewed ORM for a DAL (Database Abstraction Layer). I do not know how
it compares to SQLAlchemy, but it is independent of everything else web2py;
I've been using it in non-web projects and it is excellent, although a little
slow if you have large result sets.

~~~
lucian1900
SQLAlchemy is philosophically similar to DAL from what I'm reading, but closer
to pure set algebra than SQL. On top of that, it builds SQL-specific features
and declarative models (as objects).

------
yummyfajitas
Another thing you can do is build caching into managers.

Here is one such effort (shameless plug):
<https://github.com/stucchio/Guacamole>

~~~
nyellin
Also consider using Johnny Cache or CacheMachine backed by memcached.

Johnny Cache does automatic invalidation, but will invalidate an entire
table's cache on one write. If you have a site with a small number of writes,
this is an easy and instant win. Setup is under ten lines of configuration.

CacheMachine, on the other hand, tracks the objects that were returned for
individual queries. The queries are invalidated when one of the associated
objects changes, but _not_ when a new object is added to the db. I believe
addons.mozilla.com uses this.

<http://packages.python.org/johnny-cache/>

<https://github.com/jbalogh/django-cache-machine>

~~~
znt
I can confirm that CacheMachine is being used by at least 2 global projects
that are in production. Haven't faced any critical problems so far.

------
insin
Also worth looking at prefetch_related(), which landed in 1.4, which can be
used to alleviate common O(n) situations:

[https://docs.djangoproject.com/en/dev/ref/models/querysets/#...](https://docs.djangoproject.com/en/dev/ref/models/querysets/#django.db.models.query.QuerySet.prefetch_related)

I've also used this in the past to do something similar, grabbing the ids of
all specified related fields (related to the same model type) and pulling them
back in a single query, with the option to only grab certain columns as dicts
instead of full model instances if you're only selecting additional data
required for specific templates:

<http://djangosnippets.org/snippets/1117/>

------
irfan
_"Django makes it extremely easy to generate database queries, which is both a
good and bad thing"_

I experienced this when a junior developer made queries in a loop inside
template without realizing that it is actually going to be a lot of SQL
queries in the end.

------
yaix
Related question: is there an easy build-in way in Django to write to stdout a
count of DB queries made for a request?

~~~
stevejalim
Use django-devserver <https://github.com/dcramer/django-devserver>

~~~
GeneralMaximus
I second django-debugserver. This is one of those nifty packages you wish were
included with the default Django distribution. More than once, it has helped
me find and fix query-related issues.

Highly recommended.

------
gbog
Maybe it is just a matter of style but I really dislike when advices are
repeated three times, I regard this as disrespectful of the reader.

------
idleloops
That seems like an unholy amount of queries. Would be nice if there was an
explanation as to the why behind the queries on those paths.

