

Scaling python for high load websites - blhack
http://yougov.github.com/pycon/slides/

======
jamwt
Not sure why this is popping up years later, but mrshoe and I presented this
at PyCon in 2007. Obviously, the landscape of individual technologies has
changed since then, but the basic approaches are still true and pretty
language-agnostic (it was just natural at a Python conference to have the talk
in Python-centric terms).

~~~
tsuraan
Under what circumstances is storing static media in memcache and using a
lighttpd/nginx module to serve them going to be faster than using the web
servers' built-in sendfile support? Sendfile is a DMA operation, and as long
as the disk cache is warm (same odds as having enough RAM for memcache to keep
all media, I think), it won't hit the disk after the first time the media is
requested. Is reading from memcache and then sending to the client ever really
better than that?

~~~
jamwt
Uh, iirc, the assumption here was that the proxy and the application server
(and its resources) were not necessarily located on the same physical machine.
Serving static content off the proxy is certainly another option, but it
involves a different kind of deployment than having the application servers
still provide the (initial) request for static content.

~~~
tsuraan
Ok, that makes sense. Thanks!

------
danohuiginn
To make this readable (all in one page), click the ø link in the bottom right

------
JamesDLevine
Seems kind of generic, but applicable.

In my environment we've also observed latencies associated with some of the
appservers referenced, so we rolled our own thin one with low-overhead in
mind. We use multiprocessing as our means to saturate multiple cores, and
avoid the nuisances associated with general-purpose multithreading.

The deck covers load-balancing but doesn't cover other patterns, such as the
async variety. We increasingly use queues of various sorts. Zmq & mongrel2
offer really exciting possibilities, though we haven't tried the latter at
scale so far.

------
gpapilion
The points are probably applicable to most scaling discussions, and not
particular to python. I'm surprised there isn't any discussion of using a CDN.
Its easy to eliminate static load, and it reduces latencies considerably.

I do wish he went more into detail about DB optimizations, since this is
usually where most people struggle to add performance to their applications.

------
jinushaun
This is a pretty general high-level overview on how to scale. Nothing really
Python specific besides the web servers. I thought the how to scale Ruby
tutorials on New Relic were much more useful and in depth.

<http://railslab.newrelic.com/scaling-rails>

------
tedjdziuba
I stopped reading at "static assets in memcached, telling nginx how to get
them".

Lrn2CDN, or Lrn2LinuxPageCache

