

Ask HN: Overcoming the GIL in Python - jncraton

I've been working on a python app that I've been trying to multithread for efficiency. The Global Interpreter Lock is preventing me from using the built in threading module to increase speed.<p>I've done some searching and discovered the multiprocessing module (formerly processing) which has a similar API to threading except that it spawns multiple interpreter processes to overcome the GIL. I've also read that it is possible to write a C module that can release the GIL and then reacquire it, but I would like to avoid this if possible. I'm working on a simple physics engine for fun, and I think that writing it in C might take the fun right out of it.
======
mcav
Also check out unladen-swallow -- they aim to remove the GIL from Python
(among other goals):

<http://code.google.com/p/unladen-swallow/wiki/ProjectPlan>

~~~
jncraton
That looks like it could eventually be incredibly useful. I hope that they
write their code in a way which allows some of their improvements to be added
to Python.

------
russell
I have been interested in this issue for a long time, but haven't dug into it
enough to be authoritative.

The GIL isn't going to be fixed until someone rewrites the interpreter, a task
for a genius with time on his hands. I don't think managing the GIL in C will
do what you want. If it were easy, someone would have done it. If I were you,
I would go go with a package that implements a good message passing model a la
Erlang. It would be a good experience. Alternatively, you could look at Jython
or Iron Python which are built on top of vms with native thread support.

------
jaydub
This may be of interest:

"Stackless Python allows you to use lightweight threads (tasklets) that can be
switched with less overhead and it allows for cooperative multitasking with
the intention of making async programming easier. This helps with IO bound
applications where you can have tens of thousands of tasklets running at the
same time (try doing that with threads). It does not however allow you to take
advantage of multi-core or multi-processor resources. It simply allows you to
squeeze the most amount of work out of a single threaded process..."

[http://www.stackless.com/pipermail/stackless/2007-August/001...](http://www.stackless.com/pipermail/stackless/2007-August/001963.html)

~~~
wmf
Stackless is the opposite of what he wants, which is to _take advantage of
multi-core or multi-processor resources_.

~~~
jaydub
_Stackless is the opposite of what he wants_

The quote above does mention that "[Stackless] does not however allow you to
take advantage of multi-core or multi-processor resources. _It simply allows
you to squeeze the most amount of work out of a single threaded process..._ ".

Depending on the application and its particular implementation, actually
Stackless could be the right solution. Which is why I brought it up. In fact,
it now looks as though he might go with Stackless, just like EVE online did :)

------
stonemetal
Yep, all true. You may want to check out scipy. It provides a good base for
numerical integration and other things you are going to need. It is written in
C so perhaps you won't have to drop into C for performance reasons.

