Ask HN: Overcoming the GIL in Python

[jncraton]

I've been working on a python app that I've been trying to multithread for efficiency. The Global Interpreter Lock is preventing me from using the built in threading module to increase speed.

I've done some searching and discovered the multiprocessing module (formerly processing) which has a similar API to threading except that it spawns multiple interpreter processes to overcome the GIL. I've also read that it is possible to write a C module that can release the GIL and then reacquire it, but I would like to avoid this if possible. I'm working on a simple physics engine for fun, and I think that writing it in C might take the fun right out of it.

[functional-tree]

Also check out unladen-swallow -- they aim to remove the GIL from Python (among other goals):

http://code.google.com/p/unladen-swallow/wiki/ProjectPlan

[jncraton → functional-tree]

That looks like it could eventually be incredibly useful. I hope that they write their code in a way which allows some of their improvements to be added to Python.

[russell]

I have been interested in this issue for a long time, but haven't dug into it enough to be authoritative.

The GIL isn't going to be fixed until someone rewrites the interpreter, a task for a genius with time on his hands. I don't think managing the GIL in C will do what you want. If it were easy, someone would have done it. If I were you, I would go go with a package that implements a good message passing model a la Erlang. It would be a good experience. Alternatively, you could look at Jython or Iron Python which are built on top of vms with native thread support.

[jaydub]

This may be of interest:

"Stackless Python allows you to use lightweight threads (tasklets) that can be switched with less overhead and it allows for cooperative multitasking with the intention of making async programming easier. This helps with IO bound applications where you can have tens of thousands of tasklets running at the same time (try doing that with threads). It does not however allow you to take advantage of multi-core or multi-processor resources. It simply allows you to squeeze the most amount of work out of a single threaded process..."

http://www.stackless.com/pipermail/stackless/2007-August/001...

[wmf → jaydub]

Stackless is the opposite of what he wants, which is to take advantage of multi-core or multi-processor resources.

[jaydub → wmf]

Stackless is the opposite of what he wants

The quote above does mention that "[Stackless] does not however allow you to take advantage of multi-core or multi-processor resources. It simply allows you to squeeze the most amount of work out of a single threaded process...".

Depending on the application and its particular implementation, actually Stackless could be the right solution. Which is why I brought it up. In fact, it now looks as though he might go with Stackless, just like EVE online did :)

[jncraton → wmf]

It turns out you are absolutely correct. Thanks for pointing that out.

"Stackless python does not make use of any kind of multi-core environment it runs on."

http://stackoverflow.com/questions/377254/stackless-python-a...

[jncraton → jaydub]

I will probably end up using stackless. CCP seems to have gotten it to work quite well for EVE. I'm still really impressed by their single server MMO architecture.

[stonemetal]

Yep, all true. You may want to check out scipy. It provides a good base for numerical integration and other things you are going to need. It is written in C so perhaps you won't have to drop into C for performance reasons.