

So Pythonistas, you want to get rid of the GIL... - sausagefeet
http://functional-orbitz.blogspot.se/2012/07/so-pythonistas-you-want-to-get-rid-of.html

======
comex
> You will rewrite your code to not make use of the guarantees.

I suppose a problem is that unlike Java, Python is not fast enough to
implement low level data structures in the language itself, so you can't
easily implement non-thread-safe data structures without losing memory safety
at the C level, which is unacceptable.

Still, I have the feeling that with heavy use of biased locking, you could
stick locks everywhere without a lot of overhead in the non-contended case. Or
maybe that famous patch just needs to be revisited, tweaked to more
efficiently address the areas of highest contention... see
[http://dabeaz.blogspot.com/2011/08/inside-look-at-gil-
remova...](http://dabeaz.blogspot.com/2011/08/inside-look-at-gil-removal-
patch-of.html) for some context.

(some forms of biased locking are patented.)

------
cpeterso
Python objects only need to be thread-safe if they are accessed by multiple
threads. A proven solution that could preserve Python's atomicity guarantees
is actor-based concurrency like Erlang or Go. When Python objects are queued
to an actor's channel, a deep copy (or copy-on-write reference) could be sent
to avoid data sharing.

~~~
earl
The blog's example is stupid. Of course if you pop a list in more than one
thread the behavior is undefined. Who on earth doesn't expect that?

@jagw: ok fair, but that is going to be the very first example of things thou
shalt not do when using threads in any article, book, or course

~~~
dagw
People who have never done proper multi-threaded programming.

------
lmm
Do actual pythonistas really hate the GIL? I thought it was mostly a troll
talking point, just like the whitespace.

~~~
jforman
Why would the inability to write multi-threaded programs that are worth a damn
be a troll talking point? It meaningfully lowers the appropriateness of using
python for server code.

~~~
nakkiel
There's more than one way to do it. If you're thinking about hundreds of
threads doing chunks of work, then my comment will not be meaningful but if
you're talking about maximizing CPU utilization for a particular task, then
multiple processes is a valid and proven approach (Tornado does just that for
example, forking as many processes as you have CPUs and putting them to work).

~~~
jedbrown
The viability of separate processes is highly dependent on the problem domain
and algorithm. Threads have their place.

------
amirmansour
I'm actually interested to know what the HN community does to achieve
parallelism in Python? Do you use multiprocessing module, Stackless python, PP
module (parallel python)?

~~~
Nursie
That's usually the point at which I decide the project could really use being
rewritten in C.

If I need performance badly enough that I'm considering making major
structural changes in order to utilise multi-process or find some other way
around the GIL, I probably ought to be considering using a language that
supports what I'm trying to do. Of course my python projects tend to be fairly
small.

~~~
smeg
Or just write the fast bits in C, call it from Python via ctypes and get the
GIL unlocked and parallelism for free.

~~~
Nursie
If my projects were large enough to make that a more attractive option, sure,
that's a great plan.

Usually when I've hit this situation I'm writing something simple but
processor intensive, and it's easier for me (12 years commercial
parallel/distributed C programming, amateur python enthusiast) to go back to
home ground at this point.

------
xlevus
Could we not have a per-object lock, but only on objects the programmer
defines as requiring thread safety?

~~~
jbert
If you tag objects as being either "owned" by a thread or "shared" then you
could (in the python implementation) do something like:

    
    
      if (obj.owner == current_thread):
        # just get in and mess with it
      elsif (obj.owner == shared):
        # get in and mess with it under a lock
      else
        throw "not your obj - make this obj shared if you need that"
    

From a language perspective, you'd just need a way to declare that an object
was 'shared'. All other objs would be owned by their creating thread.

