

First Python 2.7 interpreter to use multiple cores - spazz
http://mail.python.org/pipermail/pypy-dev/2012-May/009901.html

======
sp332
To be a bit pedantic: the multiprocess module works fine for multiple cores in
regular cpython, I actually used it for a little graphics program I wrote.
Nice to see the GIL gone for multithreading though :)

~~~
timtadh
It isn't pedantic just irrelevant. Using multiprocess is an entirely different
beast than using threads. All shared values have to explicitly allocated that
way in shared memory. Since there is a separate interpreter for each process
there is no problem with internal state getting messed up. Threads are just a
harder problem for python and one which has _never_ seen an effective
solution.[1] That is why this kind of work is exciting.

[1] You may argue (correctly) that multiprocessing and using pipes to
communicate is a better solution than threads to begin with (for many
applications). However, sometimes a threaded solution is the right solution
and previously if you needed threads python didn't work for you.

~~~
slurgfest
Using multiprocess is different from using threads. But that doesn't make it
irrelevant, because it is a completely functional, already-existing way of
using multiple cores.

------
halayli
I much prefer share-nothing approach and use sockets + subprocess for
communication. It scales nicely.

~~~
freyrs3
Depends on what you're doing. Shoving a numpy array with 10^7 elements over
the wire isn't going to scale nicely at all. There are still a lot of
computing scenarios having shared memory is the best approach.

~~~
jandrewrogers
You do not need threads for shared memory. I know of quite a few high-
performance software systems that use process isolation and yet leverage
shared memory via mechanisms like mmap() for IPC.

~~~
seunosewa
Good point, but shared memory via mmap isn't very Pythonic since you have to
manage the shared memory manually, you can't allocate Python objects on the
shared region, etc.

~~~
d0mine
You don't need to use mmap explicitly. multiprocessing.Array and numpy arrays
can share the same memory that can be used from different processes e.g.,
<http://stackoverflow.com/a/9849971>

------
helpbygrace
It seems to use Software Transactional Memory scheme. I hope that it will work
better with hardware-aided transactional memory in Intel's next generation
architecture Haswell.

------
dbecker
This is incredibly interesting, and the people working on it are incredibly
smart.

But I think it's a slight oversell to call this a python 2.7 interpreter. For
instance, most scientific computing code that runs in cpython 2.7 won't run on
pypy.

It's impressive from a computer science point of view. But, pypy has a ways to
go before I'd call it a python interpreter without adding qualifications.

~~~
fijal
PyPy is officially sanctioned as python 2.7 interpreter. Cannot run scientific
code does not rule it out. This is more a problem on the scientific code being
too implementation specific (for good reasons though). This _is_ known to be a
problem to a lot of people (including me), however it does not rule pypy as a
python 2.7 interpreter (without qualifications).

~~~
dbecker
My apologies. I mistakenly lumped C extensions in as an inherent part of
python. I was wrong.

------
tocomment
I'm not following this. Does it mean the GIL is finally gone?

~~~
slurgfest
I will get excited when it actually works out to be faster than interpreters
with the GIL.

I'm a bit disappointed that Python seems to have no interest in standardizing
a lightweight-thread (greenlet, coroutine) interface that isn't an awful mess.
yield is powerful but coroutine implementations with it are baroque and
opaque. Java-style threading has a million gotchas and the interface is just a
dog. I want to be able to teach people with a week of experience to write
concurrent programs. (What about writing turtle programs with multiple turtles
running at the same time?)

If Python doesn't do it, then I want someone else to eat their lunch.

~~~
Tobu
Which of the existing coroutine implementations have you tried, and what did
you dislike about it?

~~~
jlgreco
In my humble opinion, the greatest problem with all of them is that there are
so many. Most of them are rather fine I think, but it can just be a serious
pain in the ass getting your head around all of them.

~~~
Tobu
I've only needed one (Twisted, which is actually slightly outside the space
occupied by gevent et al). You can't stop people from building more, but feel
free to bet on a popular project if you don't want to deal with choice.

~~~
jlgreco
In a world where I only work with my own code, that works. Unfortunately that
is not my reality.

------
chimeracoder
My understanding of the GIL is that its inclusion was justified by the fact
that removing it would speed up multithreaded programs at the cost of slowing
down single-threaded programs (the latter being more common than the former).

Given that, and given that the patches for removing the GIL were submitted for
an earlier version a while back (2.4 or before, I believe), is it
unfeasible/impossible to design an interpreter that detects whether a program
requires support for multiple threads and then act accordingly? Since this
currently requires the inclusion of a module, it seems like it would be
unambiguous.

~~~
sanxiyn
Detection will be non-trivial. On the other hand, it could be a simple command
line option, like specifying the number of cores you want your program to run
on.

~~~
ori_b
I'm not sure about that. If you hook thread creation into invalidating the
code that was was JITted without STM, you might be able to handle it
reasonably well.

Of course, the slow path would just use STM all the time.

~~~
sanxiyn
> ... invalidating the code that was JITted ...

I mean, that is non-trivial.

~~~
ori_b
Maybe. If you determine that you only need to do it on spawning the first
thread, and never go back, then you can probably spare the expense of throwing
out ALL of the jit cache and starting from nothing.

