

PyPy-STM: first “interesting” release - rguillebert
http://morepypy.blogspot.com/2014/07/pypy-stm-first-interesting-release.html

======
jpgvm
Very cool.

I am interested to see how this progresses. It seems to me there is lots of
optimisation possible to alleviate the cost of contention with greater number
of threads. Single thread performance will always have to suffer somewhat but
their cheap read barriers have done wonders already to bring that down to an
acceptable level. It will reach a point however where the only way to speed it
up further will be for the programmer to be able to specify STM hints for
variables.

I would like to see this in Java or C# where there is more room for extended
optimisation.

Kudos to the PyPy guys for sticking to their guns on this.

~~~
Torn
Java and C# don't have a GIL though, and have good support for threading
right?

~~~
thesz
Linked list for single processor is a job for first year student.

Linked list for multiprocessors with mutexes and locks is a PhD-level job.

Linked list for multiprocessors with STM is a job for first year student
again.

The quote is from one of Simon Peyton-Jones talks.

~~~
Torn
Thanks - that quote immediately shifted my thinking.

------
bsaul
That raises a few questions :

1- will this takes us closer to a golang like way of working in parallel (
since that means no GIL) ?

2- Why did dropbox / guido didn't join forces with pypy, since they already
had proved that their approach was very efficient performance wise ? Is it
because pypy doesn't deal with native C code at all, whereas the python LLVM
approach they chose may let the two work better ?

3- does anyone else feel like the python world is a complete mess right now ?

EDIT : didn't want to sound harsh, on the contrary. Seing so many smart people
work on separate path on great projects for so long feels like some great
energy is going to end up a waste of time.

~~~
meowface
>1- will this takes us closer to a golang like way of working in parallel (
since that means no GIL) ?

I doubt it. The asyncio module was an attempt at this but the API isn't that
great and it also isn't closely integrated with the language like goroutines
are in Golang. This is still a great improvement though, as is PyPy in
general. Basically, currently existing programs should run faster, and people
will use the multiprocessing module less and the threading module more. It may
change the way lower level parallel code is written, and library maintainers
may be able to simplify some things.

>2- Why did dropbox / guido didn't join forces with pypy, since they already
had proved that their approach was very efficient performance wise ? Is it
because pypy doesn't deal with native C code at all, whereas the python LLVM
approach they chose may let the two work better?

Probably a mix of "NIH" and also the belief that Dropbox has some of the best
Python devs (and BDFLs) out there, so surely they can create the best
solution. We'll see in the next few years if it plays out for them.

>3- does anyone else feel like the python world is a complete mess right now ?

Not that much. It's a big mess if you're trying to maintain a large open
source project and need to keep retaining compatibility from 2.6 to 3.4+, but
as a regular developer it's not too terrible. Personally I am still going to
stick with 2.7 for personal projects for years to come, though.

~~~
servercobra
For 3, even that isn't terrible in a lot of cases. All of OpenStack (AFAIK)
maintains py26, py27 and py33 support (and one of the projects I work on aims
for pypy as well, but isn't gating on it yet).

~~~
meowface
It's not that terrible, but it can be annoying. And arguably worst of all, it
can make code a lot uglier than it needs to be.

------
DannyBee
Neat stuff.

Not sure why they have "(insert LLVM snark here)" when the docs say "At the
LLVM IR level, the behavior of these special address spaces depends in part on
the underlying OS or runtime environment, and they are specific to the target
(and LLVM doesn't yet handle them correctly in some cases)."

Expecting that to work without fighting bugs seems "optimistic" to say the
least :)

------
tdicola
Awesome news. I'm kind of scared to think how much Python code is written to
assume the GIL though. I have a feeling a lot of libraries and code will have
very subtle race conditions and issues, but it's still great to have an option
for getting around the GIL.

~~~
chrisseaton
I think the big idea is that this STM only attempts to commit at points where
the GIL would have traditionally been released anyway - after so many
bytecodes or some other condition. The linearisation of the memory writes of
the different threads should be the same as if there was a GIL.

