
PyPy 1.6 Released - Full Python 2.7.1 Implementation - jparise
http://morepypy.blogspot.com/2011/08/pypy-16-kickass-panda.html
======
lliiffee
Can I take advantage of this thread to ask the HN crowd a technical question?
Some time ago, I implemented an automatic differentiation tool. Using operator
overloading on a special "autodouble" type the tool would trace the execution
of a block of numerical code. Then, some calculus would automatically happen,
and it would output and compile fast c-code that would compute the original
function and derivatives in pure c. This was great, except the c-code that was
output was freaking gigantic (like hundreds or thousands of megabytes) albeit
very simple, and so the c compiler would take _forever_ to run. Sigh.

My question is: could I leverage pypy somehow to avoid this? Can I output
RPython? Can I output whatever RPython is compiled down to instead? Can I do
this with no more than, say, a 3x penalty compared to c?

(I apologize for asking a question only marginally related to the particular
article here...)

~~~
carterschonwald
Off hand and guessing about the problem:

You might want to look at approaches that use dual numbers. Likewise, in
instead of inlining the procedures, generate a differentiated version of each
procedure with a new name. If those don't cover your problems, perhaps look at
how other autodiff tools for C do it?

~~~
lliiffee
For being offhand, those are very good guesses! Dual numbers won't be
efficient, as I want reverse-mode autodiff.

As to the multiple procedures: Well, as I was doing it, even a single
procedure can be many hundreds of megabytes large. Even for very very simple
code, however, I noticed that GCC was superlinear in code size. I suppose I
could somewhat arbitrarily break up the code into arbitrary functions. I
wonder if that would speed things up?

Other autodiff tools: Well, they basically trace execution, but then run an
interpreter instead of trying to actual generate compiled code. I wanted to
both be faster than that and have the wonderful experience of writing pure
python...

~~~
carterschonwald
it sounds like you're getting lots of code duplication. 1) try running a
common subexpression elimination process on your code before doing the
autodiffing, and create a procedure for each shared expression 2) for prim
ops, again, have a procedure created for the diffed version instead of
inlining, and sub in the procedure instead.

perhaps something like these ideas would help

If you want an example of a nice high level Auto diff lib, a nice one that
works via operator overloading is <http://hackage.haskell.org/package/ad> ,
which seems quite nice though I've not had the opportunity to use it myself.

yes, optimizing compilers such as gcc use algorithms that are superlinear in
code size when they're optimizing. Perhaps you should instead try out the
operator overloading approach (and see if you can )?

gl :-)

Aside: When I hear the phrase execution trace in the context of program
analysis, i think abstract interpretation, though I'm not sure if thats
relevant for you.

cheers!

~~~
lliiffee
It isn't exactly common subexpressions. Basically the problem is things like
matrix multiplies _always_ get unrolled.

I've used lots of operator overloading based autodiff packages for C++. They
are great, but the issue is not how the function is recorded (I used operator
overloading myself in my python package) but how it gets executed at runtime.
Unless a compiler (or JIT) is called sometime between when the operator
overloading happens and execution happens, the function is basically being
interpreted at runtime. This is what happens in, e.g. ADOL-C, SACADE, and
CPPAD, all of which come with a significant (e.g. 20x) performance penalty as
compared with hand-written derivatives.

------
voyvf
> has beta level support for loading CPython C extensions.

Is this via ctypes, or "real" support in much the same as how CPython would
behave?

I ask because this is one of the features that I've been waiting (impatiently)
for - I've run some Flask projects using PyPy and gunicorn, and _love_ how
fast it goes, but really want to be able to use the rest of my codebase, which
unfortunately does rely on some C (and Cython) extensions. (:

~~~
vgnet
It's "real" support, using your criteria. However, it's slower in PyPy than in
CPython and than what the same thing based on ctypes would be on PyPy.

If there's a pure-Python version of the C extension, it might be faster on
PyPy than the C extension support that cpyext (PyPy C-API compatibility
module) provides.

Cython-based code is currently incompatible (it goes well beyond the public
C-API), but a GSoC project to generate ctypes-based pure-Python code from
Cython is (was?) going on.

~~~
voyvf
> It's "real" support, using your criteria. However, it's slower in PyPy than
> in CPython and than what the same thing based on ctypes would be on PyPy. >
> If there's a pure-Python version of the C extension, it might be faster on
> PyPy than the C extension support that cpyext (PyPy C-API compatibility
> module) provides.

That's fine, the mere ability to run them will be nice. (:

> Cython-based code is currently incompatible (it goes well beyond the public
> C-API), but a GSoC project to generate ctypes-based pure-Python code from
> Cython is (was?) going on.

I remember that GSoC! Any idea as to what happened to it, if it was scrapped,
or whatever?

~~~
vgnet
> I remember that GSoC! Any idea as to what happened to it, > if it was
> scrapped, or whatever?

I know it made a lot of progress in Cython-land (updates at
<http://rguillebert.blogspot.com/>) but no real overview or high level status
update has been given yet (AFAIK). Since it's from GSoC 2011, it's about time
for that to happen... and for other people to start contributing code <hint,
hint> :).

------
sylvinus
I'm always blown away by the consistent performance gains they reach with each
new version. Congrats!

------
kristofferR
In general I'm very happy with my choice of Ruby/Rails instead of
Python/Django, but PyPy is one of the few things I envy Python developers for.

I wish something similar could be developed for Ruby.

~~~
irahul
> but PyPy is one of the few things I envy Python developers for.

Depending on your needs, there are others - numpy, scipy, matplotlib, nltk,
gevent.

> I wish something similar could be developed for Ruby.

Isn't <http://rubini.us/> supposed to be the PyPy for Ruby? It's not complete,
but then neither is PyPy.

~~~
stephenjudkins
Rubinius and PyPy take quite different approaches to solving the same general
problem. It's inaccurate to analogize them.

Rubinius specifies a bytecode (see <http://rubini.us/doc/en/virtual-
machine/instructions/>) and implements a VM that executes this bytecode. The
VM is written in C++. The main difference between Rubinius and other Ruby
implementations is that nearly everything else is written in Ruby. The lexer,
parser, and parts that compile the AST to bytecode is all written in pure
Ruby. Further, large parts of the standard library and Ruby language that are
implemented in lower-level languages in other implementations are instead
written as runtime-level Ruby. However, large and important parts of the
infrastructure (including GC and memory allocation) are written in C++.

PyPy takes an altogether different approach. It specifies a restricted subset
of Python known as "RPython". Basically, it's statically-typed Python. There
is also a "compiler" (also called a "JIT generator"), written in pure Python,
that compiles an arbitrary interpreter written in RPython into a fast JIT.
(See [http://morepypy.blogspot.com/2011/04/tutorial-writing-
interp...](http://morepypy.blogspot.com/2011/04/tutorial-writing-interpreter-
with-pypy.html) for an example of writing a very simple, small interpreter in
RPython.) A cool feature of RPython is that while there must a statically
RPython AST to compile, during the "eval" stage you can leverage the full
dynamic capabilities of Python.

The second part of PyPy is a full Python interpreter written in RPython. Since
RPython is a subset of Python, you can actually run this interpreter sandboxed
inside any other Python implementation, albeit very slowly. The PyPy toolchain
compiles this down to a very fast Python interpreter that features JIT
compilation, among other cool features.

The neat part of PyPy (in contrast to Rubinius) is that the toolchain part is
largely decoupled and generalized away from Python-the-language. Conceivably,
one could write ANY language in RPython (or what's compiled down to RPython)
and have a fast JIT interpreter. Experimental language features can be added
and changed (see the source code for the different garbage collectors in PyPy)
with little mucking with the low-level VM.

Rubinius is a neat, pragmatic project that could turn into one of the best
Ruby implementations around. However, it's not breaking new ground like PyPy
is.

~~~
headius
Rubinius's parser is C/++ code derived from the same Bison grammar as regular
Ruby. That's not to say it couldn't be pure Ruby, but it isn't right now.

~~~
stephenjudkins
I stand corrected. Thanks.

------
sho_hn
I'm still bummed at being stuck with the dilemma of having to chose between
CPython 3.x and PyPy. PyPy with Py3k support would rock.

------
socratic
Are there production users of PyPy?

I feel like PyPy has always been the most academically interesting Python
implementation. But has it taken away mindshare from CPython?

~~~
vgnet
Yes, there are.

Recently Quora announced it was running on PyPy[1]. Some other disclosures
were made (a Django project[2], LWN internal processing[3], tweets about
speedups in production, etc.), but the PyPy team is thinking about officially
asking for success stories in the near future[4].

[1]: [http://www.quora.com/Alex-Gaynor/Quora-product/Quora-is-
now-...](http://www.quora.com/Alex-Gaynor/Quora-product/Quora-is-now-running-
on-PyPy)

[2]: <https://convore.com/python/whos-using-pypy-in-production/>

[3]: <http://lwn.net/Articles/442268/>

[4]:
[https://bitbucket.org/pypy/extradoc/src/tip/blog/draft/succe...](https://bitbucket.org/pypy/extradoc/src/tip/blog/draft/success_stories.rst)

~~~
socratic
To me, this list seems to suggest the opposite actually.

[1] is the company where one of the main developers of PyPy works. [2] appears
to be a discussion where they are looking for anyone using PyPy in production.
[3] seems to be an article about someone experimenting with PyPy for git
processing. [4] is the developers looking for non-toy examples of production
use so that they can get more funding.

~~~
vgnet
It suggests that there aren't many production users, true. But it does prove
that there are some.

~~~
socratic
Indeed. Though two-ish production users after 7 years of development hardly
seems like a success.

By way of contrast, (1) in the Ruby world, YARV went from an alternative
implementation to the official implementation within two years, and (2) in the
JavaScript world, node.js is similarly being used in production in tons of
places after only two years. (Though I realize that those examples aren't
exact parallels.)

This isn't meant to criticize your response, rather, I find it interesting
that the Python world seems to have so many alternative implementations (PyPy,
IronPython, Jython, Cython) despite what appears to be really minority
mindshare compared to CPython.

~~~
sqrt17
* Jython is a really nice alternative if you want to use Python-the-language in a Java environment. Its suboptimal speed and the fact that it's really easy to write Java code that can be used from Jython means that you'll probably use it more as a scripting language * The relation between IronPython and .NET is probably very similar * For CPython, people have always wrapped their favorite C library and started using it - with SWIG in the old times and nowadays with Cython, which is a compiler for a subset of Python extended with types. In fact, quite a lot of useful Python libraries use Cython (whether as a glue language or as an implementation language), and it's probably in production use at a lot of places.

One of the problems of PyPy is that, while targeting Python-the-language, it
also gives up CPython compatibility by using their own PyPy VM (in the sense
of, GC'd execution environment with a JIT that's not really compatible with
other C/C++ code or even CPython). Because they're targeting their own
execution environment, they cannot profit from either the plethora of C/C++
libraries that exist or even the very decent library support that exists on
the JVM.

The PyPy people have recognized this and started to take care of the affair
with a JNI-type version of CPython's ctypes library. The problem is that very
few library wrappers (or even non-Python CPython extensions) are based on
ctypes, so that it's not useful as a replacement for CPython for most of the
people (especially those that need the speed improvements).

The reluctance of PyPy developers to start thinking about a solution for all
the existing CPython extensions has severely limited the appeal for using PyPy
in production use - having 10x gains for new code is not all that appealing
once you realize that you need to rewrite the other couple thousand lines of
(C++ or Cython) code and hope that you can optimize the PyPy code as well as
you optimized the C++ or Cython code.

Having said that, I'm really excited by the idea of a Cython variant that can
be used to wrap C/C++ code (or use Cython code) in PyPy's execution
environment. (This is the GSoC project that was referred to earlier) Or even
if I could use Cython to compile my C/C++ Python extensions for the JVM or
.NET.

~~~
nickik
Thats the problem all scripting languages have. They start slow, then people
want to be faster and start writting C code and that makes it really really
hard to have alternativ implementations. The guy implmenting Erjang had the
same problem he had to rewrite alot of C functions in Java. I think the python
comunity should recognisse what an awesome technolagy pypy is and start making
an widespreed effort and discurage C extentions. Pypy makes python-code so
much faster that alot of C extentions are not really worth it anymore.

~~~
sqrt17
Rewriting things in C, or Cython, also makes things much, much faster, without
the hassle of PyPy. And because you can do your own memory management (in the
places where it makes sense), Cython code is quite a lot better for well-
defined numerical applications than what you can get out of a GC-based
environment.

Realistically, you always want to be able to take advantage of one of the two
big ecosystems - namely the C world and the JVM world - because there are so
many libraries out there doing nontrivial things you do not have to
reimplement. Right now, writing C code that works well with generational
garbage collection (or really any kind of garbage collection that moves
objects around - i.e. all the well-performing ones) is either very tedious
(when you try to take account of objects being moved) or slow and possibly
error-prone (if you rely on JNI-style locking and unlocking of object
references).

As a result, it may actually be more attractive to build a Java bytecode JIT
into PyPy (and be able to have PyPy use Java classes within its more powerful
representation scheme) and get mindshare among the people currently using
Jython than trying to get the diehard C extension users to switch.

So much for the '"the X community should recognize what an awesome technology
Y is" is a surefire recipe for building sucky software' talk. People will do
whatever they do, and calling them idiots because they don't do what you think
is awesome doesn't lead anybody anywhere. (Though having a decent installer
and usable documentation may actually lead to more people discovering the
advantages of PyPy).

