

Wikipedia processing. PyPy vs CPython benchmark - robert-zaremba
http://rz.scale-it.pl/2013/02/18/wikipedia_processing._PyPy_vs_CPython_benchmark.html
What PyPy can do for Wikipedia processing tasks? Speedup!
======
JulianWasTaken
Can I tangentially point out without much connection to this benchmark more
than any of the others recently, that one of the great things about PyPy is
that if you have a thing and you run it on PyPy, you can usually pop in the
IRC channel and often get even _more_ tips on how to tune it to be even
faster?

There are the simple tips like "write everything in Python where possible,
don't use C extensions" like the OP noticed, but even after you've made the
decision on using PyPy there are often specific performance characteristics of
the PyPy implementation that can be really helpful to keep in mind, and it's a
great resource to try and take advantage of (human interaction with PyPy
developers like fijal who care about making things fast).

~~~
chubot
Can you give an example of a tip that would speed a particular program up?
Does PyPy have a lot of knobs to tune?

CPython doesn't seem to have that many knobs... You can tune some settings in
GC, and maybe the GIL acquiring interval, but it doesn't seem to produce
speedups in most programs.

~~~
fijal
PyPy does have a few knobs to tune, but they're not very often used (they're
nicely documented, however mostly in the source code, I'll try to pull say GC
parameters more prominently in the docs).

However, the way you _write_ python code can be tuned a lot. This is actually,
at least to an extend, our failure. JIT can make things _really_ fast, but can
get confused in places. We're trying to eliminate them as we go, but it can't
be always done.

Generators vs list comprehensions. Not using sys._getframe, sys.exc_info, etc.
not relying too much on dynamism like passing very general __kwargs and *args,
the list goes on.

Jitviewer can be a lot of help, but it's a bit buggy and too low level to be
seriously recommended.

~~~
ak217
> Generators vs list comprehensions

I'm a bit confused. Which one is faster?

~~~
robert-zaremba
list comprehensions of course!

~~~
signed0
The exclamation led me to believe you were being facetious, but
<https://bitbucket.org/pypy/pypy/wiki/JitFriendliness> agrees with you:

"As of now (Feb 2011) generators are usually slower than corresponding code
not using generators. Same goes for generator expressions. For most cases
using list comprehension is faster."

------
stefantalpalaru
>Moreover PyPy doesn’t kill my CPU as CPython does so in a meantime I could
normally use my laptop

You're not supposed to "use your laptop" during a benchmark.

~~~
rg3
Your advice is good in general. However, note the OP is reporting user time
(probably as shown by the "time" command). This time is the total CPU time
used by the process. It's not measuring how much wall clock time it took to
complete or the time used to complete involved IO operations. I think trusting
that number should be fine in this case.

~~~
sp332
It can cause problems with competing for cache space, which could have large
effects on CPU time.

~~~
rg3
You're right and I stand corrected.

------
tworats
Great to see real world use cases, and very encouraging to see PyPy performing
so well. I'll definitely be trying it on my future compute-intensive projects.

------
dbecker
I'm reminded of when I first started following the PyPy project, and I thought
it would be impressive if they could ever get close to CPython's speed.
They've done something truly amazing.

I hope the python community someday coalesces around a single version. I
generally can't take advantage of all their awesome work because the libraries
I depend on (e.g. pandas) won't run in pypy.

------
Qantourisc
I wonder what would happen if you would toss a LLVM solution at it :) Seems
like some are working on it according to google. No joy yet though.

~~~
huxley
I believe you're referring to the Unladden Swallow project, which
unfortunately is "pining for fjords". The project didn't make the progress
expected and folded.

<http://en.wikipedia.org/wiki/Unladen_Swallow>

------
kristianp
An optimisation path available to CPython is to put your inner loops into a C
Extension. I think a benchmark with this would be interesting.

~~~
fijal
except it's work. and it's not that good usually because you end up
manipulating Python objects anyway. It doesn't really work for anything that
can't be represented as a tight semi-autonomous C code with limited connection
to Python code. And then you usually can't run it on PyPy, which might or
might not be an issue.

~~~
stefantalpalaru
Unless that work was already done and available to you (lxml).

~~~
fijal
See comments on the blog. Majority of work is not in xml parsing so lxml does
not help (and you're crippled that the rest of the code does not run as fast).

