
High Performance Python Extensions: Part 1 - JohnnyLee
https://www.crumpington.com/blog/2014/10-19-high-performance-python-extensions-part-1.html
======
syllogism
Here's how I would write this, in Cython using "pure C" arrays:

[https://gist.github.com/syllog1sm/3dd24cc8b0ad925325e1](https://gist.github.com/syllog1sm/3dd24cc8b0ad925325e1)

It's getting 18,000 steps/second, in the same ballpark as your C code.

I prefer to "write C in Cython", because I find it easier to read than the
numpy code. This may be my bias, though --- I've been writing almost nothing
but Cython for about two years now.

Btw, if anyone's interested, "cymem" is a small library I have on pip. It's
used to tie memory to a Python object's lifetime. All it does is remember what
addresses it gave out, and when your Pool is garbage collected, it frees the
memory.

Edit: GH fork, with code to compile and run the Cython version:
[https://github.com/syllog1sm/python-numpy-c-extension-
exampl...](https://github.com/syllog1sm/python-numpy-c-extension-examples) . I
hacked his script quickly.

~~~
JohnnyLee
If I could have your permission, I'd like to incorporate this into a future
post in the series. I can credit you in any way that you'd like.

~~~
syllogism
That's fine, please link to
[http://honnibal.wordpress.com](http://honnibal.wordpress.com) . I'll probably
write a short note on it, I've been meaning to say more about "my way" of
using Cython.

Edit: Submitted here.
[https://news.ycombinator.com/item?id=8483872](https://news.ycombinator.com/item?id=8483872)

------
krastanov
If I understand correctly, the `pypy` people strongly encourage the use of
`cffi` instead of the CPython API, as the latter is tied too much to CPython
and does not permit efficient JITing.

~~~
Derbasti
It's not so much about efficient JITing, and more about the fact that Pypy is
written in Python and does not have a C-backend.

That said, I advocate CFFI even in CPython--CFFI is a clean way of calling C
libraries without having to

\- write interface code in C (CPython extensions) \- write something that is
not C or Python (Cython) \- use a compiler or linker (CPython extensions and
Cython)

I have personally written nontrivial CPython extensions, Cython extensions and
numerous CFFI bindings, and I vastly prefer CFFI over any of the alternatives.
With CFFI, you write your interface code in Python, and dynamically load C
libraries at runtime with no compiler/linker required. It's fast and easy.

Plus, it's all pure Python and thus one source file works across
Windows/OSX/Linux and CPython/Pypy without modification or compilation. I
really can't say enough good things about CFFI!

------
neotrinity
I recently cythonized the performance critical parts of a numpy/scipy based
project with much success.

One does not necessarily need to get you hands dirty writing c-extensions (
although it can be a good exercise to learn the CPython API ).

In cython, you just need to sprinkle some static types to the inner loops and
bump the speed up.

------
makmanalp
There are also other approaches like the HOPE jit [https://github.com/cosmo-
ethz/hope](https://github.com/cosmo-ethz/hope) and Theano which is more about
expression optimization and compilation:
[http://deeplearning.net/software/theano/](http://deeplearning.net/software/theano/)

~~~
grakic
I wrote a blog post about HOPE
[http://blog.goranrakic.com/archives/2014/10/evaluating_hope....](http://blog.goranrakic.com/archives/2014/10/evaluating_hope.html)

------
pdknsk
Have you tried using ctypes? In particular ctypes.from_buffer is such a useful
function.

[https://docs.python.org/2/library/ctypes.html](https://docs.python.org/2/library/ctypes.html)

Unfortunately it doesn't work in PyPy.

[https://bitbucket.org/pypy/pypy/issue/869/](https://bitbucket.org/pypy/pypy/issue/869/)

------
JohnnyLee
Part 2 of this series is posted here:

[https://www.crumpington.com/blog/2014/10-21-high-
performance...](https://www.crumpington.com/blog/2014/10-21-high-performance-
python-extensions-part-2.html).

Hacker News thread:

[https://news.ycombinator.com/item?id=8493366](https://news.ycombinator.com/item?id=8493366)

------
syllogism
Why not Cython?

~~~
JohnnyLee
My preference is to work in C directly. That being said, I may look into a
Cython implementation for a future post.

~~~
andreasvc
Cython lets you avoid a lot of C API boilerplate. The code to parse function
arguments is quite distracting, and especially when it comes to managing
reference counts it's error prone to do it manually.

With Cython you get to choose whether to work in a purely low-level style that
maps directly to C code, or whether you want to mix in Python code. This lets
you speed up existing code gradually.

