
Making Cython as easy as Python - hack4supper
https://github.com/Russell91/runcython
======
vog
This strategy should be more common! In the source tarball, there should be
one main program which you start. The compiling should be an implementation
detail of the main program.

The same for web applications. I never liked the idea of reintroducing a
separate build (concat/minify/compress) process whenever the sources change.
Instead, the production mode should be as simple as the debug mode: When the
main page is requested per HTTP, first rebuild the client JS app as needed,
then deliver it to the browser.

Caching the build result should be no different from any other caching of
automatically-generated resources.

I use this approach for almost all new software that I write and I'm quite
satisfied with that. The main program (e.g. "bin/mytool") is a wrapper shell
script or Python script that runs the build as needed, then starts the
program. In the simplest case, this is a wrapper around "make && ./run_tool".

Of course, the program should indicate the build with a small message to
stderr, especially if the build takes some seconds or more. Moreover, it
should be possible to trigger the build individually such that distros can
build packages from that. On the other hand, running the program with "-h" or
"\--help" is a kind of build-only process, too.

~~~
icebraining
It's practical, but on the other hand, that goes against the security practice
of not allowing the executable zone (be it memory or filesystem) to be
writeable.

I personally prefer to precompile stuff, even running "python -m compileall"
to create .pyc files, to avoid having to keep that path writeable by the user
running the program.

~~~
vog
Good point! However, if you go down that route in a web application, don't
forget to pre-compile templates, and do the same for all other cached things
that are "code-like".

More generall, I find W^X hard to enforce in typical web applications. Sure
you can mount directory read-only, disable access to /tmp and observe what
happens. But then they might use the database as template cache, or whatever.

In most (web) application I'd be happy if they had a central directory where
they write stuff into (bonus points for making that directory configurable),
instead of scattering generated files all across the source tree.

------
vegabook
Cython, Numpy, Numba (and others) are what make me skeptical that any of the
numerical computing competitors to Python (Julia, or to a lesser extent
Lua/JS/Clojure even compiled Scala etc) can displace it at. Why would you
abandon this wonderfully friendly, malleable language with easily the largest
set of options for _anything_ you might like to do, for something nominally
faster on an artificial benchmark (which inevitably ignores the above 3
libraries), when by dipping into any of these you get C-speed in such an
accessible and friendly way, most of the time beating the competitors easily
on performance.

The only real threats to Python to me come would from a properly parallelized
language that would use the GPU/multicore natively in vector form. I don't see
that anywhere yet.

~~~
pathsjs
The problem with the Python ecosystem is that there exist a lot of libraries
for scientific computation mostly incompatible with each other (blaze, numpy,
numba, numexpr, dask...). Some of them try to reintroduce types in order to
compile to native code, thereby losing the advantages of dynamic typing. In
short, it is a mess, and everyone is trying to add on this because the
existing ecosystem is big and the cost of rewriting everything is deemed too
high, so people try to fix python limitations at the library level, sometimes
making python more cumbersome to write tha a conventional statically typed
language (I am looking at theano).

I don't think that Lua is that much of an improvement, and I do not have big
hopes for the JVM. The growing number of projects trying to improve the
situation with value types, off-heap allocations and C compatibility are a
sign that the VM is not suitable for scientific computing and will not be much
better for a while.

I instead hope that Nim will take a leading role for scientific computation.
It has an easy syntax and is flexible enough to write libraries such as numpy.
At the same time it is fast and C compatibility is trivial. Of course, there
is a library ecosystem to build, but I think that this is more doable than
trying to work against the language itself. I am trying to start this effort
with [https://github.com/unicredit/linear-
algebra](https://github.com/unicredit/linear-algebra) but the road is very
long and I hope it takes off.

I cannot really comment on Julia as I do not know it welll enough

~~~
vegabook
I am intrigued by Nim. Does it have a REPL though? I think an official REPL
(not a third party hack) is mandatory for scientific/numerical computing.
Ocaml is the only compiled language I can think of which has a credible REPL,
officially supported.

~~~
pathsjs
It currently does not have a REPL, and that is its main limitation right now.
I think it is still suitable for more complex workflows where you would not do
it in the REPL anyway - think training a large neural network. I hope
something will come after 1.0 is out...

------
willvarfar
Its an aside, but recently Python 3 has got annotations. It would be nice if
cython supported annotation syntax, so that type-annotated cython was as much
as possible valid python than can also be run through an interpreter.

I understand a lot of the nuances, particularly the lack of in-function
annotations etc. My own foray into Python type annotations is obiwan
[https://pypi.python.org/pypi/obiwan/](https://pypi.python.org/pypi/obiwan/)

------
cbsmith
"Runcython aims to simplify the process of using Cython without sacrificing
scalability."

I know this is being a semantic weenie, but I hate when people use scalability
when they mean efficiency.

~~~
chuckbot
And I hate when people use efficiency when they mean speed.

Python -> Cython speeds up you program but does not change the complexity of
your algorithm.

~~~
oliwarner
How do you suppose it speeds it up then?

I'll give you "the complexity of your algorithm" likely doesn't change, but
how that algorithm is processed _is_ usually more efficient when compiled
through C, rather than running through a Python interpreter, if only because
of how it's running.

~~~
chuckbot
I agree it is faster by running it through compiled C code.

Saying "my code is more efficient than your code" might be acceptable, but
saying "my code is efficient" drives me mad when it is used as a synonym for
"my code is fast". Take a look at [1] and the note that this is not about
optimization. Efficient algorithms are usually algorithms with close to
optimal time or space complexitiy.

[https://en.wikipedia.org/wiki/Algorithmic_efficiency](https://en.wikipedia.org/wiki/Algorithmic_efficiency)

------
rplnt
Seeing how this uses mktemp, I assume the results of runcython build are not
reused. Is that correct? If sou, wouldn't it make sense to save the results?
Maybe with a switch?

Also feature requests: Windows support for run/make would be nice. Cross-
compile options for make would be awesome.

------
halosghost
This is great to see! I don't know if it was inspired by runhaskell, but it
does appear to be similar. I have to say, this kind of tool is something I
have come to feel is incredibly helpful and important for my dev workflow.

I spend most of my time developing in statically-typed, compiled-to-machine-
code languages; obviously, one of the major advantages of more dynamic or
interpreted languages is the ability to rapidly prototype. This type of tool
allows me to quickly prototype something without losing all the power, control
and other benefits of my languages of choice (in particular, Haskell these
days).

Things like this make me more amenable to python by the day! Keep up the good
work!

------
python_que
I am using Python to solve combinatorial problems, and so my code relies
heavily on the itertools library.

My question is if I would still get a considerable speed-up if I rewrite some
of my code in Cython, given my reliance on itertools?

~~~
hogu
it may be worth checking out cytoolz

~~~
mathnode
Yup. Cytoolz. Some notes on performance of Cytoolz:

[http://matthewrocklin.com/blog/work/2014/05/01/Introducing-C...](http://matthewrocklin.com/blog/work/2014/05/01/Introducing-
CyToolz/)

~~~
python_que
great thanks both for the tip.

