
Python on Wheels - donaldstufft
http://lucumr.pocoo.org/2014/1/27/python-on-wheels/
======
shadowmint
Wow, great article. I've heard of wheels for months, but I've never really
understood why I should remotely care about them before (it always seemed to
me they solved a problem that didn't exist; namely, binary distributions for
pure python packages, since you can't support c-plugin packages cleanly with
wheel anyhow...).

Turns out the answer is:

1) Yes, you can kind of support packages with c-plugins, if you're specific
and careful.

2) They make server deployments significantly faster.

That's actually pretty neat.

...but damn, they still look like a massive pain to work with.

~~~
brabram
I confirm about the pain. They should be transparent and you shouldn't need to
care about them but that's absolutely not the case. Also, it's the
responsibility of the pypi maintener to push wheels on it, but none do this,
so you end up building your own wheels while this should have already been
done on pypi and it's stupid. And one last thing:
[https://twitter.com/mitsuhiko/status/426700148409135104](https://twitter.com/mitsuhiko/status/426700148409135104)
"It turns out, Python binary wheels on Python 2 are rejected by PyPI, I
suppose because of the lack of UCS tag in the filename."

Apart from all of this pain, I'm very happy to be able to use them.

I'm really waiting for someone to build a "wheel on demand" website that would
be a proxy to pypi. Or to turn pypi into a wheel farm (I don't know how
realist this idea is).

~~~
donaldstufft
<\- PyPI Administrator.

I have plans for either turning PyPI into a build farm, or making a secondary
service that acts as a build farm for PyPI. We've just been more focused on
cleaning up other issues.

------
girvo
For a language that prides itself on simplicity, this seems really painful...

~~~
Too
It is like a farce on the "there shall only be one way to do it" mantra. Not
to mention that some of the package managers has to be installed through one
of the other packet manager, creating a very long dependency chain if you want
to install a package through the former.

~~~
pak
The more that I use Python in the context of other languages, the more the
"there should only be one [obvious] way to do it" mantra irks me.

First of all, there's the cases in which reality fails to live up to
expectations, like this packaging debacle. If there was an obvious and best
way to do everything, we wouldn't need to spend so much money and time
engineering software. We wouldn't spend so much time arguing about distributed
vs. centralized, OOP vs. declarative vs. functional, type systems, build/test
systems, etc. The mantra is arrogant on its face.

Secondly, it flies in the face of innovation. It works against building a
creative community of users. So often, when Python users point out the holes
in the ecosystem, they are simply told "well the long and outdated PEP 5991834
already establishes the obvious way to do it" or "just shim it on top of this
inadequate corner of the standard lib" or the use case itself is questioned,
which patronizes the criticizer. The mantra becomes an excuse for reinforcing
a hive mind philosophy.

In this case, I wonder if it has subtly undermined the development of a
diverse and widely supported package index. The whole point of such a package
system is to support the belief that there are actually many ways to do the
same thing, and some may be better for certain styles/teams/circumstances.
Having the opposite outlook as a design principle for a language will attract
users less inclined to contribute to or build such a diverse packaging system.
I wonder this particularly because the more "laissez-faire" languages like
Perl, Ruby, and JavaScript are precisely the ones that have such great
packaging systems and ecosystems.

------
sashk
Spent some time this week to move deployment of venvs to use wheels (we
install numpy and matplotlib). Now, instead of close to 10 minutes deployment
time, it's done in about 30 seconds. I can live with the fact, that I'll need
to make changes to the deployment process now and then, while wheel traveling
towards 1.0, but I'm happy to go that route. And you should to.

~~~
gtaylor
Isn't there a way to make a relocatable virtualenv? I'd be inclined to just do
that, compress it up and distribute the entire virtualenv (our deploy
environment is homogenous to the extreme).

~~~
travisoliphant
Conda ([http://conda.pydata.org](http://conda.pydata.org)) is a more general
packaging format than wheels and we use it to distribute the Scientific Python
Stack (including complex C-dependencies) with ease. It is more general than
wheels and solves this fundamental problem now with all the benefits without
waiting for something in the future. See these blog-posts for more
information:
[http://www.continuum.io/blog/conda_packaging](http://www.continuum.io/blog/conda_packaging)
and
[http://technicaldiscovery.blogpost.com](http://technicaldiscovery.blogpost.com)

In particular look at the `conda package` command for an easy way to bundle up
any non-conda packages across a homogeneous environment.

~~~
dcuthbertson
FYI: I think your last link should be
"[http://technicaldiscovery.blogspot.com/"](http://technicaldiscovery.blogspot.com/").
There doesn't seem to be an active site at blogpost.com.

~~~
travisoliphant
Yes, thank you!. You have the correct link.

------
e12e
A great introduction to wheels. I'm still not sure if wheels really is a good
idea though. For simple use-cases traditonal pip seems fine, for more complex
use-cases buildout seems better? Having tried to install numpy and pygame in a
virtualenv I do see a need for something better -- but I'm not convinced
wheels will solve the problem of complex c-library dependencies...?

------
travisoliphant
This is a nice post, and the Python packaging community is making strides in
the right direction.

However, it's not there yet. Armin could bring himself to say this in this
post: "It's there, it kinda works, and it's probably worth your time."

Conda is not perfect (it needs more people building conda binaries as well as
a few pull requests to add https improvements) --- but I will confidently say
"it is there, it does work now, and it is worth your time"

This is especially true with a package repository like exists at
[http://repo.continuum.io](http://repo.continuum.io) and the many personal
repositories at [http://binstar.org](http://binstar.org).

People that have used conda for deployment (especially of complex
C-dependencies in the NumPy and PyData stack) are saving time and effort
today.

------
mariocesar
I use wheels to speed up the tests for sorl-thumbnail
[https://github.com/mariocesar/sorl-
thumbnail/blob/master/.tr...](https://github.com/mariocesar/sorl-
thumbnail/blob/master/.travis.yml#L26) Initially I was into fully use wheel to
install the environment but I found that some packages didn't work, now I just
use it for just some critical packages and reduce the 12 minutes test to 3
minutes

Will love to see wheels getting more attention, really feels like a great
direction for python packaging.

~~~
travisoliphant
A lot of people in the PyData ecosystem using NumPy and Pandas are using conda
and conda packcages ([http://conda.pydata.org](http://conda.pydata.org)).
Conda packages are easy to build and many exist at repo.continuum.io and on
channels at [http://binstar.org](http://binstar.org)

------
beggi
Page is currently down, so here:
[http://webcache.googleusercontent.com/search?q=cache:http://...](http://webcache.googleusercontent.com/search?q=cache:http://lucumr.pocoo.org/2014/1/27/python-
on-wheels/)

------
overgard
Ugh. I'm so sick of the open source communities' disdain for binaries. There's
no reason we should be recompiling shit everywhere. It's a waste of time,
headspace, and processing power.

~~~
Pxtl
When you're trying to support multiple architectures and OSes and your ABI
took a decade or two to settle down, and all your tools are open-source so you
have the source anyways, I can see the instinct to just say "I'll just compile
it against the target machine" even if it's absurdly wasteful.

~~~
hackerboos
Popular binaries can be uploaded falling back to building from source.

------
raptium
I thought it was another web framework ...

~~~
r0muald
Just like Ruby on Rails :)

------
_jb
Google cache link:
[http://webcache.googleusercontent.com/search?q=cache:0rftZaF...](http://webcache.googleusercontent.com/search?q=cache:0rftZaFa274J:lucumr.pocoo.org/2014/1/27/python-
on-wheels/+&cd=1&hl=en&ct=clnk&gl=ca)

------
pippy
I've had the exact same experience as the author of this post. Eggs are great
if you just want to install a python library. Horrible if you want to modify
the library or distribute a program.

I personally use a mixture of setuptools and distutils2, and it's a confusing
mix.

------
asmeurer
"What the command is supposed to do is to collect all the dependencies and the
convert them into wheels if necessary..."

How does it convert the non-wheel packages into wheels?

------
amouat
On a slight tangent; does anyone else think that Docker is probably a better
solution than virtualenv in most cases? I never really got on withe
virtualenv.

~~~
dagw
The subset of problems that virtualenv solves that are also solvable by Docker
is pretty tiny. But sure if you have problem that can easily be solved by
Docker, then Docker is probably a better solution.

~~~
amouat
Really? I was thinking you just set up a new Docker env for each python
project, then you don't need to worry about conflicting with libs from other
projects.

What am I missing?

~~~
dagw
Docker only runs on a small subset of the platforms python runs on is the most
obvious one.

Can docker easily access the host filesystem or do you have to copy all your
data into each docker env?

Can I access my GPU for doing CUDA/OpenGL/OpenCL things from withing docker?

Does docker play nicely with GUI apps?

Calling into a running docker env from an external program (when you are using
an app that has python as scripting or plug-in language) doesn't work as far
as I know.

The overhead of setting up, tearing down an switching between several Docker
envs seems to higher than with tools like virtualenvwrapper or conda.

Now there may be workarounds for these (and all the other) problems but on the
surface they don't seem easier than virtualenv.

~~~
creack
Docker runs on any linux. There is a few dependencies but the list is reducing
very fast. You don't even need to have AUFS as you can use devicemapper or
even regular copy if you have neither.

Yes, docker can easily access host's filesystem with volumes.

Yes you can access you GPU. I am using docker for mining.

Yes Docker plays nicely with GUI apps, there are plenty of GUI usecases with
docker (docker desktop, firefox, etc..)

You can do anything you want within a python app and docker. There is a
docker-py library that gives access to all docker features.

You can take a look at this:
[https://github.com/jpetazzo/stevedore](https://github.com/jpetazzo/stevedore)
in order to easily switch envs.

~~~
dagw
Sounds like it's high time I take a closer look at docker

------
DrJ
what about pex?

