
Python Wheels - T-A
http://pythonwheels.com/
======
SEJeff
This is far and above my favorite post on python wheels from the guy who wrote
Flask, bits of Sphinx, and jinja2 to name a few, Armin Ronacher:

[http://lucumr.pocoo.org/2014/1/27/python-on-
wheels/](http://lucumr.pocoo.org/2014/1/27/python-on-wheels/)

I switched our webapp deployment from using sdists from an internal pypi
instance (yay cheeseshop!) to using wheels built on the server and the time to
install some things such as lxml, or numpy, went from 20 minutes-ish to 20-30
seconds. Python wheels are fantastic.

~~~
mattikus
How are you hosting the wheels internally? Are you still using an internal
pypi instance but with wheels instead of sdists?

I've been looking at doing something similar in our environment but there's so
many options I haven't figured out what the best and most straightforward way
might be.

~~~
benesch
Shameless plug: I had the exact same question at my last gig, and wrote up a
quick open-source tool to build wheels and dump them to an S3 bucket. [0]

Usage is as simple as

    
    
        mkwheelhouse mybucket.mycorp.co scipy numpy
    

which will automatically build and upload those wheels for your current
architecture and dump them into mybucket.mycorp.co. It builds a pip-compatible
index, too, so you can just tell pip to search your wheelhouse first before
falling back to PyPI:

    
    
        pip install --find-links mybucket.mycorp.co scipy numpy
    

If you need to build for several OSes, you can run mkwheelhouse against the
same bucket from each OS.

The downside of this approach is you can't host private packages, because you
need to enable public website hosting. (Although, VPC endpoints might have
changed this!) But the simplicity of this approach plus the massive speedup of
not needing to constantly recompile scipy was totally worth it.

[0]:
[https://github.com/whoopinc/mkwheelhouse](https://github.com/whoopinc/mkwheelhouse)

------
andrewguenther
Python packaging is such an absolute disaster. As a casual Python user, can
someone explain to me how Wheels fixing the problem?

~~~
coherentpony
As a casual python user, I've always found it fairly straightforward.

As a systems engineer, I've always found it challenging to maintain a sane
python ecosystem across multiple hosts.

~~~
e12e
As a (now casual) sysadmin, I think things have gotten a bit better.
Virtualenv (now part of python3) helps a lot. The trick, if running on Debian,
is:

1) use system python -- if you can. Now with python3.4 and 2.7 in stable,
that's the easy bit.

2) If you can't do 1), use backports and/or manually backport[1]

3) If you need python-packages that depend on c-libraries, use "apt-get build-
dep python-foo".

4) use a virtualenv, or several virtualenvs, eg:

    
    
      virtualenv -p python2.7 myvenv
      ./myvenv/bin/pip list -o #list outdated packages,
       # typically that'll mean you first need to:
      ./myvenv/bin/pip install -U pip #possibly also setuptools
    

Then install the packages you need with pip -- see 3) for tip on stuff like
pyqt that needs some (fairly hairy) packages.

This works for most everything -- except perhaps for pygame, which is
exceptionally gnarly.

[1] Manually backport, the easy way:

    
    
      sudo -s
      echo "deb-src http://httpredir.debian.org/debian\
       sid main contrib non-free" \
       >> /etc/apt/sources.list.d/sid-src.list
      apt-get update
      cd /tmp
      apt-get build-dep python3
      exit # No need to build as root
      apt-get source python3 # this assumes sid has a version
       # that'll work for you
      cd python3*
      dpkg-buildpackage -uc -us
      cd ..
      sudo dpkg -i python...deb
    

Note, this may or may not work with major packages like python -- but it often
goes a long way. If you can't get enough build-deps satisfied, things are more
complicated -- and you're _probably_ better off running testing/unstable in a
chroot, possible with help of the schroot package -- rather than fighting with
conflicting packages.

------
ambrop7
I don't personally care about Python packaging (and virtualenv) anymore when I
can use any relevant Python package with Nix, or easily make a Nix package
myself. I just hope the wheels don't make it harder to package stuff with Nix.
With eggs I think there's something like an --old-and-unmanagable flag that
makes it still output a directory structure.

For deploying software to end-users (rather than developers), bundling all
required Python code with appropriate wrapping to adjust PYTHONPATH works just
fine.

------
afarrell
You could alternately use conda, as much of the scientific computing community
does: [http://conda.pydata.org/docs/](http://conda.pydata.org/docs/)

~~~
ForHackernews
No, no no. No 'alternately'. The python community needs to pick ONE standard
and, uh... _standardize_ on it.

~~~
dvanduzer
This isn't the issue.

Wheels and Conda are almost completely orthogonal. Conda is a higher level
abstraction than pip. Wheels is operating at the lowest level of abstraction.

Multiple package management tools are relatively fine. Having multiple
competing _format_ standards (egg vs wheel vs lots of other options) _would_
be a disaster.

(For reference, there's .deb versus .rpm, but also dpkg versus apt-get versus
yum versus up2date versus...)

~~~
shoo
From a continuum.io blog post about Conda [1]:

> Having been involved in the python world for so long, we are all aware of
> pip, easy_install, and virtualenv, but these tools did not meet all of our
> specific requirements. The main problem is that they are focused around
> Python, neglecting non-Python library dependencies, such as HDF5, MKL, LLVM,
> etc., which do not have a setup.py in their source code and also do not
> install files into Python’s site-packages directory.

I haven't used Conda, but I totally get their main point here. _Python
packages sometimes depend upon non-Python packages_. Any Python-specific
packaging solution that cannot express that e.g. a Python package depends upon
a native library does not really solve this problem.

I don't regard packaging, dependency management, and isolation as programming-
language-dependent problems.

[1] [http://www.continuum.io/blog/conda](http://www.continuum.io/blog/conda)

~~~
e12e
Conda certainly sounds ambitious -- but I'm doubtful how useful it really is.
As I see it, it mainly fills two needs: the lack of a package manager on OS X
and the lack of a package manager on Windows.

Windows is getting a package manager now, and OS X arguably have the App
Store.

Everywhere else (read: Linux/BSD) -- most of what's packaged by Conda, is
already packaged by the vendor.

In short: I don't think it's the best way forward (but it may very well be the
best way for _many_ usecases/users _right now_ and in the near future).

~~~
cossatot
A big part of Conda's usefulness is not system-wide package management a la
apt or yum or whatever, but being able to have reproducible, isolated
environments for scientific computing or other times when reproducibility is
very important, and/or when you need multiple versions of packages for
compatibility with other packages.

I believe it uses virtualenv for this, but I may be wrong. Regardless, having
this feature (as well as no interference with the system Python) is one of the
killer features for me, and others I'm sure.

~~~
e12e
This is a great point. Still, whole system snapshots in the form of a vm disk
image, vagrant etc might be a better way to serve that use-case. Perhapssome
form of appkiance levering something like guix or coreOS?

------
svisser
One of the real benefits of wheel files is that the file format is
standardised:
[https://www.python.org/dev/peps/pep-0427/](https://www.python.org/dev/peps/pep-0427/).

This means the Python community can work towards standard tools and interfaces
for working with Python packages rather than trying to support many different
ways of installing / distributing packages.

------
fndrplayer13
My experience with wheels has been that it continues to be an absolute
nightmare to deal with Linux distros and we end up building from the tarballed
source as usual.

Then again, we tend to struggle a bit with Python package management in
general (who doesn't, I guess).

~~~
viraptor
Which packages specifically did you have to rebuild?

------
schmichael
> Avoids arbitrary code execution for installation. (Avoids setup.py)

This isn't much of an advantage as you're trusting code compiled on an
untrusted machine. I suppose it saves you some CPU cycles, but don't be fooled
into thinking it's safer.

~~~
geofft
There's a somewhat compelling use case for `pip install foo` not trusting the
packages, but the actual Python code that runs "import foo" trusting it. In
particular, if I'm a sysadmin of a shared machine and a user asks me to
install something systemwide, I don't want arbitrary code running as root, but
if the installed code is malicious, that's up to the unprivileged user who
chooses to use it to keep their own data safe.

It's not terribly compelling given stuff like virtualenv, though, since users
can just `pip install` stuff on their own and the experience is strictly
nicer. And it's also not compelling if there isn't an explicit promise that
this is treated as a security boundary.

~~~
schmichael
> In particular, if I'm a sysadmin of a shared machine and a user asks me to
> install something systemwide, I don't want arbitrary code running as root,
> but if the installed code is malicious, that's up to the unprivileged user
> who chooses to use it to keep their own data safe.

Ah, that makes a lot of sense, especially for applications (eg meld) and
command line utilities (eg httpie) which are nice to install systemwide as
root, but only ever run in an unprivileged user's context.

------
jedberg
Where did they get the list of the top 360 packages on pypi? I tried to find
it the other day but it looks like it was removed?

Also, the ordering on the right seems to imply they are in order of popularity
on pypi.

Does anyone know where they got their list or if they are associated with pypi
in some way?

~~~
mattikus
It looks like they're querying the rpc interface for pypi and sorting by
downloads:

[https://github.com/meshy/pythonwheels/blob/master/utils.py#L...](https://github.com/meshy/pythonwheels/blob/master/utils.py#L68-L71)

------
dr_zoidberg
Wheels solved one major issue that few people talk about: packages that needed
compiling on 64-bit Windows (which was a mess, bordering the impossible).

~~~
nvk
Stop using Windows, it's impossible to keep any level of sanity on a non POSIX
system.

~~~
dr_zoidberg
Customer uses Windows, so...

------
Spiritus
I don't get all the hate towards Python packaging. For any package that isn't
ancient (or extremely complex like NumPy or SciPy) it just works to "pip
install". And nowadays pip will even cache builds locally as a wheel so
installing something a second time takes next to no time.

Also, learn to use:

    
    
        pip install --user <package_name>
    

Disclaimer: I use Python every day, mostly on OS X and RHEL.

------
kp25
"Installation of a C extension does not require a compiler on Windows or OS X"
This is the real advantage.!

~~~
pekk
I have often seen it touted that OS X is "real Unix." To me, that would imply
the easy availability of working compilers and compile environments. Is that
not true?

~~~
Tenobrus
There's easy availability, but it requires installing the Apple Developer
Tools, which can either be done at install time (by not by default) or by
downloading them from apple. So even though python is installed by default a C
compiler might not be, and if a beginner is trying to use python for the first
time they might be daunted by having downloading several gigabytes of
developer tooling to install one python package.

------
MBlume
I don't see an explanation for the name -- is the joke that cheese comes in
wheels and Python packages come from the cheeseshop?

~~~
msellout
Yes.

------
schmichael
Are there community buildbots for OSX and Windows for people who don't own
licenses of those OSes?

~~~
shoo
Re: Windows, Appveyor provides a CI service for windows. It has been used to
test and build python packages, and produce wheel archives for windows. E.g.
sklearn uses this. I have used it myself to set up a CI build for a simple
python utility and found it straight forward to get working.

------
gmjosack
The biggest problem I've had with Python packages is that you can't use
multiple version of the same package in a Python project without a lot of
complex tricks. This becomes more and more a problem in large code bases over
time. One dependency might need requests 1.x and another 2.x in incompatible
ways.

Unfortunately Wheels don't solve this problem as it seem inherent to how
python imports modules.

I've seen people get around this by vendoring their dependencies and I've done
some tricks in the past where you provide a vendor proxy package that you can
point at a different site packages but this is brittle if your dependency has
additional dependencies or uses absolute imports. Now you're maintaining a
fork.

I'd love to hear if anyone has had more success in this realm. I love Python
but using node and rust, and having the ability to have isolated dependencies
trees has made me sad about the state of python packages.

~~~
pekk
To balance this, you might also solicit feedback about the problems that are
caused by Node's use of multiple versions of the same package in a project.

------
pyre
Anyone else have issues with pillow/PIL and wheels? I have to force that
package to _not_ install from cached wheels, because the first build is fine,
but subsequent installs from the cached wheel seem to break.

~~~
jtchang
Pillow and PIL have always gave me headaches while installing (not specific to
wheels)

------
Spiritus
Not related to wheels, but Python packaging targeted towards RHEL/CentOS
systems. This is extremely convenient:

    
    
        python setup.py bdist_rpm
    

Has to be done one a RHEL/CentOS machine obviously.

------
binaryautomata
Why is it called a python "wheel"?

Should python really have been called "rodent" with "wheels" being the "cheese
wheels" we rodents get from the "cheese shop"?

~~~
msellout
Think British comedy, not big snake. Monty Python had a sketch about cheese.
Thus wheels of cheese.

