
Python Wheels Crosses 90% - groodt
https://pythonwheels.com/
======
haberman
Today I spent at least an hour fighting with Python packaging. The more I
think about it, the more I feel that self-contained static binaries are the
way to go. Trying to load source files from all over the filesystem at runtime
is hell. Or at least it's hell to debug when it goes wrong.

I would love to see a move towards "static" binaries that package everything
together into a single, self-contained unit.

~~~
gorgoiler
In the past week my coding colleague (whose age is only just into double
digits) and I have had to:

* insert lines of code into asciidoctor Ruby to debug how the command line interface differs from the library interface;

* read through Raspberry Pi SenseHat code to figure out that the drivers won’t load via SSH (you have to plug in an HDMI cable!);

* rummage through Python’s PY Sequence List to answer the question “what kind of sort does Python actually use?”.

The problem with shipping binaries is it lowers the standard for being able to
build from source.

If a package is easier to ship as a binary because the source is hard to
distribute reliably, we give up hope of end users being able to modify and
build their own versions.

This is obviously bad for freedom. It’s also hard for debugging and
_learning_.

~~~
executesorder66
You have a 10 year old coding colleague?

~~~
gorgoiler
Not quite that young. I am a teacher with 85 teenagers under my wing, at the
last count.

An important part of school is learning how to learn. It is a constant
struggle with pupils to get them to actually think instead of googling for an
answer.

Sometimes it feels a bit like I am a grumpy oldie telling them things like _in
my day we had to look things up in books instead of using Google!_ but most of
the time it’s valuable in and of itself to go through things from first
principles.

(The sensehat bug was a case in point.)

As far as source code is concerned, it turned out to be quite tractable to
follow the Python source code at least far enough to find the timsort.txt
document.

If any of these kids go on to have technical careers, there will be plenty
enough time for googling answers later on in life. Hopefully some of them will
be writing the answers too.

~~~
fnord123
tbh the only source of truth is the code. Documentation and comments are all
lies! _shakes fist_

~~~
toyg
It's a bit like tech support, where you have to assume the user is _always_
lying.

~~~
maxbond
I once got locked out of my Amazon account because I went through a weird flow
where I ended up changing my email without clicking a confirmation link, and I
made a typo, so it was changed to a nonexistant email.

No one in tech support believed me. No one was willing to help me. They told
me to make a new account.

I've done my best not to buy from Amazon since.

Please don't be hostile to users.

~~~
toyg
That's more fraud detection than tech-support, sadly. Even if they believed
you, they might not have been authorized to help. You wouldn't want random
people to hijack your account by crying "typo" to the support guys.

~~~
dragonsngoblins
Surely they could revert it back to the previous email account though?

Possibly after you provide some kind of validation like the last few digits of
credit card tied to that account or something?

~~~
toyg
I'm almost sure Amazon used to allow you to log on with old email-password
combinations, until they got flak for it. Not sure what they do now.

------
u801e
One thing I found when converting our python application packaging from RPM to
wheels is that wheels don't properly handle the data_files parameter in the
setup call. That is, it places files under the python library directory
instead of in the absolute path as specified. This means that sample
configuration files and init scripts end up in the wrong place on the file
system. In order to get around this, we had to upload the source distribution
to our devpi instance and run pip install with the --no-binary option which
would then place those files in the correct directories.

The other issue is that there's no equivalent of the %(config) RPM spec
directive to prevent the config file from being overwritten if it already
exists on the file system.

So, for libraries, wheels are a good cross-platform packaging solution, but
not so much for applications that require configuration files and init
scripts.

~~~
raziel2p
Wheels don't, and shouldn't, replace RPM or deb packages, and I don't think
anyone advertised them as such. They replace eggs or installing from source
.py files, you should still build RPMs or debs on top of that if you want to
do any system-level stuff like set up systemd services, config files in /etc
and so on.

~~~
u801e
> Wheels don't, and shouldn't, replace RPM or deb packages

Part of the motivation for using wheels in our case was to allow for pulling
in later versions of certain python packages as part of the dependency
resolution process when installing or updating the package. Using RPM while
attempting to get the updated dependencies would mean having to repackage
every single dependency (direct and indirect) from pypi as RPMs and upload
them to our yum repo so that yum would be able to handle dependency
resolution.

> I don't think anyone advertised them as such. They replace eggs or
> installing from source .py files,

The setup method from setuptools does have the necessary features for handling
system-level files and the bdist_rpm command leveraged that when building
RPMs. It just seems that if this was a bug rather than a feature, then it
should have been noted when transitioning from egg based distributions to
wheels. Nothing in the documentation for eggs or wheels states that they don't
support deploying system files in absolute file paths.

The fact that doing something like:

pip install git+ssh://git@github.com/org/project.git@1.2.3

and

pip install project==1.2.3

have different results for where files listed in the data_files named
parameter of the setup end up on the file system suggest that the lack of
absolute path support when installing wheels is a bug rather than an
intentional change.

~~~
raziel2p
That makes sense. What I did in the past was include wheels of the project
itself and all dependencies (pip download) in the RPM/deb, then create the
virtualenv and install dependencies as a post-install script. You could
upgrade dependencies by calling pip install -U from the virtualenv, however
any time you installed the RPM/deb it would "reset" to the packaged version
(though there should be ways to prevent that, e.g. tell pip to never downgrade
packages).

------
navait
I don’t know a lot about Python tooling, but in general my experiences with
pip have been pleasant, so I appreciate all the work done by the maintainers
to make it pleasant.

------
foldr
Ah, so I've been confusing PyPI with PyPy. Someone made a great naming
decision there.

~~~
wodenokoto
A too little known fact: PyPI is pronounced Py-pee-ai, whereas PyPy is
pronounced Py-py.

If everybody knew that, I think the confusion would mostly disappear.

~~~
JTon
> A too little known fact: PyPI is pronounced Py-pee-ai

So I was reading this comment and thinking to myself, how does one even
pronounce "ai"? Language is not my strong suit. After searching the web, I
found out that "ai" is a diphthong and it sounds like "eye" or the letter "i".
Leaving that info here for others who may be confused. Sound:
[https://www.youtube.com/watch?v=uyKgPH0kmrU](https://www.youtube.com/watch?v=uyKgPH0kmrU)

~~~
wodenokoto
Just goes to show how difficult conveying sounds and pronunciation can be in
text!

I’ll keep this in mind to next time I want to write out the letter I and use
eye instead.

------
schwag09
At one point in time I created a Python package to highlight this benefit of
wheels: "Avoids arbitrary code execution for installation. (Avoids setup.py)"
\- [https://github.com/mschwager/0wned](https://github.com/mschwager/0wned)

Of course Python imports can have side-effects, so you can achieve the same
results with 'import malicious_package', but the installation avenue was
surprising to me at the time so I created a simple demo. Also consider that
'import malicious_package' is typically not run as root whereas 'pip install'
is often run with 'sudo'.

~~~
ghshephard
I thought the rule was never run `sudo` with `pip install` or you'll screw up
the permissions on your system.

~~~
sigjuice
For other reasons, it is just as bad when using python from Homebrew on macOS
where sudo isn't necessary. `pip install` will install modules in `/usr/local`
where they will get mixed with Homebrew-provided python packages. I was hoping
there would be a way to make `pip install --user` the default, but I couldn't
figure it out the last time I checked.

~~~
ghshephard
This is exactly why you want to do all (as in 100%) of your python work in a
virtual environment, so the packages are completely isolated in your
~/.virtualenvs/[ENVNAME]/lib/pythonx.x/site-packages.

Never, ever, do a pip install in your non virtualenv environment.

------
downerending
So far it's just a bit of smoke on the horizon, but I'm noticing some packages
abandoning 'pip' installs entirely in favor of 'conda'. It's a bit early to
tell if this trend will take off, but it does seem plausible.

~~~
dijksterhuis
I really hope conda stays on the user end.

Like, I understand it's great for managing package dependencies/setting up
environments etc for a single project. But I've found it's an absolute
nightmare for building docker images/generally doing build stuff when it's
involved.

Not to mention conda installs seem to take waaaay longer than pip.

apt + pip works far more sensibly and reliably in my experience.

~~~
blt
conda seems to spend a lot of time in a SAT solver or something to deal with
the known NP-complete problem of dependency resolution [1]. Maybe pip doesn't
do the same thing for requirements.txt setups?

[1] [https://research.swtch.com/version-
sat](https://research.swtch.com/version-sat)

~~~
downerending
It can be pretty bad, yes. A primary issue seems to be that conda package
authors don't always do their deps right.

Pip, on the other hand, often "wins" by not even bothering to notice the
versions of low-level libraries, etc. If that doesn't work, you get to keep
both pieces.

------
groodt
325 of top 360 Python packages are now distributed as Wheels.

------
thehiddenbrain
For anyone working on python wheels requiring to compile c/cpp/fortran, this
may be useful. See [https://scikit-build.org](https://scikit-build.org)

(Disclaimer: I am one of the maintainer)

~~~
dralley
Thanks for your work! I've used skbuild to package a couple of C library
dependencies we have that would otherwise only be available to us as RPMs.

------
usr1106
That doesn't seem to be a particular fast adoption. I remember seeing the
first wheels when working at a job I quit early 2010. So it must be over 10
years.

Edit: A web search points to 2012, so maybe it's "only" 8 years?

Edit 2: Pip came in 2008, so something changed somewhat before 2010 as I
remembered. But what did it install if not wheels?

~~~
groodt
If you were using PyPI you would have encountered source archives or Wheels.
If you were not using PyPI you could have encountered source archives or Eggs.

~~~
usr1106
> If you were using PyPI you would have encountered source archives or Wheels.

Before 2010? That's what I believed to remember. But it looks like wheels did
not exist before 2012, so that can't be true.

~~~
groodt
If you were using PyPI before Wheels were available, you would have
encountered source distributions (sdist).

------
fortran77
But snakes don't have wheels. Why not call it "skins?"

~~~
xapata
You get wheels of cheese from the Cheese Shop.
[https://www.youtube.com/watch?v=Hz1JWzyvv8A](https://www.youtube.com/watch?v=Hz1JWzyvv8A)

~~~
reedwolf
I wonder how long until nobody remembers Python isn't named after the snake
species?

~~~
kuroguro
But surely Monty Python was named after the snake species?

------
jessaustin
This seems like a really effective way to set up a page for shaming
"laggards". It would be interesting to track over time how many github issues
are just links to this page.

~~~
rlayton2
As noted on the page, but this idea (at least in the python world) started
with such a "Wall of shame" for libraries that hadn't updated to Python 3
support. I'm not sure how much progress was directly attributable to that
site, but it was fairly widely referenced.

~~~
jessaustin
Haha python is so authoritarian.

------
dingdingdang
Since pip is used to install Wheels it would probably be best to have a new
separate 3rd party meta tool to install package managers themselves to avoid
the confusion. Preferably this should have it's own additional PEP and
integrate PyPI and PyPy along with other packages that could make life simpler
for the (hopefully now happier) end user.

------
awinter-py
have always wondered why pypi doesn't generate whl files for pure-python
sdists

and why companies like travis / github aren't more active in language-level
packaging work

github gives away so much free docker time -- faster installation would save
them money directly

~~~
X-Istence
Because there is no way to know if something is pure-python or not.

Even more so with pep517 and being able to use different build systems.

~~~
awinter-py
wait then how does bdist_wheel know to make platform-dependent vs platform-
independent whl files?

~~~
icegreentea2
Your hunch is right, setuptools is able to determine if it's pure python or
not ([https://packaging.python.org/guides/distributing-packages-
us...](https://packaging.python.org/guides/distributing-packages-using-
setuptools/#pure-python-wheels))

Maybe it's not reliable enough?

In any case, I can see why PyPI wouldn't want to increase the scope of their
work. Sometimes is super valuable to just be good at one thing (distribute
what others give you).

~~~
wirrbel
I think the pypi folks just don't have the resources / capacity to implement
this. With appropriate funding this could have been done (development efforts
wouldn't be that big, but probably operational costs for the build farms). In
the past I ran (devpi builder)[[https://github.com/blue-yonder/devpi-
builder/](https://github.com/blue-yonder/devpi-builder/)] with
(devpi)[[https://pypi.org/project/devpi/](https://pypi.org/project/devpi/)] to
build wheels for all my dependencies and keep them stored in a place I have
control over.

------
ciarannolan
Could someone give a beginner's tl;dr of wheels vs. eggs?

I use Python extensively but the "Advantages of wheels" section on this site
is way over my head.

edit: Thanks everyone :)

~~~
groodt
Wheels are newer and have an official PEP (PEP 427)

Eggs contain compiled Python bytecode which requires publishing egg versions
for every version of Python. While Wheels are distribution formats that can
support more than a single Python version.

------
fouc
Is Python Wheels the equivalent of Ruby Gems? Or is it better?

~~~
groodt
Gems

~~~
quietbritishjim
Honest questions from non-user of Ruby:

* Do Ruby gems contain all binary depedencies statically linked into them?

* Is there a list of exceptions that are guaranteed to be present on target platform (see manylinux [1]).

The fact that you answered "Gems" rather than "something better" means the
answer to both must be "yes" but I wanted to check.

[1] [https://github.com/pypa/manylinux](https://github.com/pypa/manylinux)

~~~
groodt
Apologies, I think the question was edited/clarified after my comment.

Gems are more similar to Python sdist’s. Native code is compiled on the
installing host.

There is no specification like manylinux1,2010,2014 like there are for Python
Wheels.

In this sense, Python Wheels can be considered more sophisticated than Ruby
Gems.

------
dmix
That’s a great way to track tech adoption within a community.

------
daveisfera
Any news on supporting Alpine Linux with wheels?

------
arusahni
Any idea if musl support is on the horizon?

~~~
groodt
I think still a while off [https://mail.python.org/archives/list/distutils-
sig@python.o...](https://mail.python.org/archives/list/distutils-
sig@python.org/thread/H3323AXRRLJAYOY2XZKS74IOUQMJUOYD/)

