The Python wheel "ecosystem" is rather nice these days. You can create platform wheels and bundle in shared objects and binaries compiled for those platforms, automated by tools like auditwheel (Linux) and delocate-wheel (OSX). These tools even rewrite the wheel's manifest and update the shared object RPATH values and mangle the names to ensure they don't get clobbered by stuff in the system library path. The auditwheel tool converts a "linux" wheel into a "manylinux" wheel which is guaranteed by the spec to run on certain minimum Linux kernels.
Wheels are merely zips with a different extension, so worst case for projects where these automation tools fail you you can do it yourself. You simply need to be careful to update the manifest and make sure shared objects are loaded in the correct order.
On the user side, a `pip install` should automatically grab the relevant platform wheel from pypi.org, or, if there is not one available, fall back to trying to compile from source. With PEP 517 this all happens in a reproduceable, isolated build environment so if you manage to get it building on your CI pipeline it is likely to work on the end user's machine.
I'm not sure what the state of wheels on Windows is these days. Is it possible to bundle DLLs in Windows wheels and have it "just work" the way it does for (most) Linux distros and OSX?
FWICS, wheel has no cryptographic signatures at present:
The minimal cryptographic signature support in the `wheel` reference implementation was removed by dholth;
The GPG ASC signature upload support present in legacy PyPI and then the warehouse was removed by dstufft;
"e2e" TUF is not yet implemented for PyPI, which signs everything uploaded with a key necessarily held in RAM; but there's no "e2e" because packages aren't signed before being uploaded to PyPI. Does twine download and check PyPI's TUF signature for whatever was uploaded?
I honestly haven't looked at conda's fairly new package signing support yet.
FWIR, in comparison to legacy python eggs with setup.py files, wheels aren't supposed to execute code as the user installing the package.
>>> Build Python wheels for all the platforms on CI with minimal configuration.
>>> Python wheels are great. Building them across Mac, Linux, Windows, on multiple versions of Python, is not.
>>> cibuildwheel is here to help. cibuildwheel runs on your CI server - currently it supports GitHub Actions, Azure Pipelines, Travis CI, AppVeyor, CircleCI, and GitLab CI - and it builds and tests your wheels across all of your platforms
You're right, both the infrastructure and metadata for cryptographic signatures on Python packages (both wheels and sdists) isn't quite there yet.
At the moment, we're working towards the "e2e" scheme you've described by adding support for Sigstore[1] certificates and signatures, which will allow any number of identities (including email addresses and individual GitHub release workflows) to sign for packages. The integrity/availability of those signing artifacts will in turn be enforced through TUF, like you mentioned.
You can follow some of the related Sigstore-in-Python work here[2], and the ongoing Warehouse (PyPI) TUF work here[3]. We're also working on adding OpenID Connect token consumption[4] to Warehouse itself, meaning that you'll be able to bootstrap from a trusted GitHub workflow to a PyPI release token without needing to share any secrets.
We bundle DLLs in our wheels in such a way that it "just works" for the user but it kind of feels like a hack. First a main DLL library is built completely separately from the wheel. Then a binary wheel is built where the .pyd file basically just directly calls functions from the main DLL. The main DLL is then just manually included in the wheel during the build step. Any dependent DLLs can also be just manually included inside the wheel as well.
Wheels are merely zips with a different extension, so worst case for projects where these automation tools fail you you can do it yourself. You simply need to be careful to update the manifest and make sure shared objects are loaded in the correct order.
On the user side, a `pip install` should automatically grab the relevant platform wheel from pypi.org, or, if there is not one available, fall back to trying to compile from source. With PEP 517 this all happens in a reproduceable, isolated build environment so if you manage to get it building on your CI pipeline it is likely to work on the end user's machine.
I'm not sure what the state of wheels on Windows is these days. Is it possible to bundle DLLs in Windows wheels and have it "just work" the way it does for (most) Linux distros and OSX?