> Fast is good, I guess, but what does this say about Python?
I don't buy this complaint (?) about Ruff. The broader point is that Python is _so good_ that people are willing to learn Rust to make the Python experience even better.
TIL about meson-python, which appears to be the most popular method after setuptools to build C extensions. There are 80 projects out of 107,757 analyzed pyproject.toml files, and SciPy and NumPy migrated to Meson.
I use setup.py legacy because of my semi-complicated requirements for C and Cython-based extensions (including a code generation step, and per-file compilation options, and compiler-based compilation and link flags). Knowing that SciPy and NumPy managed to migrate means Meson might be the right option for me.
There's also scikit-build (https://scikit-build.readthedocs.io/en/latest/), a wrapper around CMake, like how meson-python is a wrapper around meson, and enscons (https://github.com/dholth/enscons, which is by the original wheel creator), which is a wrapper around SCons (I'm not sure how much maintenance it gets though).
It's interesting that the original post (https://chriswarrick.com/blog/2023/01/15/how-to-improve-pyth...) complained about there being too many options (whilst missing the "scientific python" and rust backends, and including wheel (which is part of setuptools now) and venv (which is primarily a library)), when much of the original push to standardise packaging came from the scientific python community (see e.g. https://pypackaging-native.github.io/), and the need to get away from a distutils/setuptools monoculture (given previous attempts at improving the situation e.g. bento).
I didn't look at scikit-build because it requires an existing CMake installation and I didn't want my customers to deal with that. (I sell a source code license.)
... oh, wait! TIL "The easiest way to get CMake is to add it to the pyproject.toml file. With pip 10 or later, this will cause the CMake Python package to be downloaded and installed when your project is built."
99 of the backends use scikit-build-core.
Only 11 of the backends analyzed used enscons.
There are too many options. "There should be one-- and preferably only one --obvious way to do it." is at odds with the "let a hundred flowers bloom" of the current build backend philosophy.
The obvious way to do it is to start with the Python documentation.
My code works with setuptools because I started this project 15 years ago, and it's been incremental changes to my setup.py ever since. I find it hard to figure out which option I should use, because so few people use these alternatives and there's very little guidance - either direct advice from PyPA or hits from DDG - about how to handle the issues I've already resolved in setuptools.
Pypackaging-native even describes the special things I need, and which I've solved with setuptools:
- how do I have per-file compilation options so I can optimize for different architectures? These are also compiler specific.
- how do I enable/disable compile-time support for OpenMP? And specify the right link flags? And provide binary wheels for macOS with OpenMP?
- how do I add a code generation step?
- how do I have the source distribution include the Cython-generated C source so my clients don't need to have Cython installed? (My experience is there's about a 20% chance my customers have an incompatible Cython installed in their environment. Distributing C source means they can build without issues.)
The fact that NumPy successfully switched to Meson tells me Meson will likely handle what I want, I just have to figure how to do it. Which I do not look forward to.
That I now know CMake is another option makes the paradox of choice rear its ugly head.
Just FYI, but you do not need to switch away from setuptools if you do not want to. `setup.py` isn't going away, rather it is no longer the interface that "build frontends" (i.e. pip, but also pypa/build) use to build the project (there is a well defined interface made up of a number of task specific functions).
Thanks. I do understand that, and even if PyPA and the setuptools maintainers wanted to get rid of it, there's such a large installed base that any transition would take at least a decade. Plus, as I learned yesterday, the cmake-based backend also depends on setuptools.
Thing is, my setup.py is full of hacks and is a nuisance to use, so I would prefer a better solution.
What kind of hacks?
In the "python setup.py install" days I processed sys.argv to handle extra compile-time parameters, like "don't compile for OpenMP". When direct use of setup.py became unsupported, I switched to environment variables to pass in the configuration.
While it works, I don't know if it's a long-term viable solution. I read about the push towards isolated environments and reproducible builds, which is not something you can get if your build depends on ephemeral environment variables which are not captured in the pyproject.toml or dependency hashes.
(Remember, I distribute a source package, not a precompiled wheel.)
The longest component in my distribution takes a minute to compile. Setuptools does not handle dependency tracking well, so even if I touch C code which only takes a second to compile, it still rebuilds the entire component.
I have a hacked-up Makefile for when I really need to work on that part of the code and want faster turn-arounds, but I would prefer to have a unified build system that really understands build dependencies.
I subclass build_ext to handle the per-file compilation args, per-compiler link args, and to add an '$ORIGIN' rpath for non-Darwin builds as I generate shared libraries which depend on other shared libraries. To make this work I had to monkeypatch setuptools implementation internals.
I know how fragile it is, and would like something better, which I could depend on being around for another 15 years.
But when I dip into the Packaging world, I feel like my needs are in a forgotten corner, with the main impetuous being the needs of enterprise systems with engineering staff - a far cry from the packaging world I started with 20 years ago where building C extensions was one of the primary concerns.
blows my mind that in 2023 PIP still exists as it is, having in other languages good examples like: npm, composer, maver, cargo, yarn, etc.. which works like a charm since several years and most of the community are happy with them (No one is happy with PIP let's be honest)
Haven't had much troble with pip to be honest. I hear a lot of complaints from all kinds of people but I have never had an issue. Issues only arise for me when people are deep in the poetry, pdm and whatever tools.
I don't buy this complaint (?) about Ruff. The broader point is that Python is _so good_ that people are willing to learn Rust to make the Python experience even better.