Great article. It’s not so bad as a user of python honestly. Venv goes a long way, and with docker its get the rest of the way there.
The problem is when you want to open-source a package. I consider myself a professional Python developer, yet I was completely overwhelmed with all ways you can manage and release a package. Hatch, poetry, rye, setup, and a million other things that can make you a waste days fighting the language.
I'm surprised there's no mention of Docker or other container-based strategies. In my world, Docker has neatly, cleanly and -- perhaps most importantly -- _portably_ addressed all of the "environment management" and "Python version management" concerns outlined in this post.
As much as I hate to admit it, it does seem the most straight-forward option to recommend. Even after all of these years of this being an obvious problem.
Which reminds me that I need to revisit devcontainers for my work. When I first tried them, I ran into a few rough edges which might have since been resolved.
In a way I'm glad it's not mentioned. As a FreeBSD user, nothing is more disheartening than to find a cool and interesting project written in a great^1 language without any packaging or distribution method other than "here's my container bruh". Like, ughh.
^1 Caveat, making a package of a Python project can be a rabbit hole can be a real pita. Virtual envs are usually good enough for most stuff until I start to need a compiler in every jail. Nowadays I'm looking almost only at Go (best) or Rust because their distribution is trivial in comparison.
I'm with you on pretty much all of what you've said. The _trivial distribution story_ is part of the reason I handed off maintenance of tmuxinator and started working on rmuxinator (spiritual successor written in Rust).
Otherwise, I often find myself needing to deploy Python/Node/Ruby at $WORK and I'd really rather not try to get a bunch of dev/prod machines (possibly running different OSes!) trying to use whatever the flavor of the day for managing virtual environments, interpreter versions, packages, etc. is.
You're right that it's not trivial if you're starting from scratch but, in theory, you only need to get it right once. Even if you did need to move to a different strategy in the future because pip stopped being included in your container's distro or something, that's a single point of failure and not something that needs to be sorted out on every dev/prod machine.
To move up a level of abstraction, though, the official Python image pretty much does The Right Thing by default (copy requirements.txt, pip install, execute some command, etc.). Of course that isn't going to work for every scenario but it's a good place to start.
It's "This thing makes no attempt to be portable or coordinate with the rest of the world, so the only way to package it is to include a copy of it's own entire world."
There is no excuse for a python script to be a hothouse flower.
It's one thing to choose to containerize something for your own operational reasons, but something else entirely for something to be so fragile that thst's the only way it even works is just you need a copy of the devs laptop.
What held me back from using Python more is the absolute madness of these tools and the ecosystem. Like, Node looks like a tidy ecosystem in comparison. I've tried almost all the tools in those venn diagrams in the article and I probably had the most luck with poetry + pyenv. But then I started having version conflicts with the packages and gave up. Most packages out there are just released to Pypi with little consideration to the right dependencies to require. Other languages I used that have really solid package management tools: Ruby, Java, Rust, Go. Node with pnpm seems to be reaching a good point.
I don't really disagree, but Python is complicated because the language existed before all of those others for the most part and got quite popular before them too. This means the various use cases for Python got spread about long before the packaging tools really started to try and catch up. It used to be a technical problem.
This means trying to fit all the various use cases for packaging has been like herding cats. It's certainly not a technical problem anymore, the newer ecosystems/languages have figured it out, but it is a community/human problem. Thankfully there is now a small dedicated team that has been working on the problem for years, and progress has been made. For the most part if you stick to a single tool(poetry or rye for example) then most of your problems disappear as long as you stick to their blessed way, whatever it is.
I do understand the historical reasons. Javascript has been kind of the same for years, also because its usage exploded right at the beginning. I'm just saying that between the tools that we have today, if you take into account the ecosystem, Python is not my first choice, even if it's used a lot for AI/statistics. I hope the team you are talking about can pull it together. It would be nice to have one package manager/environment manager, especially for a language that has a philosophy of the "one obvious way" to do things.
After this blogpost was published in 2023, uv came along:
> TL;DR: uv is an extremely fast Python package installer and resolver, written in Rust, and designed as a drop-in replacement for pip and pip-tools workflows.
> uv represents a milestone in our pursuit of a "Cargo for Python": a comprehensive Python project and package manager that's fast, reliable, and easy to use.
> As part of this release, we're also taking stewardship of Rye, an experimental Python packaging tool from Armin Ronacher. We'll maintain Rye as we expand uv into a unified successor project, to fulfill our shared vision for Python packaging.
Rye is listed. While the Astral guys continue to impress, their adoption of the project is still super young. I do not think uv changes anything over base rye yet.
They've made a ton of progress in just the last couple weeks, actually.
Still super young, like you said, but it's really good for a lot of cases and now defaults to using uv under the hood I believe, which makes it quite fast.
The problem is when you want to open-source a package. I consider myself a professional Python developer, yet I was completely overwhelmed with all ways you can manage and release a package. Hatch, poetry, rye, setup, and a million other things that can make you a waste days fighting the language.
I highly recommend this cookiecutter from Simon Willison: https://simonwillison.net/2024/Jan/16/python-lib-pypi/
I just open-sourced a project last week[0] and it took 5 minutes (after 2 days fighting other tools).
0. https://pypi.org/project/agentrun/