Packaging is hard but it's not hard enough to wholly explain pip.
Lock files alone are a proven piece of technology that pretty much just works across all modern package managers, and yet the best that pip can recommend to date is `pip freeze > requirements.txt`, which is strictly inferior than what is available in other package managers because there's no hash verification and no distinction between transitive dependencies and explicit dependencies.
That pip still doesn't have an answer for lock files is a sign of a deep problem in the Python package management world, not the natural result of packaging being hard.
My understanding is that pip developers are unpaid volunteers with other duties. That's what they frequently say at least. PSF rakes in a decent amount of coin every year but prefers to spend it on "outreach" instead of gaping wounds like this.
In short, python-dev leadership just doesn't care. Can't fix that with technical solutions.
Lockfile does not work perfectly for me because it does not lock python version. Using a lock file with different python version or even same version but different OS may fail.
That’s literally his point. Other languages have solved this. It makes no sense that it’s not solved in python. (If your not sure how look at a golang go.mod file or a rust Cargo.toml file.)
There's no reason why Python version couldn't be baked into the lock file or package manifest if pip had one. package.json does this [0]. Since all pip has is requirements.txt, it's a bit of a moot point.
> Your are not supposed to freeze into requirements.txt
ironic that pip freeze literally generates a requirements.txt with ==
constraints.txt certainly did not exist back when I was doing python.
Conversely Ruby packaging has been a solved problem for a decade, when the python community has been extremely resistant to conceptually similar solutions for the longest time on strange ideological grounds, and came around only recently.
Somehow I've managed to go all this time without ever having heard of this feature. If this is the blessed path, can you explain why the pip docs recommend freezing to requirements.txt [0]? And why does the documentation for Constraints Files in the next section talk about them as though they're for something completely different?
Here's what they say about requirements:
> Requirements files are used to hold the result from pip freeze for the purpose of achieving Repeatable Installs. In this case, your requirement file contains a pinned version of everything that was installed when pip freeze was run.
Here's what they say about constraints:
> Constraints files are requirements files that only control which version of a requirement is installed, not whether it is installed or not. ... In terms of semantics, there is one key difference: Including a package in a constraints file does not trigger installation of the package.
> ... Write a single constraints file for your organisation and use that everywhere. If the thing being installed requires “helloworld” to be installed, your fixed version specified in your constraints file will be used.
> Constraints file support was added in pip 7.1. In Changes to the pip dependency resolver in 20.3 (2020) we did a fairly comprehensive overhaul, removing several undocumented and unsupported quirks from the previous implementation, and stripped constraints files down to being purely a way to specify global (version) limits for packages.
This sounds like something vaguely similar to a lock file, but they seem to intend it to be used globally at the organization level, and they're certainly not pushing it as the answer to locking dependencies for a specific project (they specifically recommend requirements for that). Maybe you can use it that way—although this Stack Overflow answer says that the 2020 update broke it for this use case [1]—but even so it doesn't answer the fundamental need for lock files unless everyone is actually on board with using it for that, and even the pip maintainers don't seem to think you should.
Java is Maven or Gradle mostly. Neither perfectly ideal but there's clear choices by subcommunitues (e.g. Android =>Gradle) & churn is low. Not at all an awful situation.
C++ is mostly unmanaged in practice which is obviously bad (though Python is the only "modern" language where I regularly see published projects without so much as a requirements.txt so very comparable). However looking at the actual options for C++, Conan seems pretty clear cut & not awful in itself.
The package manifest for maven is pom.xml. Sure it's got quirks - I might have the wrong version of a maven plugin it's got required directives for or something - but it's always there, it's declarative, and dependencies are listed in a consistent, predictable, universally parsable format (I don't even need maven, it's XML). It's also highly configurable via .mvn at project level so if you check that in chances are you can work around many weirdnesses.
And it's always there. If your IDE didn't generate it your maven archetype did.
What's the source of truth in any random pip project that I should look to as a package manifest? requirements.txt? Setup.cfg? Pyproject.toml? Setup.py? What if they're all there? Do they all agree? Should I cascade or defer to one? Merge the trees? What if none of the above exist? Pip certainly doesn't ensure any does.
As for parsing them, requirements.txt is the most barebones near-informationless manifest out there, and setup.py is init code - good luck trying to automate reading the dep tree reliably without side effects.
The recent CocoaPods SPM migration is almost as painful as py2->3 but it seems to be a clearly understood direction with end in sight.
SPM's manifests being non-declarative is also super problematic for 3P tooling but again - it's one well defined target. Decisions are clear, if a little cumbersome. There's nowhere near the mess of indecision and reduplication you see in python projects.
Python refuses to have packages (folders don't count) or accept build steps. Everything follows from there, making it much harder to have a good solution.
You will notice that every package management solution from all your favourite languages will have its own set of tradeoffs.