To me this isn't a question of one project, it's a question of OSS project governance in general. Is there a succession plan? How do you know when more maintainers are needed? How do you promote someone from contributor to committer to releaser?
We went around adding CoCs to everything a few years ago, perhaps a similar effort could be made with governance plans? Like a boilerplate plan projects could use, hopefully leading to a new community norm of "if you are using a project with no governance plan, that's up to you, but don't say you weren't warned!"
It's only when that implicit contract breaks down that you get "is X dead?"
I work on a small project. I didn't add a CoC when the big push came about because I concluded that it was if I say it is bad it is bad, otherwise it is good - that is no CoC and so there is no point. (I did consider asking some large project - KDE for example - to be arbitrator, but I didn't bother)
To take a shot at something "more concrete":
* New releases will be cut at least on a quarterly schedule
* When the number of open issues passes 25, a new maintainer shall be added (if not sooner)
* Maintainers may leave a project at any time and are encouraged to do so if they need to move on to other things
* Maintainers may return when they have time
* Once per year, maintainers shall either reaffirm their desire to continue maintaining the project or step back from maintaining
* After 2 merged PRs, a contributor is promoted to a maintainer
This is certainly not a good governance plan for everyone, some may find it too prescriptive, but it seems like it could be a good start for projects like axios & pipenv. The idea is to prevent "is this project dead" type issues before they arise. When a project is losing maintainers & no one wants to step, it's clear that the project is waning.
Of course it's also perfectly reasonable to say "that's too much work" or "f*ck off, it's my project." This convention would help people make more informed decisions when adopting a project for use. Some will be fine with "whatever", some will want a clearer plan, both are OK.
Jazzband  seems meant to address exactly this: “a collaborative community to share the responsibility of maintaining Python-based projects.” And it looked promising, but it’s not entirely clear that the Jazzband project itself is all that active (only news status update is the launch announcement from 2015; last public Twitter activity was in 2017).
It's so easy to install dependencies, but it comes with a large hidden cost and painful lessons down the line.
Does this thing also bring in 1000x sub dependencies? How much am I REALLY going to use this. What's the actual added value here.
I've seen people bring in libraries for simply the dumbest things. The worst offender I've seen is lombok brought in for a @Logger annotation on one class. That stopped our java 6->8 migration because the version of lombok they brought in was old and for highly used library (so upgrading wasn't simple).
If you can write the used functionality in 10 minutes, you should not bring it in as a dependency.
I disagree with this; it depends on the size of the dependency, plus the risk of a problem with the dependency vs the risk of making a mistake vs the risk of code drift when the same 10-min thing gets written a dozen different ways across an org.
The dependency may change in surprising ways in the future. The dependency may not change in expected ways in the future. (ie, now you're trapped in an old version of the language, or another one of your dependencies)
The problematic part with reinventing the wheel or NIH is that you add a fixed cost of doing business to any given change. The problem with importing the world is that you add a dynamic cost to simply existing without change.
It's a tradeoff, and it's not a tradeoff that's consistent across organizations. For some orgs it's really important to be first to market. In other orgs it's important to not break shit for your existing customers.
My org is well established in our field. Nothing matters more than retaining our existing customers. Importing a new lib requires approval from legal, which requires about a personweek. In other words, if one developer could implement the same thing in less than 40 hours, the developer should implement it themself. For us, a slow moving org, I think we're at the right place. For a startup, obviously not.
I'm talking about things developers think they can do in ten minutes but that they are at high risk of getting wrong: generating SQL, parsing JSON, handling dates, etc. These are all things that have well-established libraries in most sane languages (maybe NodeJS is not sane in this regard), where everything that you save yourself from by using the library is worth even a 40 hour review process.
10 minute problems are things like
is-odd - https://www.npmjs.com/package/is-odd
is-even - https://www.npmjs.com/package/is-even
leftpad - https://www.npmjs.com/package/pad
Or bringing in large libraries to use single methods or functions from them. Such as
Bringing in gauva for max
Bringing in apache commons for StringUtils.join.
I've seen both examples in my company's code base.
A library that does JSON parsing, date handling, etc is doing a lot more heavy lifting than "is-even".
My personal opinion that some day ripples on the water made whole loop.
And last but not least: pip+requrements.txt are not the best, but pipenv doesn't add much over it. It gives you a little but introduce another level of abstraction over the same things with its own level of complexity.
I did some research and think this link is enough https://github.com/pypa/pipenv/issues/1137.
The users in that topic then continue the discussion in all of its aspects, expecting the maintainer to engage with them. I can imagine it feels to the maintainers as an energy-sucking discussion.
IMHO, you're not shaming the maintainers, you're shaming the community here.
I'm a bit confused here. Maybe the project is mismanaged, or there's some upstream issue with package distro, but it seems to be far from dead.
There are so many small and medium sized projects on Github, where you have no idea if they are maintained or not. Sometimes there are projects which are alive and kicking with multiple pull requests, maintainers promising changes or a major release and suddenly nothing. Sometimes smaller projects see no changes for months, but they simply work and don't need maintenance at all. Maybe the community moved elsewhere and and the project will only see bugfixes coming in and no more features?
The burden of figuring this all out lies with the visitor and is a annoying hassle.
I would love if Github could somehow show me on the landing page, if the project is worth investing time in. Maybe they could send a heartbeat to the maintainer and simply show the project as dead as soon as there is no response?
There are Java packages that haven't seen a commit in years and are still perfect for a task. There are npm packages that are 2 months old and terribly outdated and unmaintained.
My personal favorite metric is "time to maintainer response": how long does it take for a maintainer to respond to issues or pull requests. Not necessarily to resolve them, but triage issues or provide guidance on a PR.
If this happens quickly and with reasonable responses, projects are usually solid, assuming they have existed for a while and see decent usage.
My favorite is Scriptella (http://scriptella.org/download.html) which recently just got its first update in around 7 years.
For what it does, it just worked, and for many uses cases, updates were never an issue.
I cannot promise active feature development, but at least keep it compatible with recent JDK versions. Let me know if you have any feature requests on https://groups.google.com/forum/#!forum/scriptella or https://github.com/scriptella/scriptella-etl
The official JSON implementation is a good example. It is a mature project. It's used everywhere. If you leave an issue or submit a PR that fixes a typo, the maintainer will flat out delete your comment and tell you to buy his book to educate yourself.
Maybe this behavior is good or maybe it's bad. But maintainers are human and metrics are not going to cleanly pick up how well a project is maintained.
Who cares if something has not commit for 7 years if almost noone uses it and it has no real issues?
On the other hand, you would hope a popular library with heaps of bugs and issues is receiving lots of maintenance.
If I could not automate response to that, I'd ignore it.
My employer gets to make bullshit intrusions on my time like that, no way will I waste my life on make-work from Github.
You are not willing to give Github the information, if a project ,which you uploaded, is still active? But, will you give that information to users asking for details? Which way is more annoying to you and the users?
What I want to say is, that Github will turn into a huge graveyard in the years to come, with projects long forgotten still having a landing page like it's the best thing ever. Do you want every visitor to first contact the maintainer if the thing is still alive? Isn't that a huge waste of time?
In my opinion, Github will have to clean up the mess left behind by unmaintained projects at some point. The sooner they start, the easier it will be.
Anything that sits broken more than (say) a month can safely be considered dead. Super-stable low-level code will continue passing their tests forever, and things dependent on specific API versioning in left_pad will be marked dead quickly...
I usually find it to be a much smaller hassle than recreating the functionality myself.
There's a bug in that release though: it thinks 0^1 is NaN. That bug was fixed in or before 2018 (https://bitbucket.org/heldercorreia/speedcrunch/issues/836/0...), but there has not been a release since. There have been a number of commits this year, but not a lot.
Is the project dead or not? I honestly couldn't tell.
What would be the difference between 0 commits without a release and 669 commits without a release?
The 669 commit project is clearly not finished, but also not really dead, yet.
I've worked on atleast a few open source projects where I'd pull and build master, trying to obtain a particular bugfix, and it would fail, and I'd have to randomly rollback commits until I found a working build.
In one case the project had a seperate repo for a library that wasn't kept entirely in sync with the parent project.. had to randomly move back both repos until I found a matching pair that built and ran successfully, but after the bugfix was implemented.
Releases are important for anyone who isn't working on the repository directly -- they gaurantee the project is in a sane state.
Right now, it seems like the issues raised in  are still a problem. The release pipeline has problems, and dragging through that isn't a priority.
There's no activity in December. There was one commit in November, and that removed a Fedora version number from the readme.
Now, I could be super huffy that they haven't released in years, but this is open source software that I'm not paying for. Moreover, I have the power to choose multiple alternatives. I can run the -current release. I can make my own "pseudo-release" by just forking here and calling this "14.2.1". Or I can use a different distro.
It would be really entitled of me to demand Patrick make a new release just because I want one, especially when he doesn't owe me anything, and I have multiple alternatives. And besides, the project is still going, it's just not doing what I want it to do; and that's Pat's prerogative.
There are things I (dimly) remember genuinely liking about Slackware in terms of its philosophy, but as near as I can tell the official installation method is "install the most recent ISO, then update from there," and when the most recent ISO is three and a half years old, that's not a great look. We've now gone the longest length in Slackware's history without even a point release.
I wouldn't demand Patrick make a new release, but I don't think it's wildly unreasonable to expect one by now. Bringing it back to the original article, I think the same can be said of pipenv.
It recently finally merged its Qt5 support into master (which was in a working state since 2014) about two months ago. That might help the release process.
What doesn't help is that a consumer facing product of that scale is a nightmare to manage - it has over 2k open issues on Github but probably 95% of them are "I don't know how to do X" tech support kinds of things.
- VSCode supports poetry as a second class citizen
- Documentation is there, but only in github issues
- There is no migration path from pip and pipenv to poetry
- I can't do releases via CI because of a poetry bug.
We're close to having everything migrated to Poetry. When that day comes, we can throw out all the compatibility stuff, update the build server, and be happy. Until that day, DepHell gives us an easy compatibility layer so that we don't have to do the migration all at once. It's awesome.
Thank you so much for pointing it out. I've added it to this issue.
In any case, I strongly recommend publishing alpha/beta releases along the way, without the ceremony of an official release, so that folks can give feedback during the long months that fly by when you don't have nearly enough time to spend on the project.
Respect the needs of your conservative users by not publishing official releases until they're really ready, but trust your more engaged users to use whatever you have right now, and let you know what's working/broken, without judgment.
Python packaging in general is such a messy ecosystem
I’m sure poetry has bugs and it could have other problems (not sure), but if you look into how it works, from a software design / philosophy perspective, it’s absolutely amazing. It might be worth waiting a bit longer before you use it in a large production project. Once Heroku adds native support to the Python buildpack, and some other mainstream avenues add support, Poetry usage will increase enough to create the necessary scrutiny that will bring it up to true production-grade quality.
When you add a package in Poetry it will add the latest version of that package’s dependencies that also satisfy all the other packages (and their dependencies’ dependencies).
If you remove a package, Poetry will remove any dependencies of that package (actually uninstall them, rather than just leaving junk behind like pipenv), and will reestablish the latest versions of any other dependencies who’s versions were held back by the package you just removed or any of its dependencies...
Also, when you add a package with Poetry it doesn’t need to completely lock and install everything from scratch; only what you have just added and whatever its dependencies dictate; nothing more.
Also, you can tell Poetry to list off all the versions of your packages and the latest versions available (in case you’re like me and you pin all of your top level packages, having to Google “pypi django”, “pypi requests”, etc., to see if there are any newer versionS worth upgrading to, is a pain in the ass.
# NB: Poetry won't run arbitrary callables, so these are wrapped in a python script.
# Poetry needs the root dir to be a package (e.g. module `my_repo` must exist) in order to run scripts
# So they are just shoved into a dummy "my_repo.py" instead of going in e.g. "scripts.py"
# Adds Git hooks from `git_hooks` path (overwrites existing!)
hookup = "my_repo:hookup"
# Run formatter+linter, does not modify anything, errors if fail
lint = "my_repo:lint"
# runs `black` (formatter), modifies stuff
format = "my_repo:format"
# A list of files which failed linting. Most helpful with flake8 vim integration https://github.com/nvie/vim-flake8
lintyfiles = "my_repo:lintyfiles"
# unit tests
utest = "my_repo:utest"
# unit + integration tests in container
citest = "my_repo:citest"
cutest = "my_repo:cutest"
Not just messy. It’s probably worst in class.
I at least can’t come up with a single worse package-management story which I do know of.
Edit: I’m talking about platforms with actual package-management which sucks, not platforms with the absence of package-management all together.
Yes it's not great right now and it's not completely clear if I should recommend a new user pip+venv, pipenv, pyenv, poetry or conda and that's a big problem, Python was actually quite early to handle dependencies and packaging in a standardized and structured way. I remember that most of the projects I worked on early on in my career more or less completely relied on vendoring dependencies, if packaging systems existed they were of either very complex or the one that came with your os/distro.
However Python currently needs to catch up, we need some alignment and a clear path forward, it would also be great to have an official way of building artefacts. There's plenty of ways today but no great blessed way to recommend a newcomer.
As someone who has had to deal with Python's packaging a lot, I would say that it's really, really bad. Maybe not the worst in the world, but it is much closer to the worst than to the best.
There are 15 ways to package Python modules, none of which are feature complete (but a lot of them pretend to be). Every year or so someone decides that they need to write another packaging tool for Python, and then they give up before it is feature complete.
A tool like virtualenv is useful, but it overpromises and underdelivers (this is the overarching theme of Python packaging imo): it does not completely separate your environment from the system. E.g. virtualenv still uses the host system python packages cache, and it does something rather nasty with .so's that get copied from the host system...
And don't get me started on dependency management in Python. When the dependencies of your dependencies make breaking interface changes, you're in for a world of pain.
I find it amusing, in a sad way, that the language that prides itself on "there is one way to do it" screwed up so badly in the packaging department by not having one good way to do it. Meanwhile the language whose motto is "there is more than one way to do it" has a standard, sane way to package things, with multiple tools that work together in a coherent way. Perl got this one right.
I was always impressed by CPAN compared to most other package managers I've used. And, yeah, between Python, Perl, Node, Ruby, and even PHP, Python is by far the most inscrutable and the most prone to giving me fits when existing Python apps like Lektor just... inexplicably break after one of the Homebrew versions of Python gets updated.
osx includes a "non-framework" build of python making it hard to use matplotlib
Actually just now finding out about "venv" in the standard library introduced in python 3.3
I've built quite big projects with Maven (>2 million LOC) without issues.
So that’s apples to no oranges, I guess. And it still doesn’t leave python looking particularly good.
Much unlike how things work with python.
- It was built in to Rust from the beginning and officially sanctioned so fifteen different people don't have to build their own incomplete, buggy package managers.
- You can add plugins for things like automatically updating or adding dependencies.
- It handles projects and subprojects.
- All you have to do to install dependencies and run a project is "cargo run", reducing friction for getting into new projects.
- For the most part, it just works.
That said it isn't perfect, especially when needing custom build scripts, but it's good.
For example, I mentioned "kitware superbuild" above. It describes itself  as:
> It is basically a poor man’s package manager that you make consistently work across your target build platforms
But that’s what you would have to do youself anyway. You can’t use all fresh upstream version of everything, since they don’t all work together. So some versions you’ll have to hold off on, some other versions might require minor patching. But this is exactly what distro maintainers do.
So, what do you do when different projects need different versions of libfoo? Right, you download and build it locally inside the project tree and fiddle with makefiles to link to the local version rather than the one installed by apt-get. So basically you do your own dependency management. Good luck with that.
Well typically other people's projects I would normally rely on the distro to compile other people's projects as well; and then the distro maintainers would sort that out. That may include making a patch to allow older projects to use newer versions of a library; or if the library maintainer made breaking changes, it might mean maintaining multiple versions of the library.
Obviously if I myself need a newer version than the distro has, I may need to work around it somehow: I might have to build my own newer package, or poke the distro into updating their version of the library.
I mean, honestly, I don't build a huge number of external projects (for the reason listed above), and so I've never really run into the issue you describe. It seems to me that the "language-specific package" thing is either a side-effect of wanting basically the same dev environment in Windows and MacOS as on Linux, or of people just not being familiar with distributions and seeing their value.
That’s an untenable situation. The package which depends on the older version of libfoo is either dead, in which case you should stop using it, or it will soon be updated to use the newer version of libfoo, in which case you’ll have to wait for a newer release. This is what release management is.
So you are suggesting that every time libfoo bumps its version I have to update the dependencies on all of my projects to use the latest, find and fix all the incompatibilities, test, release and deploy a new version? Seriously?
You should read the release notes of the new version of your dependency, fix any obvious issues from that, see if your tests pass, and wait for bug reports to roll in for non-obvious things not caught by automated tests. Ideally you should do this before the new release of the dependency hits the repos of the distros most of your users use, so that it's only the enthusiasts that are hit by unexpected bugs.
Even if you could delay and batch the work together every several releases of a dependency, you're still doing the same amount of work, and it's usually simpler to keep up bit by bit than all at once.
One trick is to not use dependencies that have constant churn and frequent backward-incompatible changes, and to avoid using newly introduced features until it's clear they've stabilised. When you choose dependencies, you're choosing how much work you're signing up for, so choose wisely.
Of course you could go the alternate route and ship all dependencies bundled - but that is a way to ignore technical debt and accidentally end up with a dead project.
Also, your project should not demand a specific version of `libfoo`. If `libfoo` follows semver and a minor release breaks your project, that is a bug in `libfoo`. Deployments of production software should pin versions, but not your project itself.
What I might have is a working piece of software that is an important part of company infrastructure. For mission critical software reliability is a much more important metric than being current. Unless there is some really compelling reason to update something it should not and will not get updated. There are mission critical services out there running on software that hasn't been changed in decades and that is a good thing.
> Also, your project should not demand a specific version of `libfoo`. If `libfoo` follows semver and a minor release breaks your project, that is a bug in `libfoo`.
Who the fuck cares if it is their bug or not? I need my service working, not play blame games. And if I have a well tested version deployed, why the hell would I want to fuck with that? And if I have tested my service when linked against libfoo 22.214.171.124.whatever2 I had better make sure that this is the version I have everywhere and that any new deployments I do come with this exact version.
But if I start a new project, I might want to use libfoo 3.14.15.whocares4 because it offers features X, Y and Z that I want to use.
If you instead just like writing software and throwing it over the wall/to the winds, you are an academian in an ivory tower, and have no connection to your users in the real world.
Quite the opposite, actually. My responsibility is to my users. And that responsibility is to keep the software as stable as possible. So the only time I will consider upgrading my dependencies is when reliability requires it. If libfoo fixes some critical bug that affects my project, yes, maybe I should upgrade (although I am running the risk of introducing other regressions). If libfoo authors officially pronounce end of life for the version of libfoo I am using, maybe I should consider upgrading, even though it is safer to fork libfoo and maintain the well tested version myself. But it is irresponsible to introduce risk simply to keep up with the version drift. So if my project is used for anything important, I should strive to never upgrade anything unless I absolutely must.
It seems that the choice is whether to live on the slightly-bleeding edge (as determined by “stable” releases, etc), or to live on the edge of end-of-life, always scrambling to rewrite things when the latest dependency library is being officially obsoleted. I advocate doing the former, while you seem to prefer the latter.
The problems with the former approach are obvious (and widely seen), but there are two problems with the latter approach, too: Firstly, you are always using very old software which are not using the latest techniques, or even reasonable techniques. This can even be considered to be bugs – like using MD5 hash for example, which, while being better than what preceded it, much software were using MD5 as a be-all-and-end-all hashing algorithm; this turned out later to be a mistake. The other problem is more subtle (and was more common in older times): It’s too easy to be seduced into freezing your own dependencies, even though they are officially unsupported and end-of-lifed. The rationalizations are numerous: “It’s stable, well-tested software”, “We can backport fixes ourselves, since there won’t be many bugs.” But of course, in doing this, you condemn your own software to a slow death.
One might think that doing the latter approach is the hard-nosed, pragmatic and responsible approach, but I think this is confusing something painful with something useful. I think that doing the former approach is more work and more pain from integration, and the latter approach is almost no work, since saying “no” to upgrades is easy. It feels like it’s good since working with an old system is painful, but I think one is fooling oneself into doing the easy thing while thinking it is the hard thing.
The other reason one might prefer the former approach to the latter is that by doing the former approach, software development in general will speed up by all the fast feedback cycles. It’s not a direct benefit; it’s more of an environmental thing which benefits the ecosystem. Doing the latter approach instead slows down all feedback cycles in all the affected software packages.
Of course, having good test coverage will also help enormously with doing the former approach.
There is software out there that absolutely cannot break. Like "if this breaks, people will die". Medical software, power plant software, air traffic control software, these are obvious examples, but even trading and finance software falls into this category, if some hedge fund somewhere goes bankrupt because of a software bug real people suffer.
It doesn't matter how boring, inefficient and outdated these systems are. It doesn't matter how much pain they are to maintain and integrate. These are systems you do not fuck with. A lot of times people who do maintenance of these don't even fix known bugs in order to avoid introducing new ones and to avoid the rigorous compliance processes that has to be followed for every release. Updating the software just to bump up some related library to the latest version is simply not a thing in this context.
I am not working on anything like this. I work on a lighting automation system. If I fuck up my release, nobody is going to die (well, most likely, there are some scenarios), but if I fuck up sufficiently, a lot of people will be incredibly annoyed. So I have every version of every dependency frozen. All the way down the dependency tree. I do check for updates quite often and I make some effort to keep some things current, but some upgrades are simply too invasive to allow.
Most people’s systems are not that special that they absolutely need to do this, but it feeds one’s ego to imagine that it is. And, as I said, it feels more painful, but it’s actually easier to do this – i.e. being a hardass about new versions – than to do the legitimately hard job of integrating software and having good test coverage. It feeds the ego and feels useful and hard, but it’s actually easy; it’s no wonder it’s so very, very easy to fall into this trap. And once you’ve fallen in by lagging behind in this way, it’s even harder to climb out of it, since that would mean upgrading everything even faster to catch up. If you’re mostly up to date, you can afford to allow a single dependency to lag behind for a while to avoid some specific problem. But if you’re using all old unsupported stuff and there’s a critical security bug with no patch for your version, since the design was inherently buggy, you’re utterly hosed. You have no safety margin.
Suddenly a small security-patch forces you to essentially replace your whole stack. If your test flags any error you have no idea which of the updated subpackages that caused it, because you have replaced all of them. Eventually you accumulate so much tech debt that it's tempting to cherry-pick the security patches into your packages instead of updating them to mainline, sucking you even deeper down the tech debt trap.
Integrating often means each integration is smaller, less risky and easier to pinpoint why the failure happens. Of course this assumes you have good automatic test coverage, which i assume you do if the systems are as life-critical as parent claim them to be.
There's also a big difference between embedded and connected systems here. Embedded SW usually get flashed once and then just do whatever they are supposed to do. Such SW really is "done", there is no need to maintain it or it's dependencies because it's not connected to the internet so zero-days or other vulnerabilities are not really a thing.
Potentially relevant: https://vorpus.org/blog/why-im-not-collaborating-with-kennet...
Big, high exposure projects should probably wait to recommend interesting new projects for actual use until they are sure that the management of that project is in reliable hands with enough backup to continue it if the original creator vanishes.
Then there's also the issue of such members of the community exploiting the human tendency to worship others.
What concrete evidence leads you to the conclusion that coordinated shills are responsible for giving the project attention?
NO! This is a common misconception.
Edit: Correction it's not endorsed by the core Python team but it's recommended by the Python Packaging Authority in various places. See replies below for more info.
> The thing that made it “official” was a short tutorial  on packaging.python.org, which is the PyPA’s packaging user guide. Also of note is the Python.org domain used. It makes it sound as if Pipenv was endorsed by the Python core team. PyPA (Python Packaging Authority) is a separate organization — they are responsible for the packaging parts (including pypi.org, setuptools, pip, wheel, virtualenv, etc.) of Python
> Use Pipenv to manage library dependencies when developing Python applications. See Managing Application Dependencies for more details on using pipenv.
> Consider other tools such as pip when pipenv does not meet your use case.
That recommendation is absolutely unambiguous.
> It's an official project of PyPA still, it's just not being pushed as the "be-all-end-all" option.
Those tools are relegated to a small section at the end, whereas the main body of the text says "Pipenv is recommended for collaborative projects..."
I'd say that counts as a specific endorsement of Pipenv. I guess it's an endorsement by the "Python Packaging Authority" rather than "Python Core" but it's pretty hard for someone at the tutorial-reading level to perceive the difference.
- Use pyenv to manage python versions (mostly works pretty well)
- In the beginning, use builtin python 3 tooling to create virtual env for the project: "python3 -m venv venv"
- Whenever needed, add new libs to "requirements.in"
- Run the "pip-compile" command to generate a new "requirements.txt" with the dependency and its sub-dependencies pinned by default to the exact version
- Run "pip install -r requirements.txt" to install the new package and its sub dependencies
- Check both requirements.in and requirements.txt into version control
The big advantage is that requirements.in specifies just the packages you care about, while requirements.txt has your packages and all of the sub dependencies pinned to the exact version.
Pip-tools has been a dream. It is just a thin layer of tools on top of Pip that separates your 'abstract' requirements (eg: django<3) from your 'release' requirements (eg: django==2.2.5) and managing running pip to have your virtualenv reflect your exact requirements.
No new standard like pyproject that is partially supported (by 3rd party that is), no intention to (but failing at) being the all dominating way to do python packaging.
Just a tool that is there to help you.
edit: it also automates updating the release requirements file (within the constraints of the abstract requirements).
I personally use setup.py (my setup.py only calls setup() and all configuration is declaratively defined in setup.cfg) the pip-compile generates a version lock (requirements.txt) and that is passed between environments, so we are ensuring that the exact same dependencies are installed during deployment.
For me, if they would merge Pip-tools into Pip and call it a day, my package management issues for Python are solved.
Posting this here again in case anyone missed it. https://vorpus.org/blog/why-im-not-collaborating-with-kennet...
Is this  not an official endorsement? It certainly seems as much.
The PSF may not have officially granted them some sort of status, but as PyPA maintain pip, setuptools and warehouse, they are in fact the authority when it comes to packaging, unless the PSF comes out with a statement saying they aren't.
I prefer poetry though, since the consensus seems to be coming together on pyproject.toml rather than individual files like Pipfile. A lot of tools have already started supporting the toml file for their config, or have PRs pending.
I can't think of many Python packages that have the same issues, and Python code isn't sent to and running on a user's browser.
Am I wrong or is there a reason that a locking mechanism (other than git) is helpful in Python?
I don't use Python/pip much but as for npm: the problem is when your dependencies, direct or indirect (dependencies of dependencies), aren't "exact". You have something like "~1.2.3" or "^1.2.3". If every developer followed Semvar perfectly, never shipped regressions or new bugs when fixing a bug, and was always able to identify every breaking change then life would be perfect.
That is, however, not the world we live in. So a "lock" file respects your "fuzzy" versions ^/~ when you first run the npm install and then subsequent runs will install using the exact versions you downloaded the first time. This helps solve the "works me me"/"work on my machine" problems. The idea being if you can run it locally then the build server and production can also build/run your code.
Why on earth would anyone need all this to manage packages for their projects?
Are there any other programming languages whose package-management comes close to this level of intricacy and complexity?
“This project needs package X”
I mean how hard can that be to get right in a self-contained environment?
This whole story is just madness coupled deep denial and Stockholm-syndrome.
I think many do. Ruby, Node, Erlang/Elixir, Java, Go, Rust, dotnet, C... I’m having trouble thinking of a modern language that doesn’t have such package management mechanisms.
For many people doing anything more than writing one-off scripts, and especially for anyone who collaborates with others or shares their code, package management is so much more than just “this project needs package X”.
My criticism isn’t about having a package-management story.
It’s about having a terrible and complex one.
If you think any of the listed examples have “simple” package management tools, I question how deeply
you have used any of them. NPM has ~60 commands and hundreds of subcommands each with multiple option flags, and probably hundreds more config options. Gem/Bundle is similar, etc.
If anything, Python is trying to catch up in how complex it’s package managers can be.
For most other language-provided package-managers the software project you’re working on is the env, so you don’t need to construct or manage a venv at all.
So my point still stands.
> Why on earth would anyone need all this to manage packages for their projects?
> Are there any other programming languages whose package-management comes close to this level of intricacy and complexity?
> “This project needs package X”
> I mean how hard can that be to get right in a self-contained environment?
Packages often have subdependencies and their requirements at times may conflict. If you are very specific in the versions you want, it is more likely to cause issues in dependency resolution.
I would hardly call a dev's machine a "self-contained environment". Most developers I know work in a number of repos with varying requirements, and polluting their system libraries and packages with each project's requirements quickly pollutes the system and can lead to issues.
But a software project is.
Only that software project needs those packages.
>I mean how hard can that be to get right in a self-contained environment?
Okay, I’ll bite.
What version of package X does it need?
Is there a specific version that’s been tested with this project and is known working?
Are there specific versions of its dependencies that have been tested and are known working?
Is it needed at runtime or only at build-time?
What repository can it be found in? PyPI is not the only Python repository; private repos are common.
And as a bonus cherry on top: how easy is it to make sure you have all the project’s dependencies installed in the venv for that project, and that you don’t have packages you’re not keeping track of? This is a UX thing, but developers are human and it matters.
"It automatically creates and manages a virtualenv for your projects, as well as adds/removes packages from your Pipfile as you install/uninstall packages."
Well, thanks but automate "python -m venv myvenv" ? Add/remove packages from a "Pipfile" ? Do I have to specify dependencies somwhere else than requirements.txt ? Why ?
There must be some use cases I'm not aware of.
Requirements.txt only lists versions of your project's requirements, but Pip actually automatically installs dependencies of those requirements too. And those versions aren't listed in your requirements.txt.
https://realpython.com/pipenv-guide/#dependency-management-w... -- This page about Pipenv vs pip + virtualenv goes into more detail.
Boom, all recursive dependencies frozen to their current state.
Yes, that 'freezes' everything installed... but that's the point. If you want to update a direct dependency, you just do pip install -u <my-dep>. Any indirect dependencies are updated, and you freeze again.
Sometimes direct-dependency-a and direct-dependency-b have a conflict on which version of indirect-dependency-y you need. This is what we call doing 'actual work.'
In my experience, the issues come around when you try to build envs cross-platform. There are a lot of dependencies that have missing versions or bugs for certain platforms. This is not a pip/venv problem though--it is more of a python problem.
Is it easy to automatically do this when you want a new package version, without having to remember to do it? Is it easier to put together this process and train other developers in it and be diligent in its use? Is all of that easier than installing an application that has a similar interface to other tools, that does all that for you, and has a community of people to help with issues?
Also if you manage to drop a dependency, you don't have an easy way to remove the things from requirements.txt that are only there because they're a subdependency.
Putting it all in one requirements.txt is just too simplistic.
- One with direct dependencies (versions pinned)
- One with direct dependencies + subdependencies (pip freeze output)
Am I being too naive ? (obviously yes if such tool as pipenv exists, but I'm trying to figure why people need *.lock files).
It also adds dependency management. If one subdependency is library_a > 1.0, and another is library_b < 2.0 while also e.g. 2.1 exists, then it will try to find a version between 1.0 and 2.0. Pip doesn't do that.
So in my mind, that's what pipenv is -- pip, virtualenv, those two files, plus dependency management.
That's a lock file...
I still think this is all too convoluted though. I hope an accepted de facto standard for this will emerge at some point.
2. Because it's not at all clear that pipenv is a third party library. It's made by the same group that makes pip, so it's confusing that pip would be considered a de-facto standard but not pipenv when it's made by the same group, and under the same project in Github.
> However, to avoid recommending a tool that CPython does not provide, it is further proposed that the pip  package manager be made available by default when installing CPython 3.4 or later and when creating virtual environments using the standard library's venv module via the pyvenv command line utility.
sorry but what does freezing the dependency tree mean?
You may want this because you have a library that you shouldn't be pinning to the third decimal on a sem-ver package, but that you don't want to hiccup in CI due to a dot-release.
Or maybe you think a loose file your tooling can read, and a hyper-specific file your builder should read, is a better interface for a project.
With the added bonus that it also contains a hash of the package so if someone pushes a new version with the same version number it would complain that the hashes don't match.
My understanding is that that's kind of the point of groups like the "Python Packaging Authority". So if they're not going to merge pull requests and do maintenance on the project, that IS a problem, since they're supposed to be the official version right now.
Return the forks of developers who have publicly stated they'd like to take over maintenance of this project.
I'd bet that narrows it to less than ten.
Now-- have a look at the blog posts where these maintainers explain their plan to sustain the project going forward and choose the most persuasive one.
I'd bet it's less than one.
Short circuited, problem solved. :)
pipenv -> pippi
They could call it pippi
Bugs get reported and closed because they are fixed in master every day, wasting not only the end users time but also that of the people actively working on the project.
This means a bad experience for people who try to use it, and makes whole Python ecosystem feel a tiny bit worse too. Imagine a frustration of someone reading the blog post , spending all the time learning about system, using it, and then discovering bugs won’t ever get fixed!
It is very easy to fix by an author: just a small, 4 line commit to readme and website saying “the project is dead , go elsewhere”. This will allow people to move on - maybe to a fork, or to some other project which does similar thing.
I don't like this framing (sort of implies they aren't be honest) but regardless just switch if you're not happy with the release frequency and you have viable alternatives.
A lot of people say “just fork it” or “choose something else”, but the problem is that python is a finite community with a finite amount of energy, and a lot of this energy has been absorbed by the star power Kenneth acquired from his prior successes (namely, requests... which btw now has an even better replacement called httpx).
It’s almost like Kent needs to come out and say “I’m sorry I screwed up, here’s my towel; good night.”, so that people can move on.
About the framing of the question, I think it's because of all the flame wars in the past when people criticized pipenv and maintainers took it a bit personal.
As a result, I end up rarely going outside of the standard library. In a perverse way, being locked on an un-upgradable (due to Reasons) version of Python 2.7.5 for the foreseeable future has helped put that temptation a bit further away.
Yes, Packaging Is Hard. It is certainly beyond me. I will probably never need to package anything I wrote, much less distribute it, so many of my concerns are purely academic.
Rather than fussing over things like the walrus operator (really, c'mon), I would love to see those who steer Python buckle down on issues like this, solve them, and then relentlessly backport the solution further back than everyone thinks is reasonable.
1. Add a project scope to the README -- this gives you and volunteers grounds to close issues that are not relevant.
2. Send a personal email to your 3 top contributors -- ask if you can give them push access (even if they aren't recent contributors) at a minimum as an insurance policy.
3. Automate or at least specify your release process. You should be able to do this reliably, while drunk and high, and when you have 17 other projects needing your attention. Example: https://github.com/fulldecent/FDWaveformView/blob/master/CON...
I have been involved in a few "takeovers" to implement the above and keep great projects running. A little structure and human goes a long way. You don't need to fork and be the new dictator to keep something great moving.
pipenv brings simpler workflow to pip. pip leverages packages which are published by their authors onto pypi, which is managed by the Python Foundation.
thats not true today, conda-forge is a community led effort that have open source recipes of the packages built by anaconda inc and many more others contributed by the community. I run conda with conda-forge packages only and it works great.
Later, conda may install a different version of the same dependency as a dependency of something else. Depending on how exactly they are installed (egg, zipped egg, whether the folder has the version number in it), you either get two versions of the same package installed, with which one gets imported being arbitrary, or you get two sets of metadata, with one of them not matching what is actually installed, such that pip may think version requirements are satisfied when they are not. It's messy as anything, and the breakage can be subtle. I distribute packages to users who use conda, and my packages have dependencies that are available in conda, so this has been messing with a lot of my users' installs. I'm now just making conda packages for these projects to solve the issue.
I made this package  to try and automate the process of making conda packages out of my existing setuptools packages, I'm quite happy with it but since it is designed to serve the needs of my projects, I can't guarantee it will suit everybody's needs.