However, Kenneth abused his position with PyPA (and quickly bumped a what is a beta product to version 18) to imply Pipenv was more stable, more supported and more official than it really was.
And worse still, for anyone saying "but ts open source, you get what you pay for", Kenneth as former Python Overlord at Heroku, encouraged Heroku to place Pipenv above Pip as the default Python package manager in the Python buildpack. This decision impacted paying customers and the Python buildpack used a broken version of Pipenv for a long time. So long, most people I know just went back to Pip.
Then, lastly, when people complained he had a tizzy at reddit and twitter and got PyPA to help backtrack and say "no we didn't support it, nope, its just a thing that happened", all while the main Pipenv Github repository was held under the PyPA GitHub Org.
There's been a lot of work on Pipenv over the the last 6 months, predominantly by Dan Ryan and Tzu-Ping Chung, and it's getting stronger and stronger with each release.
If you've gone back to using pip I'd encourage you to give Pipenv another try. Introducing a lockfile is a big step forward for Python dependency management, and the team working on Pipenv are committed and doing a great job.
I don't deny that, what I am (and the article is) saying is that we were sold on Pipenv being "the officially recommended Python packaging tool from Python.org".
And PyPA didn't refute it, and Heroku didn't refute it, so the community bought it.
Yes, introducing a Lockfile is huge, and it was massively needed, and thats why when we were told "heres the official way to do it", we got excited. Then we got daily breaking updates, rude issue close messages, and a giant backtrack of "its free and still under development, why do you expect so much from us?"
I don't think that the parent disagreed with that. The point, as I understood it, was that this beta stage improvement was marketed as being ready. IOW, if pipenv was not Kenneth's project, it likely would have evolved in, to use your phrase, a straighter line.
That's kind of the problem. Why on earth are libraries and apps getting a different treatment? The JS ecosystem manages to have one tool for apps and libraries. One too for installing and publishing. All of it with lockfile support, workspace/"virtualenv" support, etc. And somehow, it's not confusing.
Adding one more tool to the stack is a really funky step forward. Yes, it brings lockfiles. Cool, although we already kind of had those (--freeze). Packaging in Python is a mess and I'm more and more in the camp that as long as Pipenv keeps flouting itself as "the better solution", all the while not covering all the basic use cases, we've gone backwards and are in even more of a mess.
This is a solved problem across a variety of popular and mainstream programming languages. I don't mean to suggest that the problem isn't complicated, but this isn't a problem that doesn't have a wealth of previously written solutions to look at for inspiration.
My Java app declares its dependencies in a build.sbt file using Scala syntax and has them cached in an Ivy directory. Yours declares them in a pom file using XML syntax and has them cached in a Maven directory. Neither of us even tries to do lockfiles and instead just has the CI server build an "uber jar" with all the dependencies bundled in.
I get why Python needs lockfiles, but goddamn, that need is a symptom of the mess of managing Python dependencies.
There's still a long way to go - I use Airflow for ETL management, and I'm using pipenv to manage it - except Pipenv can't create a lockfile because a dependency of a dependency of Airflow requires in flask-wtf <= 0.7, while another dependency of a dependency requires ==0.8
In a Maven pom.xml I can easily exclude or override the conflicting dependency manually if needed, but I can't in a Pipfile.
Well, that seems naive. Almost 1/4th of Maven Central libraries broke binary compatibility in patch updates: https://avandeursen.com/2014/10/09/semantic-versioning-in-ma...
Netflix, at least, doesn't agree that Java doesn't need lockfiles: https://github.com/nebula-plugins/gradle-dependency-lock-plu...
Yes, which is part of the reason why lots of Java projects almost never update their dependencies, because no one remembers why that version was chosen. Splitting what you want from what you have is important to communicate this.
> There's still a long way to go - I use Airflow for ETL management, and I'm using pipenv to manage it - except Pipenv can't create a lockfile because a dependency of a dependency of Airflow requires in flask-wtf <= 0.7, while another dependency of a dependency requires ==0.8
I'm not sure what you expect it would be able to do here...
Most shops I know use one of maven or gradle. I can’t think of any other serious contenders in Java (if you still use ant: please consider alternatives).
Poetry looks like it did just that though and I'm warming up to it at a very high speed.
Boot, and Lein. I'm partial to boot lately.
You wouldn't write your application to support 5 versions of Django, but you _probably_ would do so for a library.
That said, I do basically agree about `pip` existing already. We could have built a tool to manipulate `requirements.txt` files instead of introducing another format and a toolchain that is _much_ slower and brittler. Though ultimately at this point Python packaging woes feel like they are at a much lower level (the fact that libraries end up being "installed" mean that preparing all your dependencies to go out to multiple servers is a mess).
There's pip-compile from https://pypi.org/project/pip-tools/ that does exactly that. Pipenv uses its resolving mechanism if I'm not mistaken. It produces standard req.txt file with versions pinned and supports separate dev requirements. It had some bug with upgrades last time I checked though, not sure whether it's resolved, currently considering using it for projects at work.
Since its all just requirements files with pip-tools, its been fairly commitment free, in that I haven't had to substantially rework any major bits of workflow to use it - nor would I to remove it. Not sure I could say the same thing about pipenv and/or poetry.
The "library" workflow works for applications too. Put your direct dependencies in setup.py. Build wheels of everything and upload them to an internal PyPI server. Pin everything in requirements.txt.
That would be weird. Perhaps you mean 'flaunting'?
Huh? I'm not really familiar with the state of dependency management for Python/dynamic languages but... there's much more out there beyond just lockfiles. I'm a bit appalled Python is so far behind.
Much more in the field of build tooling/package management. Pinning versions is fine, but dependency resolution is another legitimate choice.
Yet, I think PyPA has not been taking the best decisions regarding Python packaging.
Your Kenneth story is not the only "weird event" in their history.
Did you know that we don't need "pyproject.toml" at all ? That there is already a production ready plain text standard to replace setup.py ?
Did you know that this standard has been perfectly working for TWO YEARS with regular setuptools, is incredibly simple to use and completly compatible with the standard "setup.py stuff" workflow (and hence the whole legacy tool stack) ?
Yep. And nobody talks about it.
Let me (re)introduce...
Oh, I know... Most people believes it's a useless file.
After all, the Python documentation seldom states to use it and for only one tiny option:
But no. Setup.cfg is awesome !!
Put one line in setup.py:
import setuptools; setuptools.setup()
Not only it has been working since 2016, but it has fantastic goodies:
version = attr: src.__version__
"license = file: LICENCE.txt"
Try it, it just works. And you can "python setup.py sdist upload" as usual with it. You don't need any new tool.
Now why did the PyPA decide to forget about this and create a new format ? The short explanation in PEP 518 is a bad joke. And why does nobody talks about it ?
When I asked the PyPA, they told me they were too invested in the new project to stop now. I don't like this answer at all: we suffered enough with python packaging during the "distutils, eggs, etc" fiasco.
setup.cfg works. It works now. It's nice. It's compatible. It does what we need.
Use it. Talk about it. Write about it.
Make sure a lot of people knows so that tool makers and PyPA finally acknowledge that there is not need for the XKCD comics about standard to be true again.
Here are some examples of my libs/apps using it in the real world, if someone needs references for how to use setup.cfg with an empty or near-empty setup.py:
Edit: I see you mention attr: src.__version__. I personally prefer doing it the other way around, with the version defined in setup.cfg and a pkg_resources snippet in __init__.py (https://github.com/HearthSim/python-hearthstone/blob/master/...).
To be honest I wish __version__ were automatically defined like that (but more reliably). Do you know if this was discussed in a PEP?
- there are things you can't do with pyproject.toml you can with setup.cfg. E.G: you can't use pyproject.toml with legacy tools, or with just a fresh python setup. This may change in the future, but would requires a lot of effort because changing setuptools is a very tedious process.
- resources (time, people, money, documentation, public attention, communication, etc) invested in creating and supporting pyproject.toml could be invested in improving setup.cfg and its ecosystem support. E.G: Why does poetry support pyproject.toml and not setup.cfg ? No technical reason. Why does nobody knows this easy way to package python libs ? No technical reason reason either.
So not only the new format brings nothing on the table, but it is also a setback, AND add clutter to a situation that was just begining to be solved. It's not just poor engineering, it's poor manners really.
I've been coding in Python for 15 years. I've lived this: https://stackoverflow.com/a/14753678
Stop the pain.
Second, the format of setup.cfg is defined in a documentation already, so there is a reference outside of configparser. Yes, the low level format is not explicitly defined (although it is implicitly): so let's do define it instead of creating a new one.
Third, it's still a much easier and saner task to rafine the definition of setup.cfg than to create a new standard. I don't even understand how this is controversial, espacially among people in the computing world, where we had those kind of problems for decades and we know the pros and cons, and consequence of this.
The "I add my little format because it's pure and better and current things suck" falacy is such a stereotype we should all be able to recognize it from miles away from nowaday.
I wouldn't call comments an edge case. The distutils documentation has a definition for comments, but I think it actually just uses configparser. setuptools just uses configparser. The pbr documentation has a slightly different definition, but I wouldn't be surprised if it just uses configparser too.
They also have different definitions of non-string types.
Even if you call those edge cases, do you think a PEP that turned edge cases into silent errors would be approved?
The most used one in the world: https://github.com/travis-ci/dpl/issues/822
Also the last time I used tox, anything complex didn't work either.
> Even if you call those edge cases, do you think a PEP that turned edge cases into silent errors would be approved?
Well the current PEP decided to turn a packaging situation that was stable into one that was not, again, after 15 years of mess with many versions of things. So you tell me.
Check the usage stats I posted in an other comment to see the problem.
Besides, yes, we do make compromise on best practices to allow peaceful transition all the time in Python. `async/await` allowing to be a variable silently. Non utf8 defaut encoding argument in open() in windows. Then... we fix it later.
Because I think you conveniently skip a lot of things I wrote in my comments. I clearly state that we would and should consider setup.cfg as a version 1 of the format. Then we would increment on that. I gave a detailed procedure on one way to do that, and there are others.
The point is, all your concerned can be addressed with a progressive transition, starting from setup.cfg. Actually we could even end up with a toml format in setup.cfg, __on the long run__, that matches exactly the current one.
While you addressed non of ours concerns. Just reject them. No will to even recognize there is a problem. It's insulting, really.
We did that during the 2/3 transition. Didn't work so well, did it ?
Oh but we do. Then we rationalize it away, because "this time...". Like we do for Big Rewrites.
It might have something to do with the fact that programming is mostly a craft you learn by doing it, so we overvalue "doing it again" because that's how we usually get better.
> Use it. Talk about it. Write about it.
Still happily using plain setuptools for library development and pip-tools for application development.
The reasoning is good, but we were just arriving to the point that every Python tool out there is either compatible with tox.ini, setup.cfg, or both (much like the JS ecosystem has tools reading from package.json).
Now we have both Pipfile and pyproject.toml on top of it!
For a language that prides itself on its stability and backwards compatibility (especially when compared to the JS ecosystem), we churn through boilerplate files harder than Google churns through instant messaging apps.
This can be done with setup.cfg. Setuptools is only a backend supporting it. You can create other ones. Poetry and pipenv could support it in a week in their authors decided so.
> The reasons for not using setup.cfg are explained in the PEP.
Those are not reasons, those are excuses. Let me quote it:
>> There are two issues with setup.cfg used by setuptools as a general format. One is that they are .ini files which have issues as mentioned in the configparser discussion above.
Not only setup.cfg does the job with the current limitations of the ini format (while pyproject.toml still doesn't with its fancy one), but python projects are not so complex they require such a rich content.
Besides, nothing prevent PyPA to says that setup.cfg format now has a version header, with the current setup.cfg being headerless version 1, then make the header mandatory for version 2 and increments it to move toward TOML if we ever reach a limitation. That's how formats grow everywhere else in the world.
>> The other is that the schema for that file has never been rigorously defined and thus it's unknown which format would be safe to use going forward without potentially confusing setuptools installations.
That's incredibly dishonest, since I gave a link to a complete documentation of the format in my previous post. Besides, it's better to actually refine the existing standard if you ever find it lacking than recreating one from scratch. While there are good reasons to do so, the later is rarely a rational engineering decision, and most often driven by ego.
>> While keeping with traditional thanks to setup.py, it does not necessarily match what the file may contain in the future
So ? How is that a problem ? A standard is not meant to be set in stone. It evolves. But it can't do so if everytime one has an itch, one reinvents the wheel.
The last sentence you quoted explains why they picked "pyproject" instead of "setup". It isn't why they picked TOML.
Also "higher-level things like key names" is half of the standard.
Besides, picking a new (even if better) serialization format is not good reason to create a whole new standard with names, convention, tooling, etc., as explained earlier.
There are sane ways to make the existing system evolves and improves incrementally, using the legacy standards that benefits from the existing situation, and allow the improvements from the new one. All that without the madness of messing with the entire community once again after 15 years of instable package management.
Yeah it's less sexy that having to create your new baby, yes it's less fun than using that new shinny format (and I say that while I __love__ TOML), and less it's less satisfying than having your name as the creator of a whole new jesus-format-saver. But that's the mature and professional things to do.
The "legacy standards" are subtly incompatible INI dialects that people recently started putting into the same files. The incompatibilities mostly don't matter because most tools just read their own sections. They do matter if you want to standardize them.
The only new tooling for TOML is a small library. A new INI dialect would need one too.
No, if you use any key, it won't work with setuptools.setup(), and just like a python code that doesn't run with cPython will never be popular, it will not be used.
Also, if you look at how poetry use pyproject.tml, they just create a custom section. So basically, they don't use your standard.
> The "legacy standards" are subtly incompatible INI dialects that people recently started putting into the same files. The incompatibilities mostly don't matter because most tools just read their own sections. They do matter if you want to standardize them.
That's kinda my point for comments and comments. Standardize the status quo, then increment from that. Not sexy. Not pure. Welcome to the real life.
Didn't you learn anything from the distutils/distribute/setuptool mess ? From the Python 2 / Python 3 breakage ?
And could you address any of my concerns instead of just attacking ? Because I'm trying to address yours with solutions. You just write short busts of "no, it's bad, we are good". That's not really giving me trust in your decisions, and it __lowers__ my confidence in pyproject.toml because the people defending it basically are not behaving like engineers trying to solve a problem, but as salesmen trying to only defend their product.
> The only new tooling for TOML is a small library. A new INI dialect would need one too.
But but we can start from a standard that works now, is used already, and is compatible with existing stacks. Instead of arriving with the theorical untested, incompatible best thing that add a layer on top of the mess.
- setup.py: 1,259,007 results (https://github.com/search?q=filename%3Asetup.py)
- setup.cfg: 165,716 results (https://github.com/search?q=filename%3Asetup.cfg)
- pyproject.toml: 2,137 results (https://github.com/search?q=filename%3Apyproject.toml)
Also, remember that setup.cfg is completly compatible with setup.py, the migration is painless. All the legacy tools work. Not the case with pyproject.toml.
However the default in poetry seems to be pyproject.toml... I'm confused.
Computing is not black and white, and perfect purity is only nice in "fizz buzz".
Now to be extra fun, poetry uses a custom section ([tool.poetry]) in pyproject.toml, not really the standard itself. What does that say about this format ?
ptest poetry add requests
I've never seen that error before.
Which version of Python do you use?
And feel free to create an issue on the issue tracker: https://github.com/sdispater/poetry/issues