Python needs a BDFL to enforce the hive mind. It is a fucking mess and it's unacceptable. Python is otherwise the most incredible language of our times. It is so far ahead of anything out there for general purpose use.
Relevant xkcd: https://xkcd.com/1987/
And there is a packaging BDFL, but there's a severe lack of time and effort in the form of volunteers to tackle a lot of the projects, big or small. Plus rolling these things out takes years and it takes even longer to gain traction as people are often reluctant to change their build tools once things work.
As an example of timelines, take what my blog post covers. PEP 518 and the concept of pyproject.toml was conceived in 2016, four years ago. Getting PEP 517 finalized and it all worked into pip didn't happen until pip 19 in January 2019, over a year ago (https://pip.pypa.io/en/stable/news/#id203). And a year later there's still enough of a need to educate folks that I wrote this blog post to help get the word out.
IOW the timelines are huge and the pay-off is way down the line. And that's if you manage to be motivated enough to do the work. Burn-out is a huge problem in the packaging world. I mean reading that you think "it is a fucking mess and it's unacceptable" doesn't motivate me and others to keep trying to improve things if that's how us volunteers will be treated for years to come while we try to rectify the situation. Add on to the fact that we aren't getting paid for this and it makes the idea of pulling out my laptop tonight to work on some packaging stuff feel rather pointless.
So please be understanding that people are working on things as best they can and that overt negativity about it all will work against you getting what you want.
Please don't over-generalize. Volunteering is not the only way FOSS is developed. E.g. the Linux kernel, Gitlab, Firefox, Golang main tools…
Python serves several huge ones: web, datascience, 3d graphics, sysadmin... all with different needs, different conventions, different “standard” tools to do this and that. The community is orders of magnitude bigger than ruby by now, and anything that requires coordination is much harder to accomplish than in a lot of other ecosystems.
Python is now on par with ecosystems like C/C++ (where different build and packaging conventions are legions) or Java (where ever 10 years they write a new build tool on top of the previous one(s) to hide the complexity, and still, like python, they struggle to make simple executables 30 years on...). It’s just a hard problem to cater for so many different needs.
Them having different conventions and standard tools is the failure, not a constraint.
What are the different needs that these ecosystems have which mean they couldn't use a common package manager? I don't believe there are any.
There's probably no logical reason why a single common package manager for these domains is impossible. But they have different priorities, and people write tools that solve their own problems. Who is going to decide which package managers are unnecessary, and convince all of the users who are happy with them to switch to something else?
So while it's fine to say you wished Python's packaging approach was as good as another languages, please do so know that it is not fair to say that if one community got something to work for their needs it means Python should have been able to as well; it's not an apples-to-apples comparison considering the trees were quite possibly planted at different times and are different varietals.
It's pretty remarkable that things have continued to function since my 17 years with Python alone have seen things like:
- Compilers becoming available on all OSs
- Linux becoming a thing
- Laptops becoming common
- Web-based email clients
As I have said a few times in comments on this thread, it's fine to feel like there's room for improvement, but trying to "stick it" to Python and its packaging ecosystem in some way for not being perfect is pointless for you and demotivating for those who are actually in a position to try to make things better for you (and even if you aren't a Python user you should care to some extent about it as Instagram runs on Python and thus it is helping to deliver you those cat photos while you're stuck at home ).
Second, there is the python foundation, which funds people to work on areas of python. This should be a priority, and can be done by paid contributors.
The fragmented ecosystem is one of the main complaints about Python packaging today. And that's really tricky, because the main way the open source world knows to solve problems is to create new tools, and that doesn't help when the problem is 'too many tools'.
not at all. from my previous experience with ruby i was using gems with compiled bindings to libraries for: mysql, memcached, json, xml, regexp and http. and probably some more i don't recall right now.
Getting there might not be the nicest experience, but most packages nowadays can and do get there.
A lot of 3d graphics houses have to roll their own solution.
This is a famous one in the cycle and a common approach-> https://github.com/nerdvegas/rez
The issue is that the environment varies a lot and there's usually a lot of C++ code that needs to be compiled to interact with the python plugins based on different environments with 500+ libraries mixing & matching...
find node_modules/ -name "node_modules" | wc -l
I think part of the reason we ended up with so many trivial libraries and dependencies in the Node ecosystem was precisely because it is so easy to create and publish packages.
I love Python as a language, but working with the packaging even just to install dependencies is like pulling teeth every time you start a new project - nevermind publishing packages yourself.
That said, PEPs 517 and 518 sound like they're going in the right direction from the article.
Compared to other languages, python has a pretty rich set of modular tools in the standard library.
Meanwhile, the pip project is full of developers who tell you that your problem is not a problem and you’re discouraging people by pointing out that they’re blame shifting: https://github.com/pypa/pip/issues/4995
I honestly have zero faith that things will get better without a radical change in leadership. Pyproject.toml is a good step in the right direction, but it’s been around for years. They had time for adding the walrus operator, but not for replacing the setuptools, venv, pip Cerberus.
I give a lot of Python professional trainings, and I get to do that regularly, in very diverse situations. It's indeed full of gotchas.
Since it's not going to be solved quickly, meanwhile, here is what works if you need to help people setuping python:
1 - Install Python correctly
The first version Python download link for windows is 32 bits. You want 64 bits, so you should actually not click on that. Also, make sure they use the latest minor release possible, as early ones can have weird bugs. Tell people to install from the app store if they are on windows 10, or give them a link to the proper (non web) installer.
Linux: python versions may not be available for their linux distro. Use EPEL for Centos or deadsnake for Ubunto. Other Linux users chose something exotic and must be able to deal with it.
Mac: brew is fine. Official installer too.
Cause: Python support is crazy good. It support 32 bits. 3.4 supports Windows XP. 2.7 supported Atari and Solaris! So there are a lot of installers, and a lot of versions. But also because the official Python website does a poor job at directing the user.
2 - Run Python correctly
This is the great lie of Python running. You cannot just use the "python" command, which is what every doc and tutorials tell you to do.
On linux and mac, tell them to use the suffixed PythonX.Y command, with X.Y being the version of python they need. E.G: python3.6.
On windows, tell them to use the "py -X.Y" command. E.G: py -3.6
Tell them anytime they see a tutorial with "python" in it, they should replace it mentally with "PythonX.Y" or "py -X.Y" depending of who you have in front of you.
Cause: people often ends up with have several versions of Python installed on the same machine, so you can't tell them to just use the "python" command. Of course, windows and unix never agreed an on naming convention. What's more, the Windows situation can lead to a PATH problem, and "python" may not be found, while "py" will always be. We are talking about providing the "py" command everywhere.
3 - Install tools correctly
Introduce them to pip. Tell them to never, ever install stuff using admin rights with it, even if told to by documentation or tutorials. No "sudo". No "run console as admin".
If you need to install a tool, such as black, mypy, pylint, etc., outside of a venv, use "--user" to install it for the current user. This requires no admin rights.
Also, don't use the pip command directly. Use "-m" so that you always know from which python you are installing the command for. E.G:
python3.6 -m pip install black --user # unix
py -3.6 -m pip install black --user # windows
Cause: installing with admin rights can destroy your python installation. Also, pip may not be in the PATH, or be attached to the wrong version of Python. It's also because we are twisting a lib packaging tool into providing programs.
This is not specific to python. Node had to introduce a whole know command, npx, to solve the same problem, and the ones bellow. It actually takes a command name, and if it doesn't exist locally, but is registered on npm, download the package that seems to contain it, install it in a temp folder with all its deps, then immediately attempt to run it in isolation.
4 - Use tools correctly
It's another lie from docs and tutorials. If you install a tool, just calling the command may fail.
If you installed a command outside of a venv like above, then you should call it using "-m". E.G:
python3.6 -m black # or py -3.6 -m black
Cause: again, you don't know how the PATH is setup, so there is no guaranty that the command will be available, or ran from the proper python version. People can have completely messed up machines and there is nothing you can do about it. So don't depend on the PATH.
5 - Use venv correctly
The solution to all those shenanigans are the venv. Once you are in a venv, you don't need to tell the version of python, you don't need "py", "-m" or "--user". You can call commands directly. You can just use "python".
But first, people need venv installed. It's installed by default on Windows and Mac, but on linux, it's often a package to install like python3-venv, python3-pip or python3-setuptool.
Do not "pip install virtualenv": people will get confused between the tools.
So, make them create a venv: "pythonX.Y -m venv name_of_the_venv" (or py -X.Y).
Show them how to use the python/pip from the venv WITHOUT activating it, so that they understand how it works and show them the that unix have ./bin and windows ./Scripts.
Then show activation, and tell them to install pytest/jupyter and Co inside. Show them that it solves all the previous problems, so that they are motivated to use them.
Tell them to NOT put their code in the venv folder. And that they can't move or rename a venv. Show them "pip freeze > requirements.txt" + "pip install -r requirements.txt" as an alternative.
Cause: historically python didn't have the venv module, and people used to pip install virtualenv, which mean we still have docs about it, and the name stuck. Having a long history makes things complicated. And once again, the differences between Windows and Unix bites us, but that's the cost of being portable. It's not Python-specific. The fact you can't move a venv is an artefact of the venv design that the community never solved.
People may want or need to use anaconda or the python embedded in a system (blender, qgis, etc). Then it's a whole different cycle of problems and solutions, so I won't talk about it here, this post is long enought.
Again, Python diversity is playing against it: you don't have one popular python distrib like with nodejs or cRuby. You also have commercial Python distribs, and some are used a lot in the corporate world.
Diversity make things more resilient and encourage innovation. But it makes thing harder.
# Pyproject and setup.cfg
They have nothing to do with all that. They are useful if you want to _produce_ a package to share your code with other devs.
My company settled on pipenv, and we regret it. It's slow and buggy and has crazy behaviour. It's not bad enough that it's worth updating all projects (or rather, there's no good enough alternative) but I regret about once a week having advocated for it.
Note: I think you made a mistake, you say that you can rename or move a venv but then you say you can't.
Being stuck at home, I currently give training remotely. If you know people that need trained in anything Python (beginner, advanced, scripting, architecture, web, data analysis, etc.), think about me :)
> If you need to install a tool, such as black, mypy, pylint, etc., outside of a venv, use "--user" to install it for the current user. This requires no admin rights
I think you should add that on Linux distros it's better to use the package manager to install libraries, because a beginner can, and will, have conflicts with (system) packages relying on a particular version of a library. Only use pip if the package is not available, while with the like of Debian, you will have old versions of libraries, I think fast paced distros like Fedora does a good job at providing up to date python libraries with the package manager.
This is not the instructions to install Python.
Those are the instructions to teach anybody in any OS with any configuration to install python and deal with 3rd party dependencies.
E.G: if you are a Windows 10 user, instructions to install python are 2 lines:
- install from the store
- run "py -X.Z" to start python
The complication comes when you have people with Windows, and Mac, and Linux, and they already have some Python installed in various ways, and then want to install stuff with pip, etc.
Which is true for Ruby, JS, PHP, etc.
E.G: if you are a Windows 10 user, instructions are 2 lines:
The difficulty is to have general instructions for all cases, including packaging. Which is a problem with every scripting languages.
I too regularly have to do the same thing, but I have absolutely no issues getting newcomers onto the python ecosystem. You download the installer, you run it, and it works. Or you apt/yum install it with no issues.
Setting up a dev environment for them though is where some hassle comes. VSCode being a particularly bad one in that regard. But with Pycharm, newcomers are on boarded in a matter of minutes.
The hidden (not deleted) comments are hidden not because they contain criticism, but because of an unnecessarily harsh tone, which is unhelpful to anyone wanting to read up on the discussion at hand, and definitely not helpful for the motivation of anybody currently doing the unthankful work of maintaining pip. They do a very difficult job, where they have to deal with 20+ years of legacy, many different operating systems, many different Python distributions, many different skill levels in their user base, and even having to fight OSS fatigue due to unconstructive criticism. Pip is as good as it is due to their hard work, and we should be thankful to them.
I reread the thread last night and asked, Was I too harsh? Am I discouraging the thankless pip team? Obviously, I am biased towards myself, but no, I don't think I am the one doing the discouraging. They are the ones doing the discouraging.
Go back through the comments and look at who is PyPA official and who's not: the non-members have ideas and members shoot them down. It's not that complicated technically--there should be a switch called "require hashes" instead of an implicit requirement for hashes once you use one. But the PyPA devs just showed zero interest in understanding the problem or helping. Their first comment was that error message was fine, nothing to fix!!!
I ask myself, would I want to contribute a patch to pip? I contribute patches occasionally to OSS. Not every day, but every couple of months when something comes up that affects me. I would never bother. Why? It's obvious from their tone that the patch would be stuck in a queue and never reviewed. (This happened to a coworker at a former job. We maintained a patched version of pip for years because they wouldn't take our change.)
In contrast, I just submitted a broken patch to Go (I broke tests and said so in my patch) and the Go team took it and fixed the tests. Why did I bother to submit a broken patch to Go? Because it was pretty clear from the tone of the discussion that if I got the ball rolling, they would push it the rest of the way over the hill and I want the issue fixed in the next version of Go, so I submitted what I had even without going through and fixing all the tests (which I didn't have the expertise to do).
Now, this is not the same. Go is supported by Google and has money and developers, yada yada. But the Python Foundation is an organization. It has a payroll. It has official members. They could ask the EU for €5m grant to work with N devs for Y years fixing Python packaging… Whatever. I'm not in charge of PyPA, so it's not my problem.
But I do object to the gaslighting in implying that we, the users, are discouraging the poor, unthanked PyPA devs. No, they don't want to be helped, so they put out the vibe to make it clear we should leave them alone.
Gentoo is the only distro on which I feel comfortable using the system interpreters because the versions are slotted (you can have multiple versions) and system pip install is effectively disabled.
So on other distros I use pyenv. But trying to explain this to non-Linux geeks is hopeless.
I've taken to recommending the following: use system python/pip to install pipx (but with --user). Use that to install "global" tools. Use virtualenvwrapper to manage projects. Use pip-tools to manage frozen environments (ie. requirements.txt). I've looked into replacing the latter with pipenv (which is awful) and poetry (which is better) but I've so far been unable to generally recommend anything else.
I don't even want to think about other platforms and conda and all that.
Replicating is just "virtualenv --python=python3 venv && . ./venv/bin/activate && pip install -r requirements-frozen.txt". Am I missing something here? Yes, to be absolutely sure, you need to have the same versions of python, pip, setuptools, gcc (for C extensions), C library dependencies, etc. on both systems that do the build, but that's the case for pretty much any build system out there if you want repeatable builds.
And if you you just don't feel like dealing with that, you can just use Docker (aka "put my machine in a tarball and ship it").
What?! Seriously? That is not how it's supposed to work. Requirements.txt is already meant to be the frozen dependencies (like the output of pip freeze). Why change that? I much prefer how pip-tools does it with a new file, requirements.in that contains the unpinned direct dependencies. I thought poetry did the same thing but with pyproject.toml?
Modern languages do pretty much the same, just have a nicer wrapper for it.
And yes, as I said, a nice wrapper for management of it is missing in python. The main ideas/approach is the same though.
edit: also, how do you manage dev dependencies with requirements.txt and setup.py?
Again, not as polished experience, but the model matches.
The pip SAT-based solver was supposed to land in 2017 or 2018 after a GSoC, and nothing came of it. Is it finally going to use a different heuristic than "#yolo" for picking which version of dependencies to install?
- discover it
- install it properly
- run it properly
- make it create a venv with the proper python version
- potentially import data from existing setup.py/cfg
- make sure you are always in the right dir when you use it
- not be on windows (poetry shell doesn't work there)
- IDE integration is not great
- people have to chose between this, pip, pip-tools, flit, pipenv, dephell, conda...
And you cannot use it if you are not "in a project". But not all venv are project related.
Ok that's a bug, probably fixed soon. Otherwise, most of these are true of many popular packaging tools in other languages.
Is that like [tool.poetry.scripts] ?
I want something such that I can run `poetry run script_name` and it will run that command line in the poetry context. Right now I either rely on my bash history or use a bash script to mimic the behavior of npm / yarn scripts. I want to turn this `poetry run python project_name/main.py` into an alias like `poetry run main` or `poetry run dev`.
Ah yes: https://xkcd.com/927/
But I agree that the current situation is a mess. The "official" solutions are suboptimal and none of the unofficial solutions, which are pretty good, have enough clout to meaningfully simplify the ecosystem (held hostage by the majority of users with just pip).
So the ecosystem is kind of frozen in place. PyPA doesn't seem to have the will/capacity to do anything more than very small, incremental changes.
By the way, Python should go beyond Go. It should single binary the someone can install without root privileges. The fact that I have hundreds of Chromebooks at my school and I can’t use any of them - school IT, correctly, doesn’t want me mucking with advanced settings.
But as I said, it takes time and effort and there's only so much of that when it's being done by volunteers in their evenings and weekends.
And as an aside, I would suggest teaching them `python -m pip` over `pip3`: https://snarky.ca/why-you-should-use-python-m-pip/.
I think one of the unfortunate things that happened is indeed the arrival of new languages. Go, Rust, Swift, whatever is taking away a bunch of senior developers to work on those. Even though Python is still "most wanted" and "most popular", imho that does not reflect the reality among senior developers who are capable of taking on large tasks like this.
Sometimes the frustration expressed by package management has more to do with build dependencies rather than the distribution itself. Like a library that has a C dependency. In this context, the world of Go is equally messy, when CGO is involved. Luckily in their case, most people do rewrite libraries in Pure Go.
Our online coding tool (and entire course library) is included in the Pack. https://next.tech/github-students and https://next.tech/github-teachers if you'd like to take a look.
You think the school IT department won't let the GP install Python but will let them install a compiler toolchain? The benchmark for "easy" is you download one binary and run it in userspace. Add any additional steps and you can't use that word anymore.
On the other hand, setting up a deploy key to PyPI and then using poetry build, poetry publish is very easy compared to using twine which I found often did daft things like upload mixed versions to PyPI unless the build directory was clean. Poetry seems to handle that a lot better.
note most of this just working around poetry for multiple packages in a single repo, with interdependencies. flip side, our pip/setuptools users building trunk from ci are unaffected.
that said there are open questions on poetry (governance/dev community), but the experience is overall pretty solid with a some minor rough edges, and the fact that it has a usable api makes those pretty minor. in contrast the only api on pip is cli arguments for subprocess/fork.
I wrote this post that compares Poetry and Pipenv if you're interested and talks about some of their differences. Poetry has both add and install commands, whereas Pipenv only has an install command, for instance.
Maybe when it has a JIT/AOT story that matches Common Lisp, Scheme, Julia, Smalltalk, JavsScript, or I am able to do refactorings with the easiness that it works on Smalltalk, I can re-consider my opinion.
Does that mean its the best at everything? Of course not. It is second best at most of the important things however, a more unique position than it sounds.
Which brings us back to "nice scripting and introduction to programming language".
Clearly no one is going to be able to convince you otherwise, so is there much point in you continuing to read this thread?
This is no small feat.
Also the ecosystem of libraries is the same.
You can't see that from your ivory tower but those things matter a lot to people on the ground.
Don't let you prejudice get in the way, it never helps :)
I see that from my historical experience.
Also we are moving goal posts here, I was quite explicit that Python is a good language for teaching programming.
What I am against is portraying as the one Language above anything else.
I can't see a world where the Python Software Foundation can't get support for this, I think they have the contacts at the big companies to get it funded.
The lead Python has on machine learning, data science, and scientific computing won't last forever (though, I think its safe to say no matter what, it ain't going anywhere.
You might be surprised at how hard it is to get money donated for this sort of thing. Large companies often have their own build infrastructure and thus build things for themselves from scratch, to the point of building their own build toolchains (e.g. look at Google and bazel, Facebook and buck). So the way they handle packaging means how the overall packaging ecosystem functions does not impact them like it does individuals or smaller companies.
This is also why companies only started funding PyPI security work about two years ago; downloading source securely from PyPI is important to big companies as that's where they grab the source that they compile. But funding pip improvements only just happened in Decembe, starting with the new pip resolver work got money (https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-rol...). And all that money came from Mozilla and the Chan Zuckerberg Initiative; two huge non-profits, not companies.
Add on to the fact that the PSF is projected to have a $627,000 USD loss this year due to PyCon US 2020 being cancelled and it makes finding funding really hard (https://pyfound.blogspot.com/2020/03/psfs-projected-2020-fin...).
Python overall very much runs on volunteers, but the packaging system especially. The core team of Python itself might generously have a total of 3 devs/week being paid to work on CPython spread across all contributors (IOW it is nowhere near as impactful as 3 actual full-time devs who aren't context-switching constantly between "work work" and "Python work"). But for Python packaging in general? If you don't count conda I think packaging has less than even 3.
But if anyone wants to donate to try and rectify this, please go to https://psfmember.org/civicrm/contribute/transact?reset=1&id... and donate to the packaging WG!
I will say, I didn't realize the funding situation was _that_ dire, especially since on Talk Python I remember specifically Michael and some others talking about the massive influx of money into Python and its ecosystem in the last year and half or so. Its not billions of dollars but it sounded like the situation had a positive uptick substantially, so, in short, I was wrong!
Has the PSF ever considered having an actual business? Like say, consulting and/or having developers work on features that businesses pay to have implemented? (A loose example of this would be Igalia, which makes surprisingly good amounts of money to implement features in web browsers on behalf of the major browser vendors)
I've always thought this would be a pretty natural way to have a steady source of revenue and the profits could all feed back into the PSF.
> is Don still there?
Yep, working on the data science features of the extension.
> on Talk Python I remember specifically Michael and some others talking about the massive influx of money into Python and its ecosystem in the last year and half or so.
As I said there have been donations for PyPI and pip just got its big donation. The CZI donation was also much bigger than pip and covered a lot of the science ecosystem.
> Has the PSF ever considered having an actual business? Like say, consulting and/or having developers work on features that businesses pay to have implemented?
There isn't a need for the PSF to have a business subsidiary to accomplish that as people can donate money for such things if they truly wanted to. It's also really hard to manage due to tax laws in the US around non-profits and such. Plus there would be start-up capital costs, etc. just like any other business. I just don't think the risk of trying to get it set up would lead to any benefit when people have always had the option to donate directly to see if something could be done.
If people want to help out financially they can go to https://donate.pypi.org for packaging donations and https://www.python.org/psf/donations/python-dev/ for the language and CPython. And if someone wanted to fund a specific thing I'm sure the PSF is happy to discuss such an idea to see how feasible it would be.
One thing I would suggest, on its face, is that any time someone has to reach you (as in, the foundation, not you specifically), that introduces friction, even the smallest amount of it, can leave lost donations on the table. If there was a simple page even, that said something like
Do you have a feature you need implemented in Python? Are there features you would like to fast track? For a simple donation, we can prioritize the work you need to grow your business
This with a nice little contact form, would eliminate that friction, rather than being in a situation where I might be reaching out to the wrong wing of the foundation, or not feeling comfortable sending a general inquiry etc.
you could also put a feature page that tracks how much money needed to implement x too, to solicit donations. I've seen this work in other communities. I think Django has gotten funding using this technique before.
Just a thought, I know you guys are doing everything you can and then some, and Python is unique in that its one of the few language communities of its size without any formal corporate benefactor.
I'm not an expert, but I'm pretty sure a non-profit can't do things like this. If the person giving the money expects a tangible benefit in return, it's not a donation.
If there's a specific pain point your employer would like to see fixed and is willing to donate money to help resolve that pain point then please go to https://donate.pypi.org/.
The conclusion has been that:
- distributing story is handled ok, but distributing program is not.
- python has some unique problems due to its popularity, diversity of user base and long history
- the py command should be available everywhere
- communication on best practices is lacking, and people mixes project deployment, dependency management, lib packaging and program distribution
- the way python deals with imports is the root of all evils, not the packaging system, which actually is a consequence of it
The best course of action to take is still not clear though, as creating yet another layer on top of the mess is probably not the right thing to do.
Looks like Canonical is paying him now.
This is a strong claim. Ruby could arguably make the same claim. So could Elixir.
For Python, it's the reversed: it's used everywhere, except for a few topics.
It's the only scripting language like that.
Yes, Python is winning the popularity battle no doubt on that, but that nothing to do to the language or the surrounding tooling itself. It's like saying VHS is an incredible format compare to Betamax(Yes I'm old :D).
I used Ruby (not Rails) and Python for many years by now. I think Ruby and its tooling are far better than Python's. But if anybody asked me which language should they learn as their first, it would be Python, just because there will be more jobs available.
I'm not super happy with package management under python but at least pip does not break every release.
I can never trust pip for on-premise installations of python code. Even the npm toolchain madness is somewhat more consolidated and organized.
By what metric(s)?
It's based on statements, and not only on expressions. This is a fundamental issue that it has yet to grow out of.
If you don't have something substantial, cited, and/or supported to say, and it is also negative, better to not write anything.
Judging from comments here and in the past, we all know packaging is one of Python’s Achilles’ heels. I spent a couple years bouncing around with pyenv/pipenv, then poetry, then back. I lovingly followed all the flamewars here and on github.
Ultimately, I realize now, I laboured under the conceit that it was desirable to have a “native” solution that did not depend on a suite like conda. I now think I was totally wrong.
I’m so much happier and more productive now that I’ve switched. For any data science use case conda is hands-down better, and for everything else it’s still more useable than any of the alternatives. I hate to say this but it’s true. The only thing I can think of that was harder to work with a conda setup was the keychain in macOS, and I’m not even sure now that that’s an issue anymore.
conda really does just work.
The main problem that Conda was originally meant to solve was that basic Python packaging (virtualenv, pip) required you to build C extensions yourself which is a huge pain (mostly getting all the C dependencies installed). But with the advent of binary wheels and (eventually, in 2016) a binary wheel standard for Linux  you can now get binary builds of most packages on most OSs by just doing a "pip install packagename" - typically this is even more likely to work than Conda in recent years.
The only limitation of virtualenv is it requires you to have already installed Python yourself. On Linux I just use the system-installed version (the Python 3 one obviously), but on Windows I use Conda to create the environments and then pip to install the packages, which works fine.
I believe both of our stories. I just always wonder how this happens that two people have such opposite experiences.
If you don’t mind my asking, what kind of stuff we’re you working on? Just curious!
I have definitely found it’s easy to bork a conda install with updates, enough to make it easier to just wipe it out and start over.
I mostly work on financial software and only conda seemed to play nicely with pandas and brokerage APIs. I should probably mention that I do a lot of this is cross-platform and conda seemed to be the only thing that worked without a lot of tweaking on Windows, Mac and Linux consistently. But the Linux experience with pyenv and virtualenv (and pipenv too, for the most part) was great. If I was Linux-only I honestly might never have switched.
If you only use conda, it works.
If you start to pip install or use venv, it will break at some point.
Add to that, that you download binaries, which you have to trust.
Another issue is, when there is no binary and it needs to compile stuff. On GNU/Linux that might work fine, but on Windows, like for some of my coworkers, you need Visual Studio (MS bundles it that way and I could not find a way around installing that whole thing) installed, for getting the required compiler stuff and even then the compilation might not work.
Conda in the past also had issues with reproducible builds of envs. I hit a case where on one OS it worked, on the other it did not.
Also I had a case where deps of deps change and when you try to build the env, suddenly it does not work any longer. That's very bad for deployments.
Since then we abandoned Conda entirely and built Python with all requirements ourselves. Envs need checksums.
I think about 50% of why I enjoy, in my soul, about Rust when I'm hacking away and personal goofs is that the dependency and project management system is all first class and pretty decent and I sure hope they don't go changing it much.
Once TOML hits 1.0 I plan to work on getting a TOML parser into Python's stdlib.
(disclaimer: I am one of the co-authors of PEP 518 that apparently handled things in way that was "a little absurd" .)
From the link. Makes sense I think.
Python has been around since 1991. It came out before installing dependencies from internet was a thing. It had support for Windows XP, Atari, Solaris. You gotta be able to pip install stuff that are written in C (numpy) or in Fortran (scipy). It had been implemented in the JVM, .net, and Python itself using pypy.
All those years, what we knew about or needed from packaging system changed, and the tooling followed. We had easy_install, then setuptools, then distutils, then distutils2, the pip + setuptools again. We had virtualenv, then venv.
You cannot break from the past easily, especially on such a popular language, with half of it's user not even being devs.
It's really purely the "designed from scratch 10 years ago" (though, only six years for Cargo).
I do really like strict-yaml, however (last time I checked) it is just a hack to simplify the large PyYaml compiled package.
fwiw version 2.0 of strictyaml will feature its own specification and parser.
without a setup.py, how do I:
from Cython.Build import cythonize
import numpy as np
At least, that's supposed to be the case; I am not 100% certain that it always is. However, even so, AFAIK setup.cfg does not directly support using third-party libraries to compute the setup options, as your setup.py does. The only workaround I'm aware of for that if you want to get rid of setup.py is to have a separate script that generates setup.cfg using whatever libraries you need. However, I don't think that's the vision of the Python developers.
It looks like there are ways to specify which build tools your package requires; I would assume Cython would be one of the tools that can be specified since it's been around a while and gets a lot of usage, but I don't know for sure.
That does not mean you should not use it anymore if you have a good reason to use it. The ones you listed are examples.
How ever, as explained in the article, you should still add the pyproject.toml
But when I first moved to pyproject.toml I didn't find its use as widespread as I hoped (flake8 being the dependency I hear about the most without support), so I started collecting a list of projects currently using it, and discussing its use, in an awesome list format. Hopefully it will be useful for other people as well:
Poetry seems the go-to tool for dependencies management and it centralizes all your package configurations alongside its dependencies in pyproject.toml.
Discussion here: https://news.ycombinator.com/item?id=22772410