
What the Heck Is Pyproject.toml? - BerislavLopac
https://snarky.ca/what-the-heck-is-pyproject-toml/
======
fermienrico
Python desperately needs a better packaging solution than pip from Python
Software Foundation. Stop all new features until this - the biggest pain point
for Python across all expertise levels from newbies to experienced Python
programmers - is _officially_ solved. No pipenv, no poetry, no conda, etc from
third party devs who sometimes get tired of the pressure [1]. It is immense.
Combine all this virtualenv stuff with great dependency management into one
tool. In the spirit of Python, there should be only one way to package, manage
dependencies and virtual environments. It should be super clean (see Go).

Python needs a BDFL to enforce the hive mind. It is a fucking mess and it's
unacceptable. Python is otherwise the most incredible language of our times.
It is so far ahead of anything out there for general purpose use.

Relevant xkcd: [https://xkcd.com/1987/](https://xkcd.com/1987/)

[1]
[https://github.com/pypa/pipenv/issues/4058](https://github.com/pypa/pipenv/issues/4058)

~~~
harikb
I teach Python to some middle school kids. Pip vs Pip3 and PYTHONPATH wasted
entirety of two class sessions.

By the way, Python should go beyond Go. It should single binary the someone
can install without root privileges. The fact that I have hundreds of
Chromebooks at my school and I can’t use any of them - school IT, correctly,
doesn’t want me mucking with advanced settings.

~~~
brettcannon
People would love to see that happen, but it requires time and effort from
people to build such a solution. Some people are trying, e.g.
[https://pyoxidizer.readthedocs.io/](https://pyoxidizer.readthedocs.io/) and
[https://briefcase.readthedocs.io/](https://briefcase.readthedocs.io/) as well
as stalwarts like PyInstaller and such.

But as I said, it takes time and effort and there's only so much of that when
it's being done by volunteers in their evenings and weekends.

And as an aside, I would suggest teaching them `python -m pip` over `pip3`:
[https://snarky.ca/why-you-should-use-python-m-pip/](https://snarky.ca/why-
you-should-use-python-m-pip/).

~~~
harikb
Thanks for the link. About the lack of resources problem though, I have a few
points to make.

I think one of the unfortunate things that happened is indeed the arrival of
new languages. Go, Rust, Swift, whatever is taking away a bunch of senior
developers to work on those. Even though Python is still "most wanted" and
"most popular", imho that does not reflect the reality among senior developers
who are capable of taking on large tasks like this.

Sometimes the frustration expressed by package management has more to do with
build dependencies rather than the distribution itself. Like a library that
has a C dependency. In this context, the world of Go is equally messy, when
CGO is involved. Luckily in their case, most people do rewrite libraries in
Pure Go.

------
talentedcoin
I don’t want to be too negative, and I realize this isn’t an option for
everyone ... but seriously, just use conda if you can.

Judging from comments here and in the past, we all know packaging is one of
Python’s Achilles’ heels. I spent a couple years bouncing around with
pyenv/pipenv, then poetry, then back. I lovingly followed all the flamewars
here and on github.

Ultimately, I realize now, I laboured under the conceit that it was desirable
to have a “native” solution that did not depend on a suite like conda. I now
think I was totally wrong.

I’m so much happier and more productive now that I’ve switched. For any data
science use case conda is hands-down better, and for everything else it’s
still more useable than any of the alternatives. I hate to say this but it’s
true. The only thing I can think of that was harder to work with a conda setup
was the keychain in macOS, and I’m not even sure now that that’s an issue
anymore.

conda really does just work.

~~~
pottertheotter
I've had the opposite experience. I used conda for several years, primarily
for data science, but regularly ran into problems with packages not working,
updates for packages bring severely delayed, and issues with conda itself.
Switched to pyenv+virtualenv 18 months ago and haven't had a single issue.

I believe both of our stories. I just always wonder how this happens that two
people have such opposite experiences.

~~~
takluyver
Just to note: this kind of difference is part of why it's so difficult to
improve packaging. There are lots of different tools, and they all work well
for some people in some situations. There's no realistic prospect of
persuading them all to use a single tool - but the fragmentation is a big part
of what makes packaging so daunting.

------
Waterluvian
Is toml even supported in the standard library? This seems just bizarre.
Python's really overtaking node for massive churn. I feel like an old guy when
peers say, "don't use virtual environment use pipenv." then someone says no no
don't even use that, use poetry. Etc.

I think about 50% of why I enjoy, in my soul, about Rust when I'm hacking away
and personal goofs is that the dependency and project management system is all
first class and pretty decent and I sure hope they don't go changing it much.

~~~
benatkin
Here's their justification for using it. It makes sense to me.

[https://www.python.org/dev/peps/pep-0518/#overview-of-
file-f...](https://www.python.org/dev/peps/pep-0518/#overview-of-file-formats-
considered)

~~~
scrollaway
It makes sense to use toml. It makes zero sense not to bring pytoml in the
stdlib before doing so. The way this was handled is a little absurd.

~~~
brettcannon
It was on purpose because TOML has not reached 1.0 yet and breaking people
using a module in the stdlib for parsing TOML because TOML itself changed
would be an absolute mess. Plus not a single packaging project said it was an
issue to either depend on or vendor a TOML parser.

Once TOML hits 1.0 I plan to work on getting a TOML parser into Python's
stdlib.

(disclaimer: I am one of the co-authors of PEP 518 that apparently handled
things in way that was "a little absurd" .)

------
hprotagonist
The thing i have yet to reliably figure out is this (i live in the science
part of python):

without a setup.py, how do I:

    
    
      from Cython.Build import cythonize 
      import numpy as np
      import versioneer
      setup(...,
           version=versioneer.get_version(),
           cmdclass=versioneer.get_cmdclass(),
           ext_modules=cythonize(extensions),
           include_dirs=[np.get_include()],
           ...)
    

I'm 100% on board with not having executable configuration files, except that
we've been happily abusing them for years now and there are a lot of edge
cases that i rely on now.

~~~
pdonis
All of those setup options are now setup.cfg options:

[https://docs.python.org/3/distutils/configfile.html](https://docs.python.org/3/distutils/configfile.html)

At least, that's supposed to be the case; I am not 100% certain that it always
is. However, even so, AFAIK setup.cfg does not directly support using third-
party libraries to compute the setup options, as your setup.py does. The only
workaround I'm aware of for that if you want to get rid of setup.py is to have
a separate script that generates setup.cfg using whatever libraries you need.
However, I don't think that's the vision of the Python developers.

~~~
kevin_thibedeau
There are lot that of options that setup.cfg doesn't support. "use_2to3" is
one.

~~~
takeda
I never needed it, but I see 2to3 options there [1], are they not working?

[1]
[https://setuptools.readthedocs.io/en/latest/setuptools.html#...](https://setuptools.readthedocs.io/en/latest/setuptools.html#options)

------
goldenshale
Our experience using pyenv, poetry and a Pyproject.toml for projects lately
has been great. I highly recommend going this route compared to dealing with
pip and pipenv.

------
carlosperate
Apart from packaging I also love the idea of consolidating configurations from
different dependencies into a single place.

But when I first moved to pyproject.toml I didn't find its use as widespread
as I hoped (flake8 being the dependency I hear about the most without
support), so I started collecting a list of projects currently using it, and
discussing its use, in an awesome list format. Hopefully it will be useful for
other people as well:

[https://github.com/carlosperate/awesome-
pyproject](https://github.com/carlosperate/awesome-pyproject)

------
yamrzou
I have found this blog post[1] quite informative regarding Python projects
packaging. It describes well how all the tools (pyenv, poetry, black, flake8,
isort, pre-commit, pytest, coverage, tox) fit together.

Poetry seems the go-to tool for dependencies management and it centralizes all
your package configurations alongside its dependencies in pyproject.toml.

[1] [https://medium.com/georgian-impact-blog/python-tooling-
makes...](https://medium.com/georgian-impact-blog/python-tooling-makes-a-
project-tick-181d567eea44)

~~~
RMPR
Still have some critical bugs here and there[1], many about resolving
dependencies. Don't get me wrong, I use it for my own current side project
github.com/rmpr/atbswp, but imho a lot of things need improvement.

[1]: github.com/python-poetry/poetry/issues

------
BerislavLopac
Announcing a new Sponsorship Program for Python Packaging:
[http://pyfound.blogspot.com/2020/04/sponsoring-python-
packag...](http://pyfound.blogspot.com/2020/04/sponsoring-python-
packaging.html)

Discussion here:
[https://news.ycombinator.com/item?id=22772410](https://news.ycombinator.com/item?id=22772410)

------
poooood
I quite like the pyproject.toml; it + poetry + dephell is (relatively
speaking) a breeze. Both poetry and dephell are quite good and they don’t not
mesh with conda. Python packaging is awful but it’s gotten way better; it’s
still maddening but it’s better.

