
Poetry – Python dependency management and packaging - reedwolf
https://python-poetry.org/
======
zelphirkalt
Poetry still has to fight with bugs, as you can see looking at the git
repository and the issues there. However, in most cases it works fine for me.

Dependency resolution might be a hard problem to solve concurrently, but it
would be great, if poetry did not only use a single core, to speed up the
process.

------
kelsolaar
Poetry is awesome, it has its quirks but really simplified our work at
[https://www.colour-science.org/](https://www.colour-science.org/). We have
dropped Conda entirely as it solved all our previous Pip dependency hell that
ended up with the adoption of Conda.

As I'm writing that we are coincidentally and unfortunately plagued by random
CI failures occurring when Poetry resolves the dependencies:
[https://github.com/actions/virtual-
environments/issues/1343](https://github.com/actions/virtual-
environments/issues/1343)

~~~
wdroz
Conda let you install non python stuffs like cudatoolkit and cudnn. Can you do
the same with Poetry?

~~~
kelsolaar
AFAIK, no, Poetry deals with Pypi, Github and urls to Python packages:
[https://python-poetry.org/docs/dependency-specification/](https://python-
poetry.org/docs/dependency-specification/)

------
Animats
Another Python packaging system? I thought we were done when "wheels" replaced
"eggs". ("Eggs", of course, replaced "setup.py" files.)

~~~
geofft
Poetry is a tool to install packages, in whatever form they're in. It's
roughly in the same space as pip/easy_install or virtualenvwrapper, but it's
really a higher-level tool than anything that is currently standard, which is
what makes it worthwhile. (It's in the same space as Cargo, if that helps.)

Wheels and eggs are packaging formats, which Poetry (and pip) installs. You
can think of poetry as like yum/dnf or apt and wheels/eggs as like .rpm or
.deb files. (You can think of pip as like the rpm or dpkg command, in the
sense that it's a lower-level tool but it also installs wheels/eggs, but that
analogy isn't perfect.) The existence of Poetry does not obsolete either
wheels or eggs.

Neither wheels nor eggs replaced setup.py files - the standard way of building
a wheel is to run "setup.py bdist_wheel". You can think of setup.py as like
Makefile. You would not say that tarballs have replaced Makefiles.

~~~
elcomet
Poetry also builds your package to create the wheel file. So it's more than
pip. Check the examples on the websites to learn more about it

~~~
geofft
Yup - I'm familiar with it, I'm just trying to come up with a loose analogy
for someone who isn't familiar with Python packaging but has heard things
second-hand.

------
Caligatio
I'm a big fan of Poetry (I have a few commits in there) but it's not without
its downsides.

At least a few months ago, you needed to install an entire compilation suite
if you wanted to install a pure-Python source distribution (sdist) that was
created with Poetry on Alpine. Somewhat by design, sdists created with Poetry
require Poetry to install. Poetry itself is dependent on the cryptography
package which, like all like all packages that use native-C extensions,
suffers from a lack of musl-compatible wheels.

~~~
geofft
Hmm, this is solvable in theory by defining a platform tag for musl such that
authors of C extensions could upload musl wheels, right?

I see some discussion at
[https://github.com/pypa/manylinux/issues/37](https://github.com/pypa/manylinux/issues/37)
but I don't totally follow why it died off.

~~~
corndoge
The pypa maintainers, in my experience, will engage with such issues but
ultimately discussion dies off and they simply ignore corner cases.
Realistically speaking if you want to ship impure wheels for anything outside
of ppc(64le), armhf, x86 or amd64 you're SOL. Python packaging is an absolute
shit show especially where impure wheels are involved. Between the manylinux
"standards" that require downloading random Centos docker images because their
libc is "probably old enough" to the setuptools vs distutils vs whatever else
with none of them having actually mature support for binary extensions (and
even less - really zero, and no docs - for raw .so's via ctypes) to the
byzantine PyPI upload procedures with its underspecified platform tags (really
a manylinux problem though), my Python packaging experience was so bad that I
ultimately gave up on the Python ecosystem as a whole. Shipping applications
and libraries with this language is simply awful to the point where I'd rather
use a compiled language because it is easier to build binaries for 10
different arches under qemu than it is to ship a fucking wheel. I'm so over
Python.

~~~
aaronbrethorst
I have absolutely no idea what you are talking about, and it makes me so happy
that I blundered into Ruby in 2007.

For comparison, with Ruby you use RVM or Rbenv plus a Gemfile with Bundler and
you’re done. There’s no debate to speak of.

The seemingly equivalent nightmare that I see in JavaScript inspires me to go
use Ember if only because its creator was instrumental in forming the modern
Ruby tooling ecosystem, and I hear that ember is equally unlikely to make me
want to stab my forehead with a fork in frustration.

~~~
sirn
RubyGem doesn't do binary distribution of C extension at all. Gem with C
extension is usually built at install time (via extconf/mkmf) which side-
stepped the whole shipping .so for 10 different arches issue but it create few
other issues of requiring build tool to be installed on production server to
install a Gem (so you're back to the Python's manylinux issue, but applies to
all arches instead of only edge cases), and any Gem that need to use pre-built
.so need to do it themselves (including differentiating between arches).

~~~
viraptor
> RubyGem doesn't do binary distribution of C extension at all

[https://github.com/rubyjs/libv8#do-i-get-a-
binary](https://github.com/rubyjs/libv8#do-i-get-a-binary) would like a word
about that. You get source or binary depending on your platform / version.

~~~
sirn
I was not aware of platform support (probably because two of my systems are
FreeBSD and linux-musl, which nobody made a binary dist for). I stand
corrected. Thank you.

------
ris
> With x, packaging in Python is a solved problem

This is the claim that has been made about every "advancement" in python
packaging for the last 20 years. Some of us have been kicking around long
enough to remember the great saviour being distutils, setuptools,
easy_install, eggs, wheels, pip, pipfiles/pip-tools... The truth being that
all people have managed to do is produce a horrific mess with an ever growing
tower of components each of which never _really_ solves the problem (and now
people want me to add "tox" to the mix too!). I have to say that packaging is
really the worst part of the python world.

I do my best these days to avoid every component of the python packaging
ecosystem I possibly can and use Nix as much as possible, which really Just
Solves The Problem.

~~~
drcongo
I agree that packaging is horrible in Python. Poetry is mostly excellent, but
I have one project where running `poetry lock` takes over an hour and I have
no idea why. I keep seeing this claim about Nix, and I keep trying to
understand it, but every time I look into it I come out more confused. Nix
appears to be an operating system (?) and packaging things _for_ Nix seems to
be something akin to a dockerfile, but one that will only work on a machine
with the Nix operating system? If I've got all of that right (which is a big
"if"), I can see how that's useful for building a system's dependencies and
deploying it to a Nix OS server, but I can never work out how any of it is
useful if I don't run Nix OS on the servers or I want to distribute a Python
package.

~~~
ris
So... Nix is just a package manager, but it's a package manager that tries to
be as generic as possible and only have the attitude that a "package" is just
"a bunch of (immutable) files that depend on the presence of another bunch of
files" and so should be able to take over the job of _all_ package mangers on
your system (no matter how much of a special snowflake language x or
application y thinks it is and deserves its own, inevitably crappy, package
manager). One of the many advantages this brings is that your system's
software installation makes sense "as a whole" and isn't the result of 7
different package managers or "auto updating" applications fighting each
other.

But because of this, Nix can only have things depend on packages it has
provided itself. It doesn't assume anything about the host system apart from
the kernel. So yes, installing Nix on an existing Linux system can feel like
installing another distribution. But you only ever have to use as much of it
as you want to. Everything lives in /nix, and to get rid of everything you
just have to `rm -rf /nix`.

NixOS _is_ a full distribution that just uses Nix for everything, including
configuration management.

------
emmanueloga_
I'm a complete outsider to the python ecosystem, from time to time I need to
write a script or use a popular python library like pygments and I used to
hate having to do anything with python because of how difficult it seemed to
install anything (cognitive overload of having 20 different ways to install
things...).

Anyway, recently I discovered miniconda and it seems like the best, most
unambiguous way to install python. It is small (in rel terms...), it works
like a charm on Windows, and I can just do something run a single command like
`conda install pygments` and I'm all set!

Just not sure why I would ever need to look into something like poetry,
although I heard conda and poetry can complement each other somehow? (as do
pip, easy_install or whatever is called and all the others haha :-p).

~~~
Spiritus
> [...] from time to time I need to write a script or use a popular python
> library like pygments and I used to hate having to do anything with python
> because of how difficult it seemed to install anything

Not sure I understand why you're having so much trouble. It's literally two
commands:

    
    
        $ python3 -m venv venv  # Create virtualenv
        $ ./venv/bin/pip install pygments
    

However! One unfortunate thing with this is that `python3 -m venv` doesn't
include the 'wheel' package in the virtualenv out-of-the-box. So it's often a
good idea to install/upgrade pip, setuptools and wheel after creating the
virtualenv:

    
    
        $ ./venv/bin/pip install -U pip setuptools wheel

~~~
emmanueloga_
"Not sure I understands why you're having so much trouble" ... proceeds to
explain a procedure that involves understanding various moving parts,
including virtualenvs, pip, setuptools, wheel, etc...

Even if I'm not gonna get deep into python, I like to feel like I understand
what's going on. With python it just involves a lot more effort than with
other ecosystems, even if you think that these things are easy to learn.

Compare with ruby: `gem install the-gem`. Done. No "virtual envs", no plethora
of similar but different tools to do slightly the same thing / whatever, no
preconditions, no step by step procedures. Simplicity is important!

~~~
ElliotH
My experience of Ruby, which is admittedly a few years old, is that people did
use multiple Ruby environments through e.g rbenv to manage different sets of
gem versions, which is what "virtual envs" are the equivalent of.

Now, I do agree the Python tooling situation is kind of a mess, but there you
can still do:

"pip install the-package"

And you're just as good, all the other tools are when you're trying to do
stuff harder like:

\- Maintain multiple development environments \- Publish new packages \- Build
binary packages \- Handle Windows

I think on Ruby you need tools beyond "gem install" to do those as well. The
point is how we achieve simplicity while keeping the ability to do those
things.

------
abalaji
Poetry is fantastic but it looks like pip is trying to create their own
competitor. [1] I would much prefer if the pypa folks and the poetry folks got
together to more tightly integrate the systems. Poetry solves a ton of
problems ranging from venv management to making packaging easier to splitting
dev and production dependencies. Consider me a huge fan.

[1]
[https://github.com/pypa/pip/issues/8511#issuecomment-6651402...](https://github.com/pypa/pip/issues/8511#issuecomment-665140265)

------
nurettin
Would have been nice if poetry shell loaded .env

I got used to that workflow with pipenv.

~~~
rajasimon
Arguing that doesn't help [https://github.com/python-
poetry/poetry/issues/337#issuecomm...](https://github.com/python-
poetry/poetry/issues/337#issuecomment-601945357)

~~~
nurettin
This seems nice, I'm down with that. Still would have been nicer if poetry had
it.

[https://github.com/python-
poetry/poetry/issues/337#issuecomm...](https://github.com/python-
poetry/poetry/issues/337#issuecomment-660549880)

------
ai_ja_nai
What's the advantage over pipenv?

~~~
zaro
Are there any advantages with pupenv at all? Pipenv definitely takes the crown
in my list of Package managers that are plain stupid.

~~~
ai_ja_nai
Well, it was the first locking package manager for Python. A definite edge
over plain pip. It always worked without issues for me. What's so bad about
it?

~~~
sirn
Pipenv tries to do so much (e.g. a pipenv command involve poking around procfs
to detect shell via shellingham to display colors), used to introduce breaking
changes between releases (before they switched the scheme to CalVer), and
takes a really long time between each releases (longer for bugs to get fixed).

Most people that switched from Pipenv to Poetry were probably done so during
the end of 2018 throughout 2019, of which Pipenv has no releases at all[1]
(non-alpha/beta version were only released in November 2018 then May 2020).

There were a rant from few years ago[2][3] regarding other issues
(performance, etc.).

[1]:
[https://news.ycombinator.com/item?id=21781421](https://news.ycombinator.com/item?id=21781421)

[2]: [https://chriswarrick.com/blog/2018/07/17/pipenv-promises-
a-l...](https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-
delivers-very-little/)

[3]:
[https://news.ycombinator.com/item?id=18612590](https://news.ycombinator.com/item?id=18612590)

------
turbinerneiter
I tried it and liked it, really simple to use, but:

* how do I use it to install i.e. service files, config files etc.? * can I use it to generate Deb/RPM packages?

I'm not sure if packaging applications is as much of goal as packaging
libraries, but I felt that it was missing stuff to do the former.

~~~
makeworld
It's not really for that. It can make Python packages for PyPI and manage deps
and that's it. Building final deployment packages like debs are out of scope.

------
dmarchand90
Previous discussion:
[https://news.ycombinator.com/item?id=20672436](https://news.ycombinator.com/item?id=20672436)

------
softinio
I tried it for the first time couple of months ago and I loved it, its my
preferred way of managing python packages given the choice. Would love to see
greater adoption.

------
merricksb
Discussed here 7 months ago:

[https://news.ycombinator.com/item?id=21779191](https://news.ycombinator.com/item?id=21779191)

------
dagenix
Poetry is far and away the most interesting solution when its comes to
managing Python dependencies.

~~~
ratherlongname
Care to elaborate for us muggles

------
midrus
[https://xkcd.com/927/](https://xkcd.com/927/)

Finally! A standard for managing dependencies in Python.

------
jayaram
I ended up with PyScaffold and never turned back again!

------
ianamartin
I'm going to sound like a very grumpy old man—because that's what I am—but if
you actually need a tool like poetry, you've really fucked up.

I have a ton of respect for the author. It's really good code and solves a
hard problem in elegant ways. But it's a problem that we shouldn't let
ourselves have.

It's like a patient talking to a therapist:

patient: I'm really depressed.

therapist: Why do you think that is?

p: Well, I've been having an affair with this woman, and my wife found out
about it.

t: How does she feel about that?

p: She's pretty angry at me. It's affecting our relationship.

t: How is it affecting your relationship?

p: Well, she doesn't want to have sex with me anymore, and things have gotten
a little weird with the kids.

t: How do you feel about not having sex with your wife?

p: It's depressing.

t: And the kids? How do you feel about them?

p: They'll grow up and understand eventually.

t: Here's a pill you can take every day that will make you feel better.

There are two ways to address this kind of situation. The first way is to give
the patient a pill to fix the symptom of depression. The second way is to fix
the behavior that's causing the depression.

Poetry (and Cargo and other package/dependency managers) are a little pill you
can take to make you feel better about stuff.

But there's another school of therapy. I'm maybe going to sound a little like
Zed Shaw here, but there's the "Don't fucking do that" school of therapy.

It's possible to fix the underlying behavior that's causing the pain instead
of taking a pill. It's harder and requires more work, sure. But in the long
term it is a better, more stable solution.

Since I'm already on a bit of a rant, I'll go ahead and say it out loud: agile
is the source of many of these types of problems. You start out with good
intentions and then one day you end up married to a thing that was only
supposed to be a proof-of-concept, but it got shipped because product team and
velocity, and now your life is hell, and that POC now has kids, and you're
legally responsible for them, and fuck it, just give me a pill, doctor.

Poetry is a brilliant solution to a problem we shouldn't create for ourselves.

~~~
LittlePeter
Can you give concrete examples how you would do <insert some scenario> without
poetry?

Examples of what I really have messed up if I have to use poetry would be also
nice. I have trouble coming up with concrete examples myself.

~~~
cturner
This is the forth-is-my-chisel vs node-js-is-my-glue-gun worldview collision.
It's bigger than vi-or-emacs, it's older than functional-vs-procedural.
There's no bridging it with contextual examples.

~~~
ianamartin
This is true, and I don't object to this description. Like I said, I'm a
grumpy old man.

~~~
cturner
Me too, loved your post.

~~~
ianamartin
Thank you.

------
kissgyorgy
I agree. Never had any problem with it so far.

