Hacker News new | comments | show | ask | jobs | submit login
Pipenv: promises a lot, delivers very little (chriswarrick.com)
271 points by BerislavLopac 5 days ago | hide | past | web | favorite | 215 comments





Pipenv is a really interesting development for Python, and I'm glad that someone was working to improve dependency locking for Python.

However, Kenneth abused his position with PyPA (and quickly bumped a what is a beta product to version 18) to imply Pipenv was more stable, more supported and more official than it really was.

And worse still, for anyone saying "but ts open source, you get what you pay for", Kenneth as former Python Overlord at Heroku, encouraged Heroku to place Pipenv above Pip as the default Python package manager in the Python buildpack. This decision impacted paying customers and the Python buildpack used a broken version of Pipenv for a long time. So long, most people I know just went back to Pip.

Then, lastly, when people complained he had a tizzy at reddit and twitter and got PyPA to help backtrack and say "no we didn't support it, nope, its just a thing that happened", all while the main Pipenv Github repository was held under the PyPA GitHub Org.


Sometimes, improvements don't happen in a straight line.

There's been a lot of work on Pipenv over the the last 6 months, predominantly by Dan Ryan and Tzu-Ping Chung, and it's getting stronger and stronger with each release.

If you've gone back to using pip I'd encourage you to give Pipenv another try. Introducing a lockfile is a big step forward for Python dependency management, and the team working on Pipenv are committed and doing a great job.


> Sometimes, improvements don't happen in a straight line.

I don't deny that, what I am (and the article is) saying is that we were sold on Pipenv being "the officially recommended Python packaging tool from Python.org".

And PyPA didn't refute it, and Heroku didn't refute it, so the community bought it.

Yes, introducing a Lockfile is huge, and it was massively needed, and thats why when we were told "heres the official way to do it", we got excited. Then we got daily breaking updates, rude issue close messages, and a giant backtrack of "its free and still under development, why do you expect so much from us?"


Another reason Heroku didn't refute Pipenv is that Pipenv's creator leads Python related things at Heroku.

Not anymore, I don’t believe.

Pipenv tries to upgrade all the versions of everything in your lockfile whenever you add a new package (not just dependencies of the package), and there's no way to disable this behavior. Right now. That's the tip of the iceberg.

You might want to check out https://github.com/pypa/pipenv/pull/3304, which Dan is actively working on to solve that.

Poor control over this sort of behavior makes Cocoapods very frustrating to deal with.

> Sometimes, improvements don't happen in a straight line.

I don't think that the parent disagreed with that. The point, as I understood it, was that this beta stage improvement was marketed as being ready. IOW, if pipenv was not Kenneth's project, it likely would have evolved in, to use your phrase, a straighter line.


Pipenv is designed solely for packaging applications, per the maintainers' admission. It's not suited for libraries and is not designed to be. If you want to replace pip, you should have a look at Poetry.

> It's not suited for libraries and is not designed to be.

That's kind of the problem. Why on earth are libraries and apps getting a different treatment? The JS ecosystem manages to have one tool for apps and libraries. One too for installing and publishing. All of it with lockfile support, workspace/"virtualenv" support, etc. And somehow, it's not confusing.

Adding one more tool to the stack is a really funky step forward. Yes, it brings lockfiles. Cool, although we already kind of had those (--freeze). Packaging in Python is a mess and I'm more and more in the camp that as long as Pipenv keeps flouting itself as "the better solution", all the while not covering all the basic use cases, we've gone backwards and are in even more of a mess.


And it isn't only Javascript that manages to have a single tool for this. Clojure has a single tool. Java has a single tool. Rust has a single tool.

This is a solved problem across a variety of popular and mainstream programming languages. I don't mean to suggest that the problem isn't complicated, but this isn't a problem that doesn't have a wealth of previously written solutions to look at for inspiration.


What's the "one tool" for Java? Is it Maven? Ivy? sbt? Gradle? You may say some of those are "build tools" rather than dependency tools, but I don't see how it's different than what we are discussing in Python.

My Java app declares its dependencies in a build.sbt file using Scala syntax and has them cached in an Ivy directory. Yours declares them in a pom file using XML syntax and has them cached in a Maven directory. Neither of us even tries to do lockfiles and instead just has the CI server build an "uber jar" with all the dependencies bundled in.


We don't need lockfiles in Java land because we generally use version ranges very carefully and rely on package developers following semver - and we certainly don't use "just whatever the latest release is, dude" like shown in the example Pipfile: https://pipenv.readthedocs.io/en/latest/basics/#example-pipf... (the asterisks)

I get why Python needs lockfiles, but goddamn, that need is a symptom of the mess of managing Python dependencies.

There's still a long way to go - I use Airflow for ETL management, and I'm using pipenv to manage it - except Pipenv can't create a lockfile because a dependency of a dependency of Airflow requires in flask-wtf <= 0.7, while another dependency of a dependency requires ==0.8

In a Maven pom.xml I can easily exclude or override the conflicting dependency manually if needed, but I can't in a Pipfile.


rely on package developers following semver

Well, that seems naive. Almost 1/4th of Maven Central libraries broke binary compatibility in patch updates: https://avandeursen.com/2014/10/09/semantic-versioning-in-ma...

Netflix, at least, doesn't agree that Java doesn't need lockfiles: https://github.com/nebula-plugins/gradle-dependency-lock-plu...


> We don't need lockfiles in Java land because we generally use version ranges very carefully

Yes, which is part of the reason why lots of Java projects almost never update their dependencies, because no one remembers why that version was chosen. Splitting what you want from what you have is important to communicate this.

> There's still a long way to go - I use Airflow for ETL management, and I'm using pipenv to manage it - except Pipenv can't create a lockfile because a dependency of a dependency of Airflow requires in flask-wtf <= 0.7, while another dependency of a dependency requires ==0.8

I'm not sure what you expect it would be able to do here...


Enforce Flask-wtf 0.8 and see if it works.

With my very small sample size, maven and gradle's lack of version locking just means it gets tacked on in terrible ways when needed.

People use sbt for Java? Thats’s...surprising.

Most shops I know use one of maven or gradle. I can’t think of any other serious contenders in Java (if you still use ant: please consider alternatives).


Right, exactly! It bugs me that we have a wealth of examples and could have just said: "Look. Here's yarn. Do that, but for Python. Don't try to be too creative."

Poetry looks like it did just that though and I'm warming up to it at a very high speed.


Clojure has two tools.. designed to be used as the single tool though.

Boot, and Lein. I'm partial to boot lately.


libraries and apps _should_ get different treatment. A library often has to coexist with other unknown code (open world assumption), an application you're controlling is effectively existing in one specific universe: the one you define (closed world assumption).

You wouldn't write your application to support 5 versions of Django, but you _probably_ would do so for a library.

That said, I do basically agree about `pip` existing already. We could have built a tool to manipulate `requirements.txt` files instead of introducing another format and a toolchain that is _much_ slower and brittler. Though ultimately at this point Python packaging woes feel like they are at a much lower level (the fact that libraries end up being "installed" mean that preparing all your dependencies to go out to multiple servers is a mess).


An internal application can exist in one specific universe. Most of the time.

The "library" workflow works for applications too. Put your direct dependencies in setup.py. Build wheels of everything and upload them to an internal PyPI server. Pin everything in requirements.txt.


> We could have built a tool to manipulate `requirements.txt` files

There's pip-compile from https://pypi.org/project/pip-tools/ that does exactly that. Pipenv uses its resolving mechanism if I'm not mistaken. It produces standard req.txt file with versions pinned and supports separate dev requirements. It had some bug with upgrades last time I checked though, not sure whether it's resolved, currently considering using it for projects at work.


Pipenv keeps flouting itself

That would be weird. Perhaps you mean 'flaunting'?


Yes, typo :)

I already managed my python deps with a lockfile - requirements.txt and requirements-frozen.txt, which about three lines of shell script took care of for me. From the article, it doesn't sound like pipenv buys me much on top of that.

It's a "standard" way to do those scripts. It gives you the NodeJS + NPM way's "npm i", by creating a virtualenv and pip installing into it. It helps with specifying prod and dev dependencies too - if I remember correctly.

I just switched away today. It still sucks

We’re giving it an honest try, but we still get burned regularly. I’m pretty disappointed.

> Introducing a lockfile is a big step forward for Python dependency management

Huh? I'm not really familiar with the state of dependency management for Python/dynamic languages but... there's much more out there beyond just lockfiles. I'm a bit appalled Python is so far behind.


Much more for what? In this context, "lockfile" means a file listing all your project dependencies with exact versions pinned. (Not semaphore-files or anything like that. It's not even a python-specific term, e.g. npm uses package-lock.json for the same purpose.) The need for something to store all dependencies with exact versions pinned exists in any language and infrastructure, are there any better solutions than store them in a file? Files are nice, they are VCS-friendly and everything.

> Much more for what?

Much more in the field of build tooling/package management. Pinning versions is fine, but dependency resolution is another legitimate choice.


I'm very glad we have the wheel and ensurepip now.

Yet, I think PyPA has not been taking the best decisions regarding Python packaging.

Your Kenneth story is not the only "weird event" in their history.

E.G:

Did you know that we don't need "pyproject.toml" at all ? That there is already a production ready plain text standard to replace setup.py ?

Did you know that this standard has been perfectly working for TWO YEARS with regular setuptools, is incredibly simple to use and completly compatible with the standard "setup.py stuff" workflow (and hence the whole legacy tool stack) ?

Yep. And nobody talks about it.

Let me (re)introduce...

Setup.cfg !

Oh, I know... Most people believes it's a useless file.

After all, the Python documentation seldom states to use it and for only one tiny option:

https://docs.python.org/3.7/distutils/configfile.html

But no. Setup.cfg is awesome !!

Put one line in setup.py:

    import setuptools; setuptools.setup()
And you can now put all the rest - yes, everything - in setup.cfg. It's perfectly documented:

https://setuptools.readthedocs.io/en/latest/setuptools.html#...

Not only it has been working since 2016, but it has fantastic goodies:

    version = attr: src.__version__ 
will include the version from __init__.py

     "options.data_files" 
replaces the MANIFEST

     "license =  file: LICENCE.txt" 
loads the licence from any text file

Try it, it just works. And you can "python setup.py sdist upload" as usual with it. You don't need any new tool.

Now why did the PyPA decide to forget about this and create a new format ? The short explanation in PEP 518 is a bad joke. And why does nobody talks about it ?

When I asked the PyPA, they told me they were too invested in the new project to stop now. I don't like this answer at all: we suffered enough with python packaging during the "distutils, eggs, etc" fiasco.

setup.cfg works. It works now. It's nice. It's compatible. It does what we need.

Use it. Talk about it. Write about it.

Make sure a lot of people knows so that tool makers and PyPA finally acknowledge that there is not need for the XKCD comics about standard to be true again.


I love setup.cfg, I've used it for years and indeed pyproject.toml is useless given that setup.cfg has existed for frickin ever at this point (and is supported by a multitude of tools). TOML is nice but its support isn't even in the stdlib which makes it very awkward to use for a core file like that.

Here are some examples of my libs/apps using it in the real world, if someone needs references for how to use setup.cfg with an empty or near-empty setup.py:

https://github.com/jleclanche/python-bna/blob/master/setup.c...

https://github.com/dj-stripe/dj-stripe/blob/master/setup.cfg

https://github.com/jazzband/django-oauth-toolkit/blob/master...

https://github.com/jazzband/django-push-notifications/blob/m...

Edit: I see you mention attr: src.__version__. I personally prefer doing it the other way around, with the version defined in setup.cfg and a pkg_resources snippet in __init__.py (https://github.com/HearthSim/python-hearthstone/blob/master/...).

To be honest I wish __version__ were automatically defined like that (but more reliably). Do you know if this was discussed in a PEP?


I don't understand the pyproject.toml hate. pyproject.toml exists specifically so you can specify your build system. Without it, you are basically forced to use setuptools/distutils as is currently the case. Hence, pyproject.toml and setup.cfg aren't at all in conflict.

- there is nothing you can do with pyproject.toml that you can't with setup.cfg. E.G: you are not forced to use setuptools to use setup.cfg. Any tool supporting pyproject.toml could support setup.cfg as easily, since it's a documented plain text format. It's a political decision.

- there are things you can't do with pyproject.toml you can with setup.cfg. E.G: you can't use pyproject.toml with legacy tools, or with just a fresh python setup. This may change in the future, but would requires a lot of effort because changing setuptools is a very tedious process.

- resources (time, people, money, documentation, public attention, communication, etc) invested in creating and supporting pyproject.toml could be invested in improving setup.cfg and its ecosystem support. E.G: Why does poetry support pyproject.toml and not setup.cfg ? No technical reason. Why does nobody knows this easy way to package python libs ? No technical reason reason either.

So not only the new format brings nothing on the table, but it is also a setback, AND add clutter to a situation that was just begining to be solved. It's not just poor engineering, it's poor manners really.

I've been coding in Python for 15 years. I've lived this: https://stackoverflow.com/a/14753678

Stop the pain.


The format of setup.cfg is whatever configparser accepts, which is different in different versions of Python.

First, it managed to work fine for setuptools for 2 years accross all recent python versions. That's because the differences are minor and edge cases. Try to use pyproject.toml in most CI toolchain just for fun...

Second, the format of setup.cfg is defined in a documentation already, so there is a reference outside of configparser. Yes, the low level format is not explicitly defined (although it is implicitly): so let's do define it instead of creating a new one.

Third, it's still a much easier and saner task to rafine the definition of setup.cfg than to create a new standard. I don't even understand how this is controversial, espacially among people in the computing world, where we had those kind of problems for decades and we know the pros and cons, and consequence of this.

The "I add my little format because it's pure and better and current things suck" falacy is such a stereotype we should all be able to recognize it from miles away from nowaday.


Using pyproject.toml just requires a recent version of pip. Which CI tools can't handle that?

I wouldn't call comments an edge case. The distutils documentation has a definition for comments, but I think it actually just uses configparser. setuptools just uses configparser. The pbr documentation has a slightly different definition, but I wouldn't be surprised if it just uses configparser too.

They also have different definitions of non-string types.

Even if you call those edge cases, do you think a PEP that turned edge cases into silent errors would be approved?


> Using pyproject.toml just requires a recent version of pip. Which CI tools can't handle that?

The most used one in the world: https://github.com/travis-ci/dpl/issues/822

Also the last time I used tox, anything complex didn't work either.

> Even if you call those edge cases, do you think a PEP that turned edge cases into silent errors would be approved?

Well the current PEP decided to turn a packaging situation that was stable into one that was not, again, after 15 years of mess with many versions of things. So you tell me.

Check the usage stats I posted in an other comment to see the problem.

Besides, yes, we do make compromise on best practices to allow peaceful transition all the time in Python. `async/await` allowing to be a variable silently. Non utf8 defaut encoding argument in open() in windows. Then... we fix it later.

Because I think you conveniently skip a lot of things I wrote in my comments. I clearly state that we would and should consider setup.cfg as a version 1 of the format. Then we would increment on that. I gave a detailed procedure on one way to do that, and there are others.

The point is, all your concerned can be addressed with a progressive transition, starting from setup.cfg. Actually we could even end up with a toml format in setup.cfg, __on the long run__, that matches exactly the current one.

While you addressed non of ours concerns. Just reject them. No will to even recognize there is a problem. It's insulting, really.

We did that during the 2/3 transition. Didn't work so well, did it ?


> we should all be able to recognize it from miles away from nowaday.

Oh but we do. Then we rationalize it away, because "this time...". Like we do for Big Rewrites.

It might have something to do with the fact that programming is mostly a craft you learn by doing it, so we overvalue "doing it again" because that's how we usually get better.


They're not in conflict. The problem is pyproject.toml is superfluous. See my comment here: https://news.ycombinator.com/item?id=18614058

I like the way you do it too, and I don't think it needs any improvements besides being shared more.

  > Use it. Talk about it. Write about it.
I have a project that converts basic setup.py files to setup.cfg files [1].

Still happily using plain setuptools for library development and pip-tools for application development.

[1]: https://github.com/gvalkov/setuptools-py2cfg


Very cool. I'll share that.

setup.cfg lets you configure setuptools with declarative syntax. pyproject.toml lets you replace setuptools with something else. The PEP explains why they didn't just reuse setup.cfg.[1]

[1] https://www.python.org/dev/peps/pep-0518/#sticking-with-setu...


The problem is that this is yet another boilerplate file.

The reasoning is good, but we were just arriving to the point that every Python tool out there is either compatible with tox.ini, setup.cfg, or both (much like the JS ecosystem has tools reading from package.json).

Now we have both Pipfile and pyproject.toml on top of it!

For a language that prides itself on its stability and backwards compatibility (especially when compared to the JS ecosystem), we churn through boilerplate files harder than Google churns through instant messaging apps.


> The point of pyproject.toml is to allow other libraries to replace setuptools.

This can be done with setup.cfg. Setuptools is only a backend supporting it. You can create other ones. Poetry and pipenv could support it in a week in their authors decided so.

> The reasons for not using setup.cfg are explained in the PEP.[1]

Those are not reasons, those are excuses. Let me quote it:

>> There are two issues with setup.cfg used by setuptools as a general format. One is that they are .ini files which have issues as mentioned in the configparser discussion above.

Not only setup.cfg does the job with the current limitations of the ini format (while pyproject.toml still doesn't with its fancy one), but python projects are not so complex they require such a rich content.

Besides, nothing prevent PyPA to says that setup.cfg format now has a version header, with the current setup.cfg being headerless version 1, then make the header mandatory for version 2 and increments it to move toward TOML if we ever reach a limitation. That's how formats grow everywhere else in the world.

>> The other is that the schema for that file has never been rigorously defined and thus it's unknown which format would be safe to use going forward without potentially confusing setuptools installations.

That's incredibly dishonest, since I gave a link to a complete documentation of the format in my previous post. Besides, it's better to actually refine the existing standard if you ever find it lacking than recreating one from scratch. While there are good reasons to do so, the later is rarely a rational engineering decision, and most often driven by ego.

>> While keeping with traditional thanks to setup.py, it does not necessarily match what the file may contain in the future

So ? How is that a problem ? A standard is not meant to be set in stone. It evolves. But it can't do so if everytime one has an itch, one reinvents the wheel.


The "existing standard" is whatever configparser accepts. configparser is part of the standard library. Different versions accept different things. The setuptools documentation covers higher-level things like key names. That isn't what they're trying to standardize.

The last sentence you quoted explains why they picked "pyproject" instead of "setup". It isn't why they picked TOML.


See my answer to your other comment.

Also "higher-level things like key names" is half of the standard.

Besides, picking a new (even if better) serialization format is not good reason to create a whole new standard with names, convention, tooling, etc., as explained earlier.

There are sane ways to make the existing system evolves and improves incrementally, using the legacy standards that benefits from the existing situation, and allow the improvements from the new one. All that without the madness of messing with the entire community once again after 15 years of instable package management.

Yeah it's less sexy that having to create your new baby, yes it's less fun than using that new shinny format (and I say that while I __love__ TOML), and less it's less satisfying than having your name as the creator of a whole new jesus-format-saver. But that's the mature and professional things to do.


Nothing from setuptools is being standardized. Tools can use any keys they want in any format they want in any file they want.

The "legacy standards" are subtly incompatible INI dialects that people recently started putting into the same files. The incompatibilities mostly don't matter because most tools just read their own sections. They do matter if you want to standardize them.

The only new tooling for TOML is a small library. A new INI dialect would need one too.


> Nothing from setuptools is being standardized. Tools can use any keys they want in any format they want in any file they want.

No, if you use any key, it won't work with setuptools.setup(), and just like a python code that doesn't run with cPython will never be popular, it will not be used.

Also, if you look at how poetry use pyproject.tml, they just create a custom section. So basically, they don't use your standard.

> The "legacy standards" are subtly incompatible INI dialects that people recently started putting into the same files. The incompatibilities mostly don't matter because most tools just read their own sections. They do matter if you want to standardize them.

That's kinda my point for comments and comments. Standardize the status quo, then increment from that. Not sexy. Not pure. Welcome to the real life.

Didn't you learn anything from the distutils/distribute/setuptool mess ? From the Python 2 / Python 3 breakage ?

And could you address any of my concerns instead of just attacking ? Because I'm trying to address yours with solutions. You just write short busts of "no, it's bad, we are good". That's not really giving me trust in your decisions, and it __lowers__ my confidence in pyproject.toml because the people defending it basically are not behaving like engineers trying to solve a problem, but as salesmen trying to only defend their product.

> The only new tooling for TOML is a small library. A new INI dialect would need one too.

But but we can start from a standard that works now, is used already, and is compatible with existing stacks. Instead of arriving with the theorical untested, incompatible best thing that add a layer on top of the mess.


Because I hate the "this ship has sailed" argument about pyproject.toml, here are some github usage stats:

- setup.py: 1,259,007 results (https://github.com/search?q=filename%3Asetup.py)

- setup.cfg: 165,716 results (https://github.com/search?q=filename%3Asetup.cfg)

- pyproject.toml: 2,137 results (https://github.com/search?q=filename%3Apyproject.toml)

Also, remember that setup.cfg is completly compatible with setup.py, the migration is painless. All the legacy tools work. Not the case with pyproject.toml.


This is interesting. Somehow I missed this, using setup.cfg make much more sense than putting everything into python code. I wonder if pip-tools will allow compiling install_requires and test_requires in setup.cfg into nice requirements.txt and dev-requirements.txt without too much magic...

Your latest blog post say good things about poetry and bad things about pyproject.toml.

However the default in poetry seems to be pyproject.toml... I'm confused.


Because I'm not dogmatic, I can regognize good tools, even if I disagree with the underlying political decision.

Computing is not black and white, and perfect purity is only nice in "fizz buzz".

Now to be extra fun, poetry uses a custom section ([tool.poetry]) in pyproject.toml, not really the standard itself. What does that say about this format ?


oh, nevermind... I'll stick to manual venv/pip:

  ptest poetry add requests
                                                                                 
[UnicodeDecodeError] 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)

Author of Poetry here!

I've never seen that error before.

Which version of Python do you use?

And feel free to create an issue on the issue tracker: https://github.com/sdispater/poetry/issues


pyproject debates aside, I love your work. I think poetry is a beautiful piece of software, the source code is very easy to read and the 2/3 compat is well done (although I would not use assert to check things).

Power tends to corrupt.

I was sorely disappointed with pipenv, and transitioned to poetry [1], with which I’ve been very satisfied. There is also some commentary in the README on the design decisions re: pipenv [3]. Contrary to the author's perspective on poetry using poetry-specific sections of pyproject.toml, that's actually the proper implementation (and expected usage) coming out of PEP-518.

I also am a big fan of pyenv [3] but that’s of course to manage Python versions (not environments)

[1] https://github.com/sdispater/poetry

[2] https://github.com/sdispater/poetry#what-about-pipenv

[3] https://github.com/pyenv/pyenv


Poetry is one of the few things that gives me hope about the mess of python packaging.

It is also great to see the author is very responsive.

My only concern is the lack of integrated "toolchain" management (what versxon of python to use, something like rustup) that is cross platform.


I agree that providing toolchains is very important.

The only non-system package manager that provides Python and its own toolchains - for Linux and macOS presently - which are used to compile every C, C++ and Fortran package, including Python itself is conda and the Anaconda Distribution.

Not doing this leads to static linking and that's inefficient and insecure.

Disclaimer: I work for Anaconda Inc.


> The only non-system package manager that provides Python and its own toolchains - for Linux and macOS presently - which are used to compile every C, C++ and Fortran package, including Python itself is conda and the Anaconda Distribution.

Nix[0] is also perfectly usable without NixOS, and provides all of that, but has far more non-Python libraries and applications packaged. It's also not constantly trying to sell you an enterprise version...

[0]: https://nixos.org/nix/


Great I'll try it out, I always meant to but never got round to it. Does it work on macOS or Windows yet? What's the oldest Linux distro upon which it will run?

Not sure we constantly try to sell our Enterprise product. You could look at it less cynically as we sell an Enterprise product to allow us to provide the Anaconda Distribution for free.


> Does it work on macOS or Windows yet?

It runs fine on macOS. It works on WSL if you disable SQLite's write-ahead log (`echo "use-sqlite-wal = false" > /etc/nix/nix.conf` before installing), but it's much slower than running it on native Linux.

> What's the oldest Linux distro upon which it will run?

It brings its own libraries, so the primary question would be what kernel you use. I haven't verified any specific version, but you'll probably be fine. You might need to disable sandboxing though, since that makes pretty elaborate use of the various namespace systems.


A lot of Nix users seem to use Mac, based on the stuff that comes up on the mailing list (discourse).

There's no "native" Windows support (yet), but I think it might work with some of the UNIX emulations (cygwin, mingw, wsl, etc.)

Not sure what the oldest working Linux version would be. However, NixOS has been around since 2003, so maybe quite old.


conda is decent, but for some reason it seems to live in a weird parallel universe where people not in science have never heard about it.

While I'm generally happy with it, some gripes:

- Using it's own package format with its own repos means that for many (most) projects you can't get all dependencies from conda, but some from pypi as well.

- And it doesn't keep track of which files belong to which package. So package X will happily scribble over files installed by package Y, and vice versa, leading to either X or Y being silently broken depending on the order they were installed in! Argh! I mean, this is something dpkg/rpm/etc. figured out decades ago, it's not rocket science.

- The dependency solver seems a bit weird. Often when upgrading an environment, it will install the same version of a package with another 'build tag', then a few days later if you upgrade again, it will downgrade back to the previous build tag. Not sure if this is the fault of the dependency solver, or whether the problem is in the packages themselves.

- Similarly, there's a lot of mutual incompatibility in the repos. E.g. dependencies on openssl versions prevent upgrading, or require removal of some package etc. I think this is not so much the fault of the conda tool itself, but rather that Anaconda Inc. needs to be more picky wrt packaging policy. Again, Linux distros have been pretty good at this. E.g. https://www.debian.org/doc/debian-policy/ , https://docs.fedoraproject.org/en-US/packaging-guidelines/ .

PS: While I have above mentioned dpkg/rpm as examples to follow, it's not like those formats don't have problems either. https://nixos.org/nix/ and https://www.gnu.org/software/guix/ are perhaps the most prominent examples of 'next generation' packaging systems solving some of the problems of the old-school dpkg/rpm approaches.


sdispater has an open issue for controlling the python version - https://github.com/sdispater/poetry/issues/621 - should make it into 1.0?

As you have experience with pyenv, pipenv and poetry, you might be able to answer: Why not conda?

I've been using conda since ~4 years now, and every single complaint lodged against any of the other package managers was never an issue with conda in the first place. And yet, it seems like there's a SEP field around it and people just ignore its existence?

In this thread, for the first time, I've seen someone mention that you might have problem porting a conda env from a Mac to Linux - never had that problem myself, but I guess it's possible; But that's easily solvable, and certainly does not require a new package manager?


That's about as long as I've been using conda. But I avoid it whenever I can because of a variety of gripes. The two biggest are the following:

1. On Lustre file systems, 'Solving environment...' can take minutes. I don't like waiting minutes to provide permission to install packages.

2. A lot of conda packages are broken. It's managed by maintainers, and unfortunately, some people maintaining believe that if it works on their machine, it will work elsewhere. Since I'm tired of ABI and runtime linking errors, I often just install from source.


conda doesn't require you to use conda packages; pip installs are perfectly integrated.

w.r.t lustre - can't comment about that; I prefer local file systems for development for various reasons (most importantly: mmaping huge local files is 10x to 100x more efficient than through networked filing systems).


Because publishing a package on PyPI is trivial compared to publishing on the Anaconda cloud. For the latter, you need to build your package for every platform and Python version you want to support. So sou need to setup some CI. This is just such an overkill for a pure-Python package. It makes a lot of sense for a project with a nontrivial C extension though, and I do build a conda package for one such project of mine. For everything else, I publish just on PyPI.

Are you aware that Conda fully supports and integrates pip?

Publish on PyPI, it's just as usable in Conda to everyone.

But assuming you for some reason insist on publishing to the system you use - the vast majority of users don't ever publish a package; what's stopping them from using Conda?

I admit I have never tried to publish anything on the Anaconda cloud, but I'm a bit surprised - I was under the impression that publishing pure python packages is simple; The requirement to do it for different python versions, though, seems perfectly warranted to me - and indeed, I ran into issues with packages on PyPI not working on specific versions (but nowhere listed as such).


There's no such requirement at all, but it's a noble goal and one every package developer should strive for.

Conda has no lockfile. The management commands are horrible to use and inconsistent. The "pip integration" is just "produce a requirements.txt and install it", so unless you're using some conda only dependency (thankfully those are few and far between) what's the point?

Conda is also annoyingly outside the python ecosystem. It seems to want to replace pip/pypi instead of working with it.


The pyenv-virtualenv extension makes all of the pyenv magic work for environments (as in virtualenvs) too.

I ran into a lot of bugs with poetry when I first moved to it (mostly with git dependencies) but they seem to have been fixed promptly, except that dependency resolution with git dependencies is still super slow.

An important thing for me is that you can disable the virtualenv management and manage it yourself (poetry makes it hard to install different python versions so I'd rather do that myself). Overall it works really well and I'd highly recommend it for general python dev.


>I also am a big fan of pyenv

I like the idea of pyenv, but in practice I find it to be pretty buggy (e.g. failing when installing older versions of python due to openssl build issues). I kind of wish it had some competition.


I'm looking into transitioning to poetry from Pipenv...my main gripes with Pipenv are 1) pyenv integration is cludgey and 2) "hard" conflicts between packages are unresolveable in pipenv and means you cannot lock your environment.

As far as #2 is concerned, does poetry allow you "ignore" or change subdependency requirements for specific packages a la maven?


Does poetry create a local vendor folder like "node_modules" or "vendor"?

That is the role of virtual env if I understand correctly what you are asking about.

"I was sorely disappointed with pipenv"

Can you flesh this out a little bit? What in particular were you disappointed with pipenv?

How does poetry do a better job?


Peotry does not handle the virtualenv at all. It deals only with the dependancy file and packaging.

So you usually couple it without something else. I use "pew".

I wish a tool would merge both.


What do you mean?

Poetry does create virtualenvs to install dependencies on a per-project basis.


Sorry, I got confused with another tool.

> This is the recommended way of installing poetry.

    curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python


Nope.

I hate python package/dependency/virtualenv management so much. Even JavaScript is preferable.

I find npm very easy to wrap my head around. I do npm install ___ and it does a lookup in its repository and downloads it and its dependencies to node_modules. If I want to start fresh, I simply delete node_modules. Everything else "just works" when I invoke node myapp.js. Punto e basta.

pipenv masquarades as the same thing, but then there is no python_modules folder to be found. Instead it downloads it to some obscure directory halfway across my computer. If I want to start fresh I guess I have to trust the tool's uninstall function? And it's unclear if I need sudo or not. Also I can no longer just run my program, I have to run it with pipenv now? Python requires too much cognitive burden for module/dependency/virtualenv management.

For me it's to the point that developing using a docker image with globally installed python modules is easier to manage and wrap my brain around than using pipenv/virtualenv/whatever.


PEP 582 is in the works to make __pypackages__ the Python equivalent of node_modules. Not sure if it will get accepted but I hope it does.

> a mechanism to automatically recognize a __pypackages__ directory and prefer importing packages installed in this location over user or global site-packages. This will avoid the steps to create, activate or deactivate "virtual environments".

https://www.python.org/dev/peps/pep-0582/


That’s a great idea. I’ve just been playing with the implementation from the PEP, and it makes things really easy.

The speed of simply extracting a bunch of wheels directly in to `__pypackages__/$PYVER/lib/` is a huge benefit. It behaves well if you symlink from a global tree of extracted packages too, like an even simpler version of pundler¹.

If others want to play with it without too much breakage, I massaged the patch on to 3.7². The 3.8 base was a little too big a change for my liking ;)

1. https://github.com/Deepwalker/pundler 2. https://gist.github.com/JNRowe/502f8772861cdd21bff5144c416e4...

Edit: It’s a third of a second to stand up a complete environment with 63 wheels in its dependency tree for the project I'm playing with right now.


That would be great! Any way to upvote or otherwise give the equivalent of a GitHub thumbs up to show support for it?

Basically it seems you are lacking only two shell commands.

    virtualenv venv
    . venv/bin/activate
Then everything works with pip

    pip install numpy

I actually prefer the pipenv way (or the virtualenvwrapper way) of putting virtualenvs into a dedicated location, so that I can wipe them off the hard disk if I want to free some space.

The real interesting and occasionally bothering point on managing your virtualenv is resolution of dependencies and compatibility ranges (hard to solve in general).


At a previous job, we used an even simpler solution: just download everything to a directory (called "deps") with

    pip install --target deps -r requirements.txt
Then running the application with

    PYTHONPATH=deps python myapp.py
This was also simple to integrate with service supervisors, since most make it easy to configure a environment variable.

Virtualenv tip: you can skip the Bash magic of calling "activate" by providing the path to the python executable.

    /hello/venvs/myproject/bin/python myscript.py
will run myscript.py exactly as if the virtualenv was active, which is easier to configure for cron jobs and services.

If this works for you, good. I think I would be cautious giving this out as advice because you can run into problems this way with regards to different python versions. The activate script uses the python version used to setup the environment.

Making a note of this, didn't know about the target flag, thanks !

This doesn’t address his problem of dependencies being “halfway across my computer.”

Yes it does. Everything is installed in the venv folder. To start fresh you just delete it [and restart bash].

That's true, but I assumed GP was referring to the fact that venvs are kept in a global, e.g. $HOME/.venvs, rather than in $(pwd)/.venv like $(pwd)/node_modules.

If you're using the commands in that post, the venv won't be kept in a global location.

Those shell commands don't do anything to publish a package, do they? Sorry, python is still lacking quite a bit of what e.g. js with npm has had for years.

See, I feel like python has a lot of those benefits and the package management doesn't need to be so complex.

Seriously, python package management can be fairly simple. On most of our machines at work, it's just "virtualenv .env && source .env/bin/activate". Then you install your packages and... everything is in one directory, like node_modules in javascript.

A clean reinstall is easy from there: remove the .env and just repeat.

I feel like pipenv violates KISS, and that a more traditional virtualenv/venv/pyenv setup is the way to go.


> A clean reinstall is easy from there: remove the .env and just repeat.

Repeat what, manually doing a bazillion `pip install`? Another nice thing about npm is the packages.json file it creates. This allows us to simply add that to version control and then all the new dev has to do is clone and run `npm install` which reads packages.json and installs everything inside of it. I'm sure there's a way to do it in python but, like everything else I bet it's a non-intuitive multistep process.

JavaScript

==========

First time installation:

    npm install express --save
Subsequent times:

    npm install
Starting clean:

    rm -rf node_modules
Running application:

    node myapp.js
Python

======

First time installation:

    virtualenv .env  

    source .env/bin/activate  

    pip3 install requests  

    pip3 freeze > requirements.txt
Subsequent times:

    virtualenv .env  

    source .env/bin/activate  

    pip install -r requirements.txt
Starting clean:

    rm -rf .env  

    [exit bash to clear environment]  

    [start bash]
Running application:

    virtualenv .env  

    source .env/bin/activate  

    python3 myapp.py
For JavaScript I typed it all out from memory. For Python I had to consult StackOverflow because I couldn't remember "pip freeze". Yes it can be made simple with use of aliases and such, but like I said in my first post, out-of-the-box cognitive burden is several times greater than, say, npm or yarn.

Your opinion here is totally valid. Npm has a lot more magic involved than doing stuff by hand with virtualenv.

And yes, aliases and bash scripts help a lot, but do increase initial overhead. I have the entire "delete create install" sequence in an alias as well as activate in another.

But envs are just files, you can even skip activate and just do `.env/bin/python` and it works. That's powerful in a linux shell because now I can just use that environment like a regular executable from anywhere, no global installs required.

I can appreciate preferring something less manual even if I don't! Ultimately, your machine, your code.


> [exit bash to clear environment]

Wat? just deactivate before you rm.

Please do not talk about things you clearly don't know well enough (which is patently the case if you can't rememer freeze and deactivate).

> Out-of-the-box cognitive burden is several times greater

You got any stats on that, or is it just your opinion? Because to me, the completely counter-intuitive --save parameter is much more painful to remember.

The venv/pip workflow is not perfect, but what you've described is not the problem.


> Wat? just deactivate before you rm.

Ok sure, but starting clean is still a 2 step process (deactivate, remove)

> Please do not talk about things you clearly don't know well enough (which is patently the case if you can't remember freeze and deactivate).

Yes, I openly admit I do not know python's package management/virtualenv stuff well enough. And I don't care to learn either, because I don't use it enough for the investment to be worth it. My point still stands that for each javascript package management task that needs accomplished, you need 2-3x as many commands to accomplish the same thing in python.

> You got any stats on that, or is it just your opinion?

My opinion, of course, but shared by the dozens of people who upvoted my OP.

> Because to me, the completely counter-intuitive --save parameter is much more painful to remember.

In what way is --save counter-intuitive? If anything pip freeze is counter intuitive. How does it know what to save? Does it just save everything you've ever installed? What if you don't want to save everything, just a few of them?

Also, --save and --save-dev allow you to segregate developer dependencies from production dependencies. Is there a way to do that with python? Again, just a guess, but it's probably going to be an unintuitive 3-4 step process that I'll no doubt find on stack overflow.


> Yes, I openly admit I do not know python's package management/virtualenv stuff well enough.

And still, you are here trying to measure the length of your "commands" with others' "commands".

> My opinion, of course, but shared by the dozens of people who upvoted my OP.

Ah great, engineering by acclamation. That usually ends well. That's how we got pipenv, btw: a popular developer stood up and declared "I'll fix it!", to general acclaim from the Powers That Be... and then things broke harder, and here we are.

> for each javascript package management task that needs accomplished you need 2-3x as many

That's precisely the perspective that led us to the mess that is pipenv: "npm is the model, we should all be like npm". Except npm fundamentally serves only a few specific needs, and was built on the lawless prairies of an ecosystem with limited aims, no stdlib, and without 28 years of accumulated legacy practices; whereas python has been pulled in every direction for literally decades, and now has to herd all that legacy into something more coherent, slowly (because this or that constituency will be ready to scream about breaking compatibility, as we've just had to endure for about 10 years with py3) and correctly - to avoid ending up in situations like the periodic breakage that happens in npm because this or that package has misbehaved.

> In what way is --save counter-intuitive?

"I've already told you to install, why should I repeat the concept? Are you really so dumb a 'manager' that you would ignore what you just installed?"

And btw, Stack Overflow says --save is actually obsolete since 2013 at least [1], so it looks like you don't know npm very well either. Maybe we should just give up and build an AI that learns development from SO, and find ourselves more meaningful jobs.

[1] https://stackoverflow.com/questions/19578796/what-is-the-sav...


> And btw, Stack Overflow says --save is actually obsolete since 2013 at least [1], so it looks like you don't know npm very well either.

"edited Sep 18 at 18:15"

`5.0.0` was introduced May 25, 2017 [1]. Most linux distributions have not picked it up yet in their repos. Ubuntu 18.04 (released this year) is still on npm 3.5. It's unreasonable to expect a developer to be familiar with the bleeding edge, especially when existing projects are locked into using older versions.

I don't know why you are being so contentious about this. I actually love Python and hate JavaScript. But in my opinion, JavaScript has the superior package management setup which is why I am rooting for PEP 582.

[1] https://github.com/npm/npm/tags?after=v5.0.2


In python you just create a virtualenv with "virtualenv whatever". From then on everything is installed in there. If you want to start fresh, you just delete the virtualenv. How is this difficult?

I don't get the node way of doing things. Why should the availability of modules be dependent on the current working directory? Seems brittle to me that your app can't find its libraries because you ran it one directory up/down from where it should be.

As for sudo, there is no situation in which you should use sudo with pip.


Node resolves relative to the path of the file requiring the module, not relative to the process cwd.

Use conda

The problem, in my view, is not that pipenv is buggy or flawed or changes too rapidly. The problem is that it was hyped-up with overblown promises and marketing and sycophantic testimonials. The author of the project describes this himself in his "letter to r/python":

> ...my fame, while certainly categorized under “cult of personality” is not necessarily accidental. It’s called marketing. I worked very hard at becoming well known within the Python community, and toiled away at it for years.

I see this as the real issue, which led to premature adoption of a tool that wasn't stable, and the subsequent backlash.

Also, I have been using virtualenv for years and if I want to freeze my dependencies, running `pip freeze > requirements.txt` is sufficient for me.


We recently switched to using poetry over pip and virtual envs. All the additional tooling poetry provides is amazing - actual dependency resolution, script entry point, building, version bumping, publishing. The author is awesome too. Also making python3 tooling possible on python 2/3 packages. I'd highly recommend trying it out.

Pipenv on the other hand was not only slower, but _failed_ to actually resolve our dependencies correctly. Additionally, it was slower. And looking up the issues, I can echo that the development team seems a bit defensive and dismissive.


Our organization uses four primary languages (Ruby, Python, JS and Elixir). The package management situation for Python is by far the weakest.

We’ve been using Pipenv, but it is atrociously slow and flawed at dependency resolution. An alternative is extremely welcome, E.g. Poetry which was mentioned above.


FWIW, resolution in Python is significantly harder than for Ruby, JS or Elixir because it requires cloning down each dependency. To really speed it up, what's needed is a registry API that provides all of the details for resolution. That's much harder for Python because of the legacy of setup.py, which allows version resolution to depend on arbitrary system considerations.

I've never really delved deeply into package management. I've been using ruby for a while and using rbenv has made it pretty painless for a while.

I've been using pipenv on my local machine but then switched to virtualenv in my ec2 deployments (because all the tutorials used it).

What makes it easier for ruby? Is there just this "registry API" that has gained enough traction that everyone uses it?


The fundamental problem with Python dependencies is that they are calculated while executing setup.py, not declared statically.

For example, the popular scikit-learn package has the following in its setup.py:

    if platform.python_implementation() == 'PyPy':
        SCIPY_MIN_VERSION = '1.1.0'
        NUMPY_MIN_VERSION = '1.14.0'
    else:
        SCIPY_MIN_VERSION = '0.13.3'
        NUMPY_MIN_VERSION = '1.8.2'
and these are used to request dependencies from the resolver/downloader. You could have more dynamic dependencies that vary based on system libraries and tools installed on the machine where you are running it, or even the current weather.

Until this is replaced by static version numbers, and all popular packages adopt it, a registry API cannot exist as it needs to run code on your machine to figure out the dependencies.


Tools such as Poetry and others expect that information to be available on PyPI, so long as you are using a newer version of setup tools and use twine to upload.

I can't speak to elixir but Ruby seems reasonable, JavaScript is a total clusterfuck. Python is dead simple.

I rarely comment here on hackernews. I've also seen several flame wars over pipenv. I also believe that python packaging and pinning is nothing but a mess. Recently I started using Pipenv and suddenly I've been having horrific issues with python to the point of me almost giving up on the language itself. I believe the issue is a mix of Pipenv, Pip and Debian. I don't have full scope view of the issue yet but without evidence I believe the issue is how Debian uses Pip at the system level, and in pip 9+ an API used by Debian changed. Pipenv seems to greedly somehow upgrade my pip and thusly hoses everything and I find myself reinstalling everything. All three are making me consider making changes to my entire stack and setup that I've been using daily for the past 5 years on many many systems.

In addition when/if I have time I'll further debug and attempt at PRs and issues to help.


My goodness, thats like a PSA to never try pipenv at all if your running on a debian system, thanks for the heads up!

I've used virtualenv, pipenv, pyenv, venv, etc.

What the hell is the difference? All that I have used have worked pretty much exactly the same as the others. All work just fine. I've never had a problem. I just use virtualenv since it's the oldest. I see no more reason to switch or try other options as long as it continues to deliver.


Have you ever used any other mature dependency management tool? Leiningen, Cargo, Composer, Yarn, just to name a few. All of them are light-years ahead of anything Python provides, which is sad.

+1

The capabilities provided by a tool like Leiningen[1] just makes everything even tangentially related to dependency management an absolute breeze.


With pip for instance, it often happens that a transitive dependency gets updated inadvertently breaking your code. This follows from the assumption that all packages follow semantic versioning perfectly and keep backward compatibility where they should. This is not the case in practice and experience has shown it is unrealistic to have that assumption. A better way is to rely on exact versions of packages (up to a single bit) and not on semantic versioning.

How would you updated something inadvertently?

If the specific version of a dependency (or subdependency) isn't pinned, then the next time the package is installed in another environment it'll get the most recent matching version. That version might break your code.

If the specific version of all subdependencies are pinned, then you have a mess on your hands of keeping track of what's actually required. You have to either manually maintain your requirements.txt, or you run the risk of removing a dependency and missing the removal of its subdependencies.

Further, you can't just upgrade everything, but dependencies might have conflicting version requirements for subdependencies.


Transitive dependencies... it seems to be common in Python libs to express dependencies using version ranges - but then version 18.1 breaks something that worked under 18.0 - which isn't what you'd expect if it followed semver.

Yeah, the problems with python packaging can not be fixed by any tool. There is simply no way to determine the dependencies of a python package without downloading it and all its dependencies. And, no accurate way without actually interpreting setup.py.

"What the hell is the difference?"

One of the big selling points of pipenv is that you can pin the versions of the packages you use and their dependencies.

None of the others do this, afaik.


`pip freeze > requirements.txt`

That's the way I did it before pipenv, but it has problems that I've outline elsewhere in this comment chain - chief amongst them, what happens when you want to remove a direct dependency? How do you identify its subdependencies and ensure they aren't being used by other dependencies as well?

This assumes semantic versioning and does not actually pin dependencies to the hash of their bytes, like a lockfile does.

`pip-compile --generate-hashes` is the best way to manage python dependencies.

https://github.com/jazzband/pip-tools https://gist.github.com/hynek/5e85706cee589a204251b333595853...


Could those hashes be different for the same package built on different architectures?

This is another good point; one that's important to me, but that doesn't seem to resonate with others.

Pipfile.lock ensures that the tarball I download for a given package release is exactly the same as the one I downloaded during development. This closes what I consider to be a fairly significant security hole.


With pip you can: pip install package==version

> Another issue with this tagline was the Python.org and official parts. The thing that made it “official” was a short tutorial [1] on packaging.python.org, which is the PyPA’s packaging user guide. Also of note is the Python.org domain used. It makes it sound as if Pipenv was endorsed by the Python core team. PyPA (Python Packaging Authority) is a separate organization — they are responsible for the packaging parts (including pypi.org, setuptools, pip, wheel, virtualenv, etc.) of Python. This made the endorsement misleading.

Hmm I'll admit I only started using pipenv because of that... "endorsement".


I just feel this article inside my bones. The headline is 100% accurate.

What bugs me is we have some excellent examples of good patterns… in the JS community. Pipenv promised to be Yarn and ended up way off target :/

More context from my previous post:

https://news.ycombinator.com/item?id=18247788


Allow me to reproduce a comment that I made in /r/programming last week on this topic.

I use pipenv in production and testing to simplify deployment on systems that don't natively support python 3.6+. When it works it is great. When it fails, or when the cli options fight each other and try to be smart but instead for a circular firing squad it is one of the most insanity inducing pieces of software I have ever used. Pipenv release have repeatedly broken CI builds for me for the past 3 months. I was so pissed with how bad it was about 9 months ago that I actually gave up trying to use it on my development machine and learned how to write gentoo ebuilds. On reflection it seems like the perfect tool for python -- if you stay on the happy path and only use it in BDFL APPROVED ways then it can be great, be woe to the fool who wanders from the light into madness.


How did this get to be such a contentious issue? While I've been focused on other languages this year, my Python side projects have all gradually moved towards pipenv. I still have a few locally shared virtual environments, and setup.py is still the default for distribution. I also sometimes embed into uwsgi, but I don't know of anyone else that bothers with that. We're using pex at work.

Overall the article seems hyperbolic, relative to the actual events cited. Packaging is a work in progress everywhere.


Curious about one criticism in particular:

  I can run pipenv shell to get a new shell which runs the activate script by default, giving you the worst of both worlds when it comes to virtualenv activation: the unwieldiness of a new shell, and the activate script, which the proponents of the shell spawning dislike.
In a project that just uses pip + a virtual environment, I'm used to activating the virtual environment when I want to run a command.

Why is `pipenv shell` any worse than that? What do "proponents of shell spawning" dislike about the activate command?


> What do "proponents of shell spawning" dislike about the activate command?

For me, it modifies my existing shell in ways that I don't fully understand. While I've never had any issues with that, I have experienced similar problems in the past with RVM and they were incredibly difficult to debug.

Using the OS's tooling to create an isolated environment seems to fit much better with the "Unix philosophy" than modifying the existing one.


What I don't really get is this: why can't you just prepend the virtualenv to the PATH and be good with it?

That is only required obviously if you're using bash scripts which can't be directed to an actual python binary.


"activate" prevents accidentally nesting environments. It also changes the prompt to show which one is active. Running "deactivate" is easier than changing the environment variables back yourself.

Unless you change state yourself in the mean time.

`deactivate` changes `$PATH` back to what it was when you ran `activate` for example, which isn’t likely to be what you want if you’ve changed it since. At least with a subshell you can know what state you’re returning to with <C-d>.

That is part of the problem with the `virtualenv` story in my eyes. It offers the illusion of isolation, but falls down in quite a few ways which are annoying when they do pop up.


The best way to manage python dependencies is with pip-compile

https://gist.github.com/hynek/5e85706cee589a204251b333595853...

    update-deps:
    	pip-compile --upgrade --generate-hashes --output-file requirements/main.txt requirements/main.in
    	pip-compile --upgrade --generate-hashes --output-file requirements/dev.txt requirements/dev.in
    
    init:
    	pip install --editable .
    	pip install --upgrade -r requirements/main.txt  -r requirements/dev.txt
    	rm -rf .tox
    
    update: update-deps init    

    .PHONY: update-deps init update

I never used pipenv but I use virtualenv+pip. Pip always resolved well all dependencies for me, so... What are the advantages of pipenv over virtualenv+pip?

For me it was the promise to be able to maintain my dependencies up to date and avoid manually handling configuration files.

Working in a team it is often useful to have your versions "pinned" down so you have a reproducible environment regardless of upstream backwards compatibility policy (just to name one example).

The Pipfile + Pifile.lock combo seemed right for the task.

Creating virtualenv was a nice plus.


What was wrong with requirements.txt? We heavily use virtualenvs and just pin package versions there. I've yet to see it fail.

updating requirements.txt is a pain, and also development dependencies vs normal ones usually results in two requirements files. lock files are a good way to do this, so borrowing from npm et al isn't a bad move.

(for what it's worth, i use pipenv and rather like it.)


Tried pipenv for the first time few weeks ago. The first thing I do is 'pipenv install celery[redis]', fairly trivial, and I hit a critical bug. The package installs fine, but the lockfile is unusable now because of the extra. So every time I install something, i need to run sed, to remove all extras from the Pipfile.lock.

And then there is the speed. pipenv is pathetically slow installing packages.

And I am thinking to myself having recently read an article about how bad npm is, "Only people who never had to deal with pythons packaging mess think npm is horrible"


People please stop engaging with pipenv and just use conda. Even critical arguments are mistakenly acting like pipenv deserves a seat at the table. Just don’t engage. It’s like the creationism of Python packaging.

I don't understand why this comment isn't higher! Honestly, using `conda` eliminates the pain of python dependency & environment management entirely. Clone a repo, do `conda env create` (from the project's `environment.yml`), then `source activate $ENV_NAME`. Straightforward. Easy. But most importantly, reliable!

It's great but can be a pain if you develop on a mac and try to replicate your env on linux because the env files are not portable.

The CLI can be confusing at times too, I always have to google how to create a new env or export it.


The exact use case I use conda for at work is developing on Mac and automatically export the same env to various linux platforms.

Why do you suggest conda doesn’twork for that? It’s one of the things conda specifically does. When recreating the same env on a different platform, it will resolve the dependencies for that platform, so there are no cross-platform issues of the underlying env unless the library being installed simply doesn’t support that platform, in which case _no_ environment manager could possibly solve that specific problem.


Don't forget to pin pipenv itself, each dot release breaks something

Oh, they don't do dot releases anymore, its "CalVer" [1] - aka. Calendar Versioning, aka. release when we feel like it. And why did they do this:

> We just switched the project over to calver, with the explicit purpose of preventing [Kenneht Reitz] from making more than one release a day

[1] https://calver.org/ [2] http://journal.kennethreitz.org/entry/r-python (Ctrl-F 'calver')


Don't worry, each CalVer release still breaks things.


I will be down-voted to hell, but sometimes I feel that KR hides behind the illness to avoid responsibility. KR told that he's just "good at marketing" and has "cult of personality" [0] and I do not see an edit that it was "KR the maniac" speaking and not the "KR the normal guy".

Personally, today I'd better use `venv + pip` instead, of `pipenv`, but `pipenv` somehow was the "official" tool for package management for months, until people started discussing it [1]. I would like to have better packaging tools in python, but `pipenv` approach seemed really strange, now I know why.

If you had manic episode and did some mistakes - go and try reverse them. Revert/change the commits. Apologise for the comments you've made. But no, let's get into the position of a victim, when someone criticises you. I believe that KR may have psychological problems and/or illnesses, but his "normal self" seems to be rather egomaniac too and takes bolder claims that he's comfortable to handle. If it would be otherwise - there would be no drama, there would be no fake "official" tools. KR could just enjoy his fame from `requests` and save his time while writing responses on his blog.

[0] http://journal.kennethreitz.org/entry/r-python

[1] https://www.reddit.com/r/Python/comments/8jd6aq/why_is_pipen...



This article has been really informative and has given me context on what the hell is the problem with pipenv in people's opinion, which I have been wondering about for a while, I'll check out Poetry soon. Reading all the context, however, has reminded me that people are mainly assholes and now I feel I need to stop using social media completely, and the Internet for that matter.

I tried pipenv, then poetry, then pip-tools [1]. pip-tools worked best for me. I control my own virtualenvs, and can compose the requirements files pip-tools compiles. It's vendored in pipenv, so it's basically the dependency engine for pipenv.

[1] https://github.com/jazzband/pip-tools


I use pipenv for the past year, there are some hiccups but overall I found it works well for my cases. I only need store Pipfile to my git repo and after 'pipenv shell' things just worked as expected for me.

The one thing about pipenv that prevents me from using it in more projects is that it refuses to not upgrate a dependency. If I install or upgrade a dependency, every dependency is upgraded. (I'm not sure what `--keep-outdated` is supposed to do, because it definitely does not keep outdated dependencies.) First of all, it's surprising and unpredictable. (Why would `pipenv upgrade requests` upgrade boto3?) It also prevents me from controlling change (and therefore controlling risk). I'm responsible for everything that gets deployed, so when I upgrade a dependency it is deliberate, isolated, and well-tested. I can't do that with pipenv.

I understand that they want to encourage keeping dependencies up-to-date, but I think the proper approach to this is npm's, where it informs me about outdated packages, but lets me do what I will with that information.

So, for the moment, I'm still in the dark ages of manually pinning everything into `requirements.txt`.


This is about to get better - check out https://github.com/pypa/pipenv/pull/3304.

I don't write libraries but pipenv works well for a simple flask application because I don't ask much of it.

All I want is a simpler file to look at compared to requirements.txt

Here is my guide for newbies like me

1. Use Pipenv in development.

2. Create a requirements.txt each time you make any changes to your pipenv. Commit all three files: pipenv, pipenv lock, and requirements.txt to git.

3. Use this requirements.txt in production.

4. ???

5. Profit

Now here is my only gripe:

I believe pipenv is meant for simple people like me. I have to say I want to use Python 3. There is no way to say I accept anything 3.5+

My understanding is that pipenv is not meant for people who actually know python inside out. My use case is that it lets me keep track of what dependencies I installed as opposed to what dependencies my dependencies installed. This should have been the ONLY problem that pipenv fixes but like the old saying goes... no project is complete until it is able to send mail (sorry I probably said it wrong).


It sounds like pip-tools[1] is what you want Pipenv to be.

1. Put your dependencies in requirements.in.

2. Run pip-compile to generate requirements.txt.

3. Commit both files.

4. Use requirements.txt in production.

You can also use setup.py or setup.cfg instead of requirements.in. This lets you build packages and specify dependencies like "Python 3.5+". requirements.in is simpler, though.

[1] https://pypi.org/project/pip-tools/


+1 for piptools. Piptools makes a lot of sense in a docker based workflow (virtualenv not needed).

https://github.com/pypa/pipenv/issues/1382

^ This was just such a glaringly terrible design decision, and the fact that they won't even acknowledge it as being a mistake is really frustrating.

npm already found the best solution (if ./node_modules exists, it is automatically used -- no need to run any sort of shell commands). All they had to do was just copy their behavior.

Oh well, maybe the _next_ python virtual environment tool will get it right... :(

Python is a great technology but it is really a shame that is hampered by bad decisions (e.g. the 2->3 fiasco).


The behavior you describe is great until you run `py.test myproject` and it proceeds to invoke all of the unit tests for tensorflow or whatever.

That's absolutely the sort of thing that can be fixed in pytest's test discovery algorithm. It's not like this isn't exactly the same fix as in the js world (= ignore node_modules in test discovery).

Sure but then you have a chicken/egg problem: pytest isn't the only testing library that does test discovery (standard library unittest does too, among tons of other 3p ones). Does everyone follow pipenv's convention? Do you write a PEP? What's the solution (maybe a pep-like is the right one to be honest), but its not an immediate change, and makes people less likely to use your tool in the interim.

Kenneth used a growth hack.


pip-tools (pip-compile, mostly) does everything I wanted out of pipenv (a sane lockfile) and is more explicit. I don't have any reason to use pipenv. No idea why pip-tools isn't super popular, honestly.

I've long freed myself from the exhausting pipenv/virtualenv/pienv/conda rat-race by using docker. The only downside is some annoying IDEs refuse to acknowledge the existence of interpreter in a container.

Why is it a rat race? I've been using pyenv+virtualenv to isolate project environments for years now and it works just fine.

can you get VSC code completion to use a python interpreter in a Docker container?

I can't! In fact that was my main motivation in posting that, maybe a wiser HNer will enlighten me.

Damn was hoping something had changed in the last 6 months when I had a look! :-)

I don’t use VSC, but assuming the interpreter is just a path to an executable file, you could just pass it a script that calls, eg ‘exec docker run -it python’.

PyCharm can!

How?

Honest/naive question - why has it been so hard to create a good package management system for Python? Couldn't someone do a straight port of one that has been proven to work well from a different language?

The problem is that (just about) all python packages express their dependencies with a script called setup.py. To figure out the full transitive closure of dependencies you have to walk the graph, downloading and executing setup.py scripts. First of all, that's slow.

But worse, setup.py is a python script with access to the full power of python. It can crash. It can be slow. It can loop forever. It can have dependencies of its own. It can require specific versions of python. It can download stuff from the internet. It can do anything.

If you wanted to implement something like npm for python, you'd have to convince all the python package maintainers to write and test package.json files for all their packages. Even if they were willing and enthusiastic about that, you'd have a chicken and egg problem because nobody could test a package.json until all their dependencies had them.

Python has a such a plethora of packaging tools because people keep trying to solve the problem by writing better code. But the real problem is lack metadata. Python will always have a lousy packaging ecosystem because it relies on setup.py.


A bit of politics, a bit of genuinely different requirements across the ecosystem. Unlike most other languages with a single packaging tool, Python serves very different niches at the same time: webdev, sysadmin, embedded, datascience, fat clients, libraries, bindings, etc etc. They all have different usecases and concepts of “packaging tool”. When a new tool emerges, it usually serves one niche better than others; the other elements of the ecosystem push back, and a new tool is built... repeat ad infinitum.

This does not happen with languages where a single niche accounts for most of the use, so a broad consensus is easier to achieve. The closest to Python is Java, but that is traditionally steered top-down (but still has competing toolchains, e.g. ant/maven/gradle).

Pipenv in particular has some additional problems that were completely self-inflicted (namely, a famous developer abusing his popularity to push an incomplete implementation as blessed).


The article doesn't mention pyenv, which I've used to great effect. It does pretty much everything I want, including having multiple environments active at once (so that tools that want to do their own virtualenv management, like tox, can find a python2.7, python3.5, python3.6, python3.7, pypy2... in PATH) and automatically activating a virtual environment when I go into a particular directory.

For a project to be easy for me to use, it doesn't need to do anything fancy to accommodate me: 1) 2 or 3 or both? 2) a setup.py or a requirements.txt. I basically always `pyenv virtualenv 3.7.1 $PROJECT && pyenv local $PROJECT` and then more or less never worry about this problem again.

When I need to _produce_ a pinned set of dependencies, it's all just pip + virtualenv so pip freeze works just fine.

The killer feature of pipenv is allegedly pinning but this has never made sense to me. Maybe I just don't understand it? But pyenv makes highly segregated environments easy, so as long as each project has an internally set of environments, everything is A-OK.


I use pyenv on some distros and it is good, but on Gentoo there is no point because Python is slotted which means you can install multiple versions and have them all available on PATH. I then use virtualenvwrapper and specify the version manually (e.g. -p python3.7). Alternatively you can always do python3.7 -m venv ...).

The annoying thing with pip freeze is it doesn't have a concept of a "world" file like emerge (Gentoo). With emerge when you install a package it gets entered into your world file but the dependencies do not. That way you always know what you actually want, rather than just incidental dependencies. I wish pip freeze did that.


I prefer to keep locally installed Pythons for regular use everywhere, though I guess some distros get the same effect by patching pip to make `--user` the default (and hence pip install xyzzy goes to my home directory instead of messing with something that belongs to the OS).

In particular, one way this has bitten me is if the OS expects a certain Python with a certain set of dependencies to be available for systems its responsible for. Isn't portage itself written in Python for example?


I used to install tools using --user, but now I install all tools using my package manager (emerge) so I don't have to worry about maintaining a ~/bin etc. So I basically never run pip outside of a virtualenv.

Portage is written in python, yes, so you can't just change your system python at will, but that's not a problem really.


Meanwhile, I've recently started using the setup.py file and its install_requires and extra_required fields for production requirements and dev requirements, respectively. I don't even use requirements.txt as thanks to setup.py I only have to install the folder itself.

Am I weird? Am I not supposed to do that? Is it bad practice?


The main issue with this is that it doesn't version lock the dependencies of your dependencies.

I've dealt with Python package management for awhile, haven't used Pipenv a lot (as my company has their own homegrown duct-tape-pip-and-virtualenv-together solution that they started building before Pipenv was available), and I've noticed a few things.

The root problem is that just having a system like pip to manage your Python dependencies is woefully insufficient, because it installs dependencies globally, you might need different versions in different projects, and you might be trying to link in some native code, so that behavior will change based on what you already have installed or what your OS is. Also, to make matters worse, you also have to manage your dependency on Python itself, since there are multiple mutually incompatible versions.

So you can use virtualenv to handle the problem of isolating one Python project from another, pyenv to handle the problem of different Python projects using different versions of Python. Then you need a system like pipenv to tie them all together. Except pipenv, itself, is in Python, so there's a bootstrapping issue with using a tool written in Python that has a dependency on a Python interpreter to indirectly manage your dependency on a Python interpreter. Sometimes if you do something ridiculous like "using a Mac where you haven't specifically installed the right version of Python via pyenv or Homebrew yet", things will break in confusing ways.

One way around this whole mess is to put everything in a Docker container, so you get to stipulate that everything runs on a particular Linux distro with a particular set of dependencies from the ground up and there's no possibility of anything else at all on your or any other machine ever polluting that dependency chain and even if you're on a Mac it'll just install dependencies and run code inside of a VM. The isolation you're trying to accomplish with virtualenv or even pyenv, whether by hand or using a tool like pipenv, is pretty much just a poor man's Docker container anyway. But this adds complexity of its own while still punting the real work off to a tool like Pip.


pipenv is not perfect but it is definitely headed in the right direction. It is a lot easier for beginners to get started with which is one of the best things I've found.

To address one of the problems, Running Scripts: basic complaint is you are starting a new shell always and you have to prefix all commands.

Prefixing commands sucks but you can get around it pretty easy with an alias.

alias pr="pipenv run" alias dm="pipenv run python manage.py"

now 'pipenv run python manage.py startapp foobanizer'

becomes 'dm startapp foobanizer'

or 'pr django-admin startapp foobanizer'

As for the shells. I really like this feature because it's a lot cleaner. With activate and deactivate you are essentially mutating the current shell which can be dangerous. Also because it loads your ENVs on every run, it's easier to switch out ENVs.


What's the recommended Python solution for deploying Python scripts/applications off-line? I am deploying several scripts on machines with no access to the internet, so I can't exactly pip install -r requirements.txt. My current solution is to cache all pip packages in a packages directory and then export PYTHONUSERBASE=$PWD/packages, but it feels like a hack. Virtualenvs are no solution because they aren't portable (hardcoded paths).

"free (as in freedom)"

Isn't that a circular definition, not the official Free Software Foundation line? Is it just a sneaky way of not actually saying "free as in speech"?


It's not circular because "freedom" can't really be used to refer to price (e.g. "freedom of beer"), so it actually disambiguates. I wouldn't personally default to assuming a political motivation for the wording in this case.

Sure it can be used to referred to price: "freedom of cost", thus "free as in beer" -vs- "free as in speech", a common expression you may have heard before, if you are aware of the Free Software Foundation.

https://jamesdixon.wordpress.com/2008/06/04/what-does-free-a...

It's not that it demonstrates a political motivation, just an ignorance of the actual free software culture they're trying to associate themselves with. It's like saying "Make America USA Again". Tautologically circular, close, but no cigar.


Same experience with pipenv. One problem is that it creates the virtualenv even if you are in sub dir. It's easy to make a mistake.

Despite disliking Clojure now, I find its (rather: Leiningen's) approach to dependency management the best possible approach: clojure.jar (language core + standard library) is a regular dependency like any other dependency. (Of course this only works because the "real" core is the JVM.)

“Not as good as pip, but it’s more reasonable than Pipenv. Also, the codebase and its layout are rather convoluted. Poetry produces packages instead of just managing dependencies, so it’s generally more useful than Pipenv.”

Can anybody confirm which codebase is being described as convoluted here?


I like and use pipenv, but the lockfile slowness thing is truly a nightmare. It's especially bad in the data science stack. On my fast MBP, I can `pipenv install numpy pandas matplotlib` and then go get lunch, and there's a fairly good chance it won't be done when I get back.

I really like pipenv, but I understand these complaints, especially the time calculating the hashes for the locks. Why is it so much slower than npm!??! But pipenv sync, and pipenv --venv Have made my deployments much easier.

It's unfortunate that more people don't know about pip-tools (it isn't even mentioned in the OP). It + virtualenv hits a sweet spot for speed, usability and utility for me.

Stop trying to make pipenv happen; it's not going to happen.

What pipenv needs more than anything is a proper back tracking dependency resolver. With that it would instantly become my package installer of choice for python.

application and library are very different, it's unrealistic to have one tool solve 99% library problems. If you disagree, please check:

1. Can it support different python version and virtual environment?

2. Can it support build packages for windows,osx,ubuntu,centos...?

3. Can it support different build tools, for example: wheel, cython...?

4. Can it manage dynamic depdency version?

5. Can it manage version, like bumpversion?


What does pipenv deliver that cannot be done with pip and virtualenv?

Skimming the comments in this thread I see firstly a lot of hate towards pipenv that I don't understand. I mean on a technical level, I don't understand what you're talking about.

Secondly, interspersed beteween the hate I could count at least 6 alternatives to pipenv.

So as a python user I don't know who to trust.

I stumbled across pipenv at a time when I was managing a "global" directory of virtualenvs instead of putting them inside each project dir.

So I could do source ~/.venvs/project/bin/activate because I was tired of having different .venvs inside project dirs.

Pipenv seemed like a welcome change and I especially liked doing pipenv run commands. I still run my dev servers with pipenv run. Keeps my environment clean.

Sourcing a virtualenv before would lock that shell to only working with one project, one environment.

I have noted some issues with pipenv, the first was in ansible deployment but it wasn't a show stopper.

The second issue I can't remember so it must have been fleeting.

As someone else pointed out, maybe it's for simple users like me who don't need to understand the advanced internals of Python.

Edit: Right! The 2nd issue was actually that I work with some services that are spread across three different git repos/projects. Maintaining a central virtualenv for multiple projects needed figuring out but I think I've resolved it by having a parent dir for the service where the virtualenv is created and then pipenv commands in the sub-dirs (git repos) use the parent virtualenv.

I've also noted some confusion in pipenv on whether it's using python3 or 2. A pipenv --two project might try to install packages using pip3 for some reason.

Lastly, a lot of the hate towards pipenv seems to be directed towards how it was launched and marketed. Which in my opinion has little to do with the tool and how it might help users with Python.

Open source has always been and will always be a wild ecosystem where the best tool floats to the top by word of mouth alone. So why be mad because someone used python.org to promote a tool when you can contribute your time and knowledge to promoting your favorite yourself.

I just don't like when there are too many options to choose from and I don't know which one is right for me. I guess time will tell. Also I don't really do CI/CD pipelines yet so maybe a lot of issues are unknown to me.


This article is just a reminder that it may be impossible to fix the nightmare that is pythons packaging system. I'll save the python rant but this kind of software just further fragments and damages pythons reputation.

Honestly it's a nightmare in every language. And I found in comparison to most other languages I've worked with (Java, Ruby, PHP, Golang) I actually feel it's done quite well.

It's a super complex topic, it needs leadership support so that you can get a reasonable percentage of the community to drill down on one solution instead of having 4-5 competing ones, but it also needs a lot of in-depth insights, which is kind of opposite to leadership which requires a top-down view.


I always felt like RubyGems is mostly done right. And not just packaging itself, it's also easy to host your own gems and combine sources with upstream for example. What is that pipenv made better than RubyGems for you?

I see your point. Python however does not have a bad reputation. It's very positively perceived.



Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: