
Python Environment - mgrayson
https://xkcd.com/1987/
======
scrollaway
Man it's frustrating to see this comic and read the comments here, because
it's all so true. Python, in its noble quest for backwards compatibility, has
accumulated so many different ways of packaging, distributing and installing
libraries and apps that it's rivaling Google's chat apps.

Pipenv is the most promising solution _today_ but is still very, very new.
It's modeled after yarn and has been officially blessed as "The One True Way"
of installing stuff by the Python documentation. It has a way to go still to
be as good as yarn (especially in terms of speed). The Python ecosystem has
never had proper declarative packages like package.json (setup.cfg can be used
to have fully declarative package metadata, but I seem to be the only one
using it that way), which is a problem for package managers.

To those suggesting it, Docker is great but you're still dealing with a
package manager inside Docker, so that's a moot point. It avoids the need for
virtualenv, kind of, but so does pipenv and it does so more reliably and
reproducibly (pipenv implements lockfiles).

~~~
alexbecker
Pipenv solves a very real problem, but it has a lot of problems of its own:

\- It's very slow: re-locking after updating 1 dependency often takes me ~1
minute.

\- It has lots of bugs. To name a few in 11.6.9: clobbers comments in the
Pipfile, inconsistently includes dependencies for other build environments in
Pipfile.lock, stores the wrong index in Pipfile.lock for packages not on PyPI.

\- They release multiple times per day, often breaking things in patch
releases.

\- Kenneth Reitz is quite unpleasant to deal with in GitHub issues, which I
often have to because of the previous 2 issues.

From what I've heard, Pipenv "has been officially blessed" only insofar as its
maintainer got commit access to PyPA's documentation and inserted a
recommendation.

~~~
kennethreitz
I apologize for not meeting perfect standards of mood at all times. I have
bipolar disorder, for what it's worth, and working on a packaging tool is very
stress-inducing.

~~~
divbzero
It’s not easy to develop for large communities, and even tougher to speak
frankly about oneself.

Keep up the good work.

------
tomkinstinch
Python environment headaches have been largely solved for me thanks to
Miniconda[1]. Not only do the environments isolate dependencies, they can
easily use different versions of Python, and can include arbitrary binary
packages too. It helps that for my work in biology an extensive number of
packages are available from the bioconda[2] channel (with many non-bio
packages from conda-forge[3]). Environments can be described via environment
files, allowing them to be transferred to collaborators, managed with source
control, or included with publications to support reproducible science. If you
like virtualenvs, you may want to give Miniconda a try. One current limitation
is that environment files cannot specify a source channel for specific
packages--they're installed from channels based on the global channel
precedence.

1\. [https://conda.io/miniconda.html](https://conda.io/miniconda.html)

2\. [https://bioconda.github.io](https://bioconda.github.io)

3\. [https://conda-forge.org](https://conda-forge.org)

~~~
icebraining
I used Conda recently and it was OK, but it's annoying that it can't install
stuff directly from PyPI. Unless you're lucky, you still have to use Pip
inside the Conda environment to install some packages.

~~~
makmanalp
A sparkle of hope is that the conda people recently announced in a github
ticket that this would be supported as a first class citizen soon.

------
dsr_
With 8 comments so far, I count seven solutions presented.

~~~
znpy
[https://xkcd.com/927/](https://xkcd.com/927/)

~~~
mfoy_
The alt-text is especially funny, now that we're in the world of USB-C and
Thunderbolt 3, etc...

Dongles for days.

------
xnyanta
Honestly, Python on OSX is the most confusing of all.

The OS comes with a python runtime, which is different from the runtime you
can download from python.org.

You can then also install it from Homebrew or from MacPorts.

I'm probably even missing something.

~~~
psychometry
Should I set up this project with conda, miniconda, virtualenv, docker, some
combination of the above...? Fuck it, I'll just use R.

~~~
RayDonnelly
You can install R and some commonly used R packages with:

conda install r-essentials

------
kissgyorgy
Still missing from the picture: pipenv, virtualenv, virtualenvwrapper, pyenv

~~~
hobofan
I think the comic should be more aptly be titled "Life before pipenv".

Pipenv is the first (popular) sane tool to make Python accessible to
programmers coming from other languages. Sure you can also just use bare
virtualenv, but then you have to figure out how it works together with
everything displayed in the picture.

~~~
nothrabannosir
Not to be glib, but your comment ironically reminds me of this:
[https://xkcd.com/927/](https://xkcd.com/927/)

No amount of adding things to the Python comic is going to simplify it.

~~~
cecilpl2
I think the intent here is to convince adherents of other tools to switch to
pipenv. Having used most of the solutions in this thread, pipenv has been the
simplest and the one that "just works" most often.

------
smilekzs
It's funny because within the comment section there are already 2 different
opinions on why this is not funny because $xyz is what every sane python dev
should use.

------
dmart
There was a time where I would have strongly related to this. Thankfully, now
I've started using pipenv whenever possible and it basically just works(TM).

I hope it becomes the standard going forward.

~~~
TomBombadildoze
pipenv doesn't solve any problems well, especially if you require multiple
versions of Python, which is _always_ a problem if you're developing
libraries. You still need something like pyenv to manage multiple versions of
Python.

If you're using pyenv, it's easy to do virtualenv with the pyenv-virtualenv
plugin, at which point you don't need Pipenv because it doesn't solve anything
(except locking, see blow) that you can't do with requirements files.
Furthermore, if you're testing under multiple versions of Python, you'd use
tox to manage your test environments anyway.

_Further_ furthermore, they authors recommend _against_ use pipenv to manage
dependencies for anything but development, so it's on you to have a copy of
runtime dependencies isolated from Pipfile. And since you have to put your
runtime requirements in setup.py anyway, there was never any way around the
issue of duplicating requirements to begin with.

The one thing Pipenv does that pip and requirements files can't is lock
dependencies down the entire tree... except it's abominably slow at that
because it has to download dependencies and walk the entire tree. There's a
debate ongoing about using pip's cache to solve the problem but there's no end
in sight.

If you're really sensitive to having the entire dependency tree frozen to
exact versions of you requirements, just `pip freeze` them occasionally. It's
way less hassle.

Packaging in Python is far better than it was a few years ago. There's still a
lot to improve but pipenv shouldn't be part of the solution.

~~~
SmirkingRevenge
> _Further_ furthermore, they authors recommend _against_ use pipenv to manage
> dependencies for anything but development, so it's on you to have a copy of
> runtime dependencies isolated from Pipfile.

I think the use-case is slightly different than that.

Pipenv is a tool that enables reproducible builds for python _app_ development
and deployment. For example, many web apps, aren't going to be packaged up
using setuptools, they are just going to be deployed somewhere (possibly
containerized first). And with pipenv you can avoid the need to vendor all
your dependancies to ensure reproducible builds.

But its not a tool that solves problems for _library_ packaging and
distribution (on say pypi). You still have to use setup.py for that, and
shouldn't be pinning your dependancies on exact package versions. But you can
still use pipenv to manage your dev environment in those cases, and for
reproducible dev environments.

------
clarkmoody
"This is not a place of honor"

[https://news.ycombinator.com/item?id=11851871](https://news.ycombinator.com/item?id=11851871)
[https://archive.is/MVVU1](https://archive.is/MVVU1)

Explanation edit:

Image alt text is _" The Python environmental protection agency wants to seal
it in a cement chamber, with pictoral messages to future civilizations warning
them about the danger of using sudo to install random Python packages."_

~~~
Depllo
Care to explain a little? The link in the discussion is a broken error page.

~~~
tofof
"This place is not a place of honor" is the first line of the most basic
signage created by a Sandia National Lab project aiming to keep future
explorers/archaeologists/looters/settlers unharmed by a nuclear waste dump.

The problem space is very difficult - it'll be something like AD 12000 before
the waste is effectively harmless. That's further in the future than all of
civilization[1] is behind us. The earliest surviving language specimens are
proto-heiroglyphics that are less than 5000 years old.

It's difficult to craft a message that won't be taken like all the myriad
curse-promising looter-deterrents guarding e.g. plundered tombs throughout the
world. It's especially unconvincing when the hazards we're trying to protect
future humans from include things like a hundredfold increased risk of
cancer.....after inhabiting the site for twenty years.

It's further difficult to attempt to ensure that the message survives that
massive amount of time. The designers came up with a linked system where
elements are supposed to reinforce and index each other, with redundancy of
information and presentation at multiple levels of complexity, with written
and pictographic forms, attempting to avoid overstatement, attempting to
ensure that protective structures cannot be usefully scrapped and re-used for
new construction, ensure that the communications resist deliberate
vandalization, etc.

1:
[https://en.wikipedia.org/wiki/History_of_the_world#Rise_of_c...](https://en.wikipedia.org/wiki/History_of_the_world#Rise_of_civilization)
claims "Though early 'cities' appeared at Jericho and Catal Huyuk around 6000
BCE,[32] the first civilizations did not emerge until around 3000 BCE in
Egypt[33] and Mesopotamia.[34]"

------
BerislavLopac
To add on to the mess, Poetry seems like a strong contestant for a well-
designed Python dependency management solution:
[https://github.com/sdispater/poetry](https://github.com/sdispater/poetry)

------
wruza
Did they forget site-packages vs dist-packages?

It’s so tempting to just apt install everything until it’s too late.

~~~
djsumdog
I forgot about that. That annoys me to no end, and that's purely due to the
Debian/Ubuntu package maintainers right?

~~~
jwilk
What's wrong with dist-packages?

Sure, it caused a lot of grief for Debian packagers (because the name "site-
packages" was hardcoded _everywhere_ ) when it was introduced back in the
Python 2.6 era; but I don't think I've ever had to worry about it as a user.

~~~
coldtea
Users that are programmers also have to package Python packages. Sometimes in
.deb form for Ubuntu too...

It's not just official Debian packages that someone needs to maintain...

------
mkopinsky
One thing that amazes me about the modern PHP ecosystem is that when it comes
to distributable PHP packages, there is only one package manager (composer)
and one central package repo (packagist).

When it comes to binaries it's still a bit of a mess (homebrew? pecl? yum?)
but the native PHP code story is clearer than any other language ecosystem
I've seen.

------
wocram
Am I the only one using Nix (the package manager) for this? It's lovely and
works for other languages and binary package management as well.

I encourage people to try it out, interactive usage looks something like `nix-
shell -p python36Packages.pyaml -p python36Packages.aiohttp --run "python
some-script-with-dependencies.py" `.

------
grx
But why.. every sane programmer uses Python 3 and pip in a virtualenvwrapper
environment, cleanly separated for non-pip packages!

~~~
quietbritishjim
I've never understood the purpose of virtualenvwrapper. I only use three
commands for administering virtual environments: "virtualenv -p python3
~/envs/foo" to make one, ". ~/envs/foo/activate" to activate it, and "rm -r
~/envs/foo" to delete it. That is really not something that needs further
simplification. What am I missing?

~~~
BerislavLopac
Virtualenvwrapper (I hate the name, and as a fish user I actually use
virtualfish instead) has as the main advantage in keeping all virtualenvs in
one place (by default ~/.virtualenvs), so you don't have to worry about the
actual location of your virtualenv. This allows for a number of useful
features like activating, listing and deleting available environments with
autocomplete, and creating a virtualenv is also quite simplified.

But yes, if you're making sure that you're always careful about passing the
location of your environments, you don't really need it.

~~~
MereInterest
It always felt like virtualenvwrapper was moving things in the wrong
direction. Why should the virtualenv be located in the home directory, instead
of in the project directory? The whole point of virtualenv is so that you
don't have environment dependencies, and each project can have its own
environment.

~~~
BerislavLopac
I'm not sure how the location of the environment affects dependencies; the
environment is isolated regardless of where it is located. Also, the
virtualenv location doesn't have to be in the home directory; it can be
anywhere your user has access to, the home is simply most convenient.

~~~
MereInterest
Using a common location means that unique names must be maintained across
multiple projects, which is a recipe for disaster.

~~~
BerislavLopac
Most definitely in automated setups, i.e. in production. (I personally never
depend on virtualenvs in production, preferring to have a fully isolated
environment either via Docker or even better a dedicated VM.)

But in development, where you manually switch between environments, a
centralised setup is great. You don't have to worry about gitignoring the
virtualenv directory, or maintaining paths in general -- a common problem with
virtualenv in your code directory is that IDEs and linters and similar tools
tend to just cut through and parse everything, unless explicitly prevented.
With virtualfish/virtualenvwrapper, the process is simply `workon {envname}`
and you have everything in place.

------
gkya
I guess what Python needs to do is to crown one solution as the official, and
then start imposing on PyPA packages to convert before a deadline or exclude.
Then they can move the non-compliant packages to a legacy archive, and build
on the one solution that the community chose, say in a poll. Granted I've only
ever interacted pip and virtualenv and though it was simple; but looking at
this, things don't have to be this complex.

~~~
pwang
The heart of the problem is that different people are doing different things
with Python, and nobody wants to pay cognitive load for what they don't want
to use.

The packagers and package tool makers of the world inherit ALL of the
technical debt from ALL of the upstream software devs. They're either the
liver or the colon. An upstream dev decided to not document which compiler
they use during dev time? That's now your problem. A compiler maker (GNU,
MSFT, etc.) decides to change how they distribute the C runtime? Congrats, now
it's your packaging system's job to know how to differentiate between Windows
7 and Windows 10 running particular versions of Visual Studio.

Nobody wants a "single simple unified solution" for packaging more than the
packaging tool makers and distro vendors. Believe me. But it's not going to
happen until software devs become take more responsibility for what they build
upstream, and how they build it.

~~~
oblio
Pypi and Python devs themselves could start doing the curation. The package
just isn't accepted into the repo if it doesn't follow X, Y, Z guidelines.

Python is big and important enough that it needs these kinds of quality
assurances. Punting on them is as bad as compromising on language design
decisions.

------
neuland
This doesn't reflect the reality of what devs should be using: virtualenv or
docker.

~~~
zodiac
Why does the official python documentation recommend something else then?

[https://packaging.python.org/guides/tool-
recommendations/#ap...](https://packaging.python.org/guides/tool-
recommendations/#application-dependency-management)

~~~
roboyoshi
This is what I learned. This is what I use. It's nice to have more options
though. Unfortunately that makes it all the more confusing.

------
kingofpandora
Maybe I just never do anything complicated with Python, but I never have any
the problems captured in this comic.

I install all my Python modules into the system's Python _using my OS 's
package manager_. I have both Python 2 and Python 3 installed but beyond
having the same module installed in each of those locations, I have never
found the need for multiple versions of the same module installed at the same
time.

On the rare occasion where my OS doesn't provide a package I need, I use pip
install --user. If it turns out to be something I'll want for the long term, I
just knock together a quick package that I can install/uninstall using
standard OS packaging tools.

~~~
coldtea
> _I install all my Python modules into the system 's Python using my OS's
> package manager._

If that's the case, then it's as you said: you "just never do anything
complicated with Python".

~~~
gkya
Most of the time the language package manager for a very popular language is
an overkill. You can get all the packages you need for web development, AI or
statistical/mathematical stuff in python from say debian packages and build
arbitrarily complex applications. And you have the additional benefit of being
able to easily package your application for many distributions that package
stable versions of popular libraries for languages like python, perl, ruby,
etc. Another additional benefit is that because most of the time you fetch
packages precompiled, you don't need to run probably insecure compilations on
your system (many popular packages for Python for example require you to
compile C extensions). And the trade-off is quite palatable: you use the APIs
hygenically, don't do things like leftpad, and sometimes forego some cutting
edge features (and many dynamic languages enable you to easily backport them
without modifying the package itself most of the time).

~~~
Chyzwar
not really the case if you work in a team or deploy your application.

~~~
gkya
Unless you streamline the development and deployment environments, that is.
You can just work in a VM, a chroot/jail, or probably use Docker (I haven't
ever used it tho, thus not sure).

~~~
Chyzwar
soner or later you will need to solve the same set of issues that virtualenv
and pipenv try to solve.

~~~
gkya
Then I can reach for virtualenv and pip. I did not mean that package
distribution and management in itself is an overkill, but that many types of
projects just don't need these tools. And if one has to use these tools for
all projects and from the get-go, then that means the library ecsosystem is
unhygienic and/or the devs are not really responsible on how they use the
APIs.

------
dguo
One of the reasons I try to use Docker for everything is because it's a cross-
language solution. It does add some complexity and has its drawbacks, but for
me, it's worth it. I'd rather have one global dependency (Docker) than several
(virtualenv for Python, nvm for Node, rbenv for Ruby, etc.), which all have
their own idiosyncrasies.

I also like using Docker because it handles non-language resources like
databases. Even installing one version of Postgres locally was painful. It's
hard to imagine trying to deal with multiple instances and versions on the
same machine (without something like Docker or Vagrant) if two projects use
9.x, and another uses 10.x.

~~~
djsumdog
I use the stock Dockerhub python container with a requirements.txt. Similar to
my Ruby code. It's a lot nicer than virtual envs for deployment.

It's still difficult to debug stuff inside your container (unless you do a
docker exec and then you're limited to the tools you installed) so it's useful
to have the same environment outside of it. `pip3 --user install` has been my
friend; and you can create new users instead of full virtulenvs if you really
need to.

Still, my python environment isn't as screwed up as this comic. System python
is always only controlled by the OS package manager. My own stuff is installed
with --user and things I need to deploy somewhere have a Dockerfile or I have
a Jenkins workflow that create rpms/debs with fpm and push them to my repo and
sign them.

------
Myrmornis
Maybe this is obvious, but to informally install executables supplied by
python packages, use a separate virtualenv for each one and place a symlink to
the venv/bin/theexecutable in a location like ~/bin that you have on your
$PATH.

~~~
icebraining
pipsi is great for that, it takes care of creating the virtualenv, etc.

------
LyndsySimon
It's almost - but not quite - this bad when using a "simple" local development
environment. As others have said it's not nearly this bad when you're
deploying to sane infrastructure, be it containerized or not.

------
avian
One more thing to add to the mess: PyBOMBS [1] A while back GNU Radio went and
reinvented the whole package manager thing on their own. Since then I
constantly have problems integrating anything GNU Radio-related with the rest
of the Python ecosystem. Random Python 2 modules being imported by Python 3?
You bet it's some random gr module in the PYTHONPATH somewhere.

[1] [https://www.gnuradio.org/blog/pybombs-the-what-the-how-
and-t...](https://www.gnuradio.org/blog/pybombs-the-what-the-how-and-the-why/)

------
jwilk
Do edges in this graph mean something, or were they just added randomly?

~~~
titanomachy
Symlinks.

------
sudders
It's funny because it's true

------
rbanffy
What's wrong with virtualenv/venv and pip?

~~~
rimunroe
Those seem great, but it doesn't help much if you haven't been using them from
the start and now find yourself with a very messy environment.

~~~
rbanffy
People should teach proper computer hygiene in school ;-)

------
sologoub
This is so true and so sad that it’s actually funny!

This issue is one of the things that drove me to utilize a dedicated VPS for
each project. This keeps environments clean, ensures project/client isolation
and allows me to utilize the exact resources needed, as opposed to worrying
about the local machine.

Once done or if the prod environment changes, I can blow away the VPS and
rebuild as needed.

------
outsideoflife
I'm using pyvenv even though it has started warning me I should move. I won't
change from it for exactly the reasons in this comic.

When I do a new OS install I will try pipenv again...but it dind't work well
for me the first time due to conflicting with something I had installed
outside of a virtualenv!

------
seanc
No docker containers?

~~~
s_kilk
That would just make the chart recursive.

------
yagyu
The 'framework build and venv on osx' thing ruined my night yesterday so this
post had great timing.

Seriously, starting from a clean osx, how do you install python? I always
liked using vex for venv management. But I would also like to have anaconda
for some tasks..

------
davidfstr
I love this comic.

I was just going through an old server of mine. Python 2 and Python 3
dependencies, installed over years, in a bit of a mess.

We may have virtualenv and the like today, but a lot of old stuff was never
setup to use it. :p

------
vram22
Guido today:

[https://mobile.twitter.com/gvanrossum/status/991083246220853...](https://mobile.twitter.com/gvanrossum/status/991083246220853248)

------
ericfrederich
Try installing pipsi such that it uses python3 to install simple scripts like
httpie or youtube-dl and you'll see how big of a mess it really is.

------
mmcnl
I use pipenv and couldn't be happier. It's dead simple and it works. It is a
bit slow. The problem presented in this xkcd has been fixed.

------
hood_syntax
I think people are missing that he seems to be aware this is a poor way of
doing things...

------
lcnmrn
sudo pip3 install everything

------
est
Need add pypy, cython, etc.

------
Analemma_
In an earlier xkcd blag post, Randall talked about messing up his Linux box so
badly that someone trying to help him fix it said "your computer looks like a
redshirt after a transporter accident". That analogy has stuck with me ever
since and I use it most often to describe Python installations.

~~~
jwilk
[https://blog.xkcd.com/2009/04/03/what-happened-to-my-
laptop/](https://blog.xkcd.com/2009/04/03/what-happened-to-my-laptop/)

------
coldtea
LOL, haven't check if this XKCD was posted today, but it was particularly
timely as I've spend the day debugging a Python + Jenkins + autotools +
setuptools + RPM spec (mock etc) setup...

------
moonbug
Anaconda. That's all you need to know.

------
cecilpl2
This thread is an instance of [https://xkcd.com/927/](https://xkcd.com/927/)

------
devy
All jokes are only funny when they are resembling to the truth but they are
not when it's far from it. So this maybe funny five years ago. But it doesn't
any more since Docker[1].

docker pull; docker run, you are golden. Even if you are going to customize
the official image, it's much less cumbersome than that xkcd cartoon.

[1]: [https://hub.docker.com/_/python/](https://hub.docker.com/_/python/)

~~~
icebraining
Sorry, can't run Docker on the Windows Server 2012 machines provided by the
client. Or on the CentOS 6 machines provided by the other.

~~~
devy
Not sure what your specific situation is, afaik it's possible. So your problem
might not be pertaining to Docker Python.

[https://www.liquidweb.com/kb/how-to-install-docker-on-
centos...](https://www.liquidweb.com/kb/how-to-install-docker-on-centos-6/)

~~~
icebraining
Yes, but you'd have to run a 2.5+ years old - and unsupported - version of
Docker (1.7.1).

~~~
devy
What version of Python you'd be running? And Windows Server 2012 is 5 years
old, no? If you are running a five years old OS, perhaps unsuported Docker
version 1.7 (released 2015) might not be your only and most import concern
right?

~~~
icebraining
We are running the latest Python 2 release, compiled from source. Windows
Server 2012 will be supported until 2023, and CentOS 6 until 2020.

