
Why I hate virtualenv and pip - dissent
http://pythonrants.wordpress.com/2013/12/06/why-i-hate-virtualenv-and-pip/
======
ubernostrum
A lot of this seems to boil down to a combination of seeing others misuse
tools and assuming that's what they're for (which is a communication/teaching
failure, not a failure of the tool), and looking at stepping-stone solutions.

Take binary packages, for example. Sure, eggs did that. Sort of. But they also
introduced a weird parallel universe where you had to stop doing normal Python
things and start doing egg things. So pip eschewed eggs.

And meanwhile the community banded together to find a way to separate the
build and install processes, the result of which is the wheel format:

[http://www.python.org/dev/peps/pep-0427/](http://www.python.org/dev/peps/pep-0427/)

Similarly, virtual environments are still being improved (with the
improvements now being integrated direct into Python itself).

And yes, you _can_ use pip's requirements files as a duplicate way to specify
dependencies. And people do that, and it's unfortunate. Because the thing
requirements files are really useful for is specifying a particular
environment that you want to replicate. That might be a known-good set of
stuff you've tested and now want to deploy on, it might be an experimental
combination of things you want others to test, etc., but that's something
setup.py's dependency system isn't good at. And repeatable/replicable
environments are certainly an important thing.

etc., etc.

~~~
shoo
i was pleased to see wheel mentioned. here is a wheel anecdote:

suppose you wanted to automate the installation of a python package to a
windows machine, but the package has binary dependencies, is irritating to
build from source, and is distributed as a pre-built .exe interactive
installer (click next, next, ...). you can `wheel convert` the exe installer
to get a binary wheel archive, then automate the installation of that wheel
archive with pip. hopefully this isn't a common scenario, but the fact that
pip + wheel make this kind of thing possible at all is very helpful.

~~~
shadowmint
I fail to see the attraction of wheel, and the uptake since it was invented
appears to be basically non-existent.

Can someone seriously explain what the point of it is?

As far as I can tell, it's: make life easier for windows python users (which I
am one of, and I don't care about it at all; you need VS to build everything
anyway; one or two wheel modules makes zero difference; it'll only make a
difference when _everything_ uses wheel, which will be never...).

------
skrebbel
As a programmer who uses a lot of Java and .NET, I was surprised by how
complex pip and virtualenv are. The complexity is hidden well enough if you
just want to get down to business, so it's not really a practical concern, but
still - copy an entire Python environment over? Why? Why can't I just have the
pip'ed libraries in a subdir somewhere and tell Python to look there?

Plain old JARs get this right, Maven gets this right, NuGet gets this right,
NPM gets this right. Why is it so complex on Python and Ruby? Some
technological aspect of the (flexibility of) the languages that need you to
basically copy over the entire world? Or just legacy of unfortunate bad design
choices in earlier days?

~~~
lifeisstillgood
because (and correct me if I am wrong) Jars (and certainly Wars) contain other
bits than the part you wrote.

let's say you are packaging up a java and a python "program". Both print Hello
world to the stdout, but use a third party spell checking package. Both the
venv and the jar will contain that third party package

All python needs to go from venv to jar is a tarball process.

that is roughly were I see us going anyway - ps anyone with good understanding
of jar files please jump in - I would love to map out the python parts against
the jar parts (ie pip install is what part of the java setup process?)

~~~
cnlwsu
> but use a third party spell checking package. Both the venv and the jar will
> contain that third party package

Jar files do not support including third party libraries. You simply cannot
put a jar in a jar. Eclipse/maven etc can explode the dependancies class files
inside of the jar, but people rarely do that for anything other then
distributing binaries of client apps (licensing becomes a massive pain btw).

war/ear files on the other hand does support including dependant jar files.

~~~
lifeisstillgood
I may have mis-spoke (could not re-read thanks to my antiprocrast) : I am
thinking of a package format, probably more similar to .deb than .jar, that
will drop the code I have written, plus the third party packages, plus the
system level packages, ie all the dependancies that can get compiled)

The idea is to divide the Operating system from the application. There is
never a good divide point but at some point we can say the chef/puppet/salt
infrastructure will give us the operating system, and the big binary blob
(.deb) will give us everything else.

Throw in configuration as a seperate function and I am going to have to lie
down in a dark room.

------
coldtea
> _Provided the developer is fully conscious that this is python-level-only
> isolation, then it is useful._

Of course we know it's python level only isolation. We're still running in an
OS with a filesystem and such. If we wanted something more we'd use jails or
something similar.

> _Full methods of isolation make virtualenv redundant_

So what? They are too heavy handed, and 99% of the time, I don't want them
anyway.

> _It is very, very easy to install something as large as a Django application
> into a prefix. Easier, I would argue, then indirectly driving virtualenv and
> messing with python shebangs._

You'd argue, but you'd lose the argument.

> _You need to preserve this behaviour right down the line if you want to run
> things in this virtualenv from the outside, like a cron job. You will need
> to effectively hardcode the path of the virtualenv to run the correct
> python. This is at least as fiddly as manually setting up your PATH
> /PYTHONPATH._

Yes, if only they gave you OTHER BENEFITS in exchange. Oh, wait.

In general, move, nothing to see here...

~~~
dissent
> You'd argue, but you'd lose the argument.

Done it already, in production. The virtualenv fanboys didn't even notice.
It's simple and elegant and works perfectly.

~~~
coldtea
The argument wasn't that it can't be done.

It was that it's "easier" ("Easier, I would argue, then indirectly driving
virtualenv and messing with python shebangs").

Also, "virtualenv fanboys"? Please, are we 16 years old?

------
kevcampb
Why I love virtualenv and pip.

We use virtualenv and pip extensively here, with virtualenvwrapper.

    
    
      mkvirtualenv <project>
      workon <project>
      pip install -r requirements.txt
    

It just works. I don't spend any time on it. Our developers don't have any
problems with it. All the other considerations in the article we either handle
as you're meant to, or understand the limitations.

Still, looking forward to some interesting comments on here.

~~~
jvdh
Did you actually read the entire article, or did you just come here to say
that?

~~~
Anderkent
It extends to deployment. Off top of my head, the script that you have to
write to trivially deploy with virtualenv is something like:

    
    
        virtualenv --no-site-packages -p$PYTHON $workdir
        "$workdir/bin/activate"
        (cd $my-package-dir && python setup.py)
        "$workdir/bin/deactivate"
        virtualenv --relocatable "$workdir"
        fpm -s dir -t deb -n "$package" -p "$package.deb" -d <system dependencies> ...
    

And you have a deb that you can just scp anywhere and it'll just work.

------
lmm
Sure, pip's imperfect; I have to install the mysql header files, woe is me.
But the cost/benefit tradeoff is better than LXC; pip gets me _most_ of the
isolation with much, much less overhead.

Is the author really claiming that it's easier to script a non-virtualenv
deployment than a virtualenv one? If so, great, do that - the _only_ reason I
deploy with virtualenv is because, guess what, that's easier to script.

Why default to --no-site-packages? Because it helps and it's easy. No, I'm not
perfectly isolated from my host system - but then the host system could have a
broken libc and then nothing, not even LXC, is going to make your isolation
system work. Just because you can't isolate perfectly doesn't mean there's no
point isolating as much as you can.

Yes, pip builds from source. That's a lot more reliable than the alternative.
The Java guys certainly _aren 't_ mocking you if they've ever done the
actually equivalent thing, i.e. deploy a library that uses JNI, which is a
complete clusterfuck.

(URLs as dependencies are indeed a bad idea; don't do that. The complaint
about pip freeze is purely derivative of the other complaints; it's wrong
because they're wrong).

~~~
dissent
Yeah, the lxc package gives you command line tools that are a close analogue
to virtualenv. Combine it with a filesystem like btrfs for copy-on-write, and
it's FAR quicker too.

I'm glad you mentioned JNI. In Java, native is the exception. In python, it's
much closer to the rule. A hell of a lot of python libraries rely on C
components which leak out of a virtualenv.

Building from source isn't reliable. It's quite hard, not to mention
relatively slow. See the great success of RPM and APT based Linux
distributions as proof of this.

------
secstate
Why pip?

pip uninstall psycopg2

or

pip install --upgrade psycopg2

But I guess with easy_install you can fake it by running with -m and then
deleting the errant egg files in lib and bin files. That's pretty easy, I
guess.

Oh but hey, you know what you can do instead? Setup a virtualenv, easy_install
everything and when it gets hopelessly out of date or munged, you can just
delete the virtualenv directory and start again.

Snark aside, I would agree with the OP that the "feature" of installing via
arbitrary URLs is an anti-pattern and encourages lazy development. Of course,
not every package we build can be posted to a public package library, so
there's always that issue with easy_install too. Sigh, what a mess we have.
Good thing I'm still able to get work done with these tools :)

~~~
mattdeboard
Good luck pip installing psycopg2 on Windows :)

~~~
secstate
Haha. Win-what? In my domain I am happily able to ignore almost all attributes
of the Windows operating system. Well, except for IE <shakes fist of to the
air>

------
lifeisstillgood
I think we should look back to the rant of Python Core Committer Hynek
Schlawack. ([https://hynek.me/articles/python-app-deployment-with-
native-...](https://hynek.me/articles/python-app-deployment-with-native-
packages/))

In short build your system and it's dependencies once and once only then pass
them around through test into live.

We have three competing needs: a reliable deployment process that can move a
binary-like blob across multiple test environments

A need for exactly reproducible environments but without dd/ghosting

A desire to keep things simple

Isolation is good - whether through the mostly isolated approach of venv, the
almost total isolation of jails/LXC or the vagrant approach. But they focus
almost entirely on binary builds - how does one pass around a python
environment without rebuilding it and it's dependencies each time ala pip?

Well by taking the running built python environments and passing them into a
package manager like apt and calling that a binary. That might mean tar
balling a venv or tar balling /use/local/python but in the end it matters that
we pass around the same basic bits.

I am working this out in pyholodeck.mikadosoftware.com and in my head - when I
have a good answer I will shout

------
viraptor
While I see where he's coming from, I really can't agree with many things he's
saying:

 _" For python packages that depend on system libraries, only the python-level
part of those packages are isolated."_

And there's nothing really bad about it. Well-written python libraries will
work with any previous version of the library they're wrapping. They will also
report incompatibilities. It's ok to use system libraries - especially if
you're advocating getting rid of virtualenv as author does.

 _" Full methods of isolation make virtualenv redundant"_

Well... no. There are times when installing a local version of some library is
required and it cannot be installed system-wide, or it will break system's yum
for example. You're not only isolating your app from the system, but also the
system tools from the app.

 _" virtualenv’s value lies only in conveniently allowing a user to
_interactively_ create a python sandbox"_

There's nothing interactive about what `tox` does for example and it's a
perfect example of why virtualenv is useful. You can have not only a
virtualenv for testing your app, but also multiple configurations for
different selected extras - all living side by side.

 _" Clearly virtualenv advocates don’t want any hidden dependencies or
incorrect versions leaking into their environment. However their virtualenv
will always be on the path first, so there’s little real danger"_

Until you want the same package that's available in the system, but your app's
version constraint is not looked at when the system's package is upgraded. Or
you want different extras selected. Or your deps are incompatible with some-
system-application deps, but you're calling it via subprocess (this is also
where changing the python path in shbang comes useful).

Venvs are definitely not perfect, but for testing and installation of apps,
they're amazingly useful. Binary libs issue is definitely annoying, but
there's a different solution for it and I'm happy to see it used more often -
don't compile extensions, but use cffi/ctypes.

------
praptak
Companion read, "Python Packaging: Hate, hate, hate everywhere" by Armin
Ronacher (June 2012): [http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-
everywhere/](http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/)

------
kyzyl
Has anybody here used conda[1] for anything significant/serious yet? I've been
using it in passing for my side and small projects but I'm still not convinced
I want to go whole hog yet.

Regardless, my experience with it so far has been... ideal. It really makes
building environments and linking/unlinking packages a breeze. I haven't
needed it for building my own packages yet, so we'll see how that goes.

[1] [http://docs.continuum.io/conda/](http://docs.continuum.io/conda/)

~~~
hogu
If anyone needs more color, Travis, the CEO of continuum and the author of
NumPy just wrote a great post explaining why conda exists, and why virtualenv
and pip aren't sufficient for the scientific python community.

[http://technicaldiscovery.blogspot.com/2013/12/why-i-
promote...](http://technicaldiscovery.blogspot.com/2013/12/why-i-promote-
conda.html)

------
INTPenis
To both commenters so far I would like to say PEP-370, a new way since 2.7 to
create virtual environments.

I started using it recently and I see no need for virtualenv anymore.

I have nothing to say about the pip issue though, never had an issue with pip
myself.

~~~
huxley
Back in 2009, Jesse Noller wrote about his experiences with PEP-370 and seems
to have felt that it is an improvement on the status-quo and a complement to
virtualenv but not a replacement:

[http://jessenoller.com/blog/2009/07/19/pep-370-per-user-
site...](http://jessenoller.com/blog/2009/07/19/pep-370-per-user-site-
packages-and-environment-stew)

------
puller
What has the author built to replace pip?

~~~
midas007
Amen.

Generic algorithm of making things better:

0\. Give it a go to fix it oneself first. Really.

1\. Failing the previous, raise the perceived deficiency with a specific and
workable proposed solution.

2\. Failing the previous, indicate what's undesirable and how, and what
behavior would be desirable.

3\. Failing the previous, put a monetary bounty on the feature, fork the
project or live with it. Rewriting from scratch has a 99.99% probability of
being several times more work than it seems.

~~~
pestaa
Replied to wrong parent?

~~~
midas007
Nope. The claim calls out that the author complained without also offering a
constructive, alternative solution.

~~~
untothebreach
While I don't agree with the author's views, I have to say that he _did_ offer
the alternative solution of lxc or vagrant to more fully isolate his python
environments.

------
knappador
The lightweight part is pretty useful. LXC is definitely overkill. I don't
want to have to bridge my graphics and networking over so that I can run
programs against different versions of libraries. Going more lightweight, if
I'm doing PYTHONPATH and PYTHONHOME, I would start scripting them, script the
installation of libraries into my virtual environment that I just recreated
badly...

\--no-site-packages has been default for a while.
[http://www.virtualenv.org/en/latest/virtualenv.html#the-
syst...](http://www.virtualenv.org/en/latest/virtualenv.html#the-system-site-
packages-option)

I don't really see the argument about compiling against system headers and
libs. Generally I do want to isolate my Python modules that are calling into
other binary libs but don't care about isolating those binary libs themselves
because their interface isn't changing when the Python wrapper for them
changes. This is unless they are part of what I'm wanting to develop/deploy
with, in which case the source will be in the virtualenv and install into the
virtualenv using the config script at worst. A frequent example ends up being
how Pygame will compile against some system libvideo.so, the behavior of which
I never change, but may Pygames might have their own API's etc, and so the
many compiled versions do have their own use.

Virtualenv is actually pretty noob friendly because one of the mistakes I see
far more frequently than the others is that users will install things using
pip system-wide that conflict with the system package manager. This can become
pretty difficult to unscrew for inexperienced Linux users.

I've been meaning to actually add some virtualenv docs because of the
frequency with which inexperienced Python and Linux users in general will
waltz in and not be able to compile something because only the old version of
Cython etc are on Ubuntu 11.blah and thus we start bringing in distribution-
specific package managers into the realm Python package management was
intended for and people try to figure out what version of Ubuntu they need
instead of figuring out that they can install everything in one place in many
instances and maintain an entire slew of projects without conflicts and
without calling on IRC when synaptics clobbers things.

~~~
dissent
There's a level between virtualenv and LXC, and that's schroot. Combine it
with a CoW filesystem and that will cover everything you mention. Although
personally, I find LXC very lightweight. Note that in my article, I did point
out that I _do_ still use virtualenv sometimes :)

------
jhull
Interesting take on this. I think everyone just accepts pip + virtualenv as
the only way to be without questioning it. You have definitely convinced me to
reexamine that on my next project.

------
d0m
Well, I really don't agree with any of the arguments in the article. I'll use
the experience card and just say that I've been using virtualenv/pip for years
and it always served me very well. Made development, testing, and deployment
easier. Even if it's hackish and that there are more robust solutions, this
one strikes a perfect good enough of quality versus time versus complexity.

------
aidos
Here's an idea, if you want to convince me there's a better way of doing
something, don't belittle me for the way I'm currently doing it.

I know pip isn't perfect. I know venv isn't perfect. They do work pretty well
though. And when you find something that works well for you in your process,
use it.

Some valid points (many of which have been on articles featured on HN before).
Shame about the tone.

------
rdtsc
Please take a look at using and building real packages for your system. RPM
and APT. These are battle tested, they handle dependencies, transitive
dependencies, multiple repo sources, {pre/post}{install/uninstall/upgrade}
scrips. They provide a transactional way to add software to your systems.

You can use pip and virtualenv as well perhaps by creating a parallel Python
install in /opt or something like that if needed. And then install that in an
RPM if needed.

But if you are installing hundreds of binary files, dlls and using
requirements.txt as the main way to specify dependencies you are probably
going to end up with a mess.

It is much harder if you have multiple OS systems to support. Installing on
Windows, RHEL/CentOS, Ubuntu and Mac OS X is hard in an easy and clean way.
But if you target a specific server platform like say CentOS 6, take a look at
RPMs.

~~~
erikb
This is pretty much what I often discuss with a friend of mine. Like you he
always tells me to use RPM/APT instead of the Python stuff. But my goal is not
that the stuff runs as clean as possible on one specific system, but that it
runs on as many OS's as possible in relatively clean manner. If virtualenv
doesn't tell my Ubuntu's APT that it installed some Python packages in
~/.virtualenv/project_one then an Ubuntu sysadmin will cry and try to kill me,
I really don't care. On the other hand with virtualenv I can make sure, that
my team leader can run the same command lines as myself and get my Python
project running on his Suse box and the QA department can run my code on their
Fedora as well. This is why I use Python and this is why I use virtualenv. I'm
not happy with how things are in Python, but if I compare the problems between
both solution paths currently I still think I'm better off with virtualenv and
co.

~~~
rdtsc
You seem to have good reasons -- multiple OS-es.

I was just saying it is good to be aware of RPMs and APT packages. Even old
and crusty setuptool has RPM support.

    
    
        $ python setup.py bdist_rpm 
    

(add deps setup.cfg and make sure to have a MANIFEST.in file to add extra data
besides python packages to the RPM package).

You seem to need multiple OS-es. That is hard and I haven't found a clean
universal solution.

It also depends on the software. We have lots of mixed,
C/C++/Node.js/Java/Python/big data files. Using virtinst/pip, then unpacking
tar.gz, make ; make install , then java's (whatever it has) then npm all to
setup a repeatable dev and test environment would be a horrible mess. That is
what RPMs packages provide for us.

~~~
lifeisstillgood
This is pretty much what hynek was on about (see my comment below) and what
the guys at Parcel are doing (Iam a johnny come lately with pyholodeck) but we
are trying to do the same thing - build once, package into apt/deb and deploy
many times.

I think the argument above (boss runs different OS) is a fallacy - you want to
deploy to the same target OS, probably in the cloud, so optimise for that
first then fiddle with different OS. I guarantee people will prefer deploying
a cloud server and logging in "just to see" and be happy with manually
bringing things up with `setup.py develop` locally.

~~~
erikb
The code I write is a framework for testers and developers, though. So it
won't run on many similar machines but on many very individual machines. But I
guess it might even be possible to transform APT/RPM packages in each other
and my guess would be that even yum can read one or the other, even if it's
not its main package format (don't know Fedora very well).

You guys really got me excited about this. If I can get any air soon, I'll try
to learn more about it!

------
natrius
Isolating an entire environment is a better idea than isolating a python
environment, but isolating a language's environment is an easier problem to
solve, so the tools for it are currently better. I doubt we'll all be doing it
this way in 2020, but it works pretty well right now.

------
ivan_ah
My friend who is a python guru uses buildout for all his apps.

Some of the recipes I have seen go more into configuration-management-like
stuff but it is cool to see a single buildout script deploy nginx, DB, deps,
and app in one go, on any linux box.

~~~
sillysaurus2
_it is cool to see a single buildout script deploy nginx, DB, deps, and app in
one go, on any linux box._

Would anyone please link to a practical, working example of this? I want to
use buildouts, but I learn from example, and there seem to be very few
examples of how to deploy a production configuration. How do the pros do it?
What are the gotchas? Is there a book I can buy? Will someone please put
together a PDF explaining all this, so that I can throw money at you?

EDIT: Arg, that's exactly what I mean... [https://github.com/elbart/nginx-
buildout](https://github.com/elbart/nginx-buildout) is an ok example just to
learn the basics of buildout, but making it "production ready" (i.e. extending
it to build postgres, etc) is left as an exercise for the reader. I was really
hoping to find a production buildout example... (But thank you, rithi! I
appreciate you took the time to dig that one up for me.)

~~~
e12e
I'm not sure I'd advocate using buildout for this, but as (you've probably
seen) there are a few recipes for stuff like haproxy[ha], varnish[va],
postgres[pg] and nginx[nx] available.

An example of how to combine many of these, for deploying a complex stack,
could be:

[http://www.martinaspeli.net/articles/an-uber-buildout-
for-a-...](http://www.martinaspeli.net/articles/an-uber-buildout-for-a-
production-plone-server)

Another somewhat complex example:

[https://github.com/kybi/buildout-openerp-
postgres](https://github.com/kybi/buildout-openerp-postgres)

You might also find this enlightening: [http://glicksoftware.com/blog/using-
haproxy-with-zope-via-bu...](http://glicksoftware.com/blog/using-haproxy-with-
zope-via-buildout)

[pg]
[https://github.com/securactive/sact.recipe.postgresql](https://github.com/securactive/sact.recipe.postgresql)

[va]
[https://github.com/collective/plone.recipe.varnish](https://github.com/collective/plone.recipe.varnish)

[ha]
[https://github.com/plone/plone.recipe.haproxy](https://github.com/plone/plone.recipe.haproxy)

[nx]
[https://github.com/collective/plone.recipe.varnish](https://github.com/collective/plone.recipe.varnish)

------
pushedx
I agree that developers using virtualenv should be aware that it does not
provude isolation for system dependancies. On two different projects, I've had
to track down a PIL bug on another developer's machine only to find that the
wrong version of libjpg or libpng was installed. Solution? Install the right
version. I've never experienced needing different versions of a system
library, but if I did, LXC with btrfs sounds like an option worth trying to
avoid the overhead of vagrant.

------
Beltiras
I think the author is just misunderstanding what virtualenv is for. I use it
to run different python package versions on the same system. With Pythonbrew
([https://github.com/utahta/pythonbrew](https://github.com/utahta/pythonbrew))
I can even run different versions of Python alongside each other. It's not
meant for easier deployment, it's meant for easier development.

------
jrockway
It depends on what you want. Ultimately, the holy grail is being able to
build, today, a bit-for-bit identical binary to whatever you are running in
production that you built a day, a week, or a month ago. That way, if you need
to debug something obscure, you're not adding extra variables to the equation.

But most people just want to be able to write two apps that use different
libraries, and for that, virtualenv is fine.

------
mathrawka
Whenever I set up a new Django project, it was always a bunch of boilerplate
work to get virtualenv setup. Then if you forgot a step you would have to look
into what is wrong.

After spending some time with Node.js, I have become spoiled by how well npm
works for everything. Albeit, if you want to mess with node versions you need
something like nvm, but it is easy to get the hang of.

Seeing something as simple as npm for python would be awesome.

~~~
leokun
There's nothing in node that can replace Django. Django does so much.

NPM is a wasteland of abandoned projects and nested npm_modules with broken
symlinks to or from a bin that'll never work on your vm's shared directory
(unless it's VMWare I guess, some kind of secret sauce they use).

Having said that, they're all ok tools that people use to build pretty cool
things. I just think there's got to be a better way, and I think it's
somewhere in between an extern and npm/pip thing where you store your
dependencies somewhere where you don't have to worry about a site being down
during deploy and versioning issues.

It's like you freeze and shrinkwrap, but by actually capturing everything you
need once your shit works and putting it somewhere where you control not
someone else.

~~~
grncdr
If you're deploying an application (really, if you're working on anything
_other_ than a library intended to be used by many other projects) check in
your dependencies. This solves a lot of problems regardless of what
language/platform you're developing on.

------
gitaarik
Well one reason to use pip over easy_install is that pip uses https by default
and easy_install doesn't

------
whoisadam
Who is dissent? Who is this 'Adam' guy?

No other posts on this 'python rants' blog, nothing else on HN?

IMO this is _astroturfing_ (
[http://en.wikipedia.org/wiki/Astroturfing](http://en.wikipedia.org/wiki/Astroturfing)
)

~~~
taybin
Astroturfing generally requires a company backing the movement. Why would a
company care enough about pip and virtualenvs and then want to hide that? I
didn't see an alternative offered besides using existing open source tools.

~~~
whoisadam
docker.io comes to mind.

~~~
nickstinemates
I actually did not know about this thread until you mentioned Docker in it.

And after reading it, I've realized I've been plain wrong in how I've been
constructing my python related Docker containers.

Full disclosure: I love pip. I love virtualenv. I use them religiously.

I also work for the Docker team.

------
hardwaresofton
I like pip.

I don't like virtual env quite as much, but maybe I haven't juggled enough
python versions at the same time (I don't often work on such disparate
codebases that virtualenv becomes a real need, and it's a little obtuse to set
up for me)...

------
rmrfrmrf
virtualenv isn't made redundant by Vagrant at all; I use them together all the
time. The remote server I deploy to is completely different from my
development machine, so I use Vagrant to simulate the remote environment.
Since I'm on shared hosting, though, I use virtualenv to make sure that my app
isn't depending on shared modules that are out of my control. It's also a LOT
more reliable than trying to deal with a userdir python build or pip --user,
where you'll probably end up fiddling with envvars anyway, except worse
because you'll be doing it by hand instead of through a script.

------
doki_pen
Binary python packages are useless IMHO. If I want binary packages, I make a
binary package for my package management system. Running pip/easy_install on a
production system is an anti-pattern.

~~~
erikb
You summarize one point I learned from this article and the comments:
deployment and development is not the same. On the development machines it
might make sense to easy_install complete package trees, but in a deployment
environment the most important concern is a clean system wide package
management and for that I need binary packages in a format that the system's
package manager understands.

Thanks.

------
OhHeyItsE
To each his own, I guess. They, together, are two of my favorite tools. Never
had a problem with either of them.

------
adnam
Oh dear. What's the right way then?

~~~
shabble
LXC + eggs + setuptools + something, I guess?

As a current member of the `pip freeze > reqs.txt' brigade, I'd be interested
in seeing a more detailed look at how to do it "right"/better.

~~~
doki_pen
Just create a setup.py and put your deps in there.

~~~
slinkp
Ideally, setup.py is for generic minimum-version-known-to-work dependencies of
a package. Pip requirements files are for very specific known-good
combinations of dependencies of an application.
[https://caremad.io/blog/setup-vs-requirement/](https://caremad.io/blog/setup-
vs-requirement/)

~~~
doki_pen
It's an anti-pattern to not pin versions in libs. setup.py is a contract.
Every version listed as a dependency should have been tested, otherwise the
package will bitrot. It should be safe to pin major versions, if you can trust
upstream packages to follow best practices with version numbers and you aren't
using any "undocumented" features of the lib. Ideally, your packages are
tested and built into binary packages before they are deployed to production
systems. If you are following that best practice, then your risk of a patch
breaking your system are minimized.

------
shn
one should hate something when that thing is something he dislikes but most
importantly it is forced on him or her. I am criticizing only the title, he
might be right about these tools, but using agitative words does not make an
argument better.

~~~
dissent
I admit I'm stirring things up a bit here. You're quite right that I'm
venting.

------
wedesoft
:+1: And the same holds for rbenv and rvm for Ruby.

------
slinkp
I’d put it a bit more broadly: The state of working with Python packages has
been painful for many years, and despite recent improvements and ongoing
heroic efforts to fix things (thanks Tarek in particular), it’s still not much
fun. Piling on hate doesn't help anybody though.

There’s some history worth considering. LXC requires Linux >= 2.6.24, released
24 January 2008. Virtualenv was released in October 2007. So LXC was hardly an
option at the time.

Virtualenv was, I think, a pretty good attempt at a pragmatic solution for
purely python-related dependency management issues at that time. I found it a
hell of a lot easier and quicker than chrooting or building a whole new python
installation (I used to do that) or using (shudder) zc.buildout. System-level
virtualization was pretty heavyweight in 2007.

I think maybe virtualenv is showing its age a bit; I agree with that the
system library isolation issue is a huge hole in the virtualenv approach. But
often, it’s enough to get work done.

As for pip vs. easy_install, anybody who was around at the time (sorry I can’t
tell from the pythonrants blog if that includes "Adam" or not) remembers that
life with easy_install was horrifically painful. It was buggy, often failed
with completely unhelpful messages, and issues with it (and setuptools more
generally) were simply not getting fixed at all. For _years_. (That is finally
changing more recently, thankfully.) Pip was intended to route around all that
(while still using setuptools internally) by doing less and by having less
painful failure modes. As one example: if you tried to easy_install a bunch of
packages, or one package with a bunch of dependencies, and one somewhere in
the middle failed because it couldn’t find a dependency, you’d end up with a
fucked environment that had half of the packages installed and half not. And
since there was no easy_uninstall, you had no easy way to clean up the mess.
And you wouldn’t even have any easy way to know what actually depended on the
dependency that failed to install. Pip took the much nicer approach of
downloading everything, resolving all dependencies, trying to tell you what
depended on something that couldn’t be found, and building everything before
installing any packages at all. So if there was a failure prior to the
installation phase, it had no effect on installed libraries at all. It’s hard
to overstate how much pain relief this provided.

For another example: setuptools and easy_install allow installing different
versions of the same package into the same python environment at the same
time. I’m not sure why that was ever considered desirable, because every time
it actually happens, it’s been a source of nothing but pain for me and didn’t
even seem to work as advertised. Pip took the opinionated approach that only
one version should be installed and if you want different versions for some
other application, just go build a separate environment for it (virtualenv or
no).

I agree that the –no-site-packages vs. –system-site-packages options to
virtualenv are problematic, but for me that’s largely because it amounts to a
binary choice between having to build all C extensions from scratch vs. having
to depend on whatever happens to be installed system-wide, which is a pretty
poor choice to have to make.

As for the statement that “there’s little real danger” because “their
virtualenv will always be on the path first”, he's forgetting (or possibly
doesn’t know) that the easy_install.pth file messes with sys.path as well. I
had several hair-tearing sessions trying to figure out why on earth the wrong
version of some package was getting imported before I realized that. At one
point I found that import behavior changed depending on which version of
setuptools or distribute I had installed. That was fun.

It’s also true that people often misuse requirements files. I don’t think
that’s the fault of the tool. Depending on the current commit of some master
or other branch is idiotic regardless of what mechanism you use. Depending on
a specific tarball URL is no more or less reliable than depending on a
specific package version being available via pypi (hasn't everybody had the
experience of somebody yanking away a package version you depended on?). The
right thing to do is probably neither, but to host all your dependencies
somewhere that you control (whether that’s your own python package index or
.deb or .rpm server or whatever).

~~~
dissent
> he's forgetting (or possibly doesn’t know) that the easy_install.pth file
> messes

The .pth files don't do anything until their own paths are added to sys.path.
It's safe to consider directories with .pth files as if they contain python
modules.

------
aerolite
ridiculous.

