
A non-magical introduction to pip and virtualenv for Python beginners - j4mie
http://dabapps.com/blog/introduction-to-pip-and-virtualenv-python/
======
JulianWasTaken
> Python actually has another, more primitive, package manager called
> easy_install, which is installed automatically when you install Python
> itself.

It's actually not, it's part of setuptools/distribute, though some Python
distributions (actually just brew that I know of) include distribute alongside
Python.

Also, while the quick skim of the rest of this looks mostly good, there's some
unnecessary advice which complicates things.

virtualenv is not necessary if the crux of the advice is "don't install
packages globally" (which is fantastic advice, second probably to the more
important advice "don't install stuff with sudo").

What beginners (and even some more experienced programmers) need to learn
about is __\--user __. You install packages with `pip install --user thing`,
and they become per-user installed (be sure to add `~/.local/bin` to your
`$PATH`). This is enough to not require sudo and for 99% of cases is actually
sufficient.

There is a rare case where you actually have packages that depend on
different, conflicting versions of another dependency. This has happened ~1
time to me.

Don't get me wrong, I like virtualenv for other reasons (relating to workflow
and maintenance), but if we're teaching people about packaging, there's no
particularly great reason to dedicate most of a tutorial on it.

~~~
pirer
How would you someone who has used pip with sudo undo the mess that has
created in his computer?

Uninstall everything and start over?

~~~
jyap
Yeah. You can use 'pip uninstall'.

~~~
pirer
I ended doing the following for those who are in the same situation:

    
    
        pip freeze > pip_list_with_sudo.txt
        sudo pip uninstall -r pip_list_with_sudo.txt
    

So now let's start doing things right with virtualenv...

------
HeyImAlex
Want to make life even easier? Check out virtualenvwrapper.

<http://virtualenvwrapper.readthedocs.org/en/latest/>

Lots of benefits, but trading 'cd path/to/my/project && source
env/bin/activate' for 'workon project_env' (with autocomplete) is alone easily
worth the five seconds it takes to check it out.

~~~
j4mie
I find it's really useful for beginners to clearly understand what's actually
happening under the covers before they start adding magical stuff on top. They
can always add magic later, if they prefer that approach.

------
Daiz
Man, global system-wide installations that require admin rights by default?
That's certainly something! Quite the stark comparison to Node.js and npm,
where everything is installed locally into the current directory (under
node_modules) by default, and "global" installation is actually a per-user
installation. Tricking pip with virtualenv seems to get you pretty close to
what you get by default with npm, albeit still somewhat more clunky. But to be
fair, most other package managing solutions seem to pale in comparison to npm
:-)

Either way, nice article. Now if only most packages weren't still for Python
2... PyPI says it has 30099 packages total, but only around 2104 of them are
for Python 3 (according to the number of entries on the "Python 3
Packages"-page[1]).

[1]
[https://pypi.python.org/pypi?:action=browse&c=533&sh...](https://pypi.python.org/pypi?:action=browse&c=533&show=all)

~~~
tomku
npm's default for -g is to install to Node's prefix, which is usually /usr or
/usr/local. If you want it to install to your home directory, you can set the
prefix to somewhere appropriate in your ~/.npmrc, which gives roughly the same
behavior as pip's --user flag.

Edit: perhaps you changed your .npmrc or set the option via npm and forgot
about it? I just checked on a fresh user, and 'npm install -g' definitely
tries to install to /usr, just like pip.

~~~
Daiz
I use Windows as my main OS, and by default npm -g installs packages to
%AppData%, which is user-specific. I guess it's different on *nix, then.

~~~
e12e
That sounds broken to me. Then it's no longer -g for global. You do run your
app under a different account than your user-account, right? Something like
"nodeuser"?

~~~
Daiz
>You do run your app under a different account

No, why would I do something like that on a development box? (Or run web stuff
on Windows servers for that matter.) And pretty much the only things I install
with -g are useful CLI tools - any code I write will have its dependencies
installed locally and listed in package.json for 'npm install'.

~~~
e12e
It wasn't clear (to me) that this was a development box. And it certainly
wasn't something npm could know -- so my point still stands. If there's a way
to install packages globally, then they should be globally available -- also
on windows. But perhaps this is documented somewhere.

As for why you would run stuff on windows, perhaps you were writing an ajax
gateway to a legacy system and it made more sense to run the node server on
the same machine as the legacy system?

(To be clear, I would pity you if that was the case, but you never know ;-)

------
almost
If you're in Brighton in the UK and you like this then maybe you'll be
interested in the one day workshop run by Jamie (author of the blog post) and
myself. This post was actually based on some of the material written by Jamie
for the course.

Next one is happening next Thursday and there are still a couple of tickets:

<http://dabapps.com/services/training/python-for-programmers>

------
wmt
"pip is vastly superior toeasy_install for lots of reasons, and so should
generally be used instead."

Unless you are using Windows, as pip doesn't support binary packages.

------
phren0logy
Nice article, but after using leiningen (the clojure solution to a similar
problem, based on maven), it's really hard to go back to something like this.
I really, really wish there was an equivalent in python (really, every
language I use).

------
tocomment
He mentions not checking the env directory into git. Why not?

In general what are the best practices for using virtualenv with version
control?

~~~
arnarbi
I find it best to keep virtual envs completely away from the project (I use
<http://virtualenvwrapper.readthedocs.org/en/latest/> which puts them by
default in ~/.virtualenvs). A virtualenv is completely machine-specific.

If your project is a package itself (i.e. it has a setup.py file), then use
that file to specify dependencies. On a new machine I check out a copy, create
a virtual env and activate it. Then in the local copy I run "pip install -e
.". This installs all the requirements from setup.py in the virtualenv, and
links the local copy of my project to it as well. Now your package is
available in the virtual env, but fully editable.

If your python project is not a package, you can install its dependencies in a
virtual env with pip. Then run "pip freeze" to generate a list of all
installed packages. Save that to a text file in your repository, e.g.
``requirements.txt``. On a different machine, or a fresh venv, you can then do
"pip install -r requirements.txt" to set everything up in one go.

~~~
turtle4
Alright, so after I set up the environment using pip and virtualenv, I see it
has python in it, etc. If I use pip freeze > requirements.txt, it lists the
packages I have installed using pip, but it doesn't list anything for the
python version itself. How do I make sure the right python version gets
captured if I don't check in the /env/ folder?

~~~
hcarvalhoalves
> How do I make sure the right python version gets captured if I don't check
> in the /env/ folder?

Document it in setup.py:

    
    
        if sys.version_info < (2, 6, 0):
            sys.stderr.write("Foo requires Python 2.6 or newer.\n")
            sys.exit(1)
    

You're using setup.py, right? ;)

------
hcarvalhoalves
I would like to just give a difference advice regarding creating virtualenvs
and installing dependencies:

When you create the virtualenv, the _current_ package you're working on
doesn't get added to site-packages, so you're forced to be at the repository
root to import the package.

The best approach is to have a proper setup.py file so you can do `python
setup.py develop`, which will link the package you're working on _into_ the
virtualenv site-packages. This way it acts as it's installed and you can
import anyway you like.

If you define your requirements on the setup.py (I think you should), you can
even skip the `pip install -r requirements.txt` step.

I've cooked up a package template that can help getting this working:

<https://github.com/hcarvalhoalves/python-package-template>

------
toukon
I'd like to know what the best practices with regards to security are for
using pip, or installing packages in general.

How do you verify package integrity? Do you simply pray that PyPI isn't
compromised at the moment, or do you download your packages from Github
instead, because the main repositories have more eyeballs on them?

How do you do security updates with pip?

I'm using apt-get at the moment which gives me security updates AFAIK, but my
need is growing for more recent versions and certain packages that aren't
accessible with apt.

~~~
thruflo
One important note is to use pip>=1.3 (included in virtualenv>=1.9) as prior
to this version, pip downloaded from pypi using http and was thus vulnerable
to man in the middle attacks.

You might also like to check out wheel, which allows you to compile signed
binary distributions that you can install using pip.

------
denzil_correa

        Python actually has another, more primitive, package manager called 
        easy_install, which is installed automatically when you install Python itself. 
        pip is vastly superior to easy_install for lots of reasons, and so should 
        generally be used instead. You can use easy_install to install pip as follows:
    

I found it quite ironic that the author says pip is "vastly superior" to
easy_install and then proceeds to install pip using easy_install.

~~~
kenny_r
It's similar to using Internet Explorer to download Chrome or Firefox.

------
riobard
Really hope someone could write a similar introduction to buildout
(<http://www.buildout.org/>)

~~~
e12e
You might enjoy:

<http://jacobian.org/writing/django-apps-with-buildout/>

Which covers a lot of the same ground, but with buildout.

------
nateleiby
Thanks for the article! I recently spent some time going through the process
of learning VirtualEnv / Pip; after looking at several tutorials, agreed that
this is one of the better ones out there. A few other things I saw in other
articles that you might like to clarify (though I understand there's a
tradeoff with simplicity/clarity).

(1) specifically state that `pip freeze` is how to create requirements file
(as folks have said in comments already) (2) add "further reading" link on
VirtualEnvWrapper, as it adds some convenience methods to ease use of
VirtualEnv (3) the "yolk" package shows you what's currently installed; it can
be helpful to `pip install yolk` then run `yolk -l` to view the different
packages installed in each of your envs. (4) when installing a package, you
can specify the version, e.g. `pip install mod==1.1`, whereas `pip install
mod` will simply install the latest

------
pbreit
For Mac, am I better off with Homebrew?

[http://docs.python-
guide.org/en/latest/starting/install/osx/...](http://docs.python-
guide.org/en/latest/starting/install/osx/#install-osx)

<https://github.com/mxcl/homebrew/wiki/Homebrew-and-Python>

~~~
thruflo
No. Use homebrew for system packages and use pip to install python packages.
It's much more flexible and doesn't rely on package managers keeping up with
releases.

In the real world, a typical pip requirements.txt file will have a mix of
package names (which pip looks for and downloads from an index like pypi), git
repo urls (eggs installed directly from a git server, eg from github) and
bleeding edge track the latest changes -e git-repo#egg=eggname urls. That you
can switch between these with ease is important, eg to switch to using your
fork of some package rather than the last official release.

------
netcraft
a good introduction - I would like to hear more about deployment with
virtualenv though - is it expected that you just document any packages with
requirements.txt and then you would create the virtualenv in the deployment
target and set everything up again? Or can you "package" a virtualenv for
deployment?

~~~
HeyImAlex
Just generate your requirements file from your env locally (pip freeze >
requirements.txt), deploy all of your files (env folder excluded) to your
sever however you want and then run 'env/bin/pip install -r requirements.txt'
on your server.

~~~
turtle4
If I use pip freeze > requirements.txt, it lists the packages I have installed
using pip, but it doesn't list anything for the python version itself. How do
I make sure the right python version gets captured if I don't check in the
/env/ folder?

------
didip
Calling env/bin/python directly, as opposed to `source activate` is very handy
for things like:

* bash command line

* cron

* or daemon manager like supervisord

~~~
thraxil
Yep. Years ago I settled on a convention of always installing a virtualenv
into a 've' directory in the project so I can just set the shebang line on
scripts (eg, django's manage.py) to "#!ve/bin/python". My Django project
template sets all that up for me automatically. So now I just have muscle
memory for typing "./manage.py ..." etc and I never have to activate
virtualenvs, mess around with virtualenvwrapper type hacks or accidently run
one project's script with a different project's virtualenv activated.

------
yew
Speaking as someone relatively new to Python (coming from an embedded
development background, mostly with C/C++): What's the standard way of
distributing client-side programs/libraries? If you only have one script you
can just put it in /usr/local/bin/ but otherwise you have to mess with the
system site-packages or modify sys.path before importing, right? I've seen a
surprisingly large number of distros that didn't check /usr/ _and_ /usr/local/
for packages.

Do you just hand the user an automatic virtualenv script? (Outside of using
one of the binary builders out there, obviously.)

------
milliams
Some of the mentioned problems with the traditional method are partially
solved with `pip install requests --user` but I understand that the bigger
problem/main reason for virtualenv isn't helped by this.

However, I was very surprised that the author didn't mention venv
(<http://docs.python.org/3.3/library/venv.html>) at all since it is basically
virtualenv but part of the Python standard library.

~~~
tbatterii
i tried pyvenv last week. and it was pretty useless without pip + distribute.
So virtualenv still wins IMO.

------
ak217
Excellent, amazingly well-written guide. This should really be put on
python.org as an "Intro to Python Package Management" or something.

------
Pxtl
VertualEnv seems less like a brilliant tool than it does a workaround for an
architectural problem in Python and pip.

That said, dependency hell is _always_ tricky, and I've had to deal with some
far uglier solutions in other platforms.

~~~
tbatterii
you aren't implying that dependency hell is a problem unique to python/pip are
you?

dealing with dependencies is always step 2 for me when learning a new language
and dependency hell seems like a universal problem. I could be wrong though.

~~~
tootie
Honestly, Maven is way easier, more powerful and works identically on Win,
Linux or Mac. The key is that Java let's you set classpath as a command line
arg and pythonpath is an environment variable.

~~~
tbatterii
IMO, buildout is more like maven for python. but again the tools solve this
problem of "dependency hell" which to me is a problem all languages have, not
just python

------
rahduro
Excellent tutorial, new to python, and I always hated when things just install
without letting you know where it's going..and next time there is some upgrade
all sorts of weird errors keep coming..Thanks a lot.

------
rvasco
Flawless explanation. As a somewhat beginner to python i must give my thanks !

------
Alex3917
Thanks for this article, it was definitely needed. I've known that I should be
using virtualenv for a while now, but actually trying to figure it out has
been somewhat daunting until now.

------
rdtsc
How does it play with OS packages?

Do I package virtualenvs in an RPM or DEB?

~~~
hoov
It's orthogonal. Create your base image, at deploy time, create a virtualenv
and install your specific artifacts.

------
mkoryak
Can someone explain the historical reasons for this problem even existing in
the first place? (the problem that virtual env solves)

~~~
HeyImAlex
Did... did you read the article? Separation of environments for projects with
different dependencies.

~~~
mkoryak
java does not have virtual envs, and i can have projects with different
dependancies. clearly this is a python problem. I was asking why it exists.

------
anonozdotcom
My mentor did not even give such a good introduction. Kudos to the author, now
I have a bright AHHA lightbulb in my head! :D

~~~
suyash
That is why they say to take multiple advice from multiple sources.

------
vram22
Interesting article.

