
Python Ecosystem - An Introduction - mnazim
http://mirnazim.org/writings/python-ecosystem-introduction/
======
nikcub
Great post. I would add:

* the site module, which is imported by default and is what is responsible for setting up the default sys.path. You can skip 'import site' by running python with the -S switch. the site module is written in python, so you can scan through it and understand how python starts up and inits.

* PYTHONSTARTUP env variable, which points to a python file that is run (like a bashrc, or AUTOEXEC.BAT, if you prefer) on interactive prompt startup. I use this to import custom paths and modules that I want to access from the REPL, such as Google App Engine

* I use pip with local repositories. clone the repos of the libs you need, and then pip install in the virtualenv from that local clone:
    
    
        $ pip install git+file:///Users/nik/.python-packages/tornado
    

(note the triple slash). Or straight from GH:

    
    
        $ pip install git+git://nikcub@github.com/nikcub/tornado
    

this can keep your versions in sync across all projects and virtualenvs and it
means no re-downloading and you can setup and update projects while offline.

* don't store the actual project inside the virtualenv. the virtualenv provides the execution context (setup and torn down using the virtualenvwrapper helper scripts). a common practice is to place all your virtualenvs into a directory like ~/.virtualenvs. you should never have to cd into this dir, access it using the wrappers and pip. (edit: also agree with comment below that you shouldn't be sudo'ing).

* just a quick add, I think it is definitely worth learning how to install python from source.

~~~
mgedmin
PYTHONSTARTUP also lets you enable things like

    
    
      * coloured prompt
      * tab-completion
      * persistent history
    

Here's mine: <http://mg.pov.lt/pythonrc>

------
d0mine
A formidable effort.

It might be matter of taste but recommendations given starting from
"Understanding the packages" and to "Install packages that need compiling" are
almost harmful.

My preference:

* you should not care what is your `sys.path` looks like. You need it for debugging if something goes horribly wrong. A tutorial might mention it but things like `sys.path.insert(0,..)` should be avoided or accompanied with a big disclaimer (don't use nuclear weapons if you care about the future)

* the same goes for `PYTHONPATH`. It is a hack that rarely needed

* don't use `sudo pip`. System packages should be managed by a system packager. Use `pip --user` or create a `virtualenv`

* `pip` can handle tarballs there is no need for `python setup.py install` in this case.

"Code Like a Pythonista: Idiomatic Python" is worth mentioning
[http://python.net/~goodger/projects/pycon/2007/idiomatic/han...](http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html)

Some third-party packages that could be listed (it is subjective):

bpython - interactive prompt; something for tests e.g., pytest, tox, selenium;
sphinx - docs; lxml - xml/html, werkzeug - if you talking about web-
development; SQLAlchemy - sql; Cython - C extension, ~ Python syntax; async.
libs e.g., gevent, Twisted.

~~~
mnazim
My indented audience was not the pure beginners. I was targeting for
developers coming to python from other platforms. Over past 2 years, if I have
learned one thing while training interns, trainees and experience devs in
Python/Django, it is that packaging confuses people a lot. apt-get vs
easy_install vs pip. That is why I choose to spend most time on package
management and virtualenv etc. I think, I should have included stronger
indications of this.

In general, your feedback is very good and important. Idiomatic Python
specifically is an important resource and should have been included.

I will incorporate your feedback as soon as I can squeeze out some time.

Big Thanks!

~~~
watmough
Great Pythonesque pun there! indented==intended, I'm sure.

~~~
mnazim
This happens to me all the time. My most favorite is when I have to type the
word "important", my mind spells "I M P O R T A N T" but my fingers
instinctively stop at "import".

~~~
mgedmin
My fingers often type "test" when I want to type "text".

------
simonw
I'd love to have one of these for Ruby. Every time I want to try out something
written in Ruby I run head-first in to the packaging problem - Debian and
Ubuntu don't appear to like shipping a working gem (presumably because it
conflicts with how apt likes to do things) and the documentation on how to
resolve the resulting inscrutable error messages isn't particularly easy to
find. The Mac is a bit better, but I still run in to problems far too often.

I'm pretty sure a "Ruby Ecosystem, An Introduction" guide is exactly what I
need.

~~~
sixtofour
The way these things get written is by keeping notes as you figure out the
bits and pieces ...

~~~
mnazim
You got it spot on. Note keeping the the way to go.

The post was extracted from our internal wiki, the content in which has
evolved over past 2 years and spread all over.

It took me about 20-25 hours, spread over 3 weeks, to put it in the shape of a
single post/tutorial.

------
kisielk
Have you thought about combining your work with Kenneth Reitz's Python Guide?
<https://github.com/kennethreitz/python-guide>

It looks like you're covering a lot of the same ground.

------
kmfrk
I would have loved to hear something about unit testing. Is nose or unittest2
the unit testing framework of choice for Python?

------
briancurtin
> Choose Python 3 only if you need to and/or fully understand the
> implications.

I would apply the "if you need to" part to Python 2. "3 if you can, 2 if you
must"

~~~
jholman
The argument (for installing 2 as a default) is that a novice to the ecosystem
will be disproportionately harmed by the inevitable Python3 experience:

Novice: "I want to do X" Internet advice: "Use package Y" Novice: "Okay
<install, install>, wtf nothing is working" <long frustrating debugging
session> Novice: "Oh, wow, this doesn't support Python3 yet. Now I have to
ignore all the internet advice and forge my own path, OR port all my code back
to 2.7! This language sucks!"

The alternative seems preferable: Stuff works, but occasionally you don't get
a whiz-bang feature (you "only" get 2.7's feature set, poor you). Five years
pass, you learn the language. Now you've gotta learn a bunch of new habits,
which sucks, but it's not as bad because now you're pretty good at Python, so
the easy parts are easy.

~~~
briancurtin
I think they should be asking "How do I do X" before choosing a version to
install. The answer might lead them towards IronPython or Jython, not a
version of CPython from python.org. Some questions might even lead people
straight to PyPy.

More and more questions have problems that can be solved by Python 3, but it's
probably a pain in the ass if you start with 3 then work out how to solve your
problem.

------
detour
"While not a software tool per se, PEP 8 is a very important resource related
to Python."

Actually, it is :)

pip install pep8

------
mnazim
Please keep the great feedback coming. I will try to incorporate as much as
possible.

I am indebted to HN community for the great feedback so far.

~~~
jnoller
As an additional supplement - here's a pile of links I've collected that might
apply: <http://jessenoller.com/good-to-great-python-reads/>

~~~
mnazim
Man. That's one hell of a collections. Thanks for sharing

------
mardiros
I think that $ sudo apt-get install python-pip is a bad idea.

You should not mix multiple packaging system on your operating system. And
more you can dammage it pip provide more recent package than your distro. And
if you upgrade a lib that have an incompatibility with a part of the system,
you can corrupt it. I have no example to give but I am sure you can find it...
Ubuntu now have many tools written in python.

You should use pip inside a virtualenv only. And, fortunatelly when you create
a virtualenv, pip is installed in it, and you don't need to use the
--distribute to have it.

~~~
wavetossed
What I tell people is that Python (as installed by apt-get, yum) is not on
your system for you to develop with. It is there because some of the system
tools are written in Python, including at least part of the apt packaging
system. It's also there for System admins to use for writing Python scripts
instead of bash scripts.

But for app development, get your own Python, manage it yourself and install
'distribute' so that you have both easy_install and pip to work with. I've
taken that to extreme by making a portable Python distro that comes in a
tarball and runs on any Linux distro, but even if you only untar the source
and run ./configure --prefix=/home/python;make

That will build a default Python with support for any shared libraries for
which you have a development version installed ( -dev version on
debian/ubuntu, -devel version on redhat/suse)

sudo make install will install it, assuming that you have write permissions on
the target prefix that you specified. You can even hide it in your home
directory with --prefix=~/tools/python272

~~~
mardiros
I don't anderstand why you want to use the --distribute to have pip and
easy_install installed.

If you do not, you also have those tools.

------
imalolz
Great post - I wish there was a unified resource for things like that for
other languages/tools.

I would only add _iPython_ \- a must for any console adventures.

~~~
MostAwesomeDude
ipython has known issues running certain code. It is not guaranteed to
function like a normal interpreter. If you insist on a fancy interpreter, use
bpython.

~~~
obtu
These issues are not known to me, could you expand?

~~~
MostAwesomeDude
References last far longer than they should, because the interpreter doesn't
always let go of references to returned results.

There used to be a problem with certain statements being executed as
expressions and printed, but I'm told that's been fixed.

The encoding is always Latin-1. Always. Hope you don't use Unicode literals.

It's not Python. It's Python plus other things. That's always a Bad Thing
because it precludes taking results from the interpreter and using them in
plain Python contexts. web2py also has this problem.

------
danso
This is fantastic. Someone could make a well-visited site by creating similar
documents for all the major languages.

------
wavetossed
You really shouldn't tell people to go ask how to install Python on
stackoverflow.com. Instead give them a few URLs to stackoverflow questions
with the answer such as this one
[http://stackoverflow.com/questions/7538834/how-to-create-
a-p...](http://stackoverflow.com/questions/7538834/how-to-create-a-python-
distribution-like-activepython)

Or even better, give them a stackoverflow search like this one
[http://stackoverflow.com/search?q=%5Bpython%5D+%22install+py...](http://stackoverflow.com/search?q=%5Bpython%5D+%22install+python%22)

P.S. I think that your wiki page is a great idea and I'm going to write a
custom one for our developer wiki.

------
krosaen
Though the article claims to be targeted at users running on linux, most of
the the info is still quite useful regardless of the platform - just figure
out how to install python and and pip and the rest is pretty platform
agnostic.

------
senthil_rajasek
Nice article. I am bookmarking it.

Also The Zen of Python can always be accessed by this Easter egg

    
    
        >>> import this
    

<http://www.python.org/dev/peps/pep-0020/>

~~~
sxephil
relax. it's a joke
[http://svn.python.org/view/python/trunk/Lib/this.py?view=mar...](http://svn.python.org/view/python/trunk/Lib/this.py?view=markup)

IMHO this _is_ more useful in the long run ...
<http://www.cs.yale.edu/quotes.html>

------
krupan
I mostly dabble with python. I learned a little reading this, and it raised
some questions for me. Is there a reason --distribute is not the default
behavior of virtualenv? Is there a plan to incorporate the stuff
virtualenvwrapper does into virtualenv (virtualenvwrapper is a pretty
cumbersome name, if for no other reason)?

------
Sukotto
Thank you for this article.

As per the Pragmatic Programmer, I thought I would learn Python this year.
It's been a tremendously _frustrating_ experience getting a workable stack
installed.

I wish the famous " _One -- and preferably only one -- obvious way to do it_ "
Python design philosophy extended to actually installing everything :(

~~~
ak217
Just curious - which OS/distribution are you on? Ubuntu and Gentoo (and
virtualenv+pip if necessary) both make the Python stack a total no-brainer for
me.

------
igorgue
Great article.

If you're on Ubuntu LTS you should install PIP from PyPI (easy_install pip),
since the system package management version is outdated and it doesn't have
the (very useful, since PyPI likes to go down) --use-mirrors install option.
That would be my only recommendation.

------
aaronh
I'm surprised nobody has mentioned pythonbrew.

<https://github.com/utahta/pythonbrew>

It's the Python version of RVM. It is higher level than even virtualenv, and
in my opinion, the most seamless way to manage Python environments.

------
RyanMcGreal
This is very good. I wish a resource like this was around when I was first
learning Python. The difficulty of getting things to work _around_ the
language has always been a pretty stark contrast to the ease of the language
itself.

------
darraghenright
Looks great. I've started turning my attention to python recently so skimming
through this I can already see lots of stuff that'll be very useful. So much
that perhaps a linked TOC at the top of the page could be an idea?

------
snthpy
Thank you, thank you, thank you. I've been looking for something like this for
ages. While Python the language is fantastic, I have found getting into the
whole environment quite tricky so your guide is fantastic.

------
fasouto
Nice article but you have a typo: updrage instead of upgrade :)

~~~
smichael
And at the end, it should be "omission" not "ommision". Thanks a lot, wish I
had this years ago.

------
eccp
Great article. Is it just me or the prepend/append examples are swapped? ie.
If you want to append TO your PYTHONPATH you should do
PYTHONPATH=$PYTHONPATH:/some/new/path

~~~
sirn
It depends on what do you mean by "append", I guess. Append _the search path_
(e.g. "try this one first"), then /path:$PYTHONPATH is correct. If you mean
append _the variable_ then your example is correct.

------
sanxiyn
I am not sure why Requests gets this much press. httplib2 has all the features
of Requests and more, and their APIs are not that different.

------
tchon
NOTE: there's brokenness lurking in urllib and email modules. The python
library modules are more often than not modeled on perfect reality rather than
a pragmatic one (e.g., complete violations of RFCs). BTW, if one reads re.py
in the dist, you'll notice that it hasn't been touched by Fredrik Lundh since
2001!

I should say I still love python. Its the most fun I've had programming next
to Scheme, and well, NodeJS is kind of fun too (in an algol way).

------
ryall
Great post! Covers everything I still didn't understand after reading LPTHW.
Thanks so much for sharing this.

------
e1ven
pip is a neat system and all, but I still don't see why to use that versus a
system package?

~~~
c4urself
if you have multiple projects on a system you may have different dependency
requirements. module with version 1 for one project and the same module with
version 2 for the other project.

~~~
d0mine
pkg_resources.require() can handle this case. Though `virtualenv` is more
explicit.

------
vaksel
would make a good idea for a startup...a Python host that has a simple
checkbox interface for installing all this stuff.

That way you can get started with coding instead of having to install
everything by yourself.

~~~
dman
<http://enthought.com/products/epd.php> (Disclaimer - I work for enthought)

------
axaroth
In the section of "The Development Environment" you should mention the
buildout (<http://www.buildout.org/>). It is a very simple way to reproduce an
enviroment.

------
lamsg
Great great great post!!!!!!!!!!!!!!!!!

------
smogzer
Nice list. I would replace django with web2py, it is closer to the python
philosophy imho.

------
xxiao
great post!

