
Python Packaging: Hate, hate, hate everywhere - uggedal
http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/
======
jnoller
_sigh_ If you want to _help_ , check out the Python-dev thread where this is
being hashed out ad-infinitum rather than flamed to death:
[http://mail.python.org/pipermail/python-
dev/2012-June/120430...](http://mail.python.org/pipermail/python-
dev/2012-June/120430.html) \- for the tl;dr crowd - yeah, things have
problems, want to fix.

~~~
the_mitsuhiko
> check out the Python-dev thread where this is being hashed out ad-infinitum
> rather than flamed to death

Where am I flaming anything to death here? The sole purpose of the post was to
point out that certain things worked in setuptools that broke with the
introduction of pip. That's neither bashing setuptools or pip, just stating
something that you can observe yourself.

It's an observation of an unfortunate effect of someone else taking the work
of someone else to make a new tool and missing a usecase that the old tool
supported with the end result of having two tools with overlapping usecases,
but neither is a superset of the other.

With some modifications virtualenv + pip works perfectly fine for us. It just
takes some extra work and it's unfortunate that it had to happen. Personal
lesson learned: try not to replace tools in the future without understanding
all the usecases of the old tool first.

> If you want to help

I think that's one of the big fallacies of open source: that you should fix it
yourself rather than saying that something is not working. I reserve the right
to feel unhappy when something does not work and I reserve the right to share
my experiences with others without having to feel guilty about not helping
out.

I tried patching pip to be able to install binary eggs but it turned out to be
impossible without replicating all the logic in easy_install because pip by
design does not follow it. Rather than making pip2 I decided against it and
looked at what the problem actually is: we want to have fast builds. So we
could solve the problem by copying over virtualenvs and fixing the paths.
Perfect? No! But it's a simple fix and solves our problem.

~~~
ianb
I didn't put binary support into pip because (a) it didn't work well on some
systems, and (b) I didn't use the other systems. I am not aware of rejected
patches to support binary packages on Windows (where it would be nice to
include), or even rejected patches on other systems (where I think it would be
misguided).

That this functionality still doesn't exist years later I think is due in part
to the problem domain – people find it easier to change parts of their
workflow than to add this functionality, and the whole problemspace is kind of
hairy which makes it hard to advance without causing regressions in somebody
else's weird workflow. (And admittedly the code is hairy too – but who wants
to fix that? I don't! No one does!)

~~~
the_mitsuhiko
> I didn't put binary support into pip because (a) it didn't work well on some
> systems, and (b) I didn't use the other systems. I am not aware of rejected
> patches to support binary packages on Windows (where it would be nice to
> include), or even rejected patches on other systems (where I think it would
> be misguided).

I did not even manage to come up with a patch that adds binary support. I
looked into it one or two months ago and the way pip finds and installs
packages is very different from how easy_install does it. It looked like too
big of a task for me to do in the time I allocated for it.

Until two months ago I did not care at all about binary distributions. It
never was an issue for me. Suddenly a requirement change and I was presented
with a problem that turned out to be tricky to fix with pip but easy as pie
with easy_install.

This is not a criticism on either pip or you as a person, just an observation
of how easy it is to miss something because it does not fit ones personal
requirements. I certainly did not care about binary eggs until I had to deploy
code to more than one machine and wanted to have the ability to quickly switch
between releases.

> And admittedly the code is hairy too – but who wants to fix that? I don't!
> No one does!

Yes. And now it's getting replaced with a new system and I hope that's not
going to make the same mistakes. I certainly don't feel like I'm up to the
task of making a replacement for distutils/setuptools.

~~~
ianb
I think you are confusing "missing a requirement" with "not prioritizing a few
other people's requirements". That pip is missing this feature has always been
documented, I always knew it was a missing requirement, but I did not have any
personal motivation to resolve that.

~~~
j-kidd
Pip is great for many many use cases, but I think it kind of get overhyped as
a replacement for easy_install. People just get upset when they encounter
situation where it doesn't live up to the hype.

~~~
slurgfest
What hype? pip can uninstall packages. This is a very hard feature to live
without for everyone. Binary eggs seem to have a devoted following but are not
a universal need.

~~~
the_mitsuhiko
> This is a very hard feature to live without for everyone.

I never ever uninstalled a package since I started using virtualenv. I always
just trash the venv every once in a while and recreate it from a
requirements.txt.

------
dice
As a sysadmin pip and its ilk annoy me as well, although for a reason which is
not mentioned in the article: It creates an entire package management system
which _is not the distribution's package manager_. Ruby's gems tick me off for
the same reason. As a sysadmin you need to decide to either manage Python and
Ruby entirely outside of the OS's native package management or try and wrap
every single Python egg and Ruby gem in an RPM/DPKG/whatever. Mixing the
management between two packaging systems is just going to cause trouble.

There's a similar problem with Perl's CPAN, which is what I think all of these
projects are aiming to emulate, but the nice thing about CPAN is that
repositories like rpmforge or EPEL already have a large number of commonly
used CPAN packages all wrapped up in RPMs already. It would be nice to see
similar community efforts centered around Python and Ruby packages. Perhaps
one day I will have enough free time to start one.

~~~
timtadh
The problem is the _distributions_ packages are often horrendously out of
date. I have stopped using them entirely even for things like Numpy and Scipy
which I would really rather use them for (as they take a long time to compile
and rely on several C libraries and a fortran compiler). It isn't my favorite
thing to have my fabric task compile all the dependencies but there isn't
really an easy way around it.[1]

In general I think pip has drastically improved my work flow and code
structures. For instance: I no longer use submodules for my library code, I
pip install the git repositories. Huge improvement, way easier to manage.

[1] Obviously, you can get around it if you do things like copy the virtualenv
and fix the paths as in the OPs post. However, I wouldn't call that "easy".

~~~
dice
It's true that the distro's packages can often be out of date: particularly
with "Enterprise" releases like RHEL/CentOS. That doesn't mean that one
shouldn't use the package manager, though. Building custom packages to
backport updates and tracking the upstream for security and bug fixes isn't
the funnest thing in the world, but it is often a necessary evil. It's
certainly better than a "compile from source then dump everything into a
tarball" approach which leaves you with no good way to track what versions of
which software are installed on which nodes.

~~~
slurgfest
It isn't a necessary evil when it isn't necessary. If you have isolation (like
virtualenv) then each app gets exactly what it needs. Life is too short to
waste it trying to make every single bit of Python use the same versions of
dependencies, pinning the version in the package manager, etc.

The universe does not revolve around lazy sysadmins

------
prodigal_erik
I would have liked to see a rationale for using virtualenv and all this
python-only stuff, when rpm/apt/msi had more mature tools and would have given
sysadmins a unified sanely manageable view of what's deployed and their cross-
language dependencies.

~~~
slurgfest
I am going to guess you are coming at this entirely from the perspective of a
sysadmin who doesn't do a lot with python.

virtualenv allows you to easily isolate dependencies so that multiple versions
can coexist in different projects without fighting. If you sysadmin
significant amounts of Python stuff then you would have already seen that
benefit. It also allows you to control PYTHONPATH more elegantly than setting
it in every script. This lets you write clean sensible imports rather than
using fragile path tricks or wrestling with relative imports.

Your project dependencies are often not exactly the same as what the
distribution gives. On one hand, distributions can be mind-bogglingly slow
about releasing updates to python libraries. On the other hand, you may not
want to force your project to use a new update of something because it may
introduce bugs. It is pretty essential to have some lead time to port if your
app is anything important. So you need control of the versions of your
dependencies.

So if your project dependencies are not exactly the same as what the
distribution uses, then you will need some kind of isolation mechanism in
order to allow your project to work as a distribution package. You could do
that, but you could also save time and just use virtualenv.

Creating platform-specific packages is nontrivial, also it is platform-
specific. If I write a library for Python I only want to specify the packaging
metadata/scripts once. I am certainly not interested in trying to get it
accepted in repos for every distribution.

~~~
rlpb
> Creating platform-specific packages is nontrivial, also it is platform-
> specific.

Unfortunately, Python-specific packages are Python-specific. I don't want to
manage one package manager for Python, one for Ruby, one for Perl and yet
another one for system libraries. And I'd prefer to have to trust only one
vendor for security updates for everything.

And anything that is not Python depends on system libraries, which are
supplied by apt or yum and live in an distro-specific namespace. How does
setuptools declare and pull in a dependency on a system library?

I understand the need for Python, Ruby etc. to have their own packaging for
cross-platform use, including on non-Unix. But system packaging tools have
their uses too. There is no easy answer, and certainly no One True Answer.

~~~
slurgfest
I never even implied that system packages do not have their uses, or that
python packages are free-standing of libc. That is just a straw man.

In fact, I agree with you for system libraries or tools which happen to use
Python and really do not need virtualenv because they are really system-
global.

Where I don't agree with you is with use cases you clearly don't understand,
where you are developing and/or running multiple python apps in which you NEED
isolation or you NEED to manage the versions of your dependencies. Your
package manager is not helping with that at all.

And if you don't understand these tools and configuration management tools,
then you should not be a sysadmin for projects which involve significant
amounts of Python or Ruby.

------
defen
Non-python programmer here. The author refers to "Python's idiotic import
system" - can someone explain why it's idiotic? How does it differ from Ruby
or Perl (both of which I am familiar with)?

~~~
kingkilr
I'd say the biggest difference between Python and Ruby's is that Python's has
a syntax, whereas Ruby's is a function call, this makes Ruby's more flexible
since you can always replace it, add new arguments, or in other ways
manipulate it. I'm not sure if that's what Armin is referring to though.

~~~
lloeki
> I'd say the biggest difference between Python and Ruby's is that Python's
> has a syntax, whereas Ruby's is a function call

This is not true on two fronts.

First, the import statement is a convenient, hookable wrapper around the
__import__ function. It's quite customizable already. See PEP302 which
references and allows implementation of PEP273 (importing modules from Zip
archives). Other points of reference include [0], [1] and [2]

Also, the biggest difference is that when you 'require', you require a
filename (i.e the arbitrary content of a file in the filesystem), but when you
'import', you import objects from a namespace into a scope. The namespace is
resolved as an item living in the filesystem. In Ruby I could require
'foo/bar' and it could create the constants Foo::Bar and Qux::Quux. I have no
way to require Qux::Quux. Rails for example tries to autoload constants based
on their names, but for all I know, the second I reference Foo::Bar, it could
actually define Qux::Quux. Ruby encourages this style of programming,
spreading extendable classes and modules around in multiple files, while
Python decides that things should be contained and well-behaving, and not
trivially pollute unrelated module namespaces.

My analysis is that by definition you just can't have both, and that either
one has a set of benefits and drawbacks.

[0]
[http://docs.python.org/library/importlib.html#importlib.impo...](http://docs.python.org/library/importlib.html#importlib.import_module)

[1] <http://docs.python.org/library/imputil.html>

[2]
[http://docs.python.org/py3k/library/importlib.html?highlight...](http://docs.python.org/py3k/library/importlib.html?highlight=importlib#importlib)

------
slurgfest
Yes, setuptools is broken. Proposal: don't use setuptools!

It's funny that the author doesn't make any mention of this, which is not new
news: <http://docs.python.org/dev/library/packaging.html>

Maybe not every use case anyone ever proposed needs to be standardized.

~~~
varikin
That isn't the takeaway I got. It was that Setuptools is broken, but so is pip
and distutils2. Pip lost very worthwhile functionality that exists in
setuptools, namely binary eggs.

~~~
slurgfest
Binary eggs aren't a good reason to trash pip, which otherwise works great. He
really didn't say how distutils2 was broken, just expressed some sort of
prejudice against it.

~~~
masklinn
> aren't a good reason to trash pip

Good thing that never happened then.

~~~
slurgfest
pip isn't broken just because it doesn't support a particular use case from
setuptools.

~~~
masklinn
Again, it would be nice if you stopped making shit up on the spot, TFA didn't
say _anywhere_ that pip is "broken" or anything even remotely close to that.

------
socratic
Does anyone know if namespace packages actually work?

I've been wanting to put up a few utility libraries on Github. I don't think
they're good enough for PyPI, and I mostly just want to be able to pip install
them and have access to a few minor functions that I use periodically.
However, I don't think they deserve their own top level library name, ideally,
they'd all be under "socratic.*" or similar. However, there seems to be some
sort of mess with init in the top level of the package, some pkgutil fix, some
sort of incomplete PEP (maybe 402?), etc. Should I just give up and name my
internal libraries "socratic_{name}" and be done with it? Or does having
multiple packages with the same namespace work?

~~~
zzzeek
it has some rough edges, sometimes when I try to build the docs for one it has
a hard time importing both packages until I kick it a few times, but they
work. I'm doing it for these two modules:

<http://pypi.python.org/pypi/dogpile.core/>
<http://pypi.python.org/pypi/dogpile.cache/>

edit: also they work fine with pip so not sure what armin's issue here is.

~~~
the_mitsuhiko
> edit: also they work fine with pip so not sure what armin's issue here is.

See this issue which ultimately made me ditch them:
<https://github.com/pypa/pip/issues/3>

------
userulluipeste
Hate, hate, hate on specific platform, not everywhere! All that trouble that
he complains about is only on *nix. For Windows is quite simple. Install
Phyton, then look here for other packages:
<http://www.lfd.uci.edu/~gohlke/pythonlibs/>

Edit: I should have said ”For Windows and ReactOS”

~~~
wesm
Programmatic deployment on Windows is still a massive problem (especially for
extension modules-- e.g. the entire scientific Python ecosystem).

~~~
samwillis
I have used Pip, Virtualenv and virtualenvwrapper-win on windows with very
little trouble... except for anything that isn't pure Python.

In order to install PIL in a virtualenv on windows I ultimately ended up
downloading the PIL windows installer, extracting the required files and
placing them in the virtualenvs site packages directory.

This is what needs fixing! Pip with access to compiled windows binaries.

Install Pip on windows: <http://stackoverflow.com/a/4921215>

Virtualenv Win: <https://github.com/davidmarble/virtualenvwrapper-win>

~~~
the_mitsuhiko
> This is what needs fixing! Pip with access to compiled windows binaries.

That's the same issue really. Pip can't install (by design) binary
distributions. Neither eggs nor .exes. So you are limited to easy_install on
windows in many cases.

~~~
j-kidd
Self hosted PyPI (with eggbasket or even a static dir listing), combined with
easy_install, is an old and unsexy solution that just works. Instead of saying
we are limited to easy_install on Windows, I say we are embraced by setuptools
on all platforms.

------
warmwaffles
Most languages are in the same boat. PHP, Java, C, etc...

------
adambyrtek
Another post on the same subject written a few years ago:
<http://www.b-list.org/weblog/2008/dec/14/packaging/>

------
aaronh
Ugh...python packaging, it makes my head hurt every time I have to figure out
how it works again. This is something that Ruby got very right, how could
Python be such a mess?

I wish it was just a simple 3-step process to fix
<http://www.youtube.com/watch?v=HR_5QFZ-kbk>

------
agravier
I'm using bento, which keeps backward compatibility with
setuptools/distribute, but aims at fixing the traditional Python packaging
tools:

<http://cournape.github.com/Bento/>

It is stable and usable.

------
smagch
I'm a python-learning guy who uses npm and homebrew.

Comparing to npm and homebrew, it seems to me python's package manager is
quite tricky.

------
danbmil99
It's all baby stuff compared to autotools.

------
vishaldpatel
Title of the post reminds me of the Player Hater's Ball.

------
berntb
Why doesn't the Python world just copy CPAN and update it for Python? You can
e.g. literally find stuff in CPAN with hundreds of dependencies -- which is a
pain, but even that works.

At least CPAN testers? Please...

If Python people insist that there is only one way to do it -- and the answer
is Python, then rewrite CPAN in Python. Copy stuff that works...

~~~
slurgfest
"Only one way to do it" is a point of language design, not an assertion that
Python is the one true way.

Python does have PyPI. Works fine. Use pip to install and uninstall packages.
Use virtualenv to isolate dependencies/PYTHONPATH, as needed.

Can you be specific about what aspect of CPAN you think should be ported?

~~~
berntb
Specific? I already mentioned CPAN Testers. And that CPAN works... (I've seen
people with years of Python experience that had problem doing simple things.
Edit: Including uninstall of packages, which you mention.)

Edit: Here you see a result of CPAN Testers for a module with complex
dependencies. Note statistics on where/how dependencies are run. Also click on
a dependency icon to see the matrix of perl/os versions.

<http://deps.cpantesters.org/?module=Catalyst;perl=latest>

~~~
slurgfest
I don't have any problem doing uninstall of packages. Use pip rather than
easy_install. You are done.

I'm not convinced that CPAN is better

~~~
berntb
Good for you, re uninstall...

(I really didn't expect Python users to admit that anything in _any_ other
environment could be better... :-) E.g. running tests at install by default is
obviously bad? :-) )

So no comment about CPAN Testers -- automatically installing dependencies,
running the test suites on different perl/os versions and then getting neat
reports? What is the better way in Python?

------
edwinnathaniel
Despite all the hate, Maven solves a lot of problems in the Java land and I
couldn't be more happier.

~~~
gvalkov
There is really no exact equivalent, but you could give buildout[1] a go if
you find yourself in need of a maven-like tool in Python land.

[1] <http://www.buildout.org/>

~~~
edwinnathaniel
Sweet, thanks for pointing out that BO is closer to Maven!

I've heard a few Python SCM tools (setuptools, distutils, fabric, scons,
buildout) but haven't gone deep into the Python land yet.

