

Trapped in python package; send food. - blasdel
http://jessenoller.com/2009/07/17/trapped-in-python-package-send-food/

======
shadytrees
Nit: setuptools does not do this. Proper setuptools packages use the
console_scripts entry point, which points to a function will never specify a
shebang. A good example is Paste Script [PS]. "paster" is linked the function
"paste.script.command:run". When this is pip-installed into a virtualenv, you
can `cat` the "paster" executable. It'll look like something like this [pr];
sure enough, the entire thing will have been automatically generated, taking
into account the changed Python binary. Slag on setuptools all you want; PEAK
got entry points nailed down perfectly.

[PS]: <http://svn.pythonpaste.org/Paste/Script/trunk/setup.py>

[pr]: <http://pastie.org/550423>

On the other hand, distutils has a `scripts=` keyword that points directly to
a file, which then gets copied and pasted into the bin directory. Since the
file has be word-for-word the script, the shebang is written directly. It
depends on env, which should normally function; but, if it doesn't, as in the
OP's case, there's no fallback. A good example of this is Mercurial [hg].
(Note: I actually haven't used raw distutils all that much, so there might be
a work-around for this. A few fruitless minutes of Google Code Search hasn't
turned up anything interesting.)

[hg]: <http://selenic.com/repo/hg/file/47ce7a3a1fb0/hg>

~~~
brodie
[http://docs.python.org/distutils/setupscript.html#installing...](http://docs.python.org/distutils/setupscript.html#installing-
scripts)

"The only clever feature is that if the first line of the script starts with
#! and contains the word “python”, the Distutils will adjust the first line to
refer to the current interpreter location. By default, it is replaced with the
current interpreter location. The --executable (or -e) option will allow the
interpreter path to be explicitly overridden."

------
radu_floricica
This reminds me of a discussion here a few days ago in which people were
happily bashing java's classpath. I don't know anything about python, but my
feeling at their comparison of java and python was that something that is way
too simple to use can't be at the same time easy to customize.

In the light of this article classpaths sound a lot like what they really are:
a working compromise everybody can live with.

~~~
jnoller
Not exactly; python already has a PYTHONPATH which acts (sort of) like java's
classpath.

------
Goladus
Why can't the hardcoded "#!.../python" lines be fixed with find+sed? I'm not
sure I understand exactly what he's trying to do.

~~~
nailer
Fixing it is good, but a nicer to start an interpreter, which handles
whichever is the preferred location to Python, is to call env, which is always
in usr bin.

#!/usr/bin/env python

Will dynamically start whatever Python is in your $PATH.

~~~
LogicHoleFlaw
You'd think /usr/bin/env would always work, but I've seen this break in
production systems where /usr/bin/env is not in the $PATH in situations like
cron scripts. It's retarded I know but it does happen.

~~~
jrockway
_/usr/bin/env is not in the $PATH_

I don't think you mean this. When you specify an absolute name, $PATH is not
even in the picture. (I do know that some systems put env in /bin. Those
systems are misconfigured.)

Anyway, shebangs are flaky by design. I just run the correct interpreter
directly, as in "/path/to/perl -Ilib bin/script.pl". There is no confusion
that way.

~~~
nailer
You're right - I think what he means is that env is not in /usr/bin. This was
the case on very old (Red Hat Linux 5.2 back in the late 90s) systems where
there was no /usr/bin/env -> /bin/env symlink.

~~~
LogicHoleFlaw
I was bit confused in my description here, sorry. /usr/bin/env existed but
when it was run under the cron user's environment it could not locate the
interpreter we required and failed silently. The shebang line worked fine
under all other users' accounts. Very frustrating. I assume this problem was
due to the cron account's $PATH being set up differently somehow. In any case
we just went back to explicitly choosing our site-specific interpreter path.

Related pet peeve: people using #!/bin/sh when they really want #!/bin/bash.
They are not interchangeable!

------
moe
This is a lot of rambling and a strange approach for what is effectively a
3-liner...

    
    
        ./configure --prefix=/opt/my_python
        make && make install
        export PATH="/opt/my_python/bin:$PATH"

~~~
jnoller
No it's not. You missed the fact that python packages that include stuff like
scripts hardcode the full, explicit prefix in the #! lines. This means you can
compile with --prefix all you want, but if you move it, everything breaks in
unexpected ways.

I agree, all you should need is a --prefix, and that scripts should obey
_whatever_ python interpreter is in /usr/bin/env - but that's _not_ the way it
works, sadly.

~~~
moe
_python packages that include stuff like scripts hardcode the full, explicit
prefix in the #! lines._

Such scripts are broken. One should fix the scripts instead of mangling the
environment. It's a one-liner:

    
    
       find /opt/myproject -type f -exec sed -i 's/#!\/usr\/bin\/python/#!\/usr\/bin\/env python/' {} \;
    

_but if you move it, everything breaks in unexpected ways._

If you move it then PYTHONPATH is your friend.

Once you have a python build for each project (which you want to have anyways)
you'll notice that you don't need virtualenv or other kludges anymore either.
Yep, it's really that easy. 4 lines of code. How many lines was yours again?

~~~
jnoller
Did you read the comment above this? Sed is the initial hack to fix it, but
it's still a hack; not a fix. Eventually I punted on it for the Linux dist,
but I'm probably going to be stuck with the sed fix for OS/X.

As for pythonpath - that has nothing to do with moving the entire directory
path, including the interpreter, /bin dir, etc.

~~~
moe
Sorry, I just can't follow where your exact problem is.

I have 9 python projects under /opt/foo* right now. Each of them has their own
python in /opt/foo*/sys/python. I su to the respective user who has PATH and
PYTHONPATH set appropiately in its profile - and everything works as expected.

I also don't follow why you consider sed a "hack" here. A hardcoded hashbang
line is clearly a Bug - but gladly one that is trivially fixed with that one-
liner. On the other side we have your "fix" that involves permanently mangling
the environment in strange ways. In my book you are the one proposing the hack
here.

~~~
jnoller
I'm not proposing mangling anything here; in fact I dropped the mangled
environment (as it was a hack, and added too much space) and swapped to a
simpler (and cleaner) virtualenv built in /opt.

I still haven't fixed the build something and upload it for people, because of
the hardcoded shebang, and yes - sed could fix that.

~~~
moe
_Also, you have to run the sed command over the tree every time you install a
new package which installs a binary_

How often do your projects depend on something else, written in python, that
installs a binary? For me that number is zero. iirc all of my python
dependencies are libraries. And if I stumbled into something that comes with a
binary, with a broken hashbang line, then I'd be rather motivated to not only
fix it, but to also feed the fix upstream. If the project is useful enough for
me to use it then those 10 minutes of my time for a brown-paper-bag micropatch
are a no-brainer.

Well, to each their own. - But I still think this is a solution looking for a
problem, with the potential to create more problems than it solves.

~~~
jnoller
Not exactly - for example, I depend on fabric, PIP, easy_install, nose, pylint
and others. In this case, it's not a matter of fixing upstream on a per-
project basis, it's fixing distutils as this is "magical" behavior I (and
others) have seen, and that's not a pile of gators I (or I suspect you) would
want to wade into. For example read the following threads:

[http://mail.python.org/pipermail/python-
dev/2009-July/090259...](http://mail.python.org/pipermail/python-
dev/2009-July/090259.html) [http://mail.python.org/pipermail/python-
dev/2009-July/090356...](http://mail.python.org/pipermail/python-
dev/2009-July/090356.html)

And so on. I agree wholeheartedly though - library-only dependencies are
stupid simple.

~~~
moe
I just checked and interestingly I also depend on easy_install (which I don't
even use, but put it in there someday), nose and pylint. None of them gave me
problems. I say if fabric and PIP are broken then fix them.

easy_install is irrelevant anyways, nobody in their right mind would use that
on a production system.

~~~
jnoller
I'm surprised having the hardcoded paths didn't bite you - of course, how
frequently are you moving the interpreter, or installing multiple versions of
the interpreter (with the same dependencies) with a shared bin dir?

