

Fixing the Python subprocess interface - daenz
http://amoffat.github.com/pbs

======
tswicegood
FWIW, re-replaying the stack trace to figure out what was imported and re-
implementing it is a horrible idea. There are much better ways to do this type
of import voodoo, specifically the import hooks that Python ships with. Here's
an example of their use inside a small side project of mine:
[https://github.com/tswicegood/maxixe/blob/master/maxixe/__in...](https://github.com/tswicegood/maxixe/blob/master/maxixe/__init__.py)

All that said, this is horribly un-Pythonic. A much better route to so is
something like envoy[1] which simply wraps `subprocess` in a sane API.

[1]: <https://github.com/kennethreitz/envoy>

~~~
tswicegood
Just to clarify. I applaud this type of development for people trying to learn
various parts of Python like playing with the stack, but the idea of this
being used in the wild scares me a bit.

~~~
steve-howard
It's like operator overloading for novel syntax in C++. It worked all right
for iostream, but if you've ever seen boost::spirit you can see that even very
smart people can make very weird things happen by trying to kludge features
into new syntax.

------
samps
It's odd that the import mechanism is abused here to make objects "out of thin
air". The fact that "from pbs import ffmpeg" works only if ffmpeg is actually
on the path is somewhat surprising.

I think the more comfortable (and Pythonic?) way to do this would be to
explicitly create these command objects:

    
    
       >>> import pbs
       >>> ffmpeg = pbs.Command('ffmpeg') # or '/usr/bin/ffmpeg', perhaps
       >>> result = ffmpeg(...)
    

[Edit: I really like the concept of using Python's positional and keyword
arguments to construct a shell command, though. Great insight there.]

~~~
daenz
Hi! Author here, there are a few ways to use it, including your suggestion
(did you make your suggestion up or were you pulling from the docs?):

    
    
        # magical, designed only for single shell scripts
        from pbs import *
        ffmpeg()
    
        # less magical
        from pbs import ffmpeg
        ffmpeg()
    
        # or
        import pbs
        pbs.ffmpeg()
    
        # no magic
        import pbs
        ffmpeg = pbs.Command(pbs.which("ffmpeg")) # command takes full path
        ffmpeg()
    

I tried to cover the main use cases adequately. My goal was to ease a pain
point myself and others have experienced, that other popular packages don't
address well unfortunately.

If it's helped anyone like it's helped me, I'm happy and glad to share!

~~~
samps
Wow -- no, I entirely missed this (and just happened to end up with the same
name). Thanks for clarifying.

------
nas
Neat but way too magical for my taste. The code to figure out what to do in
the case of 'from .. import *' is particularly ugly.

Perhaps the commands should accessible from an object you import. That's
slightly more typing but more explicit and would not require ugly magic. E.g.

    
    
      from pbs import sh
      print sh.ifconfig('eth')
    

If it's not clear, the 'sh' object could override __getattr__ or
__getattribute__ and wrap commands as necessary.

~~~
dman
Anything that encourages a from ... import * usage is evil irrespective of its
implementation.

~~~
petsos
Why?

~~~
ulope
Because it leads to namespace pollution and hard-to-track-down bugs.

It's especially bad in this case where adding an executable in the system can
suddenly shadow any built-in name (e.g. imagine someone adds a "print" or
"sys" executable).

~~~
gbog
Refactoring is also made much harder with import * if you have cascading
modules. I have a pylint hook forbidding these before committing to our hg
repo

------
jmtulloss
So many people saying this is "too much magic". Whatever, I'm into it. The
idea of commands being functions that can just pass their output to other
functions is intuitive, and passing arguments as, well, arguments is as well.

It might not be pythonic, it might be a crime against Guido and everything he
represents, but it's pretty awesome.

~~~
gbog
I have the feeling you misunderstood the problem with magic. In fact, having a
function calling shell command is ok. The problem is to know, from reading the
code, where it comes from. In python there is this idea of transparency, the
determinism, which is with it feels enlightening, like in the docs cartoon.
Anything obscuring that is said to be unpythonic.

------
scott_s
I use Python for shell scripting a lot. Ignoring all of the issues people have
brought up here, I really like how function composition is piping:

    
    
      # sort this directory by biggest file
      print sort(du("*", "-sb"), "-rn")
    
      # print the number of folders and files in /etc
      print wc(ls("/etc", "-1"), "-l")
    

The reason that I like that method over, say, envoy's [1] method is that
_envoy.run('uptime | pbcopy')_ has what I consider code in strings. When I'm
writing a script, the programs I'm calling, and how they interact, is part of
the "code" to me. I would prefer that they're at the language level, and not
represented as strings.

[1] I only just learned about envoy in this thread. Thanks! Sadly, one of the
places where I run my Python scripts is a location where I can't install my
own packages, and I don't want to deal with using my own install of Python, so
I tend to just implement something like this:

    
    
      def checked_exec(seq):
        p = Popen(seq, stdout=PIPE, stderr=PIPE)
        stdout, stderr = p.communicate()
        if p.returncode != 0:
            print 'err: ' + stderr
            print "'" + seq + "' failed."
            sys.exit()
        return stdout

~~~
ulope
As long as the package you need doesn't include a C extension (which most
don't) you can just ship it with your code (license permitting ofc.) - just
add the path to the libary to sys.path. It's not a very clean solution but can
be a real life saver when you have to work on "broken" systems.

~~~
scott_s
Sorry, I wasn't clear. This is a system I log into every day, and I don't want
to maintain my own install of Python and related packages on it. It's too much
overhead. I'd rather just use the default Python, even though it's old.

~~~
rat87
For simple libs I just dump the py file/folder in the same folder as the
script and import.

------
kenneth_reitz
This module is a huge hack. It's a neat hack, though :)

I've written what I feel is a much better solution to this problem: Envoy.

<https://github.com/kennethreitz/envoy>

It's pythonic and makes far fewer assumptions about both your code and what
you're running.

~~~
mafro
+1 for Envoy. It's a really nice replacement for subprocess (which is awful).

Clint and Requests are worth a look too.. Thanks Kenn!

------
j2labs
I like Kenneth Reitz's envoy too. He describes it as "Python Subprocesses for
Humans™", similar to how he describes Requests, which he also wrote.

<https://github.com/kennethreitz/envoy>

------
bryze
Can we please stop titling postings like this? How about "An Alternative to
the Python Subprocess Interface". Fixing something implies that it's broken or
inadequate, and from my limited experience, Subprocess is already a major
improvement over os.system. I'm not saying this idea doesn't have value, I'm
saying that the way it's framed lacks the requisite humility it ought to.

------
ggchappell
This is a cute little hack, and quite possibly a very useful one. (And if it
isn't useful, then it is certainly interesting and instructive.)

However:

Please, please, _please_ don't use or recommend things like "from pbs import *
". Namespace pollution is bad enough when importing a documented collection of
functions. Importing functions that are named based on whatever happens to be
in my path at the moment ... that's seriously scary.

But "import pbs" looks like fun. :-) And "pbs.ls" isn't _that_ much to type.

As The Zen of Python says:

] Namespaces are one honking great idea -- let's do more of those!

P.S. Hmmm ... but does "import pbs" work? Haven't tried it.

------
ulope
Wow this is so frighteningly magical and will break in many entertaining ways.

For a sane alternative I'd recommend Kenneth Reitz' awesome Envoy
(<https://github.com/kennethreitz/envoy>)

------
trun
Reminds me of something I saw not too long ago...

<https://github.com/JulienPalard/Pipe>

It would be really cool (though admittedly less Pythonic) to combine the infix
notation provided by the Pipe library to allow more shell-like function
chaining.

Instead of this...

    
    
      print wc(ls("/etc", "-1"), "-l")
    

You would have this...

    
    
      print ls("/etc", "-1") | wc("-l")

~~~
daenz
There's a few ideas floating around here
<https://github.com/amoffat/pbs/issues/6> on how to implement it, but nothing
looks really feasible. If you have any insights, I welcome them :)

~~~
fmoralesc
I wrote some code for that a while ago: <https://gist.github.com/1300342>

------
stephen_mcd
The value/cost of a "hack" is offset by what it provides. If something can be
implemented at the same cost in a less brittle and more future-proof way, then
by all means label it horrible.

If you think "horrible" hacks are a slight against something that provides an
amazing level of functionality to an end user, then you've lost sight of what
we're coding for.

------
mLewisLogic
If you don't like the import *, don't use it. Python supports it, so why
shouldn't this library? (not that it's a great idea)

If you'd like to handle missing system executables, catch the exception. You
should be writing in that style anyways.

Honestly, this cleans up a ton of system scripting code, making it way more
readable/maintainable.

Maybe the code could be cleaned up, but this is the direction that Python
should be heading. Abstract away the complications when possible, keep low-
level stuff around for when it's absolutely needed.

Beautiful is better than ugly.

------
jayferd
__Magic!!!!! __

I code in Ruby for a living, and this is even a bit much magic for me. I
suppose you can use it in magic-less mode with `from pbs import Command`, and
set up your command set manually.

------
cobbal
It states that the lines

    
    
      curl("http://duckduckgo.com/", "-o page.html", "--silent")
      curl("http://duckduckgo.com/", "-o", "page.html", "--silent")
    

are equivalent. This worries me, I would much rather always have it be one
argument corresponding to exactly one shell argument. Here it looks like in
some (maybe all?) cases arguments are split on spaces, which means always
having to be extremely cautious about escaping, something a good abstraction
shouldn't force you to deal with.

~~~
mgedmin
Exactly. If you look at the sources, you'll see that the arguments are all
joined into a single string with spaces, then split back into separate words
using shlex.split().

So cat("filename with spaces in it") will fail, but cat("'filename with spaces
in it') ought to succeed.

It's a neat experiment, but using this module in production would not be a
great idea.

------
VMG
Very impressive, but the use of globals and the dynamic lookup mechanism are a
little scary. Looking at the source there seems to be some magic involved like
hacking the interpreter.

I'd feel more comfortable if it only exported one variable.

------
ishi
That's pretty brilliant, well done.

------
bwarp
This is basically a dynamic DSL. A confusing muddle of syntax and semantics.
Will result in pain. Usually when least expected.

------
y4m4
<https://github.com/Harshavardhana/pbs> \- Refactored the code into more
Python library like and still trying to fix the command line import problem.

------
simonw
This looks great, but I really don't like the fact that I can't fire up a
Python shell and try it out interactively. Having to run pbs.py itself to get
a different kind of shell is uncomfortable.

~~~
daenz
This is fixed on master as of version 0.4, just fyi. It has some limitations
(no star import) but otherwise works as expected.

------
petsos
Why not use pipes for piping? e.g. print du("*", "-sb") | sort("-rn")

~~~
rplnt
Would break syntax too much and would make even less sense to python user
reading the code. But there is several packages which does this (piping using
pipes), maybe you could use that.

------
akx
The import hack is the only thing I don't really like (I'd prefer the `from
pbs import sh` syntax proposed elsewhere) -- and without reading the code...
is piping really piping, or will the function composition example actually
consume the first command result fully first?

Anyway, this is a great idea. :)

------
singingwolfboy
Too much magic! Too much magic!!

------
ajuc
With this, maybe having python as a shell will be possible :)

