
Working with Python subprocess - llambda
http://jimmyg.org/blog/2009/working-with-python-subprocess.html
======
Pewpewarrows
As someone who to this day still cringes every time they need to work with the
subprocess module, do yourself a favor and use Kenneth Reitz's excellent
wrapper library called Envoy:

<https://github.com/kennethreitz/envoy>

For reference, he's the same guy that did the Requests module, which is pretty
much the defacto standard for avoiding urllib hell.

~~~
shuzchen
As someone who has used (and contributed code) to envoy, I must give warning
that envoy is still very much a work in progress. Don't go in there expecting
the same quality of the Requests module.

------
falcolas
A few notes from a recent attempt to use subprocess in the middle of a few
generators:

p.communicate() only works if the output of the program can fit in memory. If
it can't, python will gladly crash your box for you.

The buffer size of the os pipes on Linux are 64k. If you don't do your reads
and writes in this size, it's possible to deadlock yourself.

Reads on the pipes has to be done in a non-blocking fashion, using os.read(),
if you're dealing with streams of data that are larger than the memory you
have available to yourself.

You can chain multiple subprocesses together directly, if you feed the stdout
pipe to stdin on the next process.

    
    
        p1 = subprocess.Popen(['cat', 'foo'], stdout=subprocess.PIPE)
        p2 = subprocess.Popen(['grep', 'cat'], stdin=p1.stdout, stdout=subprocess.PIPE)
        p2.communicate()
    

select.poll() works on the os.pipes which are contained in the Popen objects.

------
ajross
I'm unmasking myself as a curmudgeon, but my blood boils when I see python
people trying to do this stuff. The subprocess API is just _awful_. Compare
the clean shell syntax at the top of this document to the wordy, awful dreck
at the bottom required to do the same stuff in python.

Likewise compare it to the perl syntax that does the same thing (and happens
to look at lot like the shell does, with good reason).

Python is a great language, but sometimes the aesthetic taste of its adherents
just boggles me.

 _Edit: I see Pewpewarrows above pointing out that there's a third party
module which fixes some of the nonsense in subprocess. And indeed, it looks
fairly clean at first glance. So perhaps I should retract my blanket
condemnation of its adherents' taste._

~~~
krupan
I felt the same (well, not quite blood boiling, but annoyed) with regular
expressions in python. Why so awkward? But now, I don't know, one thing the
somewhat cumbersome subprocess and regular expression support helped me to
learn was that I didn't want or need to spawn a subprocess or use a regex for
every little thing. There are good python libraries for manipulating files and
directories, globbing, and so forth. For strings, you can get a very long way
using split, join, slicing, and the "in" operator. I like my programs more
this way. They look like, you know, programs, instead of hacked together shell
scripts.

~~~
kstrauser
Out of curiosity, what do you dislike in Python regexes? I switched from Perl
to Python years ago and didn't notice much of a difference, other than that
you have to write `value = re.sub(pattern, repl, value)` instead of `$value =~
s/pattern/repl/;`.

~~~
ajross
You have to write a lot more than that. Python's strings don't map well to
regex syntax, forcing some odd quoting conventions and/or escapes. Python
forces you to instantiate the regex at runtime instead of parse time, so if
you want a compiled regex you need a separate bit of code to create it and
manage its scope. Perl's captures pop out in automatic variables that can be
directly used and interpolated, making simple parsing tasks (e.g. "print "$2
$1\n" if /.../") into sequences of easy-to-read one liners instead of page
long "parsers".

All of that is the kind of thing that makes _Python_ people's blood boil, of
course, but it's absolutely the reason that regex handling is just so much
natural in perl.

------
rollypolly
In regards to "Accessing Return Values, poll() and wait()":

Another easy way to get a returncode without polling is to call communicate().

returncode will be set when it returns from communicate().

------
gaius
Article was written 3 years ago; perhaps things are better now?

