

Get a Python subprocess' output without buffering - thelinuxkid
https://gist.github.com/thelinuxkid/5114777

======
adamtj
This is complex and inefficient. Using os.read() instead of the fileobject's
read() method is better.

It's inefficient to read a single byte at a time. The problem with
proc.stdout.read() is that, say, proc.stdout.read(4096) will block until it
has all 4096 bytes or the pipe has been closed. "for line in proc.stdout"
blocks until a full line has been read. That's the "buffering" this is trying
to avoid.

Using os.read() is better. It acts like more like socket.recv(). It will block
until data is available, but then return all data in the OS buffer up to a
size of 4096 bytes. The key is that, unlike a python fileobject's read()
method, os.read() will return less than 4096 bytes if that's all that is
available.

For example:

    
    
      fd = getattr(proc, stream).fileno()
      data = os.read(fd, 4096)
      while data:
          yield data
          data = os.read(fd, 4096)
    

Edit: Now that I look more closely, "unbuffered()" actually buffers until it
can produce an entire line. There's no reason to reinvent the wheel:

    
    
      for line in proc.stdout:
          print line

