On Nov 27, 4:30 am, Nick Craig-Wood <[EMAIL PROTECTED]> wrote: > MonkeeSage <[EMAIL PROTECTED]> wrote: > > Couple of things. You should use poll() on the Popen instance, and > > should check it explicitly against None (since a 0 return code, > > meaning exit successfully, will be treated as a false condition the > > same as None). Also, in your second example, you block the program > > when you call readlines on the pipe, since readlines blocks until it > > reaches eof (i.e., until pipe closes stdout, i.e., process is > > complete). Oh, and you don't have to split the input to the args > > option yourself, you can just pass a string. > > Though passing an array is good practice if you want to avoid passing > user data through the shell.
Well, he was setting shell=True, but I guess being explicit (about that) is better than implicit. ;) > > So, putting it all together, you want something like: > > > import subprocess, time > > > cmd = "cat somefile" > > proc = subprocess.Popen(args=cmd, shell=True, > > stdout=subprocess.PIPE, stdin=subprocess.PIPE, > > stderr=subprocess.STDOUT, close_fds=True) > > > while 1: > > time.sleep(1) > > if proc.poll() != None: > > break > > else: > > print "waiting on child..." > > > print "returncode =", proc.returncode > > This works fine unless the command generates a lot of output (more > than 64k on linux) when the output pipe will fill up and the process > will block until it is emptied. > > If you run the below with `seq 10000` then it works fine but as > written the subprocess will block forever writing its output pipe > (under linux 2.6.23). > > #------------------------------------------------------------ > import subprocess, time > > cmd = """ > for i in `seq 20000`; do > echo $i > done > exit 42 > """ > > proc = subprocess.Popen(args=cmd, shell=True, > stdout=subprocess.PIPE, stdin=subprocess.PIPE, > stderr=subprocess.STDOUT, close_fds=True) > > while 1: > time.sleep(1) > if proc.poll() != None: > break > else: > print "waiting on child..." > > print "returncode =", proc.returncode > lines = 0 > total = 0 > for line in proc.stdout: > lines += 1 > total += len(line) > print "Received %d lines of %d bytes total" % (lines, total) > #------------------------------------------------------------ > > So you do need to read stuff from your subprocess, but there isn't a > way in the standard library to do that without potentially blocking. > > There are a few solutions > > 1) use the python expect module (not windows) > > http://pexpect.sourceforge.net/ > > 2) set your file descriptors non blocking. The following recipe shows > a cross platform module to do it. > > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554 > > Or just do it with the fcntl module > > 3) Use a thread to read stuff from your subprocess and allow it to > block on proc.stdout.read() > > Here is an example of 2) > > #------------------------------------------------------------ > import subprocess, time, os > from fcntl import fcntl, F_GETFL, F_SETFL > from errno import EAGAIN > > cmd = """ > for i in `seq 100000`; do > echo $i > done > exit 42 > """ > > proc = subprocess.Popen(args=cmd, shell=True, > stdout=subprocess.PIPE, stdin=subprocess.PIPE, > stderr=subprocess.STDOUT, close_fds=True) > > # Set non blocking (unix only) > fcntl(proc.stdout, F_SETFL, fcntl(proc.stdout, F_GETFL) | os.O_NONBLOCK) > > def read_all(fd): > out = "" > while 1: > try: > bytes = fd.read(4096) > except IOError, e: > if e[0] != EAGAIN: > raise > break > if not bytes: > break > out += bytes > return out > > rx = "" > while 1: > time.sleep(1) > if proc.poll() != None: > break > else: > print "waiting on child..." > rx += read_all(proc.stdout) > > rx += read_all(proc.stdout) > print "returncode =", proc.returncode > lines = 0 > total = 0 > for line in rx.split("\n"): > lines += 1 > total += len(line) > print "Received %d lines of %d bytes total" % (lines, total) > #------------------------------------------------------------ > > Which runs like this on my machine > > $ python subprocess-shell-nb.py > waiting on child... > waiting on child... > waiting on child... > waiting on child... > waiting on child... > waiting on child... > waiting on child... > waiting on child... > returncode = 42 > Received 100001 lines of 488895 bytes total > > -- > Nick Craig-Wood <[EMAIL PROTECTED]> --http://www.craig-wood.com/nick Nice. Thanks for the recipe link too. Regards, Jordan -- http://mail.python.org/mailman/listinfo/python-list
