On May 24, 2013 7:06 AM, "Luca Cerone" <luca.cer...@gmail.com> wrote: > > Hi everybody, > I am new to the group (and relatively new to Python) > so I am sorry if this issues has been discussed (although searching for topics in the group I couldn't find a solution to my problem). > > I am using Python 2.7.3 to analyse the output of two 3rd parties programs that can be launched in a linux shell as: > > program1 | program2 > > To do this I have written a function that pipes program1 and program2 (using subprocess.Popen) and the stdout of the subprocess, and a function that parses the output: > > A basic example: > > from subprocess import Popen, STDOUT, PIPE > def run(): > p1 = Popen(['program1'], stdout = PIPE, stderr = STDOUT) > p2 = Popen(['program2'], stdin = p1.stdout, stdout = PIPE, stderr = STDOUT)
Could you provide the *actual* commands you're using, rather than the generic "program1" and "program2" placeholders? It's *very* common for people to get the tokenization of a command line wrong (see the Note box in http://docs.python.org/2/library/subprocess.html#subprocess.Popen for some relevant advice). > p1.stdout.close() > return p2.stdout > > > def parse(out): > for row in out: > print row > #do something else with each line > out.close() > return parsed_output > > > # main block here > > pout = run() > > parsed = parse(pout) > > #--- END OF PROGRAM ----# > > I want to parse the output of 'program1 | program2' line by line because the output is very large. > > When running the code above, occasionally some error occurs (IOERROR: [Errno 0]). Could you provide the full & complete error message and exception traceback? > However this error doesn't occur if I code the run() function as: > > def run(): > p = Popen('program1 | program2', shell = True, stderr = STDOUT, stdout = PIPE) > return p.stdout > > I really can't understand why the first version causes errors, while the second one doesn't. > > Can you please help me understanding what's the difference between the two cases? One obvious difference between the 2 approaches is that the shell doesn't redirect the stderr streams of the programs, whereas you /are/ redirecting the stderrs to stdout in the non-shell version of your code. But this is unlikely to be causing the error you're currently seeing. You may also want to provide /dev/null as p1's stdin, out of an abundance of caution. Lastly, you may want to consider using a wrapper library such as http://plumbum.readthedocs.org/en/latest/ , which makes it easier to do pipelining and other such "fancy" things with subprocesses, while still avoiding the many perils of the shell. Cheers, Chris -- Be patient; it's Memorial Day weekend.
-- http://mail.python.org/mailman/listinfo/python-list
