On 06/28/2010 05:48 AM, Dave Pawson wrote:
> Main queries are: Ease of calling out to bash to use something like
> imageMagick or Java? Ease of grabbing return parameters? E.g. convert
> can return both height and width of an image. Can this be returned to
> the Python program? Can Python access the exit status of a program?
Sure. I've created a module called runcmd that does 90% of what I want
(easy access to stdout, stderr, error code). I've attached it to this
e-mail. Feel free to use it; this post puts my code into the public domain.
> I'd prefer the advantages of using Python, just wondering if I got so
> far with the port then found it wouldn't do something?
Python really isn't a shell scripting language. So there are things
that Bash does much better, such as spawning processes and piping them
together. I've tried over the years to create a pythonic library that
would let me do that, but haven't found a good syntax that I like.
It turns out, though, that much of what I use piping for in Bash is to
run external processes to do things that I could use python modules for.
For example, I typically pipe stuff to cut a lot to get certain fields.
For example:
ps ax | cut -c1-5
In python I could simply take the output of "ps ax" and use python's
own, superior, cutting routines (using my module):
(err, stdout, stderr) = runcmd.run( [ 'ps', 'ax' ] )
for x in stdout.split('\n'):
print x.strip().split()[0]
Sure it's a couple more lines in this case, but in other cases, python's
abilities make it simpler than bash. A great document on how you can
exploit python's abilities (particularly generators) to replace bash
pipelines is here: http://www.dabeaz.com/generators/
#!/usr/bin/python
import subprocess
import sys
import fcntl
import os
import select
import StringIO
"""
This module contains a single function run() that spawns a
process with arguments, and returns it's errcode, stdout,
and stderr.
It currently has a number of deficiencies, including the lack
of a way to kill the child.
"""
def _make_non_blocking(fd):
"""
Takes a file descriptor and sets it to non-blocking IO mode,
allowing us to use the posix select() on it. Used by this
module to allow us to read from a process as it writes to
stderr and stdout
@type fd: number
@param fd: file descriptor number to set to nonblocking
"""
fl = fcntl.fcntl(fd, fcntl.F_GETFL)
fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)
def run(cmd, **kwargs):
"""
Runs a command using the subprocess.Popen class, and
returns the err, stdout, and stderr of the process.
Optionally it can print to stdout the process's stderr
and stdout, and can also call callbacks when stdout and
stderr is available from the process. The following
kwargs are useful:
- print_stdout - if set to True will print process stdout
to stdout as data arrives
- print_stderr - if set to True will print process stderr
to stdout as data arrives
- stdin - Text to feed to process's stdin
- stdout_callback - function to call with stdout data
- stderr_callback - function to call with stderr data
@type cmd: text or list
@param cmd: the command to run, use a list if args are passed
@type kwargs: keyword args
@param kwargs: optional keyword arguments (see above)
@rtype: tuple
@return: Tuple of error code, stdout text, stderr text
"""
# Set flags depending on kwargs
if "print_stdout" in kwargs:
print_stdout=True
else:
print_stdout=False
if "print_stderr" in kwargs:
print_stderr=True
else:
print_stderr=False
if "stdout_callback" in kwargs:
stdout_callback=kwargs["stdout_callback"]
print "setting stdout callback"
else:
stdout_callback=None
if "stdout_callback" in kwargs:
stderr_callback=kwargs["stderr_callback"]
print "setting stderr callback"
else:
stderr_callback=None
if "shell" in kwargs:
shell=kwargs["shell"]
else:
shell=False
# create process object, set up pipes
child = subprocess.Popen(cmd,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
bufsize=0,
shell=shell
)
# give the process any stdin that we might have, then
# close stdin to prevent deadlocks
if "stdin" in kwargs:
child.stdin.write(kwargs["stdin"])
child.stdin.close()
# get output pipes, make them non-blocking
fo=child.stdout
fe=child.stderr
_make_non_blocking(fo.fileno())
eof=False
stdout=StringIO.StringIO()
stderr=StringIO.StringIO()
# Loop, checking for data on process's stdout and stderr
# until we've reached an eof condition
to_check = [fo,fe] * (not eof)
while True:
# check for data
ready = select.select(to_check,[],[],.05)
# process any stdout
if fo in ready[0]:
data=fo.read()
if data == '':
eof =True
else:
if print_stdout:
sys.stdout.write(data)
sys.stdout.flush()
#print "got data"
#print data
if stdout_callback:
stdout_callback(data)
stdout.write(data)
stdout.flush()
# process any stderr
if fe in ready[0]:
data=fe.read()
if not data == '':
if print_stderr:
sys.stdout.write(data)
sys.stdout.flush()
#print data,
if stderr_callback:
stderr_callback(data)
stderr.write(data)
stderr.flush()
if eof:
break
# clean up child and get its errcode
err=child.wait()
# Get the data in string form
stdout=stdout.getvalue()
stderr=stderr.getvalue()
return (err, stdout, stderr)
if __name__=="__main__":
"""
Random test cases. I think you pass a command as the first arg
and the second arg is used as stdin.
"""
lines=sys.stdin.readlines()
if lines:
err, stdout, stderr = run(sys.argv[1:],stdin=''.join(lines),
print_stdout=True)
else:
err, stdout, stderr = run(sys.argv[1:], print_stdout=True)
print "Process ended with %d" % err
print "Standard out:"
print stdout
print stdout.split('\n')
print "Standard err:"
print stderr
--
http://mail.python.org/mailman/listinfo/python-list
