Feature Requests item #1191964, was opened at 2005-04-28 13:40 Message generated for change (Comment added) made by josiahcarlson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1191964&group_id=5470
Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Josiah Carlson (josiahcarlson) Assigned to: Peter Åstrand (astrand) Summary: asynchronous Subprocess Initial Comment: It would be terribly nice if the Popen class in the subprocess module would allow a programmer to easily say "send some data right now (if I have some to send) and receive whatever information is available right now". Essentially the equivalent of asyncore.loop(count=1), only that it returns the data received, instead of placing it in a buffer. Why would this functionality be useful? Currently, when using the subprocess module with pipes, the interaction with a pipe is limited to "send data if desired, close the subprocess' stdin, read all output from the subprocess' stdout, ...". Certainly one can pull the pipes out of the Popen instance, and perform the necessary functions on posix systems (with select or poll), but the additional magic on WIndows is a little less obvious (but still possible). There is a post by Paul Du Bois with an example using win32pipe.PeekNamedPipe: http://groups-beta.google.com/group/comp.lang.python/msg/115e9332cc1ca09d?hl=en And a message from Neil Hodgeson stating that PeekNamedPipe works on anonymous pipes: http://mail.python.org/pipermail/python-dev/2000-May/004200.html With this modification, creating Expect-like modules for any platform, as well as embedded shells inside any GUI with a text input widget, becomes easy. Heck, even Idle could use it rather than a socket for its interactive interpreter. ---------------------------------------------------------------------- >Comment By: Josiah Carlson (josiahcarlson) Date: 2006-12-31 10:04 Message: Logged In: YES user_id=341410 Originator: YES Unless you use PeekNamedPipe on Windows, there is no guarantee that the pipe will return *any* data until the process ends, even if you say pipe.read(1) and there is 1k of data already sent. Note that the two async reading methods that I provide (recv() and recv_err()) take a 'maximum bytes' argument. Just like I have written a 'send_all()' utility function, I (or you) can easily write a 'recv_exact()' utility function that receives an exact number of bytes before returning. That functionality would be more or less required for one of the expected use-cases I specify in the recipe, "writing a multi-platform 'expect' module". Stick with async calls (with the utility recv_exact()) until you need to use the .communicate() method. ---------------------------------------------------------------------- Comment By: Benjamin (ooooooooo) Date: 2006-12-31 07:19 Message: Logged In: YES user_id=1680023 Originator: NO If there were a blocking "read x bytes" type call, could you not do some non-blocking stuff afterwards? If you use the "read" method, with a byte limit, directly on the stdout member of Popen, that could return with your bytes, and then you do some non-blocking stuff afterwards. That's the external view of course, I suspect that you're saying there's some internal lower-level reason this is currently not possible. Thanks for your interim class BTW, it is proving to be very useful. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2006-12-30 15:21 Message: Logged In: YES user_id=341410 Originator: YES The way subprocess is currently written, if you were to use a blocking semantic, you would no longer be able to use an asynchronous semantic afterwards (it will read the result until there is nothing more to read). Note that this is the case whether it is mode or method based. ---------------------------------------------------------------------- Comment By: Benjamin (ooooooooo) Date: 2006-12-30 13:45 Message: Logged In: YES user_id=1680023 Originator: NO I would also like to see this feature. I'm using Josiah Carlson's recipe for the time being. I'm agnostic about whether the asynchronicity is a "mode" or just a case of using different functions. However, if the former is chosen, then one should be able to switch modes at will, because sometimes I want blocking and sometimes I don't. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2005-09-21 13:55 Message: Logged In: YES user_id=341410 I've implemented this as a subclass of subprocess.Popen in the Python cookbook, available here: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554 ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2005-09-21 13:51 Message: Logged In: YES user_id=341410 I've implemented this as a subclass of subprocess.Popen in the Python cookbook, available here: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554 Essentially this is a request for inclusion in the standard library for Python 2.5 . ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2005-06-26 12:47 Message: Logged In: YES user_id=341410 How about if subprocesses have 3 new methods, send(input), recv(maxlen), and recv_stderr(maxlen). send(input) would perform like socket.send(), sending as much as it currently can, returning the number of bytes sent. recv(maxlen) and recv_stderr(maxlen) would recieve up to the provided number of bytes from the stdout or stderr pipes respectively. I currently have an implementation of the above on Windows and posix. I include the context diff against revision 1.20 of subprocess.py in current CVS. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2005-05-28 17:15 Message: Logged In: YES user_id=341410 I suppose I should mention one side-effect of all this. It requires three more functions from pywin32 be included in the _subprocess driver; win32file.ReadFile, win32file.WriteFile, and win32pipe.PeekNamedPipe . Read and Peek are for reading data from stdout and stderr, and Write is for the support for partial writes to stdin. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2005-05-28 16:22 Message: Logged In: YES user_id=341410 I've got a version of subprocess that has this functionality with pywin32. Making it work on *nix systems with usable select is trivial. About the only question is whether the functionality is desireable, and if so, what kind of API is reasonable. Perhaps adding an optional argument 'wait_for_completion' to the communicate method, which defaults to True for executing what is currently in the body of communicate. If the 'wait_for_completion' argument is untrue, then it does non-waiting asynchronous IO, returning either a 2 or 3-tuple on completion. If input is None, it is a 2-tuple of (stdout, stderr). If input is a string, it is a 3-tuple of (bytes_sent, stdout, stderr). How does that sound? ---------------------------------------------------------------------- Comment By: Dave Schuyler (parameter) Date: 2005-04-28 16:38 Message: Logged In: YES user_id=473331 More control of sub-processes would be great. Several times in the past few years I've done stuff with os.system, popenN, or spawn* and it would be useful to have even more cross-platform consistency and support in this area. Anyway, this is just a vote to encurage other developers. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1191964&group_id=5470 _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
