[issue12328] multiprocessing's overlapped PipeConnection on Windows

Roundup Robot added the comment: New changeset bdad874195d6 by Victor Stinner in branch '3.4': Isuse #12328, #20978: Add _winapi.WAIT_ABANDONED_0 symbol, needed by http://hg.python.org/cpython/rev/bdad874195d6 New changeset 2e4692a762d5 by Victor Stinner in branch 'default': (Merge 3.4) Issue #1

[issue12328] multiprocessing's overlapped PipeConnection on Windows

Antoine Pitrou added the comment: One of our buildbots seems to have recurring issues with the timeout issues. I don't know if the machine is very loaded: == FAIL: test_wait_integer (test.test_multiprocessing.TestWait) --

[issue12328] multiprocessing's overlapped PipeConnection on Windows

Antoine Pitrou added the comment: Committed, thanks! -- resolution: -> fixed stage: patch review -> committed/rejected status: open -> closed ___ Python tracker ___ ___

[issue12328] multiprocessing's overlapped PipeConnection on Windows

Roundup Robot added the comment: New changeset 75c27daa592e by Antoine Pitrou in branch 'default': Issue #12328: Fix multiprocessing's use of overlapped I/O on Windows. http://hg.python.org/cpython/rev/75c27daa592e -- ___ Python tracker

[issue12328] multiprocessing's overlapped PipeConnection on Windows

sbt added the comment: Updated patch addressing Antoine's comments. -- Added file: http://bugs.python.org/file24737/pipe_poll_fix.patch ___ Python tracker ___ __

[issue12328] multiprocessing's overlapped PipeConnection on Windows

sbt added the comment: Updated patch against 2822765e48a7. -- Added file: http://bugs.python.org/file24730/pipe_poll_fix.patch ___ Python tracker ___ ___

[issue12328] multiprocessing's overlapped PipeConnection on Windows

sbt added the comment: > Hmm, I tried to apply the latest patch to the default branch and it > failed. It also seems the patch was done against a changeset > (508bc675af63) which doesn't exist in the repo... I will do an updated patch against a "public" changeset. (I usually use a patch fo

[issue12328] multiprocessing's overlapped PipeConnection on Windows

Antoine Pitrou added the comment: Hmm, I tried to apply the latest patch to the default branch and it failed. It also seems the patch was done against a changeset (508bc675af63) which doesn't exist in the repo... -- ___ Python tracker

[issue12328] multiprocessing's overlapped PipeConnection on Windows

sbt added the comment: I have done an updated patch. (It does *not* switch to using bytes oriented pipes as I suggested in the previous message.) The patch also adds a wait() function with signature wait(object_list, timeout=None) for polling multiple objects at once. On Unix it is jus

[issue12328] multiprocessing's overlapped PipeConnection on Windows

sbt added the comment: Quite honestly I don't like the way that polling a pipe reads a partial message from the pipe. If at all possible, polling should not modify the pipe. I think the cleanest thing would be to switch to byte oriented pipes on Windows and create PipeIO which subclasses Raw

[issue12328] multiprocessing's overlapped PipeConnection on Windows

Antoine Pitrou added the comment: > > It seems to me that ERROR_OPERATION_ABORTED is a "true" error, and so > > should raise an exception. > > I guess so, although we do expect it whenever poll() times out. What > exception would be appropriate? BlockingIOError? TimeoutError? I would say eith

[issue12328] multiprocessing's overlapped PipeConnection on Windows

sbt added the comment: > It seems to me that ERROR_OPERATION_ABORTED is a "true" error, and so > should raise an exception. I guess so, although we do expect it whenever poll() times out. What exception would be appropriate? BlockingIOError? TimeoutError? --

[issue12328] multiprocessing's overlapped PipeConnection on Windows

Antoine Pitrou added the comment: > There are three "expected" error conditions: > > ERROR_OPERATION_ABORTED: operation stopped by CancelIo(Ex)() > ERROR_MORE_DATA: operation complete, but only got part of the message > ERROR_IO_INCOMPLETE: operation still has not finished > > In the win32 api

[issue12328] multiprocessing's overlapped PipeConnection on Windows

sbt added the comment: > I have the feeling that if we have to call GetLastError() at the > Python level, then there's something wrong with the APIs we're > exposing from the C extension. > I see you check for ERROR_OPERATION_ABORTED. Is there any situation > where this can happen? There are

[issue12328] multiprocessing's overlapped PipeConnection on Windows

Antoine Pitrou added the comment: > Here is an updated patch (pipe_poll_fix.patch) which should be applied > on top of sigint_event.patch. > > It fixes the problems with PipeConnection.poll() and Queue.empty() and > makes PipeListener.accept() use overlapped I/O. This should make all > the pip

[issue12328] multiprocessing's overlapped PipeConnection on Windows

Antoine Pitrou added the comment: I've re-added the fast uncontended path in semaphore.c and committed sigint_event.patch. Now I'm gonna take a look at pipe_poll_fix.patch... -- stage: -> patch review versions: -Python 2.6, Python 2.7, Python 3.1, Python 3.2

[issue12328] multiprocessing's overlapped PipeConnection on Windows

Roundup Robot added the comment: New changeset 08953a04b2e6 by Antoine Pitrou in branch 'default': Issue #12328: Under Windows, refactor handling of Ctrl-C events and http://hg.python.org/cpython/rev/08953a04b2e6 -- nosy: +python-dev ___ Python track

[issue12328] multiprocessing's overlapped PipeConnection on Windows

sbt added the comment: Here is an updated patch (pipe_poll_fix.patch) which should be applied on top of sigint_event.patch. It fixes the problems with PipeConnection.poll() and Queue.empty() and makes PipeListener.accept() use overlapped I/O. This should make all the pipe releated blocking

[issue12328] multiprocessing's overlapped PipeConnection on Windows

sbt added the comment: I have noticed a few more problems. * Because poll() isn't thread safe on Windows, neither is Queue.empty(). Since a queue's pipe will never contain empty messages, this can be fixed easily by using (a wrapper for) win32.PeekNamedPipe(). * PipeListener/PipeClient have

[issue12328] multiprocessing's overlapped PipeConnection on Windows

sbt added the comment: sigint_event.patch is a patch to make _multiprocessing.win32.WaitForMultipleObjects interruptible. It applies directly on to default. The patch also adds functions _PyOS_SigintEvent and _PyOS_IsMainThread which are implemented in signalmodule.c and declared in intrcheck.

[issue12328] multiprocessing's overlapped PipeConnection on Windows

sbt added the comment: > You are right, we need a manual reset *or* we must ensure that every > user of _PyOS_SigintEvent only does so from the main thread. On second thoughts, even using an auto-reset event, resetting the event before waiting is unavoidable. Otherwise you are liable to catch

[issue12328] multiprocessing's overlapped PipeConnection on Windows

Antoine Pitrou added the comment: > > Hmm, it seems to me that it should be done in _poll() instead. > > Otherwise, recv() will not be interruptible, will it? > > Or maybe WaitForMultipleObjects() should be changed to also wait on > sigint_event if called by the main thread. Indeed, this may

[issue12328] multiprocessing's overlapped PipeConnection on Windows

sbt added the comment: > Hmm, it seems to me that it should be done in _poll() instead. > Otherwise, recv() will not be interruptible, will it? Or maybe WaitForMultipleObjects() should be changed to also wait on sigint_event if called by the main thread. > Also, after looking at this again,

[issue12328] multiprocessing's overlapped PipeConnection on Windows

Antoine Pitrou added the comment: > pipe_interruptible.patch is a patch to support to making poll() > interruptible. It applies on top of pipe_poll_2.patch. Hmm, it seems to me that it should be done in _poll() instead. Otherwise, recv() will not be interruptible, will it? Also, after lookin

[issue12328] multiprocessing's overlapped PipeConnection on Windows

sbt added the comment: pipe_interruptible.patch is a patch to support to making poll() interruptible. It applies on top of pipe_poll_2.patch. I am not sure what the guarantees are for when KeyboardInterrupt will be raised. I would have done it a bit differently if I knew a good way to test w

[issue12328] multiprocessing's overlapped PipeConnection on Windows

sbt added the comment: > Also, what is the rationale for the following change: > > -elif timeout == 0.0: > +elif timeout == 0.0 and nleft != 0: > return False If PeekNamedPipe() returns (navail, nleft) there are 3 cases: 1) navail > 0: just return True

[issue12328] multiprocessing's overlapped PipeConnection on Windows

Antoine Pitrou added the comment: Hello, Thanks for the patch. >I would suggest making it a programming error for the overlapped >object to be deallocated while the operation is still pending, and >to print a RuntimeError if that happens. (This does not prevent us >from still

[issue12328] multiprocessing's overlapped PipeConnection on Windows

sbt added the comment: The attached patch hopefully fixes problems (1)-(5), but I have never used overlapped I/O before. test_pipe_poll.py passes with these changes. -- keywords: +patch Added file: http://bugs.python.org/file22350/pipe_poll.patch _

[issue12328] multiprocessing's overlapped PipeConnection on Windows

New submission from sbt : There are some problems with the new Windows overlapped implementation of PipeConnection in the default branch. 1) poll(0) can return False when an empty string is in the pipe: if the next message in the pipe is b"" then PeekNamedPipe() returns (0, 0) which is ind