[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Andrey Vlasovskikh added the comment: > On closer look your patch is also ignoring SystemExit. I think it's > beneficial to honor SystemExit, so a user could use this as a means to > replace the current process with a new one. Yes, SystemExit should cancel all the tasks that are currently in the queue. I guess my patch doesn't handle this properly. > If we keep that behavior, the real problem here is that the result handler > hangs if the process that reserved a job is gone, which is going to be > handled by #9205. Should we mark it as a duplicate? Yes, I think that #9205 covers this issue, so #8296 may be marked as a duplicate. -- ___ Python tracker <http://bugs.python.org/issue8296> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Changes by Andrey Vlasovskikh : -- nosy: +vlasovskikh ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
New submission from Andrey Vlasovskikh : multiprocessing.Pool methods map, imap, etc. are said to be able to normally handle exceptions. But it seems that it is true only for synchronous exceptions inside their first func arguments. When (typically during a long-running parallel map) a user hits ^C, an asynchronous KeyboardInterrupt isn't handled properly and leads to the interpreter hangup. More precisely, children processes become (on Linux), so the only way to terminate the whole program is to issue the KILL signal. As stopping a program with ^C while running potentially long parallel computations is probably quite a common scenario, the interpreter should not hang up in such a case. I'm using Python 2.6.5 (r265:79063, Mar 23 2010, 04:44:21) [GCC 4.4.3] on linux2. I've also tried to use the current multiprocessing.pool module from the current (2.7) trunk with my 2.6.5 installation, but the bug persists. -- components: Library (Lib) messages: 102219 nosy: vlasovskikh severity: normal status: open title: multiprocessing.Pool hangs when issuing KeyboardInterrupt type: crash versions: Python 2.6, Python 2.7 ___ Python tracker <http://bugs.python.org/issue8296> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Andrey Vlasovskikh added the comment: Yes, here is my test case. -- Added file: http://bugs.python.org/file16743/test_pool_keyboardinterrupt.py ___ Python tracker <http://bugs.python.org/issue8296> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Andrey Vlasovskikh added the comment: Yes, I've come up with the same solution by myself, but it cannot cover all the cases of the bug. It works only for cases when ^C is hit during a call to the users' function: http://stackoverflow.com/questions/1408356/keyboard-interrupts-with-pythons-multiprocessing-pool/2561809#2561809 If the user is "lucky", he may hit ^C during getting or putting data into the queues in multiprocessing.pool.worker. To reproduce such a case, you may insert `sleep(10)` before `task = get()` or `put((job, i, result))`, for example. I've encountered such cases just by running test examples several times. -- ___ Python tracker <http://bugs.python.org/issue8296> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Andrey Vlasovskikh added the comment: Despite of several workarounds available on the Web, the problem persists. Almost any exception that is rised in `worker` function while putting or getting tasks from queues result in Pool hang up. Currently, `worker` is only aware of Exception descendants rised inside of the map function parameter. I've written a unit test that checks if KeyboardInterrupts are handled normally. The source code may be included in `Lib/test/test_multiprocessing.py`. -- Added file: http://bugs.python.org/file16784/test_map_keyboard_interrput.py ___ Python tracker <http://bugs.python.org/issue8296> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Changes by Andrey Vlasovskikh : Removed file: http://bugs.python.org/file16743/test_pool_keyboardinterrupt.py ___ Python tracker <http://bugs.python.org/issue8296> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Andrey Vlasovskikh added the comment: Here is a patch that fixes this problem. Basically, it catches all the BaseExceptions that could happen during: a) getting a task from the `inqueue`, b) calling a user function, c) putting a task into the `outqueue`. The exception handler puts the exception to the `outqueue`. It can be cleanly applied on top of revision 78790. -- keywords: +patch Added file: http://bugs.python.org/file16785/fix-sigint.diff ___ Python tracker <http://bugs.python.org/issue8296> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1230540] sys.excepthook doesn't work in threads
Change by Andrey Vlasovskikh : -- keywords: +patch pull_requests: +8116 stage: needs patch -> patch review ___ Python tracker <https://bugs.python.org/issue1230540> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1230540] sys.excepthook doesn't work in threads
Andrey Vlasovskikh added the comment: I've added a PR with a patch I developed during the EuroPython 2018 sprint. I've fixed this issue in a way that is more or less consistent with how '_thread' threads interact with sys.excepthook, but I haven't added the log line they print to sys.stderr about an unhandled exception in a thread: Unhandled exception in thread started by at 0x108c840d0> I can add these messages if they are needed. I'd like to mention the issue I discussed with Victor Stinner at the sprints (adding Victor to the nosy list as he requested). The fix could possibly break the de facto contract that has been there for ages that 'theading' threads don't invoke sys.excepthook on uncaught exceptions. If the fact that sys.execepthook doesn't work with threading is considered a feature, then an alternative solution could be a new threading-specific hook: threading.excepthook. -- nosy: +vlasovskikh, vstinner ___ Python tracker <https://bugs.python.org/issue1230540> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1230540] sys.excepthook doesn't work in threads
Andrey Vlasovskikh added the comment: > Would it be possible to modify the default implementation of sys.excepthook > to have a different output when it's not called from the main thread? Mimick > the current traceback from threads. I agree it's a good idea to mimic what `_thread` does in case of both handled and unhandled exceptions. Why would you want to do it in `sys.excepthook`? The code for handling unhandled exceptions in `_thread` threads is located in `_threadmodule.c` itself, so there is no dependency from `sys` to `_thread`. By following the same logic, the code for handling unhandled exceptions in `threading` should be located in `threading.py`, like it does now. So the only thing that's missing in the pull request compared to `_thread` is the output: Unhandled exception in thread started by at 0x7f6769ca8e18> for all the exceptions: not caught by the sys.excepthook **and** caught by the sys.exceptook. > An alternative would be to introduce a new hook to log exceptions in threads, > ex: sys.threadexcepthook. That hoook would take a 4th parameter. But Andrey > Vlasovskikh didn't like this idea. I'm overall OK with this idea, but I would prefer to make the existing `sys.excepthook` API applicable to all the exceptions: for the main thread (works well), for `_thread` threads (works well) and for `threading` threads (doesn't work). -- ___ Python tracker <https://bugs.python.org/issue1230540> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
