New submission from Alexis:
I am launching a process inside a pool worker, using the multiprocessing module.
After a while, a deadlock append when I am trying to join the process.
Here is a simple version of the code:
import sys, time, multiprocessing
from multiprocessing.pool import ThreadPool
def main():
# Launch 8 workers
pool = ThreadPool(8)
it = pool.imap(run, range(500))
while True:
try:
it.next()
except StopIteration:
break
def run(value):
# Each worker launch its own Process
process = multiprocessing.Process(target=run_and_might_segfault,
args=(value,))
process.start()
while process.is_alive():
sys.stdout.write('.')
sys.stdout.flush()
time.sleep(0.1)
# Will never join after a while, because of a mystery deadlock
process.join()
def run_and_might_segfault(value):
print(value)
if __name__ == '__main__':
main()
And here is a possible output:
~ python m.py
..0
1
........8
.9
.......10
......11
........12
13
........14
........16
........................................................................................
As you can see, process.is_alive() is alway true after few iterations, the
process will never join.
If I CTRL-C the script a get this stacktrace:
Traceback (most recent call last):
File
"/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/pool.py",
line 680, in next
item = self._items.popleft()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "m.py", line 30, in <module>
main()
File "m.py", line 9, in main
it.next()
File
"/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5
/lib/python3.5/multiprocessing/pool.py", line 684, in next
self._cond.wait(timeout)
File
"/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5
/lib/python3.5/threading.py", line 293, in wait
waiter.acquire()
KeyboardInterrupt
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File
"/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5
/lib/python3.5/multiprocessing/popen_fork.py", line 29, in poll
pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
Using python 3.5.1 on macos, also tried with 3.5.2 with same issue.
Same result on Debian.
I tried using python 2.7, and it is working well. May be a python 3.5 issue
only?
Here is the link of the stackoverflow question:
http://stackoverflow.com/questions/39884898/large-amount-of-multiprocessing-process-causing-deadlock
----------
components: Library (Lib)
messages: 278221
nosy: Hadhoke
priority: normal
severity: normal
status: open
title: Possible deadlock after many multiprocessing.Process are launch
type: behavior
versions: Python 3.5
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue28382>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
