New submission from Brian Candler:
Probably best demonstrated by example.
~~~~~~~~~~~~~~~~
import multiprocessing
class Myerror(ValueError):
def __init__(self,a):
self.a = a
def __str__(self):
return repr(self.a)
def foo(arg):
raise Myerror(arg)
#foo("1") #<= this works fine, raises exception as expected
#But this breaks:
pool = multiprocessing.Pool(2)
pool.map(foo, ["1","2","3"])
~~~~~~~~~~~~~~~~
The result seen:
~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 504, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 353, in
_handle_results
task = get()
TypeError: ('__init__() takes exactly 2 arguments (1 given)', <class
'__main__.Myerror'>, ())
~~~~~~~~~~~~~~~~
At this point the application hangs. Worse: pressing ctrl-C shows a traceback
and KeyboardInterrupt, but the worker keeps getting restarted, so it's
impossible to stop. You have to go to another shell and do somthing like
killall python
to terminate the program.
A real-world example (which caused me to track this down) is a
CalledProcessError raised by subprocess.check_call
~~~~~~~~~~~~~~~~
import multiprocessing
import subprocess
def foo(arg):
subprocess.check_call("nonexistent", shell=True)
#raise subprocess.CalledProcessError(127, "nonexistent")
pool = multiprocessing.Pool(2)
pool.map(foo, ["1","2","3"])
~~~~~~~~~~~~~~~~
which fails in the same way:
~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 504, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 353, in
_handle_results
task = get()
TypeError: ('__init__() takes at least 3 arguments (1 given)', <class
'subprocess.CalledProcessError'>, ())
~~~~~~~~~~~~~~~~
Behaviour tested on:
python 2.7.3 on Ubuntu 12.04
python 2.7.1 on OSX 10.7.5
Workaround: re-raise a parameter-less exception instead.
try:
...
except Exception as e:
raise RuntimeError
----------
components: Library (Lib)
messages: 176416
nosy: candlerb
priority: normal
severity: normal
status: open
title: multiprocessing fails to raise exception with parameters
type: behavior
versions: Python 2.7
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue16558>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
