New submission from Freek Dijkstra <fr...@macfreek.nl>:
I came across three limitation in the multiprocessing module that were not
handled correctly.
Attached is a file that reproduces the errors in minimal code. I tested them
with Python 2.6.5 and 3.1.2.
Expected result:
multiprocessing.Pool's promises a map function where each result is returned
transparently to the main process (despite that the calculation was done in a
subprocess)
Actual result:
Not all values returned by a subprocess are returned transparently.
I expected multiprocessing to handle these cases gracefully by yielding an
Exception in the Main process.
The cases I found are:
1) A multiprocessing worker can not return (return, not raise!) an Exception.
If this is attempted, the result handler thread in the Pool calls the exception
with no arguments,
which might raise an error if multiple arguments are required:
TypeError: ('__init__() takes exactly 2 arguments (1 given)', <class
'__main__.MyException'>, ())
2) A multiprocessing worker can not return an hashlib Object.
If this is attempted, pickle returns an error:
PicklingError: Can't pickle <type '_hashlib.HASH'>: attribute lookup
_hashlib.HASH failed
3) A multiprocessing worker can not return an Object which overrides
__getattr__, and accesses a variable from self in __getattr__.
If this is attempted, Python 2.6 crashes with a bus error:
Program terminated by uncaught signal #10 after 1.56 seconds.
Python 3.1 yields the error:
RuntimeError: maximum recursion depth exceeded while calling a Python object
----------
components: Library (Lib)
files: multiprocessingbugs.py
messages: 113816
nosy: macfreek
priority: normal
severity: normal
status: open
title: Limitations in objects returned by multiprocessing Pool
type: crash
versions: Python 2.6, Python 3.1
Added file: http://bugs.python.org/file18507/multiprocessingbugs.py
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue9592>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
