New submission from Wojciech Walczak:
While using multiprocessing.Manager() to send data between processes I've
noticed that one of the traceback messages is not very informative:
Traceback (most recent call last):
File "age_predict.py", line 39, in <module>
train_data, train_target, test_data = prepare_data()
File "age_predict.py", line 30, in prepare_data
train_data = q.get(True, 10000)
File "<string>", line 2, in get
File "/usr/lib/python2.7/multiprocessing/managers.py", line 777, in
_callmethod
raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:
---------------------------------------------------------------------
Unserializable message: ('#RETURN', [A WHOLE LOT OF DATA GOES HERE]
The real problem here is that my machine is running out of memory, but the
traceback message shadows this information and reports "Unserializable message"
instead.
After a simple path (see: attached file) the traceback message is as follows:
Traceback (most recent call last):
File "age_predict.py", line 39, in <module>
train_data, train_target, test_data = prepare_data()
File "age_predict.py", line 30, in prepare_data
train_data = q.get(True, 10000)
File "<string>", line 2, in get
File "/usr/lib/python2.7/multiprocessing/managers.py", line 775, in
_callmethod
raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:
---------------------------------------------------------------------
Unserializable message: Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/managers.py", line 288, in
serve_client
send(msg)
MemoryError: out of memory
Here's another example (see: http://bugs.python.org/issue6766):
This python3 code:
from multiprocessing import Manager
manager = Manager()
ns_proxy = manager.Namespace()
evt_proxy = manager.Event()
ns_proxy.my_event_proxy = evt_proxy
print(ns_proxy.my_event_proxy)
Produces following traceback message:
Traceback (most recent call last):
File "test.py", line 6, in <module>
print(ns_proxy.my_event_proxy)
File "/cpython/Lib/multiprocessing/managers.py", line 1032, in __getattr__
return callmethod('__getattribute__', (key,))
File "/cpython/Lib/multiprocessing/managers.py", line 748, in _callmethod
raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:
----------------------------------------------------------------------
Unserializable message: ('#RETURN', <threading.Event object at 0x7f9484aff278>)
----------------------------------------------------------------------
...and after applying the attached patch it becomes clearer what's going on:
Traceback (most recent call last):
File "test.py", line 6, in <module>
print(ns_proxy.my_event_proxy)
File "/cpython/Lib/multiprocessing/managers.py", line 1032, in __getattr__
return callmethod('__getattribute__', (key,))
File "/cpython/Lib/multiprocessing/managers.py", line 748, in _callmethod
raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:
----------------------------------------------------------------------
Unserializable message: Traceback (most recent call last):
File "/cpython/Lib/multiprocessing/managers.py", line 276, in serve_client
send(msg)
File "/cpython/Lib/multiprocessing/connection.py", line 206, in send
self._send_bytes(ForkingPickler.dumps(obj))
File "/cpython/Lib/multiprocessing/reduction.py", line 50, in dumps
cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class '_thread.lock'>: attribute lookup
lock on _thread failed
This patch doesn't break any tests.
----------
components: Library (Lib)
files: managers_tb_msg.patch
keywords: patch
messages: 215934
nosy: wojtekwalczak
priority: normal
severity: normal
status: open
title: Uninformative error message in multiprocessing.Manager()
type: behavior
versions: Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 3.5
Added file: http://bugs.python.org/file34785/managers_tb_msg.patch
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue21201>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
