On 11/21/2013 12:01 PM, John Ladasky wrote:
This is a case where you need to dig into the code (or maybe docs) a bit
File ".../evaluate.py", line 81, in evaluate
> result = pool.map(evaluate, bundles) File
"/usr/lib/python3.3/multiprocessing/pool.py", line 228, in map
> return self._map_async(func, iterable, mapstar, chunksize).get()
The call to _map_async gets a blank MapResult (a subclass of
ApplyResult), queues tasks to fill it in, and returns the filled in
result. This call is designed to always return as task exceptions are
caught and assigned to MapResult._value in both ApplyResult._set and
MapResult._set.
result = MapResult(self._cache, chunksize, len(iterable), callback,
error_callback=error_callback)
self._taskqueue.put((((result._job, i, mapper, (x,), {})
for i, x in enumerate(task_batches)), None))
return result
It is the subsequent call to get() that 'fails', because it raises
the caught exception.
> File "/usr/lib/python3.3/multiprocessing/pool.py", line 564, in get
> raise self._value
ValueError: operands could not be broadcast together with shapes (1,3) (4)
Notice that no line of numpy appears in the traceback? Still, there
are three things that make me think that this error is coming from
numpy.
It comes from one of your tasks as the 'result', and your tasks use numpy.
If I could only see the line of numpy code which is generating the
ValueError, I would have a better chance of spotting the bug in my
code.
Definitely.
> So, WHY isn't there any reference to numpy in my traceback?
I suspect that raising the exception may replace its __traceback__
attribute. Anyway, there are three things I might try.
1. Use 3.3.3 or latest 3.4 to see if there is any improvement in output.
I vaguely remember a tracker issue that might be related.
2. _map_async takes an error_callback arg that defaults to None and
which is passed on to MapResult. When _value is set to an exception,
"error_callback(_value)" is called in ._set() before the later .get()
re-raises it. pool.map does not allow you to set either the (success)
callback or the error_callback, but pool.map_async does (this is the
difference between the two methods). So switch to the latter so you can
pass a function that uses the traceback module to print (or log) the
traceback attached to _value, assuming that there is one.
3. If that does not work, wrap the current body of your task function in
try: <current suite>
except exception as e:
<use traceback module to add traceback to message>
raise e <or a new exception>
--
Terry Jan Reedy
--
https://mail.python.org/mailman/listinfo/python-list
