I'm running into a problem in the multiprocessing module.
My code is running four parallel processes which are doing network access
completely independently of each other (gathering data from different remote
sources). On rare circumstances, the code blows up when one of my processes
has do start doing some error recovery. I strongly suspect it is because there
is a time-out that isn't being caught in the multiprocessing lib, and that in
turn is exposing the TypeError. Note that the error is "cannot concatenate
'str' and 'NoneType' objects and it is occurring way down in the
multiprocessing library.
I'd really appreciate it if someone more knowledgeable about multiprocessing
could confirm (or refute) my suspicion and then tell me how to fix things up.
I'm running python 2.7.5 on a Mac OS-X 10.8.5
The traceback I get is:
TypeError: cannot concatenate 'str' and 'NoneType' objects
File "/Users/wrw/Dev/Python/Connection_Monitor/Version3.0/CM_Harness.py", line
20, in <module>
my_pool = pool.map(monitor, targets) # and hands off to four targets
File
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py",
line 250, in map
return self.map_async(func, iterable, chunksize).get()
File
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py",
line 554, in get
raise self._value
To save you-all some time:
The "get" function at line 554 in pool.py (which is in the multiprocessing lib)
is:
def get(self, timeout=None):
self.wait(timeout)
if not self._ready:
raise TimeoutError
if self._success:
return self._value
else:
raise self._value
And the map function (also in pool in the multiprocessing lib) is:
def map(self, func, iterable, chunksize=None):
'''
Equivalent of `map()` builtin
'''
assert self._state == RUN
return self.map_async(func, iterable, chunksize).get()
Finally, my code that calls all this is pretty simple (note that the targets
are dummies here):
#!/usr/bin/env python
""" Harness to call multiple parallel copies
of the basic monitor program
"""
targets = ["www.sdsc.edu", "www.ncsa.edu", "www.uiuc.edu", "www.berkeley.edu"]
pool = Pool(processes=4) # start 4 worker processes
my_pool = pool.map(monitor, targets) # and hands off to four targets
TiA
Bill
--
https://mail.python.org/mailman/listinfo/python-list
