Kyle Stanley <aeros...@gmail.com> added the comment:
Due to the way pickle works, it's not presently possible to serialize wrapped
functions directly, at least not in a way that allows you to pass it to
executor.submit() within the inner function (AFAICT). I'm also not certain what
it would involve to provide that, or if it would be feasible to do so in a
manner that would be backwards compatible.
In the meantime, this is a decent work-around:
```
from concurrent import futures
import random
class PPEWrapper:
def __init__(self, func, num_proc=4):
self.fn = func
self.num_proc = num_proc
def __call__(self, *args, **kwargs):
with futures.ProcessPoolExecutor() as executor:
fs = []
for i in range(self.num_proc):
fut = executor.submit(self.fn, *args, **kwargs)
fs.append(fut)
result = []
for f in futures.as_completed(fs):
result.append(f.result())
return result
def _get_random_int():
return random.randint(0, 100)
# it's often quite useful anyways to have a parallel and non-parallel version
# (for testing and devices that don't support MP)
get_random_int = PPEWrapper(_get_random_int, num_proc=4)
if __name__ == "__main__":
result = get_random_int()
print(result)
```
This doesn't allow you to use the decorator syntax, but it largely provides the
same functionality. That being said, my familiarity with the pickle protocol
isn't overly strong, so the above is mostly based on my own recent
investigation. There could very well be a way to accomplish what you're looking
for in a way that I was unable to determine.
----------
nosy: +aeros, alexandre.vassalotti, pitrou
_______________________________________
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue41577>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
