I have a program where I am currently using a
concurrent.futures.ThreadPoolExecutor to run multiple tasks
concurrently. These tasks are typically I/O bound, involving access to
local databases and remote REST APIs. However, these tasks could
themselves be split into subtasks, which would also benefit from
concurrency.
What I am hoping is that it is safe to use a
concurrent.futures.ThreadPoolExecutor within the tasks. I have coded up
a toy example, which seems to work. However, I'd like some confidence
that this is intentional. Concurrency is notoriously tricky.
I very much hope this is safe, because otherwise it would not be safe to
use a ThreadPoolExecutor to execute arbitrary code, in case it also used
concurrent.futures to exploit concurrency.
Here is the toy example:
|importconcurrent.futures definner(i,j):returni,j,i**j
defouter(i):withconcurrent.futures.ThreadPoolExecutor(max_workers=5)asexecutor:futures
={executor.submit(inner,i,j):j forj inrange(5)}results =[]forfuture
inconcurrent.futures.as_completed(futures):results.append(future.result())returnresults
defmain():withconcurrent.futures.ThreadPoolExecutor(max_workers=5)asexecutor:futures
={executor.submit(outer,i):i fori inrange(10)}results =[]forfuture
inconcurrent.futures.as_completed(futures):results.extend(future.result())print(results)if__name__
=="__main__":main()|
I have previously posted this on Stack Overflow, but didn't get any
replies. Apologies if you are seeing this twice.
https://stackoverflow.com/questions/44989473/nesting-concurrent-futures-threadpoolexecutor
--
https://mail.python.org/mailman/listinfo/python-list
