[issue29842] Make Executor.map work with infinite/large inputs correctly

Change by Leonard Lausen : -- nosy: +leezu ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pytho

[issue29842] Make Executor.map work with infinite/large inputs correctly

David Lukeš added the comment: Any updates on this? Making Executor.map lazier would indeed be more consistent and very useful, it would be a shame if the PR went to waste :) It's a feature I keep wishing for in comparison with the older and process-only multiprocessing API. And eventually, y

[issue29842] Make Executor.map work with infinite/large inputs correctly

Change by Thomas Grainger : -- keywords: +patch pull_requests: +17948 stage: -> patch review pull_request: https://github.com/python/cpython/pull/18566 ___ Python tracker ___

[issue29842] Make Executor.map work with infinite/large inputs correctly

Josh Rosenberg added the comment: Noticed unresolved comments (largely on documentation) on the PR and since I'm sprinting this week I finally had the time to address them. I merged the latest master into the PR, hope that's considered the correct way to approach this. -- __

[issue29842] Make Executor.map work with infinite/large inputs correctly

Change by Josh Rosenberg : -- versions: +Python 3.8 -Python 3.7 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe

[issue29842] Make Executor.map work with infinite/large inputs correctly

Josh Rosenberg added the comment: In response to Max's comments: >But consider the case where input is produced slower than it can be processed >(`iterables` may fetch data from a database, but the callable `fn` may be a >fast in-memory transformation). Now suppose the `Executor.map` is call

[issue29842] Make Executor.map work with infinite/large inputs correctly

Josh Rosenberg added the comment: In any event, sorry to be a pain, but is there any way to get some movement on this issue? One person reviewed the code with no significant concerns to address. There have been a duplicate (#30323) and closely related (#34168) issues opened that this would a

[issue29842] Make Executor.map work with infinite/large inputs correctly

Changes by Klamann : -- nosy: +Klamann ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/

[issue29842] Make Executor.map work with infinite/large inputs correctly

Max added the comment: Correction: this PR is useful for `ProcessPoolExecutor` as well. I thought `chunksize` parameter handles infinite generators already, but I was wrong. And, as long as the number of items prefetched is a multiple of `chunksize`, there are no issues with the chunksize opti

[issue29842] Make Executor.map work with infinite/large inputs correctly

Max added the comment: I'm also concerned about this (undocumented) inconsistency between map and Executor.map. I think you would want to make your PR limited to `ThreadPoolExecutor`. The `ProcessPoolExecutor` already does everything you want with its `chunksize` paramater, and adding `prefet

[issue29842] Make Executor.map work with infinite/large inputs correctly

Changes by Josh Rosenberg : -- title: Executor.map should not submit all futures prior to yielding any results -> Make Executor.map work with infinite/large inputs correctly ___ Python tracker