I have very often wanted versions of itertools.product(), itertools.permutations(), and itertools.combinations() that don't concretize their iterators.
They very much violate the spirit of itertools, and hard-lock my machine when I forget the bad design and try infinite iterators. I don't think we necessarily need variations for every sensible ordering of produced items. But not freezing works be a good start. On Fri, Mar 19, 2021, 4:06 PM Alastair Stanley via Python-ideas < [email protected]> wrote: > This is closely related to https://bugs.python.org/issue40230 - > "Itertools.product() Out of Memory Errors." > `itertools.product()` completely consumes all input iterables before > yielding any values, which can cause memory issues in certain (extreme) > cases. > > I recently needed to partially iterate through the product of two very > large iterators which were themselves power sets of some input values. For > unlucky inputs these iterators were enormous (but not infinite) and > `itertools.product()` used up all the memory on my machine, even though I > sometimes only needed the first few values. > > My solution was to write my own version of `itertools.product()` that > takes generators instead of iterables. This allowed me to recreate and loop > through the required iterables multiple times without ever having to store > all their values. Product tuples are produced immediately and the iteration > can be completed for large inputs without runaway memory consumption. > > I'm proposing that equivalent functions accepting functions/generators: > maybe `fproduct()`, `fcombinations()` etc, be added to `itertools`. > > e.g. simplified `fproduct` for 2 inputs: > > ``` > def fproduct(f1, f2): > for a in f1(): > for b in f2(): > yield (a, b) > ``` > > Would others find this useful? Are there any drawbacks I'm missing? > > Best, > Alastair > _______________________________________________ > Python-ideas mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/RFX7C2YYHIR4SLUSXTPVADM4XJERITLU/ > Code of Conduct: http://python.org/psf/codeofconduct/ >
_______________________________________________ Python-ideas mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/32GXJNX2M52XIV2L2GJK6L2F5JAGKJ36/ Code of Conduct: http://python.org/psf/codeofconduct/
