Re: Reducing "yield from" overhead in recursive generators

[Antoon Pardon]

Op 18/03/2022 om 04:14 schreef Rathmann:

...

```python
def recursive_generator_driver(g):
 """g is a generator object, which is likely to call other generators"""
 stack = [g]
 while True:
 g = stack[-1]
 try:
 val = next(g)
 if isinstance(val, types.GeneratorType):
 # Yielded value represented a recursive call, so push
 stack.append(val)
 else:
 # Regular value for iterator, give to caller
 yield val
 except StopIteration:
 stack.pop()
 if len(stack) == 0:
 return
```

and the modified tree traverser is just

```python
def inorder_traverse_mod(node):
 """ONLY for use with recursive_generator_driver"""
 k, l, r = node
 if l:
 yield inorder_traverse_mod(l) # NOT yield from
 yield k
 if r:
 yield inorder_traverse_mod(r) # NOT yield from
```

...

So what do people think of this hack/technique?  Is it

A) A terrible idea?  (Please be specific.)
B) Already well-known (and I just missed it.)
C) Potentially useful for the niche case of deeply recursive
generators?


I would go with B' + C. It seems there is a resemblance with the trampoline 
technique
in order to eliminate tail recursion. I know the people of macropy have written 
a
decorating macro that would eliminate tail recursion from such decorated 
functions.
Maybe if you contact them they can be interested in making a similar decorating 
macro
for use with such recursive decorators.

--
Antoon Pardon.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Reducing "yield from" overhead in recursive generators

[Rathmann]

On Friday, March 18, 2022 at 2:07:45 AM UTC-7, Antoon Pardon wrote:
> Op 18/03/2022 om 04:14 schreef Rathmann: 
> > 
> > So what do people think of this hack/technique? Is it 
> > 
> > A) A terrible idea? (Please be specific.) 
> > B) Already well-known (and I just missed it.) 
> > C) Potentially useful for the niche case of deeply recursive 
> > generators?
> I would go with B' + C. It seems there is a resemblance with the trampoline 
> technique 
> in order to eliminate tail recursion. I know the people of macropy have 
> written a 
> decorating macro that would eliminate tail recursion from such decorated 
> functions. 
> Maybe if you contact them they can be interested in making a similar 
> decorating macro 
> for use with such recursive decorators. 
> 
> -- 
> Antoon Pardon.

Thank you Antoon.  I was not aware of macropy.  It certainly seems
like a macro could automate the process of switching the "yield from"
to yield forms.  That is already pretty easy to do by hand, though.

The driver itself could be packaged as a decorator using just the
facilities available in regular Python implementations.

**Much** more ambitious would be a macro to automate the
transformation of a generator from (general) recursive to iterative
form, so as to avoid the need for this technique entirely.

The other challenge/question would be to see if there is a way to implement
this basic approach with lower overhead.  So far, the savings don't
kick in until the recursion hits a depth of 12 or so (even more with
CPython), and many use cases have an expected recursion depth
shallower than that.
-- 
https://mail.python.org/mailman/listinfo/python-list

Compiling and Linking pre-built Windows Python libraries with C++ files on Linux for Windows

[Ankit Agarwal]

Hi,

This is a very specific question. I am trying to figure out whether or not
I can use pre-built python libraries and headers on Windows in a MinGW
build on Linux. Essentially I have some python and C++ code which interface
via cython and pybind. I want to build a self contained C++ binary for
Windows with the MinGW compiler that runs on Linux. For both Cython and
PyBind, they need to compile with the python headers and link against the
python DLL in order to run on Windows.

I know that the python DLL specifically are compiled with the MSVC
compiler, however since it is in C, the ABI between the DLL should be
compatible with MinGW, and I should be able to import and link against it.
My question is will this work, or will there be some other problem that I
might run into.

I am using python 3.7 by the way for this.

Let me know what you think,

Thanks :)

Ankit

*Ankit Agarwal*

Software Engineer

an...@applied.co

+1 630.506.4914

Applied Intuition, Inc. 
blog.applied.co
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Reducing "yield from" overhead in recursive generators

[Greg Ewing]

On 19/03/22 9:40 am, Rathmann wrote:

The other challenge/question would be to see if there is a way to implement
this basic approach with lower overhead.


My original implementation of yield-from didn't have this problem,
because it didn't enter any of the intermediate frames -- it just
ran down a chain of C pointers to find the currently active one.

At some point this was changed, I think so that all the frames
would show up in the traceback in the event of an exception.
This was evidently seen as more important than having efficient
resumption of nested generators.

I'm not sure exactly how it works now, but I believe it involves
re-executing the YIELD_FROM bytecode in each generator down the
chain whenever a resumption occurs. This is nice and simple, but
not very efficient.

Maybe another way could be found of preserving the traceback,
such as temporarily splicing the chain of frames into the call
stack and then resuming the last one.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Reducing "yield from" overhead in recursive generators

[Chris Angelico]

On Sat, 19 Mar 2022 at 14:06, Greg Ewing  wrote:
>
> On 19/03/22 9:40 am, Rathmann wrote:
> > The other challenge/question would be to see if there is a way to implement
> > this basic approach with lower overhead.
>
> My original implementation of yield-from didn't have this problem,
> because it didn't enter any of the intermediate frames -- it just
> ran down a chain of C pointers to find the currently active one.
>
> At some point this was changed, I think so that all the frames
> would show up in the traceback in the event of an exception.
> This was evidently seen as more important than having efficient
> resumption of nested generators.
>
> I'm not sure exactly how it works now, but I believe it involves
> re-executing the YIELD_FROM bytecode in each generator down the
> chain whenever a resumption occurs. This is nice and simple, but
> not very efficient.
>
> Maybe another way could be found of preserving the traceback,
> such as temporarily splicing the chain of frames into the call
> stack and then resuming the last one.
>

This is a fundamentally difficult problem with async/await too, so if
there's any good solution, it would be worth carrying across.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list