Re: [Python-Dev] Please reconsider PEP 479.
Guido van Rossum wrote: Hm, that sounds like you're either being contrarian or Chris and I have explained it even worse than I thought. I'm not trying to be contrary, I just think the PEP could explain more clearly what you're trying to achieve. The rationale is too vague and waffly at the moment. Currently, there are cases where list(x for x in xs if P(x)) works while [x for x in xs if P(x)] fails (when P(x) raises StopIteration). With the PEP, both cases will raise some exception That's a better explanation, I think. -- Greg ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On 26 November 2014 at 18:30, Greg Ewing wrote: > Guido van Rossum wrote: >> >> Hm, that sounds like you're either being contrarian or Chris and I have >> explained it even worse than I thought. > > I'm not trying to be contrary, I just think the PEP could > explain more clearly what you're trying to achieve. The > rationale is too vague and waffly at the moment. > >> Currently, there are cases where list(x for x in xs if P(x)) works while >> [x for x in xs if P(x)] fails (when P(x) raises StopIteration). With the >> PEP, both cases will raise some exception > > That's a better explanation, I think. The other key aspect is that it changes the answer to the question "How do I gracefully terminate a generator function?". The existing behaviour has an "or" in the answer: "return from the generator frame, OR raise StopIteration from the generator frame". That then leads to the follow on question: "When should I use one over the other?". The "from __future__ import generator_stop" answer drops the "or", so it's just: "return from the generator frame". Raising *any* exception inside the generator, including StopIteration, then counts as non-graceful termination, bringing generators into line with the PEP 343 philosophy that "hiding flow control in macros makes your code inscrutable", where here, the hidden flow control is relying on the fact that a called function raising StopIteration will currently always gracefully terminate generator execution. The key downside is that it means relatively idiomatic code like: def my_generator(): ... yield next(it) ... Now needs to be written out explicitly as: def my_generator(): ... try: yield next(it) except StopIteration return ... That's not especially easy to read, and it's also going to be very slow when working with generator based producer/consumer pipelines. After thinking about that concern for a while, I'd like to suggest the idea of having a new builtin "allow_implicit_stop" decorator that swaps out a GENERATOR code object that has the new "EXPLICIT_STOP" flag set for one with it cleared (attempting to apply "allow_implicit_stop" to a normal function would be an error). Then the updated version of the above example would become: @allow_implicit_stop def my_generator(): ... yield next(it) ... Which would be semantically equivalent to: def my_generator(): try: ... yield next(it) ... except StopIteration return but *much* faster (especially if used in a producer/consumer pipeline) since it would allow a single StopIteration instance to propagate through the entire pipeline, rather than creating and destroying new ones at each stage. Including such a feature in the PEP would also make the fix to contextlib simpler: we'd just update it so that contextlib._GeneratorContextManager automatically calls "allow_implicit_stop" on the passed in generator functions. Single-source Python 2/3 code would also benefit in a 3.7+ world, since libraries like six and python-future could just define their own version of "allow_implicit_stop" that referred to the new builtin in 3.5+, and was implemented as an identity function in other versions. Cheers, Nick. P.S. While I'm less convinced this part is a good idea, if "allow_implicit_stop" accepted both generator functions *and* generator objects, then folks could even still explicitly opt in to the "or stop()" trick - and anyone reading the code would have a name to look up to see what was going on. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On 26 November 2014 at 21:44, Petr Viktorin wrote: > On Wed, Nov 26, 2014 at 12:24 PM, Nick Coghlan wrote: >> Now needs to be written out explicitly as: >> >> def my_generator(): >> ... >>try: >> yield next(it) >> except StopIteration >> return >> ... > > It could also be written as: > > def my_generator(): > try: > ... > yield next(it) > ... > except StopIteration > return > > i.e. put the try-block around the whole body, not just the individual > yield. This emulates what's happenning in current Python, and it would > be faster than individual try blocks. Not appreciably though - the potentially slow part that the status quo avoids is the "catch old exception, return from frame, create and raise new exception" step, and that only happens once per generator termination, regardless of how many times the try/except/return pattern appears in the body of a particular generator. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Wed, Nov 26, 2014 at 10:24 PM, Nick Coghlan wrote: > The other key aspect is that it changes the answer to the question > "How do I gracefully terminate a generator function?". The existing > behaviour has an "or" in the answer: "return from the generator frame, > OR raise StopIteration from the generator frame". That then leads to > the follow on question: "When should I use one over the other?". > > The "from __future__ import generator_stop" answer drops the "or", so > it's just: "return from the generator frame". If I understand you correctly, you agree that this is a benefit, correct? > The key downside is that it means relatively idiomatic code like: > > def my_generator(): > ... > yield next(it) > ... > > Now needs to be written out explicitly as: > > def my_generator(): > ... >try: > yield next(it) > except StopIteration > return > ... > > That's not especially easy to read, and it's also going to be very > slow when working with generator based producer/consumer pipelines. I'm not sure how often the ease-of-reading concern will come up, but I can at least benchmark the performance of it. I have two virtually-identical builds of CPython 3.5, one with and one without the POC patch. There's no guarantee that this will properly match the performance of the final product, as there'll likely be some additional checks (especially when there's a __future__ directive to concern ourselves with), but it's a start. yield from: https://github.com/Rosuav/GenStopIter/blob/485d1/perftest.py explicit loop: https://github.com/Rosuav/GenStopIter/blob/c071d/perftest.py The numbers are pretty noisy, but I'm seeing about a 5% slowdown in the 'yield from' version, with a recursion depth of 100 generators. (Obviously less percentage slowdown with less depth, as other factors have more impact.) Rewriting the loop to use an explicit try/except and while loop roughly doubles the time taken (so clearly 'yield from' is implemented very efficiently), and also destroys any meaning in the numbers - the two interpreters come out effectively equal. My preliminary conclusion is that there is some impact, but it's unlikely to be significant in the real world. Do you have a more real-world code example to try? > After thinking about that concern for a while, I'd like to suggest the > idea of having a new builtin "allow_implicit_stop" decorator that > swaps out a GENERATOR code object that has the new "EXPLICIT_STOP" > flag set for one with it cleared (attempting to apply > "allow_implicit_stop" to a normal function would be an error). > > Then the updated version of the above example would become: > > @allow_implicit_stop > def my_generator(): > ... > yield next(it) > ... > > Which would be semantically equivalent to: > > def my_generator(): >try: >... > yield next(it) > ... > except StopIteration > return > > but *much* faster (especially if used in a producer/consumer pipeline) > since it would allow a single StopIteration instance to propagate > through the entire pipeline, rather than creating and destroying new > ones at each stage. If the issue is performance, I would prefer to see something done with a peephole optimizer instead: if there's a "try... except StopIteration: return" construct, it's optimized away down to a magic flag. That way, the code is guaranteed to be correct in all cases, with no hidden behavioral changes, and this is _just_ a performance optimization. I'd still rather see the exception-catching scope narrowed as much as possible, though, which means not having something that's semantically equivalent to wrapping the whole generator in "try... except StopIteration: pass". > P.S. While I'm less convinced this part is a good idea, if > "allow_implicit_stop" accepted both generator functions *and* > generator objects, then folks could even still explicitly opt in to > the "or stop()" trick - and anyone reading the code would have a name > to look up to see what was going on. That would look somewhat thus, then: use_an_iterable(allow_implicit_stop(x+1 for x in itr if x<10 or stop())) I'm definitely not convinced that this would improve matters. However, I don't currently have any recommendation for an "or stop()" replacement, other than "refactor it into an explicitly-written generator function and call it, then you can use return statements". Suggestions welcomed. ChrisA ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Wed, Nov 26, 2014 at 12:24 PM, Nick Coghlan wrote: > On 26 November 2014 at 18:30, Greg Ewing wrote: >> Guido van Rossum wrote: >>> >>> Hm, that sounds like you're either being contrarian or Chris and I have >>> explained it even worse than I thought. >> >> I'm not trying to be contrary, I just think the PEP could >> explain more clearly what you're trying to achieve. The >> rationale is too vague and waffly at the moment. >> >>> Currently, there are cases where list(x for x in xs if P(x)) works while >>> [x for x in xs if P(x)] fails (when P(x) raises StopIteration). With the >>> PEP, both cases will raise some exception >> >> That's a better explanation, I think. > > The other key aspect is that it changes the answer to the question > "How do I gracefully terminate a generator function?". The existing > behaviour has an "or" in the answer: "return from the generator frame, > OR raise StopIteration from the generator frame". That then leads to > the follow on question: "When should I use one over the other?". > > The "from __future__ import generator_stop" answer drops the "or", so > it's just: "return from the generator frame". > > Raising *any* exception inside the generator, including StopIteration, > then counts as non-graceful termination, bringing generators into line > with the PEP 343 philosophy that "hiding flow control in macros makes > your code inscrutable", where here, the hidden flow control is relying > on the fact that a called function raising StopIteration will > currently always gracefully terminate generator execution. > > The key downside is that it means relatively idiomatic code like: > > def my_generator(): > ... > yield next(it) > ... > > Now needs to be written out explicitly as: > > def my_generator(): > ... >try: > yield next(it) > except StopIteration > return > ... It could also be written as: def my_generator(): try: ... yield next(it) ... except StopIteration return i.e. put the try-block around the whole body, not just the individual yield. This emulates what's happenning in current Python, and it would be faster than individual try blocks. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/26/14, Nick Coghlan wrote: > On 26 November 2014 at 04:04, Guido van Rossum wrote: > > On Tue, Nov 25, 2014 at 9:49 AM, Chris Angelico wrote: > >> > >> On Wed, Nov 26, 2014 at 4:45 AM, Isaac Schwabacher > >> wrote: > >> > Yield can also raise StopIteration, if it's thrown in. The current > >> > interaction of generator.throw(StopIteration) with yield from can't be > >> > emulated under the PEP's behavior, though it's not clear that that's a > >> > problem. > >> > > >> > >> Hrm. I have *absolutely* no idea when you would use that, and how > >> you'd go about reworking it to fit this proposal. Do you have any > >> example code (production or synthetic) which throws StopIteration into > >> a generator? > > > > > > Sounds like a good one for the obfuscated Python contest. :-) > > > > Unless the generator has a try/except surrounding the yield point into which > > the exception is thrown, it will bubble right out, and PEP 479 will turn > > this into a RuntimeError. I'll clarify this in the PEP (even though it > > logically follows from the proposal) -- I don't think there's anything to > > worry about. > > This is actually the edge case that needs to be handled in contextlib > - a StopIteration raised by the with statement body gets thrown into > the generator implementing the context manager. My current porting > recommendation is to catch the RuntimeError & look at __cause__ to see > if it's the StopIteration instance that was thrown in, but an > alternative option would be to just call gen.close() in that case, > rather than gen.throw(exc). If this is the current contextlib implementation, does it break if the yield statement is replaced with yield from another context manager generator? ijs ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On 11/26/2014 12:24 PM, Nick Coghlan wrote: Now needs to be written out explicitly as: def my_generator(): ... try: yield next(it) except StopIteration return ... To retrieve a single value from an iterator, one can use the for/break/else idiom: def my_generator(): ... for val in it: yield val break else: return ... In general, catching and raising StopIteration feels like something that should rarely be done by normal code. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Thu, Nov 27, 2014 at 2:55 AM, Hrvoje Niksic wrote: > To retrieve a single value from an iterator, one can use the for/break/else > idiom: > > def my_generator(): > ... > for val in it: > yield val > break > else: > return > ... While that does work, it's not really much more "normal" than a try/except. A for loop implies iteration; having a loop with a hard "break" at the end of it messes with expectations. ChrisA ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/26/14, Nick Coghlan wrote:
> On 26 November 2014 at 04:04, Guido van Rossum wrote:
> > On Tue, Nov 25, 2014 at 9:49 AM, Chris Angelico wrote:
> >>
> >> On Wed, Nov 26, 2014 at 4:45 AM, Isaac Schwabacher
> >> wrote:
> >> > Yield can also raise StopIteration, if it's thrown in. The current
> >> > interaction of generator.throw(StopIteration) with yield from can't be
> >> > emulated under the PEP's behavior, though it's not clear that that's a
> >> > problem.
> >> >
> >>
> >> Hrm. I have *absolutely* no idea when you would use that, and how
> >> you'd go about reworking it to fit this proposal. Do you have any
> >> example code (production or synthetic) which throws StopIteration into
> >> a generator?
> >
> >
> > Sounds like a good one for the obfuscated Python contest. :-)
> >
> > Unless the generator has a try/except surrounding the yield point into which
> > the exception is thrown, it will bubble right out, and PEP 479 will turn
> > this into a RuntimeError. I'll clarify this in the PEP (even though it
> > logically follows from the proposal) -- I don't think there's anything to
> > worry about.
>
> This is actually the edge case that needs to be handled in contextlib
> - a StopIteration raised by the with statement body gets thrown into
> the generator implementing the context manager. My current porting
> recommendation is to catch the RuntimeError & look at __cause__ to see
> if it's the StopIteration instance that was thrown in, but an
> alternative option would be to just call gen.close() in that case,
> rather than gen.throw(exc).
This actually leads to a good example of why the PEP is necessary:
```
In [1]: import contextlib
In [2]: @contextlib.contextmanager
...: def transact():
...: print('setup transaction')
...: try:
...: yield from subgenerator()
...: except:
...: print('rollback transaction')
...: raise
...: else:
...: print('commit transaction')
...: finally:
...: print('clean up transaction')
...:
In [3]: def subgenerator():
...: print('setup subgenerator')
...: try:
...: yield
...: except:
...: print('subgenerator failed')
...: raise
...: else:
...: print('subgenerator succeeded')
...: finally:
...: print('clean up subgenerator')
...:
In [4]: with transact():
...: next(iter([]))
...:
setup transaction
setup subgenerator
subgenerator failed
clean up subgenerator
commit transaction # BAD NOT GOOD BIG FAIL
clean up transaction
```
ijs
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On 11/26/2014 03:24 AM, Nick Coghlan wrote: > > After thinking about that concern for a while, I'd like to suggest the > idea of having a new builtin "allow_implicit_stop" decorator that > swaps out a GENERATOR code object that has the new "EXPLICIT_STOP" > flag set for one with it cleared (attempting to apply > "allow_implicit_stop" to a normal function would be an error). This solution is incomplete -- it's saying that if there is one guarded location, all locations must be guarded, and that is not the case; any other call outside the try/except block has the potential to raise StopIteration. -- ~Ethan~ signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 26 November 2014 at 16:24, Isaac Schwabacher wrote: > This actually leads to a good example of why the PEP is necessary: [...] Oh! If that's the current behaviour, then it probably needs to go into the PEP as a motivating example. It's far more convincing than most of the other arguments I've seen. Just one proviso - is it fixable in contextlib *without* a language change? If so, then it loses a lot of its value. Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Wed, Nov 26, 2014 at 8:54 AM, Paul Moore wrote: > On 26 November 2014 at 16:24, Isaac Schwabacher > wrote: > > This actually leads to a good example of why the PEP is necessary: > [...] > > Oh! If that's the current behaviour, then it probably needs to go into > the PEP as a motivating example. It's far more convincing than most of > the other arguments I've seen. Just one proviso - is it fixable in > contextlib *without* a language change? If so, then it loses a lot of > its value. > It's hard to use as an example because the behavior of contextlib is an integral part of it -- currently for me the example boils down to "there is a bug in contextlib". Maybe it would have been caught earlier with the change in the PEP, but when using it as a motivating example you have to show the code containing the bug, not just a demonstration. If you want to try though, I'm happy to entertain a pull request for the PEP. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 26 November 2014 at 17:19, Guido van Rossum wrote: > It's hard to use as an example because the behavior of contextlib is an > integral part of it -- currently for me the example boils down to "there is > a bug in contextlib" Hmm, fair point. I was assuming that the bug in contextlib can't be fixed with the current language behaviour (and I'd personally be OK with the example simply adding a comment "this can't be fixed without changing Python as proposed in the PEP"). But I'm not sure how true that is, so maybe it's not quite as compelling as it seemed to me at first. Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/26/2014 08:54 AM, Paul Moore wrote: > On 26 November 2014 at 16:24, Isaac Schwabacher wrote: >> This actually leads to a good example of why the PEP is necessary: > [...] > > Oh! If that's the current behaviour, then it probably needs to go into > the PEP as a motivating example. It's far more convincing than most of > the other arguments I've seen. Just one proviso - is it fixable in > contextlib *without* a language change? If so, then it loses a lot of > its value. No value is lost. The PEP exists because this mistake is so easy to make, and can be so hard to track down. -- ~Ethan~ signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] [RELEASE] Python 2.7.9 release candidate 1
I'm pleased to announce the first release candidate of Python 2.7.9, which will be the next bugfix release in the Python 2.7 series. Despite technically being a maintenance release, Python 2.7.9 will include several majors changes from 2.7.8: - The "ensurepip" module has been backported to Python 2.7. - Python 3's ssl module has been backported to Python 2.7. - HTTPS certificates are now verified by default using the system's certificate store. - SSLv3 has been disabled by default due to the POODLE attack. Downloads are at https://www.python.org/downloads/release/python-279rc1/ Application and library authors are encouraged test Python 2.7.9 release candidate 1 with their code. This is especially important for 2.7.9 due to significant changes mentioned above. Please report bugs to https://bugs.python.org/ Python 2.7.9 final is currently scheduled for December 10th. Enjoy, Benjamin 2.7 release manager on behalf on python-dev and all of Python's contributors ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Wed, Nov 26, 2014 at 3:24 AM, Nick Coghlan wrote: > On 26 November 2014 at 18:30, Greg Ewing > wrote: > > Guido van Rossum wrote: > >> > >> Hm, that sounds like you're either being contrarian or Chris and I have > >> explained it even worse than I thought. > > > > I'm not trying to be contrary, I just think the PEP could > > explain more clearly what you're trying to achieve. The > > rationale is too vague and waffly at the moment. > > > >> Currently, there are cases where list(x for x in xs if P(x)) works while > >> [x for x in xs if P(x)] fails (when P(x) raises StopIteration). With the > >> PEP, both cases will raise some exception > > > > That's a better explanation, I think. > It's now in the PEP. > The other key aspect is that it changes the answer to the question > "How do I gracefully terminate a generator function?". The existing > behaviour has an "or" in the answer: "return from the generator frame, > OR raise StopIteration from the generator frame". That then leads to > the follow on question: "When should I use one over the other?". > > The "from __future__ import generator_stop" answer drops the "or", so > it's just: "return from the generator frame". > That's now also in the PEP. > Raising *any* exception inside the generator, including StopIteration, > then counts as non-graceful termination, bringing generators into line > with the PEP 343 philosophy that "hiding flow control in macros makes > your code inscrutable", where here, the hidden flow control is relying > on the fact that a called function raising StopIteration will > currently always gracefully terminate generator execution. > Right. > The key downside is that it means relatively idiomatic code like: > > def my_generator(): > ... > yield next(it) > ... > I probably considered this an upside of generators when they were introduced. :-( > Now needs to be written out explicitly as: > > def my_generator(): > ... >try: > yield next(it) > except StopIteration > return > ... > > That's not especially easy to read, and it's also going to be very > slow when working with generator based producer/consumer pipelines. > I want to consider this performance argument seriously. Chris did a little benchmark but I don't think he compared the right things -- he showed that "yield from" becomes 5% slower with his patch and that a while loop is twice as slow as "yield from" with or without his patch. I have no idea why his patch would slow down "yield from" but I doubt it's directly related -- his change only adds some extra code when a generator frame is left with an exception, but his "yield from" example code ( https://github.com/Rosuav/GenStopIter/blob/485d1/perftest.py) never raises (unless I really don't understand how the implementation of "yield from" actually works :-). I guess what we *should* benchmark is this: def g(depth): if depth > 0: it = g(depth-1) yield next(it) else: yield 42 vs. the PEP-479-ly corrected version: def g(depth): if depth > 0: it = g(depth-1) try: yield next(it) except StopIteration: pass else: yield 42 This sets up "depth" generators each with a try/except, and then at the very bottom yields a single value (42) which pops up all the way to the top, never raising StopIteration. I wrote the benchmark and here are the code and results: https://gist.github.com/gvanrossum/1adb5bee99400ce615a5 It's clear that the extra try/except setup aren't free, even if the except is never triggered. My summary of the results is that the try/except setup costs 100-200 nsec, while the rest of the code executed in the frame takes about 600-800 nsec. (Hardware: MacBook Pro with 2.8 GHz Intel Core i7.) Incidentally, the try/except cost has come down greatly from Python 2.7, where it's over a microsecond! I also tried a variation where the bottommost generator doesn't yield a value. The conclusion is about the same -- the try/except version is 150 nsec slower. So now we have a number to worry about (150 nsec for a try/except) and I have to think about whether or not that's likely to have a noticeable effect in realistic situations. One recommendation follows: if you have a loop inside your generator, and there's a next() call in the loop, put the try/except around the loop, so you pay the setup cost only once (unless the loop is most likely to have zero iterations, and unlikely to have more than one). > After thinking about that concern for a while, I'd like to suggest the > idea of having a new builtin "allow_implicit_stop" decorator that > swaps out a GENERATOR code object that has the new "EXPLICIT_STOP" > flag set for one with it cleared (attempting to apply > "allow_implicit_stop" to a normal function would be an error). > > Then the updated version of the above example would become: > > @allow_implicit_stop > def m
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
Can you summarize that in a self-contained form for inclusion in the PEP? (That was a rhetorical question. :-) On Wed, Nov 26, 2014 at 12:17 PM, Isaac Schwabacher wrote: > On 14-11-26, Guido van Rossum wrote: > > On Wed, Nov 26, 2014 at 8:54 AM, Paul Moore wrote: > > > > > On 26 November 2014 at 16:24, Isaac Schwabacher wrote: > > > > This actually leads to a good example of why the PEP is necessary: > > > [...] > > > > > > Oh! If that's the current behaviour, then it probably needs to go into > > > the PEP as a motivating example. It's far more convincing than most of > > > the other arguments I've seen. Just one proviso - is it fixable in > > > contextlib *without* a language change? If so, then it loses a lot of > > > its value. > > > > It's hard to use as an example because the behavior of contextlib is an > integral part of it -- currently for me the example boils down to "there is > a bug in contextlib". Maybe it would have been caught earlier with the > change in the PEP, but when using it as a motivating example you have to > show the code containing the bug, not just a demonstration. > > How is this a bug in contextlib? The example behaves the way it does > because gen.throw(StopIteration) behaves differently depending on whether > gen is paused at a yield or a yield from. What *should* > contextlib.contextmanager do in this instance? It has faithfully forwarded > the StopIteration raised in the protected block to the generator, and the > generator has forwarded this to the subgenerator, which has elected to fail > and report success. The bug is in the subgenerator, because it fails to > treat StopIteration as an error. But the subgenerator can't in general be > converted to treat StopIteration as an error, because clearly it's used in > other places than as a nested context manager (otherwise, it would itself > be decorated with @contextlib.contextmanager and accessed as such, instead > of yielded from). And in those places, perhaps it needs to simply allow > StopIteration to bubble up. And can we factor out the error checking so > that we don't have to duplicate subgenerator? Well... yes, but it's tricky > because we'll introduce an extra yield from in the process, so we have to > put the handling in the subgenerator itself and wrap the > *non*-context-manager uses. > > ijs > > > If you want to try though, I'm happy to entertain a pull request for the > PEP. > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 14-11-26, Guido van Rossum wrote: > On Wed, Nov 26, 2014 at 8:54 AM, Paul Moore wrote: > > > On 26 November 2014 at 16:24, Isaac Schwabacher wrote: > > > This actually leads to a good example of why the PEP is necessary: > > [...] > > > > Oh! If that's the current behaviour, then it probably needs to go into > > the PEP as a motivating example. It's far more convincing than most of > > the other arguments I've seen. Just one proviso - is it fixable in > > contextlib *without* a language change? If so, then it loses a lot of > > its value. > > It's hard to use as an example because the behavior of contextlib is an > integral part of it -- currently for me the example boils down to "there is a > bug in contextlib". Maybe it would have been caught earlier with the change > in the PEP, but when using it as a motivating example you have to show the > code containing the bug, not just a demonstration. How is this a bug in contextlib? The example behaves the way it does because gen.throw(StopIteration) behaves differently depending on whether gen is paused at a yield or a yield from. What *should* contextlib.contextmanager do in this instance? It has faithfully forwarded the StopIteration raised in the protected block to the generator, and the generator has forwarded this to the subgenerator, which has elected to fail and report success. The bug is in the subgenerator, because it fails to treat StopIteration as an error. But the subgenerator can't in general be converted to treat StopIteration as an error, because clearly it's used in other places than as a nested context manager (otherwise, it would itself be decorated with @contextlib.contextmanager and accessed as such, instead of yielded from). And in those places, perhaps it needs to simply allow StopIteration to bubble up. And can we factor out the error checking so that we don't have to duplicate subgenerator? Well... yes, b ut it's tricky because we'll introduce an extra yield from in the process, so we have to put the handling in the subgenerator itself and wrap the *non*-context-manager uses. ijs > If you want to try though, I'm happy to entertain a pull request for the PEP. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On 27 Nov 2014 06:35, "Guido van Rossum" wrote: > > On Wed, Nov 26, 2014 at 3:24 AM, Nick Coghlan wrote: >> After thinking about that concern for a while, I'd like to suggest the >> idea of having a new builtin "allow_implicit_stop" decorator that >> swaps out a GENERATOR code object that has the new "EXPLICIT_STOP" >> flag set for one with it cleared (attempting to apply >> "allow_implicit_stop" to a normal function would be an error). >> >> Then the updated version of the above example would become: >> >> @allow_implicit_stop >> def my_generator(): >> ... >> yield next(it) >> ... >> >> Which would be semantically equivalent to: >> >> def my_generator(): >>try: >>... >> yield next(it) >> ... >> except StopIteration >> return >> >> but *much* faster (especially if used in a producer/consumer pipeline) >> since it would allow a single StopIteration instance to propagate >> through the entire pipeline, rather than creating and destroying new >> ones at each stage. > > > I think we can put a number to "much faster" now -- 150 nsec per try/except. > > I have serious misgivings about that decorator though -- I'm not sure how viable it is to pass a flag from the function object to the execution (which takes the code object, which is immutable) and how other Python implementations would do that. But I'm sure it can be done through sheer willpower. I'd call it the @hettinger decorator in honor of the PEP's most eloquent detractor. :-) I agree with everything you wrote in your reply, so I'll just elaborate a bit on my proposed implementation for the decorator idea. What I was thinking is that to implement the __future__ import, we're going to need a code object flag. For the sake of discussion, let's assume that flag is called "EXPLICIT_STOP". When StopIteration escapes from a generator frame, we'll check for that flag and if it's set, StopIteration will be converted to RuntimeError. The details will differ for other implementations, but they're going to need something at least approximately equivalent to handle the transition period. The implicit stop decorator would then check the flags on the code object attached to the passed in function. If GENERATOR wasn't set, that would be an immediate ValueError, while if EXPLICIT_STOP wasn't set, the generator function would be passed through unmodified. However, if EXPLICIT_STOP *was* set, the generator function would be replaced by a *new* generator function with a *new* code object, where the only change was to clear the EXPLICIT_STOP flag. The core part of the idea is actually keeping the runtime check for the EXPLICIT_STOP flag on generator code objects even *after* setting the flag becomes the default compile time behaviour. The builtin (or itertools?) decorator to revert to the current behaviour on a per-generator basis then becomes a convenient way to access (and document!) that functionality without messing about with your own dynamic code object creation, and without coming up with new syntax for it (since new syntax can't be emulated on older versions of Python, or made implicit in other decorators like contextlib.contextmanager). Regards, Nick. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 27 Nov 2014 03:58, "Paul Moore" wrote: > > On 26 November 2014 at 17:19, Guido van Rossum wrote: > > It's hard to use as an example because the behavior of contextlib is an > > integral part of it -- currently for me the example boils down to "there is > > a bug in contextlib" > > Hmm, fair point. I was assuming that the bug in contextlib can't be > fixed with the current language behaviour (and I'd personally be OK > with the example simply adding a comment "this can't be fixed without > changing Python as proposed in the PEP"). But I'm not sure how true > that is, so maybe it's not quite as compelling as it seemed to me at > first. The "contextlib only" change would be to map StopIteration in the body of the with statement to gen.close() on the underlying generator rather than gen.throw(StopIteration). (That's backwards incompatible in its own way, since it means you *can't* suppress StopIteration via a generator based context manager any more) This is actually the second iteration of this bug: the original implementation *always* suppressed StopIteration. PJE caught that one before Python 2.5 was released, but we didn't notice that 3.3 had brought it back in a new, more subtle form :( It's worth noting that my "allow_implicit_stop" idea in the other thread wouldn't affect subgenerators - those would still convert StopIteration to RuntimeError unless explicitly silenced. Regards, Nick. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
You can use the README here: https://github.com/Rosuav/GenStopIter On Wed, Nov 26, 2014 at 1:57 PM, Isaac Schwabacher wrote: > > Can you summarize that in a self-contained form for inclusion in the PEP? > > > > (That was a rhetorical question. :-) > > Sure. Is it on GitHub? ;D > > ijs > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Wed, Nov 26, 2014 at 3:15 PM, Nick Coghlan wrote: > > On 27 Nov 2014 03:58, "Paul Moore" wrote: > > > > On 26 November 2014 at 17:19, Guido van Rossum wrote: > > > It's hard to use as an example because the behavior of contextlib is an > > > integral part of it -- currently for me the example boils down to > "there is > > > a bug in contextlib" > > > > Hmm, fair point. I was assuming that the bug in contextlib can't be > > fixed with the current language behaviour (and I'd personally be OK > > with the example simply adding a comment "this can't be fixed without > > changing Python as proposed in the PEP"). But I'm not sure how true > > that is, so maybe it's not quite as compelling as it seemed to me at > > first. > > The "contextlib only" change would be to map StopIteration in the body of > the with statement to gen.close() on the underlying generator rather than > gen.throw(StopIteration). (That's backwards incompatible in its own way, > since it means you *can't* suppress StopIteration via a generator based > context manager any more) > > This is actually the second iteration of this bug: the original > implementation *always* suppressed StopIteration. PJE caught that one > before Python 2.5 was released, but we didn't notice that 3.3 had brought > it back in a new, more subtle form :( > > It's worth noting that my "allow_implicit_stop" idea in the other thread > wouldn't affect subgenerators - those would still convert StopIteration to > RuntimeError unless explicitly silenced. > You've lost me in this subthread. Am I right to conclude that the PEP change doesn't cause problems for contextlib(*), but that the PEP change also probably wouldn't have helped diagnose any contextlib bugs? (*) Except perhaps that some old 3rd party copy of contextlib may eventually break if it's not updated. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Thu, Nov 27, 2014 at 9:53 AM, Nick Coghlan wrote: > The implicit stop decorator would then check the flags on the code object > attached to the passed in function. If GENERATOR wasn't set, that would be > an immediate ValueError, while if EXPLICIT_STOP wasn't set, the generator > function would be passed through unmodified. However, if EXPLICIT_STOP *was* > set, the generator function would be replaced by a *new* generator function > with a *new* code object, where the only change was to clear the > EXPLICIT_STOP flag. Is it possible to replace the code object without replacing the function? Imagine if you have multiple decorators, one of which retains a reference to the function and then this one which replaces it - the order of decoration would be critical. OTOH, I don't know that anyone would retain references to __code__. ChrisA ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
> Can you summarize that in a self-contained form for inclusion in the PEP? > > (That was a rhetorical question. :-) Sure. Is it on GitHub? ;D ijs ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
The design just copies the code object with one flag set differently. Code objects are immutable but they can be copied (though the interface to do that is kind of hidden). On Wed, Nov 26, 2014 at 4:03 PM, Chris Angelico wrote: > On Thu, Nov 27, 2014 at 9:53 AM, Nick Coghlan wrote: > > The implicit stop decorator would then check the flags on the code object > > attached to the passed in function. If GENERATOR wasn't set, that would > be > > an immediate ValueError, while if EXPLICIT_STOP wasn't set, the generator > > function would be passed through unmodified. However, if EXPLICIT_STOP > *was* > > set, the generator function would be replaced by a *new* generator > function > > with a *new* code object, where the only change was to clear the > > EXPLICIT_STOP flag. > > Is it possible to replace the code object without replacing the > function? Imagine if you have multiple decorators, one of which > retains a reference to the function and then this one which replaces > it - the order of decoration would be critical. OTOH, I don't know > that anyone would retain references to __code__. > > ChrisA > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Thu, Nov 27, 2014 at 11:33 AM, Guido van Rossum wrote: > The design just copies the code object with one flag set differently. Code > objects are immutable but they can be copied (though the interface to do > that is kind of hidden). Yes, but the proposal as written spoke of replacing the generator *function*, which has broader consequences. If it's simply replacing the __code__ attribute of that function, it ought to be safe, I think? ChrisA ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
No, that was a figure of speech. The proposed decorator returns a new function object that references a new code object. The original function and code object are unchanged. On Wed, Nov 26, 2014 at 4:38 PM, Chris Angelico wrote: > On Thu, Nov 27, 2014 at 11:33 AM, Guido van Rossum > wrote: > > The design just copies the code object with one flag set differently. > Code > > objects are immutable but they can be copied (though the interface to do > > that is kind of hidden). > > Yes, but the proposal as written spoke of replacing the generator > *function*, which has broader consequences. If it's simply replacing > the __code__ attribute of that function, it ought to be safe, I think? > > ChrisA > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Thu, Nov 27, 2014 at 11:50 AM, Guido van Rossum wrote: > No, that was a figure of speech. The proposed decorator returns a new > function object that references a new code object. The original function and > code object are unchanged. Then it has a potentially-confusing interaction with decorators like Flask's app.route(), which return the original function unchanged, but also save a reference to it elsewhere. The order of decoration determines the effect of the @hettinger decorator; there will be two functions around which are almost, but not entirely, identical, and it'd be very easy to not notice that you decorated in the wrong order. ChrisA ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
Well, that's just a general problem with decorator ordering. On Wed, Nov 26, 2014 at 4:57 PM, Chris Angelico wrote: > On Thu, Nov 27, 2014 at 11:50 AM, Guido van Rossum > wrote: > > No, that was a figure of speech. The proposed decorator returns a new > > function object that references a new code object. The original function > and > > code object are unchanged. > > Then it has a potentially-confusing interaction with decorators like > Flask's app.route(), which return the original function unchanged, but > also save a reference to it elsewhere. The order of decoration > determines the effect of the @hettinger decorator; there will be two > functions around which are almost, but not entirely, identical, and > it'd be very easy to not notice that you decorated in the wrong order. > > ChrisA > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Thu, Nov 27, 2014 at 12:01 PM, Guido van Rossum wrote: > Well, that's just a general problem with decorator ordering. Indeed. I was hoping it could be avoided in this instance by just altering __code__ on an existing function, but if that's not possible, we fall back to what is, after all, a known and documented concern. ChrisA ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
A decorator with a side effect *elsewhere* (like the route registrations) is acceptable; one with a side effect *on the decorated function* is questionable, and instead the decorator should behave "functionally", i.e. return a new object instead. On Wed, Nov 26, 2014 at 5:07 PM, Chris Angelico wrote: > On Thu, Nov 27, 2014 at 12:01 PM, Guido van Rossum > wrote: > > Well, that's just a general problem with decorator ordering. > > Indeed. I was hoping it could be avoided in this instance by just > altering __code__ on an existing function, but if that's not possible, > we fall back to what is, after all, a known and documented concern. > > ChrisA > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Thu, Nov 27, 2014 at 12:11 PM, Guido van Rossum wrote: > A decorator with a side effect *elsewhere* (like the route registrations) is > acceptable; one with a side effect *on the decorated function* is > questionable, and instead the decorator should behave "functionally", i.e. > return a new object instead. Okay! I learn something new every day. :) Did not know that... so it's a good thing I don't write decorators. I withdraw the suggestion. ChrisA ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please reconsider PEP 479.
On Wed, Nov 26, 2014 at 2:53 PM, Nick Coghlan wrote: > On 27 Nov 2014 06:35, "Guido van Rossum" wrote: > [...] > > > I think we can put a number to "much faster" now -- 150 nsec per > try/except. > > > > I have serious misgivings about that decorator though -- I'm not sure > how viable it is to pass a flag from the function object to the execution > (which takes the code object, which is immutable) and how other Python > implementations would do that. But I'm sure it can be done through sheer > willpower. I'd call it the @hettinger decorator in honor of the PEP's most > eloquent detractor. :-) > > I agree with everything you wrote in your reply, so I'll just elaborate a > bit on my proposed implementation for the decorator idea. > This remark is ambiguous -- how strongly do you feel that this decorator should be provided? (If so, it should be in the PEP.) (I'm snipping the rest of what you said, since I understand it: the flag on the code object even has a name in the PEP, it's REPLACE_STOPITERATION -- although I could imagine renaming it to GENERATOR_STOP to match the __future__.) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Thu, Nov 27, 2014 at 8:57 AM, Isaac Schwabacher wrote: >> Can you summarize that in a self-contained form for inclusion in the PEP? >> >> (That was a rhetorical question. :-) > > Sure. Is it on GitHub? ;D Thanks Isaac, I've incorporated your edits. https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-0479.txt ChrisA ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Python 3.4.2 MSI fails to install and run on Windows For Devices
Hello, Following recent discussion of keeping Windows matters to the main dev list, I hope this message will be found useful. Please note that this is not a bug report, mostly a FYI. So, I tried to install Python 3.4.2 on an Intel Galileo board shipped with Windows For Devices. Actually, I don't know if that's correct name for this version of Windows, or rather it's Windows On Devices or Windows IoT. Either way it's essentially a headless Windows, with landing page at http://windowsondevices.com/ . So, I couldn't install python-3.4.2.msi, because that Windows comes without msiexec. Trying to install it via a redistributable (WindowsInstaller-KB893803-v2-x86.exe) fails without any error. At which point I started to think that it may be because running any GUI apps there is not supported. I then installed the MSI with wine (I'm using Linux host), uploaded the installed tree to the board via FTP, and also collected any dependent DLLs from the system folder. python.exe exits immediately in start, just as WindowsInstaller-KB893803-v2-x86.exe. Ironically, that board was gotten to test and better support Windows port of MicroPython. It starts up and executes quick manual command fine, and the reason I needed CPython there is to run MicroPython's testsuite which depends on it. As I mentioned, I don't consider this bug in CPython Windows build, it's rather bug in the discussed Windows version, and I'll try to figure out where to report it properly. But somehow I feel that resolving the underlying issue (running msiexec, [arbitrary] other applications) may take a while, and until then, Python is not usable on this board ;-I. So, until this issue is figured out, MicroPython already supports this embedded Windows version better than CPython ;-). -- Best regards, Paul mailto:[email protected] ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
