[Python-Dev] context manager - generator interaction?
The interaction shown below feels like a bug, or at least very much a
trap for the unwary. It's some sort of interaction between a context
manager and a generator, both of which can raise StopIteration. The
code is excised from a real application, so it's a bit artificial
looking. Nevertheless, it represents an entirely natural evolution of
some code (whereby the call to a context manager was added deep within
the code). Since it seems to work with open as the context manager,
but not with my own null context manager defined with
context_manager(), it feels like this is probably a bug in
contextmanager().
I did look in the archives, and it's possible this has been raised
before, but I wasn't clear on that. So apologies in advance if this is
already understood.
This is using the stock distribution of python 2.5 on Linux 2.4.
Comments?
Bob Sidebotham
bash-2.05b$ cat test.py
from __future__ import with_statement
def gen():
return# Should cause stop iteration
yield # Just to make this a generator, for this example
# This works as expected.
try:
gen().next() # Should immediately raise StopIteration
except StopIteration:
print "Part 1: should reach here!"
else:
print "Part 1: shouldn't reach here!"
# Now try the same thing, but with a context manager wrapping the
generator call.
# This also works as expected.
try:
with open("/etc/passwd") as pw:
gen().next()
except StopIteration:
print "Part 2: should reach here!"
else:
print "Part 2: shouldn't reach here!"
# Finally, try this with our own home grown null context manager.
# This does not work as expected.
from contextlib import contextmanager
@contextmanager
def ctxt():
yield
try:
with ctxt():
gen().next()
except StopIteration:
print "Part 3: should reach here!"
else:
print "Part 3: shouldn't reach here!"
bash-2.05b$ python test.py
Part 1: should reach here!
Part 2: should reach here!
Part 3: shouldn't reach here!
bash-2.05b$
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] context manager - generator interaction?
On Thu, Apr 05, 2007, Bob Sidebotham wrote: > > The interaction shown below feels like a bug, or at least very much a > trap for the unwary. Please go ahead and file a bug report on SourceForge -- that way there is a record. -- Aahz ([EMAIL PROTECTED]) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] context manager - generator interaction?
Isn't this violating the rule that a try/except should only enclose
the smallest expression where the exception is expected?
BTW this is probably more appropriate for c.l.py -- even if there's a
change to the language to come out of the discussion, it should be
held there until the use case is well established. But my suspicion is
that you get what you ask for with an overly-wide try block.
--Guido
On 4/5/07, Bob Sidebotham <[EMAIL PROTECTED]> wrote:
> The interaction shown below feels like a bug, or at least very much a
> trap for the unwary. It's some sort of interaction between a context
> manager and a generator, both of which can raise StopIteration. The
> code is excised from a real application, so it's a bit artificial
> looking. Nevertheless, it represents an entirely natural evolution of
> some code (whereby the call to a context manager was added deep within
> the code). Since it seems to work with open as the context manager,
> but not with my own null context manager defined with
> context_manager(), it feels like this is probably a bug in
> contextmanager().
>
> I did look in the archives, and it's possible this has been raised
> before, but I wasn't clear on that. So apologies in advance if this is
> already understood.
>
> This is using the stock distribution of python 2.5 on Linux 2.4.
>
> Comments?
>
> Bob Sidebotham
>
> bash-2.05b$ cat test.py
> from __future__ import with_statement
>
> def gen():
> return# Should cause stop iteration
> yield # Just to make this a generator, for this example
>
> # This works as expected.
> try:
> gen().next() # Should immediately raise StopIteration
> except StopIteration:
> print "Part 1: should reach here!"
> else:
> print "Part 1: shouldn't reach here!"
>
> # Now try the same thing, but with a context manager wrapping the
> generator call.
> # This also works as expected.
> try:
> with open("/etc/passwd") as pw:
> gen().next()
> except StopIteration:
> print "Part 2: should reach here!"
> else:
> print "Part 2: shouldn't reach here!"
>
> # Finally, try this with our own home grown null context manager.
> # This does not work as expected.
> from contextlib import contextmanager
>
> @contextmanager
> def ctxt():
> yield
>
> try:
> with ctxt():
> gen().next()
> except StopIteration:
> print "Part 3: should reach here!"
> else:
> print "Part 3: shouldn't reach here!"
>
> bash-2.05b$ python test.py
> Part 1: should reach here!
> Part 2: should reach here!
> Part 3: shouldn't reach here!
> bash-2.05b$
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] context manager - generator interaction?
At 03:54 PM 4/5/2007 -0400, Bob Sidebotham wrote: >The interaction shown below feels like a bug, or at least very much a >trap for the unwary. It's some sort of interaction between a context >manager and a generator, both of which can raise StopIteration. The >code is excised from a real application, so it's a bit artificial >looking. Nevertheless, it represents an entirely natural evolution of >some code (whereby the call to a context manager was added deep within >the code). Since it seems to work with open as the context manager, >but not with my own null context manager defined with >context_manager(), it feels like this is probably a bug in >contextmanager(). It actually appears to be a quirk in the "with" machinery itself, that only affects generator context managers because of their need to handle StopIteration specially. If you rewrite your context manager as a class with __enter__ and __exit__, the problem will go away. Checking what was happening via pdb, I found that the contextmanager's __exit__ is being called with a "value" of None, rather than with an exception *instance*. This fouls up GeneratorContextManager.__exit__'s special handling for StopIteration -- which was apparently never tested by any unit tests. I can see a few different ways to work around the problem in GeneratorContextManager itself, the simplest of which is to add these two lines right before the self.gen.throw() call: if type is StopIteration and value is None: value = type() It seems to me that this is a bit of a kludge, but trying to make it so that __exit__ always gets called with an instance is much more of a pain since it appears that the None a performance enhancement for simple errors like StopIteration, that avoids creating an instance. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] context manager - generator interaction?
Guido van Rossum wrote: > Isn't this violating the rule that a try/except should only enclose > the smallest expression where the exception is expected? Yeah, and I keep finding myself wanting to hyperlink to that "rule" in the official docs, but it only seems to be written down in developer's heads. Can we get that into the Language Ref somewhere? Maybe on the http://docs.python.org/ref/try.html page? Robert Brewer System Architect Amor Ministries [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] context manager - generator interaction?
"Robert Brewer" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] | Guido van Rossum wrote: | > Isn't this violating the rule that a try/except should only enclose | > the smallest expression where the exception is expected? | | Yeah, and I keep finding myself wanting to hyperlink to that "rule" in | the official docs, but it only seems to be written down in developer's | heads. Can we get that into the Language Ref somewhere? Maybe on the | http://docs.python.org/ref/try.html page? Submit a doc patch (normal text without Latex markup is fine) to SF with the sentence you would like, the location, and reason. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] context manager - generator interaction?
> Guido van Rossum wrote: >> Isn't this violating the rule that a try/except should only enclose >> the smallest expression where the exception is expected? > > Yeah, and I keep finding myself wanting to hyperlink to that "rule" in > the official docs, but it only seems to be written down in developer's > heads. Can we get that into the Language Ref somewhere? Maybe on the > http://docs.python.org/ref/try.html page? I'll add a snippet to the tutorial with advice on try/except. I don't think it should go in the main documentation because it is really more of a suggestion or technique than an actual rule. Also, there are plenty of other situations where the "rule" doesn't apply. Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
