Bugs item #1186195, was opened at 2005-04-19 12:02
Message generated for change (Comment added) made by bcannon
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1186195&group_id=5470
Category: Parser/Compiler
Group: AST
Status: Open
Resolution: None
Priority: 5
Submitted By: Brett Cannon (bcannon)
Assigned to: Nick Coghlan (ncoghlan)
Summary: [AST] genexps get scoping wrong
Initial Comment:
test_genexps is failing because it is unable to find a
global defined in a genexp that is returned. Here is
the problem simplified:
def f(n): return (i for i in xrange(n))
list(f(10))
Leads to ``SystemError: no locals when loading 'xrange'``.
Comparing Python 2.4 bytecode:
1 0 LOAD_CONST 1 (<code
object <generator expression> at 0x3931e0, file
"<stdin>", line 1>)
3 MAKE_FUNCTION 0
6 LOAD_GLOBAL 0 (xrange)
9 LOAD_FAST 0 (n)
12 CALL_FUNCTION 1
15 GET_ITER
16 CALL_FUNCTION 1
19 RETURN_VALUE
20 LOAD_CONST 0 (None)
23 RETURN_VALUE
to AST bytecode:
1 0 LOAD_CLOSURE 0 (n)
3 BUILD_TUPLE 1
6 LOAD_CONST 1 (<code
object <generator expression> at 0x5212e8, file
"<stdin>", line 1>)
9 MAKE_CLOSURE 0
12 LOAD_NAME 0 (xrange)
15 LOAD_DEREF 0 (n)
18 CALL_FUNCTION 1
21 GET_ITER
22 CALL_FUNCTION 1
25 RETURN_VALUE
26 LOAD_CONST 0 (None)
29 RETURN_VALUE
makes it obvious something is off (no peepholer; turned
it off in my build of 2.4).
Looks like extraneous work is being done in making a
closure. Seems like it will still work, though.
Plus the usage of LOAD_NAME is wrong in the AST;
LOAD_NAME gets an object from the local namespace based
on its name instead of offset.
----------------------------------------------------------------------
>Comment By: Brett Cannon (bcannon)
Date: 2005-04-20 23:13
Message:
Logged In: YES
user_id=357491
OK, so basically what I figured out earlier today. I
noticed that both genexps in 2.4 and the AST branch
generated the same bytecode, so the only difference was how
it was handling passing in arguments to the generator for
its initial creation. I figured there was some jiggering in
the compiler to note that the scoping in the genexp should
not affect how variables were handled for its arguments.
And as you said, the AST branch just doesn't take this into
consideration as it should.
I will see if I get to it, but I doubt I will (leaving town
Friday for the weekend). I think this has justified
fleshing out Python/compile.txt with a more thorough
discussion of how the symbol table is generated. If you
have any specific points you think should go in there, Nick
(or anyone for that matter), please email them to me. Even
explaining how the whole thing works would be a help. =)
----------------------------------------------------------------------
Comment By: Nick Coghlan (ncoghlan)
Date: 2005-04-20 15:13
Message:
Logged In: YES
user_id=1038590
The compiler stage does some fancy footwork so that the
outermost iterable is evaluated immediately, and passed as
an argument to the genexp block. The symbol table generation
needs similar footwork, but it doesn't have it.
That is, the symbol table for the outermost iterable is
being created in the genexp scope, instead of in the
containing function's scope, but the actual execution is
happening in the function scope. Hilarity ensues. . .
My original testing didn't pick this up because I was using
(1, 2, 3) as the outermost iterable - no name lookups in the
outermost iterable to be affected!
If you haven't figured out a fix for this part of the
problem before Sunday, I should be able to get to it then.
The location of the enter_scope and exit_scope in the symbol
table pass for genexps needs adjusting so that the outermost
iterable is NOT part of the new scope.
----------------------------------------------------------------------
Comment By: Brett Cannon (bcannon)
Date: 2005-04-19 17:30
Message:
Logged In: YES
user_id=357491
OK, figured out why the closure thing is happening. 'n' is
being detected as a free variable. This leads to
PyCode_GetNumFree() to return a non-0 value. In
compiler_make_closure() this automatically triggers the
LOAD_CLOSURE/.../MAKE_CLOSURE chunk of bytecode instead of
LOAD_CONST/MAKE_FUNCTION.
So, how to make 'n' be detected as a local instead a free
variable ...
----------------------------------------------------------------------
Comment By: Brett Cannon (bcannon)
Date: 2005-04-19 17:09
Message:
Logged In: YES
user_id=357491
Some playing with gdb has turned up some clues. So
LOAD_NAME is emitted by compiler_nameop(). How this
partially works is that it gets the scope for an argument
and then based on that emits the proper load, store, or delete.
So why is xrange() coming out at the NAME scope? Well,
turns out it is not being found in a particular scope by
PyST_GetScope() and is thus returning 0 by default. This
means that NAME becomes the scope for xrange().
So it looks like the symtable creation is screwing up and
not making xrange() a global like it should. This might be
a side-effect of the while closure thing above. Not sure,
though.
----------------------------------------------------------------------
Comment By: Brett Cannon (bcannon)
Date: 2005-04-19 12:03
Message:
Logged In: YES
user_id=357491
Initially assigned to Nick since he did the genexp patch and
in hopes he might know what is going on off the top of his
head. Otherwise assign to me.
I have a sneaking suspicion that the symtable code overall
is slightly busted.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1186195&group_id=5470
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
