Bugs item #1153622, was opened at 2005-02-28 11:48
Message generated for change (Comment added) made by tjreedy
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1153622&group_id=5470
Category: Documentation
Group: Python 2.4
Status: Open
Resolution: None
Priority: 5
Submitted By: Mattias Engdegård (yorick)
Assigned to: Jeremy Hylton (jhylton)
Summary: eval does not bind variables in lambda bodies correctly
Initial Comment:
eval() does not bind variables in lambda expressions
correctly:
>>>def f(g): return eval('lambda x: g(x)')
>>>f(lambda y: y * 2)(17)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<string>", line 1, in <lambda>
NameError: global name 'g' is not defined
The docs say this about eval():
# If both dictionaries are omitted, the expression is
# executed in the environment where eval is called.
and using plain local variables work as expected:
>>>def h(d): return eval('d(10)')
>>>h(lambda y: y * 2)
20
Also, if locals() is presented as the global dict to
eval(), it works:
>>>def f(g): return eval('lambda x: g(x)', locals(),
locals())
>>>f(lambda y: y * 2)(17)
34
but this does not allow the expression to reference
global variables of course.
----------------------------------------------------------------------
Comment By: Terry J. Reedy (tjreedy)
Date: 2005-03-03 22:46
Message:
Logged In: YES
user_id=593130
" def f(x): eval('x') works as expected and
def f(g): eval('lambda x: g(x)') does not. Why?"
For the same reason
def f(g,s): return eval(s)
f(lambda x: x, 'lambda x: g(x)')(1)
and
def g(x): return lambda: eval('x')
do not 'work'. eval is a builtin C function whose behavior
depends on its arguments (including the somewhat magical
defaulting to globals(), local()) and not on its lexical position.
" Both are evaluated at the same time and use
the same environment to look up their variables."
No, the lambda in your second example introduces a new local
namespace that is different from the one passed in.
" The fact that the 'g' variable in the second case is not evaluated
immediately does not affect its scoping"
The delay and change of scoping are correlated. Evaluation is
delayed because g is inside a lambda function def which
introduces a new local scope which does not contain g, even
though the original one did.
----------------------------------------------------------------------
Comment By: Terry J. Reedy (tjreedy)
Date: 2005-03-03 21:55
Message:
Logged In: YES
user_id=593130
After more carefully reading the eval doc in Lib Ref 2.1, I agree
that it needs improvement. My suggestions (assuming that my
experiment-based inferences are correct):
In
"The expression argument is parsed and evaluated as a Python
expression (technically speaking, a condition list) using the
globals and locals dictionaries as global and local name space."
insert "in a new top-level environment" before 'using'. This says
right-off, even if indirectly, that lexical scoping does not work
across the eval call.
In
"If the locals dictionary is omitted it defaults to the globals
dictionary."
insert "only" after "If'. If both are omitted, it does not so default.
Add a comma after 'omitted', as in the next sentence.
In
"If both dictionaries are omitted, the expression is executed in
the environment where eval is called."
replace "the expression ... is called", which I believe to be
incorrect as well as misleading, with something like "they default
to current globals() and locals()." This parallels the previous
sentence and is, I believe, more correct. Within a function,
locals() is not the current local namespace, and the following
shows that the difference can make a visible difference:
>>> def g1(): return op.setitem(locals(), 'x', 1), x
...
>>> g1()
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 1, in g1
NameError: global name 'x' is not defined
>>> def h1(): return eval("op.setitem(locals(), 'x', 1), x")
...
>>> h1()
(None, 1)
After
"The return value is the result of the evaluated expression. "
add something like
It does not depend on the lexical position of the eval call and
hence the expression should not contain names that require
lexical scoping reaching outside the eval call to be valid."
Note that eval scope blocking, the OP's pseudobug, does not
require a lambda within the eval-ed expression:
>>> def f(x): return lambda: x
...
>>> f(1)()
1
>>> def g(x): return lambda: eval('x')
...
>>> g(1)()
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 1, in <lambda>
File "<string>", line 0, in ?
NameError: name 'x' is not defined
This might be another example for the PEP.
----------------------------------------------------------------------
Comment By: Raymond Hettinger (rhettinger)
Date: 2005-03-03 10:29
Message:
Logged In: YES
user_id=80475
Jeremy, would you update the pep and docs to cover the OP's
examples.
----------------------------------------------------------------------
Comment By: Mattias Engdegård (yorick)
Date: 2005-03-03 06:16
Message:
Logged In: YES
user_id=432579
>No, this issue is not specific to either eval or lambda:
Right, so let me rephrase: The bug occurs when explicitly
evaluating a lambda expression or function definition
statement using eval or exec. (This is an artifact of
Python's strong separation of statements and expressions.)
If this is done "by design", why cannot I find anything
anywhere describing this? If this is just a documentation
oversight, please say so, but then I would also like to have
an explanation of the behaviour.
The fact remains that
def f(x): eval('x')
works as expected and
def f(g): eval('lambda x: g(x)')
does not. Why? Both are evaluated at the same time and use
the same environment to look up their variables. The fact
that the 'g' variable in the second case is not evaluated
immediately does not affect its scoping, because that is how
lambda expressions work.
>If you want Python to work
>differently, write a PEP or a patch, or raise the question in
>the newsgroup/mailing list.
www.python.org told me that this is the place to report bugs
in Python. If that is wrong, we should change the web site.
----------------------------------------------------------------------
Comment By: Terry J. Reedy (tjreedy)
Date: 2005-03-02 17:59
Message:
Logged In: YES
user_id=593130
No, this issue is not specific to either eval or lambda:
>>> def f(g):
... exec 'def h(x): return g(x)'
... return h
...
>>> f(lambda y: y * 2)(17)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<string>", line 1, in h
NameError: global name 'g' is not defined
It is specific to creating a function at top-level in a separate
execution environment, as done, by design, by both eval and
exec, and with either a def statement or lambda abbreviation
thereof.
In Python, lexical scoping works because nested functions are
compiled along with the outer function, so that scoped variables
can be identified and both functions adjusted so that the coupling
works. In particular, the scoped variable has to not be deleted
when the outer function returns. Eval/exec compile their string
only later, when the function is called.
"it works that way because it is the way it works".
Those are your words, not mine. If you want Python to work
differently, write a PEP or a patch, or raise the question in the
newsgroup/mailing list. I'm done discussing it here.
----------------------------------------------------------------------
Comment By: Mattias Engdegård (yorick)
Date: 2005-03-02 11:42
Message:
Logged In: YES
user_id=432579
No, this issue is specific to eval of lambda expressions.
Please read my problem description. Please refer to the
Python documentation if you are confused with how standard
function declaration or lexical scoping works.
----------------------------------------------------------------------
Comment By: Branko (bbange)
Date: 2005-03-01 19:20
Message:
Logged In: YES
user_id=1230541
Obviously I forgot a statement in my previous comment's
code example. So here's the complete version:
n=2
def f(x): return n*x
del n
f(2)
# the Python implementation will result in a name error here.
But what should happen if Python had bound variable n at the
time of f's definitionf?
# let's define n again
n=3
f(2)
# the implementation will return 6, but how about your
expected implementation?
----------------------------------------------------------------------
Comment By: Branko (bbange)
Date: 2005-03-01 19:15
Message:
Logged In: YES
user_id=1230541
I think this issue is not special for eval and can be also
reproduced with a def statement. The point is that at function
definition time Python does not do any variable binding
concerning variables not local to the function. Instead Python
looks for that variable in the namespace of the module in
which the function was created at the time the function is
executed. Python determines that module by evaluating the
variable __module__ at function definition time and
remembers it by setting the function attribute with the same
name. That's why only the variable __module__ is relevant at
function definition time. Simply put, Python does only do a
module level variable binding at function definition time. This
is simple and sensible. If you don't agree consider this:
n=2
def f(x): return n*x
del n
f(2)
# the Python implementation will result in a name error here.
But what should happen if Python had bound variable n at the
time of f's definitionf?
# let's define n again
f(2)
# the implementation will return 6, but how about your
expected implementation?
As you see, changing the implementation would either make
Pythons semantics more complicated or would remove much
of Pythons dynanism.
----------------------------------------------------------------------
Comment By: Mattias Engdegård (yorick)
Date: 2005-03-01 13:26
Message:
Logged In: YES
user_id=432579
What you are saying is "it works that way because it is the
way it works". I see no reason at all for this odd behaviour
other than bug-compatibility. I find nothing at all in the
documentation supporting this behaviour either; please
inform me if I have missed something.
All other languages supporting eval and lexical scoping
(Lisp, Scheme, Perl, Ruby, etc) work in the expected way. I
have no problems if Python wants to be different for
whatever reason, but it should be documented.
I did a quick Google in comp.lang.python but could not find
anything that supported this "exception" or gave a rational
explanation. Kindly direct me to any resource you know of
that could help enlighten me on this issue.
># From your comments, I suspect you expect 0.
Of course not. I know very well how lexical scoping works,
so please don't put words in my mouth.
None of your examples have anything to do with scoping. As
we both know, it is not the _values_ of the variables that
is important for variable binding, it is their identity;
which variable is chosen, not what they happen to contain at
the time the lambda expression is evaluated.
----------------------------------------------------------------------
Comment By: Terry J. Reedy (tjreedy)
Date: 2005-03-01 12:29
Message:
Logged In: YES
user_id=593130
Whoops. eval('x') == x as code snippets has an exception,
which is the one tripping you up. When the eval is within a
function definition (and lambda expressions are abbreviated
simple function definitions) and 'x' contains a function definition,
then the body of the contained function definition does not have
access, when it is run, to the locals of the containing function
(the lexical scope), whereas it will when x is compiled directly *as
part of the containing function body*. eval('x') removes x from
that part of its context. eval only has the simple two-level
globals/locals environment, which can be anything the caller
passes in, so it compiles x as if it were top-level code. Hence
free variables in contained functions are looked up in the global
passed to eval when the evaled function is called.
This issue has been discussed on the Python newsgroup/mailing
list more than once. If my explanation is not clear, you might be
able to find others in Google c.l.p archives. Do consider that
core functions which have been debugged for over a decade are
unlike to have many bugs left, although the docs are still being
improved.
While Python's scoping is lexical, its free variable binding is late.
Consider
>>> def f():
... x = 0
... def g(): print x
... x = 1
... return g
...
>>> f()()
# What gets printed? 0 or 1?
# From your comments, I suspect you expect 0.
# Irregardless, it is
1
Similarly
>>> f()()
1
>>> d={'x': 0}
>>> h=eval('lambda: x', d, d)
>>> h()
0
>>> d['x'] = 1
>>> h()
# now what gets printed?
1
----------------------------------------------------------------------
Comment By: Mattias Engdegård (yorick)
Date: 2005-03-01 04:11
Message:
Logged In: YES
user_id=432579
>Variables in Python functions are resolved
>when the function is *called*, not when it is defined.
I'm not sure what you mean by that, since Python obeys
lexical scoping, not dynamic.Consider:
def f(x): lambda y: x + y
When the inner lambda expression above is evaluated, x
inside the lambda body is bound to the parameter of the call
of f, even if x+y is not evaluated until that function is
called.
So since
def f(x): return eval('x')
fetches its definition of x from the lexical variable x, why
shouldn't
def f(g): return eval('lambda x: g(x)')
fetch its definition of g from the lexical variable g? A
lambda expression is just a way of delaying evaluation,
*not* delaying how variables are bound --- this is done
immediately.
----------------------------------------------------------------------
Comment By: Terry J. Reedy (tjreedy)
Date: 2005-03-01 00:30
Message:
Logged In: YES
user_id=593130
I am 99.5% sure that this is 'invalid' (a Resolution category) and
should be closed. With the default environment, eval('x') is the
same as unquoted x. Variables in Python functions are resolved
when the function is *called*, not when it is defined. There is no
resolution for g in the default globals. Eval does not change this.
The NameError is exactly correct.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1153622&group_id=5470
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
