Re: [Python-Dev] variable name resolution in exec is incorrect
Perhaps the next step is to re-open the issue? If it is seen as a bug,
it would be great to see a fix in 2.6+ - a number of options which
will not break backward compatibility have been put forward - cheers,
Colin
On Thu, May 27, 2010 at 9:05 PM, Reid Kleckner wrote:
> On Thu, May 27, 2010 at 11:42 AM, Nick Coghlan wrote:
>> However, attaining the (sensible) behaviour Colin is requesting when such
>> top level variable references exist would actually be somewhat tricky.
>> Considering Guido's suggestion to treat two argument exec like a function
>> rather than a class and generate a closure with full lexical scoping a
>> little further, I don't believe this could be done in exec itself without
>> breaking code that expects the current behaviour.
>
> Just to give a concrete example, here is code that would break if exec
> were to execute code in a function scope instead of a class scope:
>
> exec """
> def len(xs):
> return -1
> def foo():
> return len([])
> print foo()
> """ in globals(), {}
>
> Currently, the call to 'len' inside 'foo' skips the outer scope
> (because it's a class scope) and goes straight to globals and
> builtins. If it were switched to a local scope, a cell would be
> created for the broken definition of 'len', and the call would resolve
> to it.
>
> Honestly, to me, the fact that the above code ever worked (ie prints
> "0", not "-1") seems like a bug, so I wouldn't worry about backwards
> compatibility.
>
> Reid
>
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bugfix releases should not change APIs
On Fri, 28 May 2010 20:31:02 -0400 Steve Holden wrote: > I think it shows how developers can "get worked over" if they are > insufficiently vigilant. > > 1) I completely agree, and adduce as evidence the fact that something > like this always seems to happen when the rule is broken; Well it's true that the change may have been unfortunate (this is the second bug report we get about it, the first one was from Holger Krekel IIRC), its exceptional nature had also been discussed on this mailing-list, and supported by Guido. It is not the product of oversight. What it does teach us is that Python 3.1 sees some real use, and we have entered a phase where backwards compatibility will become as important as it was in the 2.x line. > 2) But we may wish to release 3.1.2.1(?) which backports fixes from the > 3.2 line but retains the file store semantics (which I am assured will > be easy in the glorious reign of Hg). I think this would be worse, as in "even more confusing". We would have a 3.1.2 with changed behaviour, a 3.1.2.1 with reverted behaviour, and a 3.2 with changed behaviour again. Now that we have inflicted this pain on our users, let's not inflict more pain on them. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
On 29/05/10 20:20, Colin H wrote: Perhaps the next step is to re-open the issue? If it is seen as a bug, it would be great to see a fix in 2.6+ - a number of options which will not break backward compatibility have been put forward - cheers, A new feature request requesting a "closure" mode for compile() in 3.2 would probably be the best way forward. Once that is done, then the question of if or when to change the default behaviour for auto-compiled code in exec and/or dis can be considered. It definitely isn't a bug fix though - it's worked this way for years, and while the existing semantics can certainly be surprising, they're far from being buggy (as Thomas said, prior to the introduction of lexical scoping all Python namespaces worked this way). Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On May 28, 2010, at 11:31 PM, Nick Coghlan wrote: On 29/05/10 10:19, Jesse Noller wrote: In my opinion, it is high time for the std lib to pay more attention to the final Zen: Namespaces are one honking great idea -- let's do more of those! Yes, your suggestion for how to move things is the way we would want to do it, and in the back of my head, what we should do long term - just not right now. Yep, this is what I have been saying as well. 1. Using concurrency.futures rather than a top level futures module resolves the potential confusion with __future__ and stock market futures without inventing our own name for a well established computer science concept. 2. With the concurrency package in place following PEP 3148, we can separately consider the question of if/when/how to move other concurrency related modules (e.g. threading, multiprocessing, Queue) into that package at a later date. Since this topic keeps coming up, some reasoning along these lines should go into PEP 3148. I'll type something up this weekend and shoot it to Brian for inclusion. I was hoping to be able to keep it out of the futures pep itself, but it seems that won't work :) Jesse ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 29/05/10 22:46, Jesse Noller wrote: On May 28, 2010, at 11:31 PM, Nick Coghlan wrote: Since this topic keeps coming up, some reasoning along these lines should go into PEP 3148. I'll type something up this weekend and shoot it to Brian for inclusion. I was hoping to be able to keep it out of the futures pep itself, but it seems that won't work :) Well, punting on whether or not we actually *do* part 2 is still fine. As Eric pointed out, there are issues with unpickling that make the wisdom of following through with renaming any existing modules fairly questionable. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Implementing PEP 382, Namespace Packages
Last night Barry Warsaw, Jason Coombs, and I met to work on implementing PEP 382. As part of my research, I came across this email from Martin: http://mail.python.org/pipermail/python-dev/2009-May/089316.html In it he says that PEP 382 is being deferred until it can address PEP 302 loaders. I can't find any follow-up to this. I don't see any discussion in PEP 382 about PEP 302 loaders, so I assume this issue was never resolved. Does it need to be before PEP 382 is implemented? Are we wasting our time by designing and (eventually) coding before this issue is resolved? Eric. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Implementing PEP 382, Namespace Packages
In it he says that PEP 382 is being deferred until it can address PEP 302 loaders. I can't find any follow-up to this. I don't see any discussion in PEP 382 about PEP 302 loaders, so I assume this issue was never resolved. Does it need to be before PEP 382 is implemented? Are we wasting our time by designing and (eventually) coding before this issue is resolved? Yes, and yes. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Implementing PEP 382, Namespace Packages
Am 29.05.2010 21:06, schrieb P.J. Eby: At 08:45 PM 5/29/2010 +0200, Martin v. Löwis wrote: In it he says that PEP 382 is being deferred until it can address PEP 302 loaders. I can't find any follow-up to this. I don't see any discussion in PEP 382 about PEP 302 loaders, so I assume this issue was never resolved. Does it need to be before PEP 382 is implemented? Are we wasting our time by designing and (eventually) coding before this issue is resolved? Yes, and yes. Is there anything we can do to help regarding that? You could comment on the proposal I made back then, or propose a different solution. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On Fri, May 28, 2010 at 17:12, Steven D'Aprano wrote: > On Sat, 29 May 2010 08:28:46 am Vinay Sajip wrote: > >> I've not seen this mentioned, but on such a long thread I might have >> missed it: we already have a "__future__" module, as in >> >> from __future__ import with_statement >> >> and to my mind, this is a potential point of confusion with the term >> "future". > [...] >> I'm not sure of the benefit of a "concurrent" namespace, since it >> wouldn't contain the existing concurrency stuff. I think it's >> something to consider only for a big reorg which would break backward >> compatibility. IMO it would make more sense to leave this module as a >> top-level module for now (a sibling to "threading", >> "multiprocessing"). > > I have suggested a way to move the existing concurrency stuff without > breaking backwards compatibility, and Terry Reedy asked if it would > work. I haven't seen any responses, either positive or negative. > > For the record, my suggestion was: > > for each concurrency modules: > move it into the concurrency package > add a top level module with the same name containing: > # e.g. for threading > from concurrency.threading import * > > Then in some future Python version, each top level module gets a > PendingDeprecation warning, followed by a Deprecation warning some time > later, and eventually in the indefinite future removal of the top level > module. This was the procedure we used for about a month for Python 2.6 in order to help renamed modules migrate to their new names in Python 3. The issue that came up (and forced use to revert this approach and fully rely on 2to3) was anything pickled by the older interpreters is not going to be happy with that shift. Luckily the stuff being moved most likely does not contain things that have been pickled and stored to disk for ages and thus would break in a transition. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Implementing PEP 382, Namespace Packages
On Sat, May 29, 2010 at 12:29, "Martin v. Löwis" wrote:
> Am 29.05.2010 21:06, schrieb P.J. Eby:
>>
>> At 08:45 PM 5/29/2010 +0200, Martin v. Löwis wrote:
In it he says that PEP 382 is being deferred until it can address PEP
302 loaders. I can't find any follow-up to this. I don't see any
discussion in PEP 382 about PEP 302 loaders, so I assume this issue was
never resolved. Does it need to be before PEP 382 is implemented? Are we
wasting our time by designing and (eventually) coding before this issue
is resolved?
>>>
>>> Yes, and yes.
>>
>> Is there anything we can do to help regarding that?
>
> You could comment on the proposal I made back then, or propose a different
> solution.
[sorry for the fundamental PEP questions, but I think PEP 382 came
about while I was on my python-dev sabbatical last year]
I have some questions about the PEP which might help clarify how to
handle the API changes.
For finders, their search algorithm is changed in a couple of ways.
One is that modules are given priority over packages (is that
intentional, Martin, or just an oversight?). Two, the package search
requires checking for a .pth file on top of an __init__.py. This will
change finders that could before simply do an existence check on an
__init__ "file" (or whatever the storage back-end happened to be) and
make it into a list-and-search which one would hope wasn't costly, but
in same cases might be if the paths to files is not stored in a
hierarchical fashion (e.g. zip files list entire files paths in their
TOC or a sqlite3 DB which uses a path for keys will have to list
**all** keys, sort them to just the relevant directory, and then look
for .pth or some such approach). Are we worried about possible
performance implications of this search? I say no, but I just want to
make sure people we are not and people are aware about the design
shift required in finders. This entire worry would be alleviated if
only .pth files named after the package were supported, much like
*.pkg files in pkgutil.
And then the search for the __init__.py begins on the newly modified
__path__, which I assume ends with the first __init__ found on
__path__, but if no file is found it's okay and essentially an empty
module with just module-specific attributes is used? In other words,
can a .pth file replace an __init__ file in delineating a package? Or
is it purely additive? I assume the latter for compatibility reasons,
but the PEP says "a directory is considered a package if it **either**
contains a file named __init__.py, **or** a file whose name ends with
".pth"" (emphasis mine). Otherwise I assume that the search will be
done simply with ``os.path.isdir(os.path.join(sys_path_entry,
top_level_package_name)`` and all existing paths will be added to
__path__. Will they come before or after the directory where the *.pth
was found? And will any subsequent *.pth files found in other
directories also be executed?
As for how "*" works, is this limited to top-level packages, or will
sub-packages participate as well? I assume the former, but it is not
directly stated in the PEP. If the latter, is a dotted package name
changed to ``os.sep.join(sy_path_entry, package_name.replace('".",
os.sep)``?
For sys.path_hooks, I am assuming import will simply skip over passing
that as it is a marker that __path__ represents a namsepace package
and not in any way functional. Although with sys.namespace_packages,
is leaving the "*" in __path__ truly necessary?
For the search of paths to use to extend, are we limiting ourselves to
actual file system entries on sys.path (as pkgutil does), or do we
want to support other storage back-ends? To do the latter I would
suggest having a successful path discovery be when a finder can be
created for the hypothetical directory from sys.path_hooks.
OK, I *think* that's all of my clarification questions when it comes
to the PEP. =) Now, on to API discussion.
The PEP (seems to) ask finders to look for a .pth file(s), calculate
__path__, and then get a loader for the __init__. You could have
finders grow a find_namespace method which returns the contents of the
requisite .pth file(s). Import could then take that, calculate
__path__, and then use that new search path to find a loader for the
__init__ (I am assuming there is an __init__ file somewhere). That's
straight-forward and makes supporting .pth files additive for finders.
The trick then becomes how the heck you get the new __path__ value
into the module through the loader as up to this point it has
calculated __path__ on its own. You could slightly abuse load_module's
semantics for reloading and stick the namespace module into
sys.modules before calling the loader for __init__ and change the
semantics definition such that if __path__ is already defined you
don't change it. Unfortunately that seems rather messy in the face of
reloads that want a fresh __path__.
Another possibility is to have the loader add the new paths, but to
provid
Re: [Python-Dev] Implementing PEP 382, Namespace Packages
At 09:29 PM 5/29/2010 +0200, Martin v. Löwis wrote: Am 29.05.2010 21:06, schrieb P.J. Eby: At 08:45 PM 5/29/2010 +0200, Martin v. Löwis wrote: In it he says that PEP 382 is being deferred until it can address PEP 302 loaders. I can't find any follow-up to this. I don't see any discussion in PEP 382 about PEP 302 loaders, so I assume this issue was never resolved. Does it need to be before PEP 382 is implemented? Are we wasting our time by designing and (eventually) coding before this issue is resolved? Yes, and yes. Is there anything we can do to help regarding that? You could comment on the proposal I made back then, or propose a different solution. Looking at that proposal, I don't follow how changing *loaders* (vs. importers) would help. If an importer's find_module doesn't natively support PEP 382, then there's no way to get a loader for the package in the first place. Today, namespace packages work fine with PEP 302 loaders, because the namespace-ness is really only about setting up the __path__, and detecting that you need to do this in the first place. In the PEP 302 scheme, then, it's either importers that have to change, or the process that invokes them. Being able to ask an importer the equivalents of os.path.join, listdir, and get_data would suffice to make an import process that could do the trick. Essentially, you'd ask each importer to first attempt to find the module, and then asking it (or the loader, if the find worked) whether packagename/*.pth exists, and then processing their contents. I don't think there's a need to have a special method for executing a package __init__, since what you'd do in the case where there are .pth but no __init__, is to simply continue the search to the end of sys.path (or the parent package __path__), and *then* create the module with an appropriate __path__. If at any point the find_module() call succeeds, then subsequent importers will just be asked for .pth files, which can then be processed into the __path__ of the now-loaded module. IOW, something like this (very rough draft): pth_contents = [] module = None for pathitem in syspath_or_parent__path__: importer = pkgutil.get_importer(pathitem) if importer is None: continue if module is None: try: loader = importer.find_module(fullname) except ImportError: pass else: # errors here should propagate module = loader.load_module(fullname) if not hasattr(module, '__path__'): # found, but not a package return module pc = get_pth_contents(importer) if pc is not None: subpath = os.path.join(pathitem, modulebasename) pth_contents.append(subpath) pth_contents.extend(pc) if '*' not in pth_contents: # got a package, but not a namespace break if pth_contents: if module is None: # No __init__, but we have paths, so make an empty package module = # new module object w/empty __path__ modify__path__(module, pth_contents) return module Obviously, the details are all in the 'get_pth_contents()', and 'modify__path__()' functions, and the above process would do extra work in the case where an individual importer implements PEP 382 on its own (although why would it?). It's also the case that this algorithm will be slow to fail imports when implemented as a meta_path hook, since it will be doing an extra pass over sys.path or the parent __path__, in addition to the one that's done by the normal __import__ machinery. (Though that's not an issue for Python 3.x, since this can be built into the core __import__). (Technically, the 3.x version should probably ask meta_path hooks for their .pth files as well, but I'm not entirely sure that that's a meaningful thing to ask.) The PEP 302 questions all boil down to how get_pth_contents() is implemented, and whether 'subpath' really should be created with os.path.join. Simply adding a get_pth_contents() method to the importer protocol (that returns None or a list of lines), and maybe a get_subpath(modulename) method that returns the path string that should be used for a subdirectory importer (i.e. __path__ entry), or None if no such subpath exists. Adding finer-grained methods is probably a waste of time, as there aren't likely to be many use cases for asking an *importer* to fetch files (vs. a loader). (In my case, of course, I'd use the pkgutil-style approach of augmenting importers or loaders that don't natively implement a needed method, that still allows third parties to register their own support for a fourth party's loader or importer type.) ___ Python-Dev mailing list [email protected] http://mail.python.org/ma
