Re: [Python-Dev] Remove typing from the stdlib
On 4 November 2017 at 03:53, Nick Coghlan wrote: > If I understand correctly, a lot of the complexity in the current > typing.py implementation is there to make isinstance and issubclass do > something "useful" at runtime, and to allow generics to be used as > base classes. > > If it wasn't for those design goals, then "typing.List[int]" could > just return a lightweight instance of a regular class rather than a > usable Python class definition. +1 to this. > If I'm right about that, then PEP 560's proposal to allow types to > implement a hook that says "Replace me with this other object for > runtime subclassing purposes" may be enough to let you delete most of > the current code in the typing module - you'd just need to have > isinstance and issubclass respect that new hook as well, and define > the hooks as returning the relevant ABCs. That would seem ideal to me. Lukasz Langa said: > So, the difference is in perceived usability. It's psychological. Please, let's not start the "not in the stdlib isn't an issue" debate again. If I concede it's a psychological issue, will you concede that the fact that it's psychological doesn't mean that it's not a real, difficult to solve, problem for some people? I'm also willing to concede that it's a *minority* problem, if that helps. But can we stop dismissing it as a non-existent problem? Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations
On 4 November 2017 at 00:40, Guido van Rossum wrote: > IMO the inability of referencing class-level definitions from annotations on > methods pretty much kills this idea. If we decided we wanted to make it work, I think the key runtime building block we would need is a new kind of cell reference: an IndirectAttributeCell. Those would present the same interface as a regular nonlocal cell (so it could be stored in __closure__ just as regular cells are, and accessed the same way when the function body is executed), but internally it would hold two references: - one to another cell object (__class__ for this use case) - an attribute name on the target object that get/set/del operations on the indirect cell's value should affect As Python code: class IndirectAttributeCell: def __new__(cls, cell, attr): self._cell = cell self._attr = attr @property def cell_contents(self): return getattr(self._cell.cell_contents, self._attr) @cell_contents.setter def cell_contents(self, value): setattr(self._cell.cell_contents, self._attr, value) @cell_contents.deleter def cell_contents(self): delattr(self._cell.cell_contents, self._attr) The class body wouldn't be able to evaluate the thunks (since `__class__` wouldn't be set yet), but `__init_subclass__` implementations could, as could class decorators. It would require some adjustment in the compiler as well (in order to pass the class level attribute definitions down to these implicitly defined scopes as a new kind of accessible external namespace during the symbol analysis pass, as well as to register the use of "__class__" if one of the affected names was referenced), but I think it would work at least at a technical level (by contrast, every other idea I came up with back when I was working on the list comprehension change was sufficiently flawed that it fell apart within a few hours of starting to tinker with the idea). As an added bonus, we could potentially also extend the same permissive name resolution semantics to the implicit scopes used in comprehensions, such that it was only the explicitly defined scopes (i.e. lambda expressions, function definitions, and nested classes) that lost implicit access to the class level variables. Cheers, Nick. P.S. If we subsequently decided to elevate expression thunks to a first class language primitive, they shouldn't need any further semantic enhancements beyond that one, since the existing scoping rules already give the desired behaviour at module and function scope. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Guarantee ordered dict literals in v3.7?
Hello, would it be possible to guarantee that dict literals are ordered in v3.7? The issue is well-known and the workarounds are tedious, example: https://mail.python.org/pipermail/python-ideas/2015-December/037423.html If the feature is guaranteed now, people can rely on it around v3.9. Stefan Krah ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations
I'm very worried about trying to come up with a robust implementation of this in under 12 weeks. By contrast, the stringification that Łukasz is proposing feels eminently doable. On Sat, Nov 4, 2017 at 6:51 AM, Nick Coghlan wrote: > On 4 November 2017 at 00:40, Guido van Rossum wrote: > > IMO the inability of referencing class-level definitions from > annotations on > > methods pretty much kills this idea. > > If we decided we wanted to make it work, I think the key runtime > building block we would need is a new kind of cell reference: an > IndirectAttributeCell. > > Those would present the same interface as a regular nonlocal cell (so > it could be stored in __closure__ just as regular cells are, and > accessed the same way when the function body is executed), but > internally it would hold two references: > > - one to another cell object (__class__ for this use case) > - an attribute name on the target object that get/set/del operations > on the indirect cell's value should affect > > As Python code: > > class IndirectAttributeCell: > def __new__(cls, cell, attr): > self._cell = cell > self._attr = attr > > @property > def cell_contents(self): > return getattr(self._cell.cell_contents, self._attr) > > @cell_contents.setter > def cell_contents(self, value): > setattr(self._cell.cell_contents, self._attr, value) > > @cell_contents.deleter > def cell_contents(self): > delattr(self._cell.cell_contents, self._attr) > > The class body wouldn't be able to evaluate the thunks (since > `__class__` wouldn't be set yet), but `__init_subclass__` > implementations could, as could class decorators. > > It would require some adjustment in the compiler as well (in order to > pass the class level attribute definitions down to these implicitly > defined scopes as a new kind of accessible external namespace during > the symbol analysis pass, as well as to register the use of > "__class__" if one of the affected names was referenced), but I think > it would work at least at a technical level (by contrast, every other > idea I came up with back when I was working on the list comprehension > change was sufficiently flawed that it fell apart within a few hours > of starting to tinker with the idea). > > As an added bonus, we could potentially also extend the same > permissive name resolution semantics to the implicit scopes used in > comprehensions, such that it was only the explicitly defined scopes > (i.e. lambda expressions, function definitions, and nested classes) > that lost implicit access to the class level variables. > > Cheers, > Nick. > > P.S. If we subsequently decided to elevate expression thunks to a > first class language primitive, they shouldn't need any further > semantic enhancements beyond that one, since the existing scoping > rules already give the desired behaviour at module and function scope. > > -- > Nick Coghlan | [email protected] | Brisbane, Australia > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Guarantee ordered dict literals in v3.7?
This sounds reasonable -- I think when we introduced this in 3.6 we were worried that other implementations (e.g. Jython) would have a problem with this, but AFAIK they've reported back that they can do this just fine. So let's just document this as a language guarantee. On Sat, Nov 4, 2017 at 10:30 AM, Stefan Krah wrote: > > Hello, > > would it be possible to guarantee that dict literals are ordered in v3.7? > > > The issue is well-known and the workarounds are tedious, example: > >https://mail.python.org/pipermail/python-ideas/2015- > December/037423.html > > > If the feature is guaranteed now, people can rely on it around v3.9. > > > > Stefan Krah > > > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations
If type annotations are treated like implicit lambdas, then that's a first step to something similar to Lisp's "special forms". A full generalization of that would allow, for example, logging.debug to not evaluate its args unless debugging is turned on (I use a logging.debug wrapper that allows lambdas as args, and evaluates them if debugging is turned on). Maybe a better question is whether we want "special forms" in Python. It complicates some things but simplifies others. But things that satisfy Lisp programmers might not make Python programmers happy. ;) On 4 November 2017 at 09:42, Guido van Rossum wrote: > I'm very worried about trying to come up with a robust implementation of > this in under 12 weeks. By contrast, the stringification that Łukasz is > proposing feels eminently doable. > > On Sat, Nov 4, 2017 at 6:51 AM, Nick Coghlan wrote: > >> On 4 November 2017 at 00:40, Guido van Rossum wrote: >> > IMO the inability of referencing class-level definitions from >> annotations on >> > methods pretty much kills this idea. >> >> If we decided we wanted to make it work, I think the key runtime >> building block we would need is a new kind of cell reference: an >> IndirectAttributeCell. >> >> Those would present the same interface as a regular nonlocal cell (so >> it could be stored in __closure__ just as regular cells are, and >> accessed the same way when the function body is executed), but >> internally it would hold two references: >> >> - one to another cell object (__class__ for this use case) >> - an attribute name on the target object that get/set/del operations >> on the indirect cell's value should affect >> >> As Python code: >> >> class IndirectAttributeCell: >> def __new__(cls, cell, attr): >> self._cell = cell >> self._attr = attr >> >> @property >> def cell_contents(self): >> return getattr(self._cell.cell_contents, self._attr) >> >> @cell_contents.setter >> def cell_contents(self, value): >> setattr(self._cell.cell_contents, self._attr, value) >> >> @cell_contents.deleter >> def cell_contents(self): >> delattr(self._cell.cell_contents, self._attr) >> >> The class body wouldn't be able to evaluate the thunks (since >> `__class__` wouldn't be set yet), but `__init_subclass__` >> implementations could, as could class decorators. >> >> It would require some adjustment in the compiler as well (in order to >> pass the class level attribute definitions down to these implicitly >> defined scopes as a new kind of accessible external namespace during >> the symbol analysis pass, as well as to register the use of >> "__class__" if one of the affected names was referenced), but I think >> it would work at least at a technical level (by contrast, every other >> idea I came up with back when I was working on the list comprehension >> change was sufficiently flawed that it fell apart within a few hours >> of starting to tinker with the idea). >> >> As an added bonus, we could potentially also extend the same >> permissive name resolution semantics to the implicit scopes used in >> comprehensions, such that it was only the explicitly defined scopes >> (i.e. lambda expressions, function definitions, and nested classes) >> that lost implicit access to the class level variables. >> >> Cheers, >> Nick. >> >> P.S. If we subsequently decided to elevate expression thunks to a >> first class language primitive, they shouldn't need any further >> semantic enhancements beyond that one, since the existing scoping >> rules already give the desired behaviour at module and function scope. >> >> -- >> Nick Coghlan | [email protected] | Brisbane, Australia >> > > > > -- > --Guido van Rossum (python.org/~guido) > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > pludemann%40google.com > > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Guarantee ordered dict literals in v3.7?
+1, as Guido correctly recalls, this language guarantee will work well with Jython when we get to the point of implementing 3.7+. On Sat, Nov 4, 2017 at 12:35 PM, Guido van Rossum wrote: > This sounds reasonable -- I think when we introduced this in 3.6 we were > worried that other implementations (e.g. Jython) would have a problem with > this, but AFAIK they've reported back that they can do this just fine. So > let's just document this as a language guarantee. > > On Sat, Nov 4, 2017 at 10:30 AM, Stefan Krah wrote: > >> >> Hello, >> >> would it be possible to guarantee that dict literals are ordered in v3.7? >> >> >> The issue is well-known and the workarounds are tedious, example: >> >>https://mail.python.org/pipermail/python-ideas/2015-Decembe >> r/037423.html >> >> >> If the feature is guaranteed now, people can rely on it around v3.9. >> >> >> >> Stefan Krah >> >> >> >> ___ >> Python-Dev mailing list >> [email protected] >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido% >> 40python.org >> > > > > -- > --Guido van Rossum (python.org/~guido) > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > jbaker%40zyasoft.com > > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations
On 5 November 2017 at 02:42, Guido van Rossum wrote: > I'm very worried about trying to come up with a robust implementation of > this in under 12 weeks. By contrast, the stringification that Łukasz is > proposing feels eminently doable. I'm far from confident about that, as the string proposal inherently breaks runtime type annotation evaluation for nested function and class definitions, since those lose access to nonlocal variable references (since the compiler isn't involved in their name resolution any more). https://www.python.org/dev/peps/pep-0563/#resolving-type-hints-at-runtime is essentially defining a completely new type annotation specific scheme for name resolution, and takes us back to a Python 1.x era "locals and globals only" approach with no support for closure variables. Consider this example from the PEP: def generate(): A = Optional[int] class C: field: A = 1 def method(self, arg: A) -> None: ... return C X = generate() The PEP's current attitude towards this is "Yes, it will break, but that's OK, because it doesn't matter for the type annotation use case, since static analysers will still understand it". Adopting such a cavalier approach towards backwards compatibility with behaviour that has been supported since Python 3.0 *isn't OK*, since it would mean we were taking the step from "type annotations are the primary use case" to "Other use cases for function annotations are no longer supported". The only workaround I can see for that breakage is that instead of using strings, we could instead define a new "thunk" type that consists of two things: 1. A code object to be run with eval() 2. A dictionary mapping from variable names to closure cells (or None for not yet resolved references to globals and builtins) Correctly evaluating the code object in its original context would then be possible by reading the "cell_contents" attributes of the cells stored in the mapping and injecting them into the globals namespace used to run the code. This would actually be a pretty cool new primitive to have available (since it also leaves the consuming code free to *ignore* the closure cells, which is what you'd want for use cases like callback functions with implicitly named parameters), and retains the current eager compilation behaviour (so we'd be storing compiled code objects as constants instead of strings). If PEP 563 were updated to handle closure references properly using a scheme like the one above, I'd be far more supportive of the proposal. Alternatively, in a lambda based proposal that compiled code like the above as equivalent to the following code today: def generate(): A = Optional[int] class C: field: A = 1 def method(self, arg: (lambda: A)) -> None: ... return C X = generate() Then everything's automatically fine, since the compiler would correctly resolve the nonlocal reference to A and inject the appropriate closure references. In such a lambda based implementation, the *only* tricky case is this one, where the typevar is declared at class scope: class C: A = Optional[int] field: A = 1 def method(self, arg: A) -> None: ... Now, even without the introduction of the IndirectAttributeCell concept, this is amenable to a pretty simple workaround: A = Optional[int] class C: field: A = 1 def method(self, arg: A) -> None: ... C.A = A del A But I genuinely can't see how breaking annotation evaluation at class scope can be seen as a deal-breaker for the implicit lambda based approach without breaking annotation evaluation for nested functions also being seen as a deal-breaker for the string based approach. Either way, there are going to be changes needed to the compiler in order for it to still generate suitable references at compile time - the only question would then be whether they're existing cells stored in a new construct (a thunk to be executed with eval rather than via a regular function call), or a new kind of cell stored on a regular function object (implicit access to class attributes from implicitly defined scopes in the class body). Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Guarantee ordered dict literals in v3.7?
On 5 November 2017 at 04:35, Guido van Rossum wrote: > This sounds reasonable -- I think when we introduced this in 3.6 we were > worried that other implementations (e.g. Jython) would have a problem with > this, but AFAIK they've reported back that they can do this just fine. So > let's just document this as a language guarantee. When I asked Damien George about this for MicroPython, he indicated that they'd have to choose between guaranteed order and O(1) lookups given their current dict implementation. That surprised me a bit (since PyPy and CPython both *saved* memory by switching to their guaranteed order implementations, hence the name "compact dict representation"), but my (admittedly vague) understand is that the presence of a space/speed trade-off in their case has something to do with MicroPython deliberately running with a much higher chance of hash collisions in general (since the data sets it deals with are naturally smaller). So if we make the change, MicroPython will likely go along with it, but it may mean that dict lookups there become O(N), and folks will be relying on "N" being consistently small due to memory constraints (but some typically O(N) algorithms will still become O(N^2) when run on MicroPython). I don't think that situation should change the decision, but I do think it would be helpful if folks that understand CPython's dict implementation could take a look at MicroPython's dict implementation and see if it might be possible for them to avoid having to make that trade-off and instead be able to use a naturally insertion ordered hashmap implementation. Cheers, Nick. P.S. If anyone does want to explore MicroPython's dict implementation, and see if there might be an alternate implementation strategy that offers both O(1) lookup and guaranteed ordering without using additional memory, the relevant files seem to be: * https://github.com/micropython/micropython/blob/77a48e8cd493c0b0e0ca2d2ad58a110a23c6a232/py/obj.h#L339 (C level hashmap/ordered array structs) * https://github.com/micropython/micropython/blob/master/py/map.c (C level hashmap/ordered array implementation) * https://github.com/micropython/micropython/blob/master/py/objdict.c (Python dict wrapper around the mapping impl) The current behaviour is that the builtin dict uses the hashmap algorithms, while collections.OrderedDict uses the ordered array algorithms. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
