Re: [Python-Dev] [Python-ideas] PEP 9 - plaintext PEP format - is officially deprecated

[Barry Warsaw]

On Jan 11, 2016, at 03:25 PM, anatoly techtonik wrote:

>On Wed, Jan 6, 2016 at 2:49 AM, Barry Warsaw  wrote:
>
>> reStructuredText is clearly a better format
>
>Can you expand on that? I use markdown everywhere

reST is better than plain text.  Markdown is not a PEP format option.

>> all recent PEP submissions have been in reST for a while now anyway.
>
>Is it possible to query exact numbers automatically?

Feel free to grep the PEPs hg repo.

>What is the tooling support for handling PEP 9 and PEP 12?

UTSL.  Everything is in the PEPs hg repo.

Cheers,
-Barry


pgpPeyIqhBzlY.pgp
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 509: Add a private version to dict

[Victor Stinner]

Hi,

After a first round on python-ideas, here is the second version of my
PEP. The main changes since the first version are that the dictionary
version is no more exposed at the Python level and the field type now
also has a size of 64-bit on 32-bit platforms.

The PEP is part of a serie of 3 PEP adding an API to implement a
static Python optimizer specializing functions with guards. The second
PEP is currently discussed on python-ideas and I'm still working on
the third PEP.

Thanks to Red Hat for giving me time to experiment on this.


HTML version:
https://www.python.org/dev/peps/pep-0509/


PEP: 509
Title: Add a private version to dict
Version: $Revision$
Last-Modified: $Date$
Author: Victor Stinner 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 4-January-2016
Python-Version: 3.6


Abstract


Add a new private version to builtin ``dict`` type, incremented at each
change, to implement fast guards on namespaces.


Rationale
=

In Python, the builtin ``dict`` type is used by many instructions. For
example, the ``LOAD_GLOBAL`` instruction searchs for a variable in the
global namespace, or in the builtins namespace (two dict lookups).
Python uses ``dict`` for the builtins namespace, globals namespace, type
namespaces, instance namespaces, etc. The local namespace (namespace of
a function) is usually optimized to an array, but it can be a dict too.

Python is hard to optimize because almost everything is mutable: builtin
functions, function code, global variables, local variables, ... can be
modified at runtime. Implementing optimizations respecting the Python
semantics requires to detect when "something changes": we will call
these checks "guards".

The speedup of optimizations depends on the speed of guard checks. This
PEP proposes to add a version to dictionaries to implement fast guards
on namespaces.

Dictionary lookups can be skipped if the version does not change which
is the common case for most namespaces. The performance of a guard does
not depend on the number of watched dictionary entries, complexity of
O(1), if the dictionary version does not change.

Example of optimization: copy the value of a global variable to function
constants.  This optimization requires a guard on the global variable to
check if it was modified. If the variable is modified, the variable must
be loaded at runtime when the function is called, instead of using the
constant.

See the `PEP 510 -- Specialized functions with guards
`_ for the concrete usage of
guards to specialize functions and for the rationale on Python static
optimizers.


Guard example
=

Pseudo-code of an fast guard to check if a dictionary entry was modified
(created, updated or deleted) using an hypothetical
``dict_get_version(dict)`` function::

UNSET = object()

class GuardDictKey:
def __init__(self, dict, key):
self.dict = dict
self.key = key
self.value = dict.get(key, UNSET)
self.version = dict_get_version(dict)

def check(self):
"""Return True if the dictionary entry did not changed."""

# read the version field of the dict structure
version = dict_get_version(self.dict)
if version == self.version:
# Fast-path: dictionary lookup avoided
return True

# lookup in the dictionary
value = self.dict.get(self.key, UNSET)
if value is self.value:
# another key was modified:
# cache the new dictionary version
self.version = version
return True

# the key was modified
return False


Usage of the dict version
=

Specialized functions using guards
--

The `PEP 510 -- Specialized functions with guards
`_ proposes an API to support
specialized functions with guards. It allows to implement static
optimizers for Python without breaking the Python semantics.

Example of a static Python optimizer: the astoptimizer of the `FAT
Python `_ project
implements many optimizations which require guards on namespaces.
Examples:

* Call pure builtins: to replace ``len("abc")`` with ``3``, guards on
  ``builtins.__dict__['len']`` and ``globals()['len']`` are required
* Loop unrolling: to unroll the loop ``for i in range(...): ...``,
  guards on ``builtins.__dict__['range']`` and ``globals()['range']``
  are required


Pyjion
--

According of Brett Cannon, one of the two main developers of Pyjion,
Pyjion can also benefit from dictionary version to implement
optimizations.

Pyjion is a JIT compiler for Python based upon CoreCLR (Microsoft .NET
Core runtime).


Unladen Swallow
---

Even if dictionary version was not explicitly mentionned, optimization

Re: [Python-Dev] PEP 509: Add a private version to dict

[Maciej Fijalkowski]

Hi Victor.

You know that pypy does this stuff without changing and exposing
python semantics right? We have a version dict that does not leak
abstractions to the user.

In general, doing stuff like that where there is a public API that
leaks details of certain optimizations makes it harder and harder for
optimizing compilers to do their job properly, if you want to do
something slightly different.

Can we make this happen (as you noted in the prior art) WITHOUT
changing ANY of the things exposed to the user?

On Mon, Jan 11, 2016 at 6:49 PM, Victor Stinner
 wrote:
> Hi,
>
> After a first round on python-ideas, here is the second version of my
> PEP. The main changes since the first version are that the dictionary
> version is no more exposed at the Python level and the field type now
> also has a size of 64-bit on 32-bit platforms.
>
> The PEP is part of a serie of 3 PEP adding an API to implement a
> static Python optimizer specializing functions with guards. The second
> PEP is currently discussed on python-ideas and I'm still working on
> the third PEP.
>
> Thanks to Red Hat for giving me time to experiment on this.
>
>
> HTML version:
> https://www.python.org/dev/peps/pep-0509/
>
>
> PEP: 509
> Title: Add a private version to dict
> Version: $Revision$
> Last-Modified: $Date$
> Author: Victor Stinner 
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 4-January-2016
> Python-Version: 3.6
>
>
> Abstract
> 
>
> Add a new private version to builtin ``dict`` type, incremented at each
> change, to implement fast guards on namespaces.
>
>
> Rationale
> =
>
> In Python, the builtin ``dict`` type is used by many instructions. For
> example, the ``LOAD_GLOBAL`` instruction searchs for a variable in the
> global namespace, or in the builtins namespace (two dict lookups).
> Python uses ``dict`` for the builtins namespace, globals namespace, type
> namespaces, instance namespaces, etc. The local namespace (namespace of
> a function) is usually optimized to an array, but it can be a dict too.
>
> Python is hard to optimize because almost everything is mutable: builtin
> functions, function code, global variables, local variables, ... can be
> modified at runtime. Implementing optimizations respecting the Python
> semantics requires to detect when "something changes": we will call
> these checks "guards".
>
> The speedup of optimizations depends on the speed of guard checks. This
> PEP proposes to add a version to dictionaries to implement fast guards
> on namespaces.
>
> Dictionary lookups can be skipped if the version does not change which
> is the common case for most namespaces. The performance of a guard does
> not depend on the number of watched dictionary entries, complexity of
> O(1), if the dictionary version does not change.
>
> Example of optimization: copy the value of a global variable to function
> constants.  This optimization requires a guard on the global variable to
> check if it was modified. If the variable is modified, the variable must
> be loaded at runtime when the function is called, instead of using the
> constant.
>
> See the `PEP 510 -- Specialized functions with guards
> `_ for the concrete usage of
> guards to specialize functions and for the rationale on Python static
> optimizers.
>
>
> Guard example
> =
>
> Pseudo-code of an fast guard to check if a dictionary entry was modified
> (created, updated or deleted) using an hypothetical
> ``dict_get_version(dict)`` function::
>
> UNSET = object()
>
> class GuardDictKey:
> def __init__(self, dict, key):
> self.dict = dict
> self.key = key
> self.value = dict.get(key, UNSET)
> self.version = dict_get_version(dict)
>
> def check(self):
> """Return True if the dictionary entry did not changed."""
>
> # read the version field of the dict structure
> version = dict_get_version(self.dict)
> if version == self.version:
> # Fast-path: dictionary lookup avoided
> return True
>
> # lookup in the dictionary
> value = self.dict.get(self.key, UNSET)
> if value is self.value:
> # another key was modified:
> # cache the new dictionary version
> self.version = version
> return True
>
> # the key was modified
> return False
>
>
> Usage of the dict version
> =
>
> Specialized functions using guards
> --
>
> The `PEP 510 -- Specialized functions with guards
> `_ proposes an API to support
> specialized functions with guards. It allows to implement static
> optimizers for Python without breaking the Python semantics.
>
> Example of a static Python optimizer: the astoptimizer of the `FAT
> Pytho

Re: [Python-Dev] PEP 509: Add a private version to dict

[Victor Stinner]

Le 11 janv. 2016 8:09 PM, "Maciej Fijalkowski"  a écrit :
> Hi Victor.
>
> You know that pypy does this stuff without changing and exposing
> python semantics right? We have a version dict that does not leak
> abstractions to the user.

The PEP adds a field to the C structure PyDictObject. Are you asking me to
hide it from the C structure?

The first version of my PEP added a public read-only property at Python
level, but I changed the PEP. See the alternatives section for more detail.

Victor

> In general, doing stuff like that where there is a public API that
> leaks details of certain optimizations makes it harder and harder for
> optimizing compilers to do their job properly, if you want to do
> something slightly different.
>
> Can we make this happen (as you noted in the prior art) WITHOUT
> changing ANY of the things exposed to the user?
>
> On Mon, Jan 11, 2016 at 6:49 PM, Victor Stinner
>  wrote:
> > Hi,
> >
> > After a first round on python-ideas, here is the second version of my
> > PEP. The main changes since the first version are that the dictionary
> > version is no more exposed at the Python level and the field type now
> > also has a size of 64-bit on 32-bit platforms.
> >
> > The PEP is part of a serie of 3 PEP adding an API to implement a
> > static Python optimizer specializing functions with guards. The second
> > PEP is currently discussed on python-ideas and I'm still working on
> > the third PEP.
> >
> > Thanks to Red Hat for giving me time to experiment on this.
> >
> >
> > HTML version:
> > https://www.python.org/dev/peps/pep-0509/
> >
> >
> > PEP: 509
> > Title: Add a private version to dict
> > Version: $Revision$
> > Last-Modified: $Date$
> > Author: Victor Stinner 
> > Status: Draft
> > Type: Standards Track
> > Content-Type: text/x-rst
> > Created: 4-January-2016
> > Python-Version: 3.6
> >
> >
> > Abstract
> > 
> >
> > Add a new private version to builtin ``dict`` type, incremented at each
> > change, to implement fast guards on namespaces.
> >
> >
> > Rationale
> > =
> >
> > In Python, the builtin ``dict`` type is used by many instructions. For
> > example, the ``LOAD_GLOBAL`` instruction searchs for a variable in the
> > global namespace, or in the builtins namespace (two dict lookups).
> > Python uses ``dict`` for the builtins namespace, globals namespace, type
> > namespaces, instance namespaces, etc. The local namespace (namespace of
> > a function) is usually optimized to an array, but it can be a dict too.
> >
> > Python is hard to optimize because almost everything is mutable: builtin
> > functions, function code, global variables, local variables, ... can be
> > modified at runtime. Implementing optimizations respecting the Python
> > semantics requires to detect when "something changes": we will call
> > these checks "guards".
> >
> > The speedup of optimizations depends on the speed of guard checks. This
> > PEP proposes to add a version to dictionaries to implement fast guards
> > on namespaces.
> >
> > Dictionary lookups can be skipped if the version does not change which
> > is the common case for most namespaces. The performance of a guard does
> > not depend on the number of watched dictionary entries, complexity of
> > O(1), if the dictionary version does not change.
> >
> > Example of optimization: copy the value of a global variable to function
> > constants.  This optimization requires a guard on the global variable to
> > check if it was modified. If the variable is modified, the variable must
> > be loaded at runtime when the function is called, instead of using the
> > constant.
> >
> > See the `PEP 510 -- Specialized functions with guards
> > `_ for the concrete usage of
> > guards to specialize functions and for the rationale on Python static
> > optimizers.
> >
> >
> > Guard example
> > =
> >
> > Pseudo-code of an fast guard to check if a dictionary entry was modified
> > (created, updated or deleted) using an hypothetical
> > ``dict_get_version(dict)`` function::
> >
> > UNSET = object()
> >
> > class GuardDictKey:
> > def __init__(self, dict, key):
> > self.dict = dict
> > self.key = key
> > self.value = dict.get(key, UNSET)
> > self.version = dict_get_version(dict)
> >
> > def check(self):
> > """Return True if the dictionary entry did not changed."""
> >
> > # read the version field of the dict structure
> > version = dict_get_version(self.dict)
> > if version == self.version:
> > # Fast-path: dictionary lookup avoided
> > return True
> >
> > # lookup in the dictionary
> > value = self.dict.get(self.key, UNSET)
> > if value is self.value:
> > # another key was modified:
> > # cache the new dictionary version
> > self.version = version
> >  

Re: [Python-Dev] PEP 509: Add a private version to dict

[Maciej Fijalkowski]

On Mon, Jan 11, 2016 at 9:56 PM, Victor Stinner
 wrote:
> Le 11 janv. 2016 8:09 PM, "Maciej Fijalkowski"  a écrit :
>> Hi Victor.
>>
>> You know that pypy does this stuff without changing and exposing
>> python semantics right? We have a version dict that does not leak
>> abstractions to the user.
>
> The PEP adds a field to the C structure PyDictObject. Are you asking me to
> hide it from the C structure?
>
> The first version of my PEP added a public read-only property at Python
> level, but I changed the PEP. See the alternatives section for more detail.
>
> Victor

I asked you to hide it from python, read the wrong version :-)

Cool!
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 509: Add a private version to dict

[Gregory P. Smith]

On Mon, Jan 11, 2016 at 8:50 AM Victor Stinner 
wrote:

> Hi,
>
> After a first round on python-ideas, here is the second version of my
> PEP. The main changes since the first version are that the dictionary
> version is no more exposed at the Python level and the field type now
> also has a size of 64-bit on 32-bit platforms.
>
> The PEP is part of a serie of 3 PEP adding an API to implement a
> static Python optimizer specializing functions with guards. The second
> PEP is currently discussed on python-ideas and I'm still working on
> the third PEP.
>
> Thanks to Red Hat for giving me time to experiment on this.
>
>
> HTML version:
> https://www.python.org/dev/peps/pep-0509/
>
>
> PEP: 509
> Title: Add a private version to dict
> Version: $Revision$
> Last-Modified: $Date$
> Author: Victor Stinner 
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 4-January-2016
> Python-Version: 3.6
>
>
> Abstract
> 
>
> Add a new private version to builtin ``dict`` type, incremented at each
> change, to implement fast guards on namespaces.
>
>
> Rationale
> =
>
> In Python, the builtin ``dict`` type is used by many instructions. For
> example, the ``LOAD_GLOBAL`` instruction searchs for a variable in the
> global namespace, or in the builtins namespace (two dict lookups).
> Python uses ``dict`` for the builtins namespace, globals namespace, type
> namespaces, instance namespaces, etc. The local namespace (namespace of
> a function) is usually optimized to an array, but it can be a dict too.
>
> Python is hard to optimize because almost everything is mutable: builtin
> functions, function code, global variables, local variables, ... can be
> modified at runtime. Implementing optimizations respecting the Python
> semantics requires to detect when "something changes": we will call
> these checks "guards".
>
> The speedup of optimizations depends on the speed of guard checks. This
> PEP proposes to add a version to dictionaries to implement fast guards
> on namespaces.
>
> Dictionary lookups can be skipped if the version does not change which
> is the common case for most namespaces. The performance of a guard does
> not depend on the number of watched dictionary entries, complexity of
> O(1), if the dictionary version does not change.
>
> Example of optimization: copy the value of a global variable to function
> constants.  This optimization requires a guard on the global variable to
> check if it was modified. If the variable is modified, the variable must
> be loaded at runtime when the function is called, instead of using the
> constant.
>
> See the `PEP 510 -- Specialized functions with guards
> `_ for the concrete usage of
> guards to specialize functions and for the rationale on Python static
> optimizers.
>
>
> Guard example
> =
>
> Pseudo-code of an fast guard to check if a dictionary entry was modified
> (created, updated or deleted) using an hypothetical
> ``dict_get_version(dict)`` function::
>
> UNSET = object()
>
> class GuardDictKey:
> def __init__(self, dict, key):
> self.dict = dict
> self.key = key
> self.value = dict.get(key, UNSET)
> self.version = dict_get_version(dict)
>
> def check(self):
> """Return True if the dictionary entry did not changed."""
>
> # read the version field of the dict structure
> version = dict_get_version(self.dict)
> if version == self.version:
> # Fast-path: dictionary lookup avoided
> return True
>
> # lookup in the dictionary
> value = self.dict.get(self.key, UNSET)
> if value is self.value:
> # another key was modified:
> # cache the new dictionary version
> self.version = version
> return True
>
> # the key was modified
> return False
>
>
> Usage of the dict version
> =
>
> Specialized functions using guards
> --
>
> The `PEP 510 -- Specialized functions with guards
> `_ proposes an API to support
> specialized functions with guards. It allows to implement static
> optimizers for Python without breaking the Python semantics.
>
> Example of a static Python optimizer: the astoptimizer of the `FAT
> Python `_ project
> implements many optimizations which require guards on namespaces.
> Examples:
>
> * Call pure builtins: to replace ``len("abc")`` with ``3``, guards on
>   ``builtins.__dict__['len']`` and ``globals()['len']`` are required
> * Loop unrolling: to unroll the loop ``for i in range(...): ...``,
>   guards on ``builtins.__dict__['range']`` and ``globals()['range']``
>   are required
>
>
> Pyjion
> --
>
> According of Brett Cannon, one of the t

Re: [Python-Dev] PEP 509: Add a private version to dict

[Victor Stinner]

2016-01-12 0:07 GMT+01:00 Gregory P. Smith :
>> Changes
>> ===
>>
>> (...)
>
> Please be more explicit about what tests you are performing on the values.
> setitem's "if the value is different" really should mean "if value is not
> dict['key']".  similarly for update, there should never be equality checks
> performed on the values.  just an "is" test of it they are the same object
> or not.

Ok, done. By the way, it's also explained below: values are compared
by their identify, not by their content.

For best dict efficiency, we can not implement this micro-optimization
(to avoid a potential branch misprediction in the CPU) and always
increase the version. But for guards, the micro-optimization can avoid
a lot of dictionary lookups, especially when a guard watches for a
large number of keys.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 509: Add a private version to dict

[Andrew Barnert via Python-Dev]

On Jan 11, 2016, at 15:24, Victor Stinner  wrote:
> 
> 2016-01-12 0:07 GMT+01:00 Gregory P. Smith :
>>> Changes
>>> ===
>>> 
>>> (...)
>> 
>> Please be more explicit about what tests you are performing on the values.
>> setitem's "if the value is different" really should mean "if value is not
>> dict['key']".  similarly for update, there should never be equality checks
>> performed on the values.  just an "is" test of it they are the same object
>> or not.
> 
> Ok, done. By the way, it's also explained below: values are compared
> by their identify, not by their content.
> 
> For best dict efficiency, we can not implement this micro-optimization
> (to avoid a potential branch misprediction in the CPU) and always
> increase the version. But for guards, the micro-optimization can avoid
> a lot of dictionary lookups, especially when a guard watches for a
> large number of keys.

Are you saying that d[key] = d[key] may or may not increment the version, so 
any optimizer can't rely on the fact that it doesn't?

If so, that seems reasonable. (The worst case in incrementing the version 
unnecessarily is that you miss an optimization that would have been safe, 
right?).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com