[Python-Dev] pickle and copy discrepancy
The pickle and the copy modules use the same protocol. The reconstruct the object by data returned by the __reduce_ex__/__reduce__ method, but does it in different and incompatible way. In general case the result of __reduce__ includes: 1) The class of the object and arguments to __new__(). 2) The state passed to __setstate__() (or a dict of attributes and possible a tuple of __slots__ values). 3) An iterator of list items that should be appended to the object by calling extend() or append(). 4) An iterator of key-value value pairs that should be set in the object by calling update() or __setitem__(). The difference is that the copy module sets object's state before adding items and key-value pairs, but the pickle module sets object's state after adding items and key-value pairs. If append() or __setitem__() depend on the state of the object, the pickling is incompatible with the copying. The behaviour of copy was changed in issue1100562 [1] (see also issue1099746 [2]). But this caused a problem with other classes (see issue10131 [3]). Changing either pickle or copy for sure will break existing code. But keeping current discrepancy makes existing code not correct and makes too hard to write correct code that works with both pickle and copy. For sure most existing code for which it is matter is not correct. The behaviour of default reducing method for dicts/lists subclasses is not documented [4]. We should choose in what direction we have to break backward compatibility. The behaviour of the copy module looks more natural. It allows to work correctly most naive implementations (as in [2]). The pickle module is more used and breaking it can cause more harm. But the order of object reconstruction is determined at pickling time, thus already pickled data will be reconstructed with old order. The change will only affect new pickles. [1] http://bugs.python.org/issue1100562 [2] http://bugs.python.org/issue1099746 [3] http://bugs.python.org/issue10131 [4] http://bugs.python.org/issue4712 ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 514: Python environment registration in the Windows Registry
Hi Steve, I have looked into this PEP to see what we need to do on Enthought side of things. I have a few questions: 1. Is it recommended to follow this for any python version we may provide, or just new versions (3.6 and above). Most of our customers still heavily use 2.7, and I wonder whether it would cause more trouble than it is worth backporting this to 2.7. 2. The main issue for us in practice has been `PythonPath` entry as used to build `sys.path`. I understand this is not the point of the PEP, but would it make sense to give more precise recommendations for 3rd party providers there ? IIUC, the PEP 514 would recommend for us to do the following: 1. Use HKLM for "system install" or HKCU for "user install" as the root key 2. Register under "\Software\Python\Enthought" 3. We should patch our pythons to look in 2. and not in "\Software\Python\PythonCore", especially for `sys.path` constructions. 4. When a python from enthought is installed, it should never register anything in the key defined in 2. Is this correct ? I am not clear about 3., especially on what should be changed. I know that for 2.7, we need to change PC\getpathp.c for sys.path, but are there any other places where the registry is used by python itself ? Thanks for working on this, David On Sat, Feb 6, 2016 at 9:01 PM, Steve Dower wrote: > I've posted an updated version of this PEP that should soon be visible at > https://www.python.org/dev/peps/pep-0514. > > Leaving aside the fact that the current implementation of Python relies on > *other* information in the registry (that is not specified in this PEP), > I'm still looking for feedback or concerns from developers who are likely > to create or use the keys that are described here. > > > > PEP: 514 > Title: Python registration in the Windows registry > Version: $Revision$ > Last-Modified: $Date$ > Author: Steve Dower > Status: Draft > Type: Informational > Content-Type: text/x-rst > Created: 02-Feb-2016 > Post-History: 02-Feb-2016 > > Abstract > > > This PEP defines a schema for the Python registry key to allow third-party > installers to register their installation, and to allow applications to > detect > and correctly display all Python environments on a user's machine. No > implementation changes to Python are proposed with this PEP. > > Python environments are not required to be registered unless they want to > be > automatically discoverable by external tools. > > The schema matches the registry values that have been used by the official > installer since at least Python 2.5, and the resolution behaviour matches > the > behaviour of the official Python releases. > > Motivation > == > > When installed on Windows, the official Python installer creates a > registry key > for discovery and detection by other applications. This allows tools such > as > installers or IDEs to automatically detect and display a user's Python > installations. > > Third-party installers, such as those used by distributions, typically > create > identical keys for the same purpose. Most tools that use the registry to > detect > Python installations only inspect the keys used by the official installer. > As a > result, third-party installations that wish to be discoverable will > overwrite > these values, resulting in users "losing" their Python installation. > > By describing a layout for registry keys that allows third-party > installations > to register themselves uniquely, as well as providing tool developers > guidance > for discovering all available Python installations, these collisions > should be > prevented. > > Definitions > === > > A "registry key" is the equivalent of a file-system path into the > registry. Each > key may contain "subkeys" (keys nested within keys) and "values" (named and > typed attributes attached to a key). > > ``HKEY_CURRENT_USER`` is the root of settings for the currently logged-in > user, > and this user can generally read and write all settings under this root. > > ``HKEY_LOCAL_MACHINE`` is the root of settings for all users. Generally, > any > user can read these settings but only administrators can modify them. It is > typical for values under ``HKEY_CURRENT_USER`` to take precedence over > those in > ``HKEY_LOCAL_MACHINE``. > > On 64-bit Windows, ``HKEY_LOCAL_MACHINE\Software\Wow6432Node`` is a > special key > that 32-bit processes transparently read and write to rather than > accessing the > ``Software`` key directly. > > Structure > = > > We consider there to be a single collection of Python environments on a > machine, > where the collection may be different for each user of the machine. There > are > three potential registry locations where the collection may be stored > based on > the installation options of each environment:: > > HKEY_CURRENT_USER\Software\Python\\ > HKEY_LOCAL_MACHINE\Software\Python\\ > HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\\ > > Environments are uniquely identifie
Re: [Python-Dev] PEP 514: Python environment registration in the Windows Registry
On 1 March 2016 at 11:37, David Cournapeau wrote: > I am not clear about 3., especially on what should be changed. I know that > for 2.7, we need to change PC\getpathp.c for sys.path, but are there any > other places where the registry is used by python itself ? My understanding from the earlier discussion was that you should not patch Python at all. The sys.path building via PythonPath is not covered by the PEP and you should continue as at present. The new keys are all for informational purposes - your installer should write to them, and read them if looking for your installations. But the Python interpreter itself should not know or care about your new keys. Steve can probably clarify better than I can, but that's how I recall it being intended to work. Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pickle and copy discrepancy
On 03/01/2016 03:14 AM, Serhiy Storchaka wrote: The difference is that the copy module sets object's state before adding items and key-value pairs, but the pickle module sets object's state after adding items and key-value pairs. If append() or __setitem__() depend on the state of the object, the pickling is incompatible with the copying. Aren't there tests to ensure the unpickled/copied object are identical to the original object? Under which circumstances would they be different? -- ~Ethan~ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 514: Python environment registration in the Windows Registry
On 01Mar2016 0524, Paul Moore wrote: On 1 March 2016 at 11:37, David Cournapeau wrote: I am not clear about 3., especially on what should be changed. I know that for 2.7, we need to change PC\getpathp.c for sys.path, but are there any other places where the registry is used by python itself ? My understanding from the earlier discussion was that you should not patch Python at all. The sys.path building via PythonPath is not covered by the PEP and you should continue as at present. The new keys are all for informational purposes - your installer should write to them, and read them if looking for your installations. But the Python interpreter itself should not know or care about your new keys. Steve can probably clarify better than I can, but that's how I recall it being intended to work. Paul Yes, the intention was to not move sys.path building out of the PythonCore key. It's solely about discovery by external tools. If you want to patch your own distribution to move the paths you are welcome to do that - there is only one string literal in getpathp.c that needs to be updated - but it's not a requirement and I deliberately avoided making a recommendation either way. (Though as discussed earlier in the thread, I'm very much in favour of deprecating and removing any use of the registry by the runtime itself in 3.6+, but still working out the implications of that.) Cheers, Steve ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 514: Python environment registration in the Windows Registry
On Tue, Mar 1, 2016 at 5:46 PM, Steve Dower wrote: > On 01Mar2016 0524, Paul Moore wrote: > >> On 1 March 2016 at 11:37, David Cournapeau wrote: >> >>> I am not clear about 3., especially on what should be changed. I know >>> that >>> for 2.7, we need to change PC\getpathp.c for sys.path, but are there any >>> other places where the registry is used by python itself ? >>> >> >> My understanding from the earlier discussion was that you should not >> patch Python at all. The sys.path building via PythonPath is not >> covered by the PEP and you should continue as at present. The new keys >> are all for informational purposes - your installer should write to >> them, and read them if looking for your installations. But the Python >> interpreter itself should not know or care about your new keys. >> >> Steve can probably clarify better than I can, but that's how I recall >> it being intended to work. >> Paul >> > > Yes, the intention was to not move sys.path building out of the PythonCore > key. It's solely about discovery by external tools. > Right. For us, continuing populating sys.path from the registry "owned" by python.org official installers is more and more untenable, because every distribution writes there, and this is especially problematic when you have both 32 bits and 64 bits distributions in the same machine. > If you want to patch your own distribution to move the paths you are > welcome to do that - there is only one string literal in getpathp.c that > needs to be updated - but it's not a requirement and I deliberately avoided > making a recommendation either way. (Though as discussed earlier in the > thread, I'm very much in favour of deprecating and removing any use of the > registry by the runtime itself in 3.6+, but still working out the > implications of that.) > Great, I just wanted to make sure removing it ourselves do not put us in a corner or further away from where python itself is going. Would it make sense to indicate in the PEP that doing so is allowed (neither recommended or frowned upon) ? David ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 514: Python environment registration in the Windows Registry
On 01Mar2016 1137, David Cournapeau wrote: If you want to patch your own distribution to move the paths you are welcome to do that - there is only one string literal in getpathp.c that needs to be updated - but it's not a requirement and I deliberately avoided making a recommendation either way. (Though as discussed earlier in the thread, I'm very much in favour of deprecating and removing any use of the registry by the runtime itself in 3.6+, but still working out the implications of that.) Great, I just wanted to make sure removing it ourselves do not put us in a corner or further away from where python itself is going. Would it make sense to indicate in the PEP that doing so is allowed (neither recommended or frowned upon) ? I was hoping not, but I suspect the question will come up again, so best to address it once in the official doc. Cheers, Steve ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
