Re: [Python-Dev] An interesting exception handling quirk

[Nick Coghlan]

On 20 October 2013 16:08, Armin Rigo  wrote:
> Hi Nick,
>
> On Sat, Oct 19, 2013 at 1:41 PM, Nick Coghlan  wrote:
>> recreating the *exact* exception subclass check from
>> Python is actually difficult these days.
>
> Can't it be done roughly like that?
>
>def __exit__(self, typ, val, tb):
> try:
> raise typ, val
> except self.exceptions:
> return True
> except:
> return False

In Python 3, you have to use "raise type if val is None else val"
instead, and you then have to deal with the fact that raise will
overwrite an already set __context__ on the exception value
(contextlib.ExitStack actually has to work around that when unwinding
the stack by restoring a more appropriate __context__ value). But yes,
actually reraising it does let you reproduce the exception matching.

That said, it actually occurs to me now that the current behaviour
(overwriting an already set __context__) could arguably be considered
a bug, since we don't overwrite an already set __traceback__.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 453 (ensurepip) updated

[Nick Coghlan]

I have posted the latest version of PEP 453 to python.org. It is
available in the usual place:

http://www.python.org/dev/peps/pep-0453/

Three significant changes have been made since the last posted version:

* removed the proposal to change the script installation directory on
Windows, due to a backwards compatibility issue identified for Windows
package installers created with earlier versions of Python (and
vice-versa when attempting to use installers created with Python 3.4
on older versions)

* noted the current certificate verification concerns for the requests
project, and made resolution of that a requirement for inclusion of
ensurepip in the final release of Python 3.4

* added an integration timeline, including a December 29th deadline
for the inclusion of pip 1.5 (or a subsequent maintenance release)
that includes a resolution of the certificate verification issues in
requests

Rather than posting the whole document again, I have just quoted the
most relevant sections for these changes:



Explicit bootstrapping mechanism




Security considerations
---

The design in this PEP has been deliberately chosen to avoid making any
significant changes to the trust model of CPython for end users that do
not subsequently run the command ``pip install --upgrade pip``.

The installers will contain all the components of a fully functioning
version of Python, including the ``pip`` installer. The installation
process will *not* require network access, and will *not* rely on
trusting the security of the network connection established between
``pip`` and the Python package index.

Only users that choose to use ``pip`` to communicate with PyPI will
need to pay attention to the additional security considerations that come
with doing so.

However, the core CPython team will still assist with reviewing and
resolving at least the `certificate update management issue
`__ currently
affecting the ``requests`` project (and hence ``pip``), and may also be
able to offer assistance in resolving other identified security concerns
[#cert-verification]_.



Integration timeline


If this PEP is accepted, the proposed time frame for integration of ``pip``
into the CPython release is as follows:

* as soon as possible after the release of 3.4.0 alpha 4

  * Documentation updated and ``ensurepip`` implemented based on a
pre-release version of ``pip`` 1.5.

  * All other proposed functional changes for Python 3.4 implemented,
including the installer updates to invoke ``ensurepip``.

* by November 20th (3 days prior to the scheduled date of 3.4.0 beta 1)

  * ``ensurepip`` updated to use a ``pip`` 1.5 release candidate.

  * PEP 101 updated to cover ensuring the bundled version of ``pip`` is up
to date.

* by November 24th (scheduled date of 3.4.0 beta 1)

  * As with any other new feature, all proposed functional changes for
Python 3.4 must be implemented prior to the beta feature freeze.

* by December 29th (1 week prior to the scheduled date of 3.4.0 beta 2)

  * ``requests`` certificate management issue resolved
  * ``ensurepip`` updated to the final release of ``pip`` 1.5, or a
subsequent maintenance release (including a suitably updated vendored
copy of ``requests``)

(See PEP 429 for the current official scheduled dates of each release. Dates
listed above are accurate as of October 20th, 2013.)

If there is no final or maintenance release of ``pip`` 1.5 with a suitable
updated version of ``requests`` available by one week before the scheduled
Python 3.4 beta 2 release, then implementation of this PEP will
be deferred to Python 3.5. Note that this scenario is considered unlikely -
the tentative date for the ``pip`` 1.5 release is currently December 1st.

In future CPython releases, this kind of coordinated scheduling shouldn't be
needed: the CPython release manager will be able to just update to the latest
released version of ``pip``. However, in this case, some fixes are needed in
``pip`` in order to allow the bundling to work correctly, and the
certificate update mechanism for ``requests`` needs to be improved, so the
``pip`` 1.5 release cycle needs to be properly aligned with the CPython 3.4
beta releases.



Appendix: Rejected Proposals


Changing the name of the scripts directory on Windows
-

Earlier versions of this PEP proposed changing the name of the script
installation directory on Windows from "Scripts" to "bin" in order to
improve the cross-platform consistency of the virtual environments created
by ``pyvenv``.

However, Paul Moore determined that this change was likely backwards
incompatible with cross-version Windows installers created with previous
versions of Python, so the change has been removed from this PEP
[#windows-incompatibility]_.




-- 
Nick Coghlan   |   ncogh...@g

Re: [Python-Dev] PEP 454 (tracemalloc): new minimalist version

[Victor Stinner]

2013/10/19 Charles-François Natali :
>> get_object_trace(obj) is a shortcut for
>> get_trace(get_object_address(obj)). I agree that the wrong size
>> information can be surprising.
>>
>> I can delete get_object_trace(), or rename the function to
>> get_object_traceback() and modify it to only return the traceback.
>>
>> I prefer to keep the function (modified for get_object_traceback).
>> tracemalloc can be combined with other tools like Melia, Heapy or
>> objgraph to combine information. When you find an interesting object
>> with these tools, you may be interested to know where it was
>> allocated.
>
> If you mean modify it to return only the trace, then that's fine.
> As for the name, traceback does indeed sound less confusing than
> trace, but we should just make sure that the names are consistent
> across the API (i.e. always use "trace" or "always use "traceback",
> not both).

I'm not sure that you understood my proposition. Extract of the doc:

Example of trace: (32, (('x.py', 7), ('x.py', 11))). The memory block
has a size of 32 bytes and was allocated at x.py:7, line called from
line x.py:11.

A trace is (size, traceback). Currently, get_trace(address),
get_object_trace(object), get_traces() return traces.

You look to be confused by get_object_trace() returning a trace
because the size may be smaller than the total size.

I proposed to rename get_object_trace() to get_object_traceback() and
modify it to only return the traceback:  (('x.py', 7), ('x.py', 11)).

But get_object_trace/traceback() can also be removed, because it's
trivial to reimplement it using get_object_address() and get_trace().
It's maybe less suprising to get a wrong size using
get_trace(get_object_address(obj)).

> Well, I can certainly find a use-case for get_object_trace(): even if
> it uses get_trace() internally, I'm not convinced that the later is
> useful.
>
> If we cannot come up with a use case for working with raw addresses,
> I'm tempted to just keep get_object_trace() public, and make
> get_object_address() and get_trace() private.
> In short, don't make any address-manipulating function public.

I would like to keep functions using addresses. Usecase: call
get_traces() to get all traces, and then (maybe after the process
exited) use it to get the trace of different objects using
get_object_address().

> [Filter.match*]
> IIUC, you only use those match methods for tests and internally for
> code factorization: IMO, that's a hint they shouldn't be made public.

Ok, I made the methods private.

>> I already used other tools like Melia and Heapy, and it's convinient
>> to get access to raw data to compute manually my own view. I don't
>> want to force users to use the high-level API (Snapshot).
>>
>> Is it a problem to have two API (low-level like get_stats() and
>> high-level like Snapshot) for similar use cases? What do you suggest?
>
> I didn't understand your above explanation: could get_stats() be
> implemented atop a snapshot, or not?

get_stats() cannot be implemented on top of Snapshot.

> Humm...
> I didn't see any metric example at
> http://www.haypocalc.com/tmp/tracemalloc/library/tracemalloc.html
> Is it me?

http://www.haypocalc.com/tmp/tracemalloc/library/tracemalloc.html#metric
"Example of metrics: Resident Set Size (RSS) memory of a process,
memory in bytes used by Python, number of Python objects, etc."

Did you expect something else for examples?

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Recall: cpython: Rename contextlib.ignored() to contextlib.ignore().

[Mark.Favas]

Favas, Mark (CSIRO IM&T, Kensington) would like to recall the message, 
"cpython: Rename contextlib.ignored() to contextlib.ignore().".
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore().

[Mark.Favas]

Or perhaps call it "contextlib.ignore_first(Exception)"?


Mark Favas
iVEC@CSIRO Director
A/Director, Technical Operations, iVEC
Advanced Scientific Computing
CSIRO IM&T


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 451: ModuleSpec

[Eric Snow]

On Fri, Oct 11, 2013 at 1:37 PM, Eric Snow  wrote:
> Any comments?  Usually silence implies no disapproval. ;)  PEP 451 did
> go through several rounds of review on import-sig, so I'm not going to
> stress over low feedback at this point.  However, I'd particularly
> appreciate knowing if there are any objections to ModuleSpec for 3.4.
> Otherwise I'd like to get pronouncement on the PEP.  Thanks!

Okay, no objections and the PEP has not changed.  I'd like
pronouncement on PEP 451.

Guido, if you feel so inclined as to delegate and either of them is
willing, Brett and Nick are both familiar with the PEP, having been
heavily involved in earlier discussions.  Thanks!

-eric
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] A C implementation of OrderedDict

[Eric Snow]

If anyone is interested in having a (faithful) C implementation of
OrderedDict for the 3.4 release, I have a patch up [1].  My interest
in having the patch applied is relative to proposals that won't apply
to 3.4, so I haven't felt the need to advance the patch.  However, in
case anyone else would find it useful for 3.4, I figured I would point
it out.

While the patch isn't all that complicated, it is large and my C/C-API
experience isn't proportional to that size.  So I don't feel
comfortable about moving ahead with the patch without at least one
thorough review.  Thanks.

-eric

[1] http://bugs.python.org/issue16991
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 451: ModuleSpec

[Guido van Rossum]

How about if Brett and Nick both agree that it's good, I approve it?
If either one wants more discussion I'll appoint them the PEP-BDFL.

On Sun, Oct 20, 2013 at 8:54 AM, Eric Snow  wrote:
> On Fri, Oct 11, 2013 at 1:37 PM, Eric Snow  
> wrote:
>> Any comments?  Usually silence implies no disapproval. ;)  PEP 451 did
>> go through several rounds of review on import-sig, so I'm not going to
>> stress over low feedback at this point.  However, I'd particularly
>> appreciate knowing if there are any objections to ModuleSpec for 3.4.
>> Otherwise I'd like to get pronouncement on the PEP.  Thanks!
>
> Okay, no objections and the PEP has not changed.  I'd like
> pronouncement on PEP 451.
>
> Guido, if you feel so inclined as to delegate and either of them is
> willing, Brett and Nick are both familiar with the PEP, having been
> heavily involved in earlier discussions.  Thanks!
>
> -eric



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Python 3.4.0a4: Coming In For A Bumpy Landing

[Larry Hastings]

3.4.0a4 is tagged and I'm in the process of releasing it.  But it's 
going to be, let's say, more "alpha-quality" than the previous alphas.


Known problems:

 * There's a reference count leak in the compiler.
 * asyncio test suite sometimes times out, which takes... an hour.
 * asyncio test suit fails on a couple of platforms.


I don't think any of these are bad enough to slip the schedule. However, 
if you disagree, pipe up, feel free to try and change my mind, it's not 
too late.



//arry/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.4.0a4: Coming In For A Bumpy Landing

[Benjamin Peterson]

2013/10/20 Larry Hastings :
>
>
> 3.4.0a4 is tagged and I'm in the process of releasing it.  But it's going to
> be, let's say, more "alpha-quality" than the previous alphas.
>
> Known problems:
>
> There's a reference count leak in the compiler.

This is fixed, so you could just move the tag up a bit.


-- 
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Python 3.4.0a4: Coming In For A Bumpy Landing

[Ned Deily]

On Oct 20, 2013, at 15:57 , Benjamin Peterson  wrote:

> 2013/10/20 Larry Hastings :
>> 
>> 
>> 3.4.0a4 is tagged and I'm in the process of releasing it.  But it's going to
>> be, let's say, more "alpha-quality" than the previous alphas.
>> 
>> Known problems:
>> 
>> There's a reference count leak in the compiler.
> 
> This is fixed, so you could just move the tag up a bit.


Unfortunately, the Windows and OS X installers have already been manufactured 
so we would either have to redo them or document that there is a discrepancy 
between the source and binary releases.  Probably not a big deal in any case 
for an alpha release.

--
  Ned Deily
  n...@acm.org -- []


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Python 3.4.0a4: Coming In For A Bumpy Landing

[Benjamin Peterson]

2013/10/20 Ned Deily :
>
> On Oct 20, 2013, at 15:57 , Benjamin Peterson  wrote:
>
>> 2013/10/20 Larry Hastings :
>>>
>>>
>>> 3.4.0a4 is tagged and I'm in the process of releasing it.  But it's going to
>>> be, let's say, more "alpha-quality" than the previous alphas.
>>>
>>> Known problems:
>>>
>>> There's a reference count leak in the compiler.
>>
>> This is fixed, so you could just move the tag up a bit.
>
>
> Unfortunately, the Windows and OS X installers have already been manufactured 
> so we would either have to redo them or document that there is a discrepancy 
> between the source and binary releases.  Probably not a big deal in any case 
> for an alpha release.

Ah, I didn't realize the release process had gotten that far. Carry on.


-- 
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Logging Module format

[Hasan Diwan]

I've been using the logging module recently and noticed the default format
doesn't timestamp log entries. I've not figured out how to change the
format after initialization. This is python 2.7, on Mac OS X. Help, anyone?
-- H

-- 
Sent from my mobile device
Envoyé de mon portable
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Logging Module format

[R. David Murray]

On Sun, 20 Oct 2013 16:34:30 -0700, Hasan Diwan  wrote:
> I've been using the logging module recently and noticed the default format
> doesn't timestamp log entries. I've not figured out how to change the
> format after initialization. This is python 2.7, on Mac OS X. Help, anyone?

This mailing list is for developing Python itself.  For help on using
python, please post to python-list.  There are a lot of smart people
hanging out there who are happy to help.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] [RELEASED] Python 3.4.0a4

[Larry Hastings]

On behalf of the Python development team, I'm very pleased to announce
the fourth and final alpha release of Python 3.4.

This is a preview release, and its use is not recommended for
production settings.

Python 3.4 includes a range of improvements of the 3.x series, including
hundreds of small improvements and bug fixes.  Major new features and
changes in the 3.4 release series so far include:

* PEP 435, a standardized "enum" module
* PEP 436, a build enhancement that will help generate introspection
   information for builtins
* PEP 442, improved semantics for object finalization
* PEP 443, adding single-dispatch generic functions to the standard library
* PEP 445, a new C API for implementing custom memory allocators
* PEP 446, changing file descriptors to not be inherited by default
   in subprocesses
* PEP 450, the new "statistics" module
* PEP 3156, the new "asyncio" module, a new framework for asynchronous I/O

To download Python 3.4.0a4 visit:

http://www.python.org/download/releases/3.4.0/


Python 3.4.0a4 has several known issues:

* The Python compiler has a small memory leak.
* The "asyncio" module test suite fails on some platforms.
* I/O conducted by the "asyncio" module may, rarely,
  erroneously time out.  The timeout takes one hour.

Please consider trying Python 3.4.0a4 with your code and reporting any
new issues you notice to:

 http://bugs.python.org/


Enjoy!

--
Larry Hastings, Release Manager
larry at hastings.org
(on behalf of the entire python-dev team and 3.4's contributors)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Coding practice for context managers

[Raymond Hettinger]

Two of the new context managers in contextlib are now wrapped in pass-through 
factory functions.  The intent is to make the help() look cleaner.  This 
practice does have downsides however.  

The usual way to detect whether something is usable with a with-statement is to 
check the presence of the __enter__ and __exit__ methods.   Wrapping the CM in 
a pass-through function defeats this and other forms of introspection.

Also, the help() output itself is worse-off.  When you run help on a CM(), 
you're trying to find-out what happens on entry and what happens on exit.  If 
those methods had docstrings, the question would be answered directly.   The 
wrapper (intentionally) hides how it works.  

Since I teach intermediate and advanced python classes to experienced Python 
users, I've become more sensitive to problems this practice will create.  
Defeating introspection can make the help look nicer, but it isn't a clean 
coding practice and is something I hope doesn't catch on.

To the extent there is a problem with the output of help(), I think efforts 
should be directed at making help() better.   A lot of work needs to be done on 
that end -- for example abstract base classes also don't look great in help().

There are a couple of other minor issues as well.  One is that the wrapper 
function hides the class, making it harder to do type checks such as 
"isinstance(x, suppress)".  The other issue is that wrappers cause extra 
jumping around for people who are tracing code through a debugger or using a 
visualization tool such as pythontutor.   These aren't terribly important 
issues, but it does support the notion that usually the cleanest code is the 
best code.

In short, I recommend that efforts be directed at improving help() rather than 
limiting introspection by way of less clean coding practices.


Raymond


 current code for suppress() 

class _SuppressExceptions:
"""Helper for suppress."""
def __init__(self, *exceptions):
self._exceptions = exceptions

def __enter__(self):
pass

def __exit__(self, exctype, excinst, exctb):
return exctype is not None and issubclass(exctype, self._exceptions)

# Use a wrapper function since we don't care about supporting inheritance
# and a function gives much cleaner output in help()
def suppress(*exceptions):
"""Context manager to suppress specified exceptions

After the exception is suppressed, execution proceeds with the next
statement following the with statement.

 with suppress(FileNotFoundError):
 os.remove(somefile)
 # Execution still resumes here if the file was already removed
"""
return _SuppressExceptions(*exceptions)


 current help() output for suppress() 

Help on function suppress in module contextlib:

suppress(*exceptions)
Context manager to suppress specified exceptions

After the exception is suppressed, execution proceeds with the next
statement following the with statement.

 with suppress(FileNotFoundError):
 os.remove(somefile)
 # Execution still resumes here if the file was already removed

 current help() output for closing() with does not have a function 
wrapper 

Help on class closing in module contextlib:

class closing(builtins.object)
 |  Context to automatically close something at the end of a block.
 |  
 |  Code like this:
 |  
 |  with closing(.open()) as f:
 |  
 |  
 |  is equivalent to this:
 |  
 |  f = .open()
 |  try:
 |  
 |  finally:
 |  f.close()
 |  
 |  Methods defined here:
 |  
 |  __enter__(self)
 |  
 |  __exit__(self, *exc_info)
 |  
 |  __init__(self, thing)
 |  
 |  --
 |  Data descriptors defined here:
 |  
 |  __dict__
 |  dictionary for instance variables (if defined)
 |  
 |  __weakref__
 |  list of weak references to the object (if defined)




___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Coding practice for context managers

[Ethan Furman]

On 10/20/2013 07:42 PM, Raymond Hettinger wrote:


In short, I recommend that efforts be directed at improving help() rather than 
limiting introspection by way of less clean coding practices.


+1

We also have the option of adding a custom __dir__ to change what help() 
displays.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] A C implementation of OrderedDict

[Raymond Hettinger]

On Oct 20, 2013, at 9:21 AM, Eric Snow  wrote:

> If anyone is interested in having a (faithful) C implementation of
> OrderedDict for the 3.4 release, I have a patch up [1].  My interest
> in having the patch applied is relative to proposals that won't apply
> to 3.4, so I haven't felt the need to advance the patch.  However, in
> case anyone else would find it useful for 3.4, I figured I would point
> it out.
> 
> While the patch isn't all that complicated, it is large and my C/C-API
> experience isn't proportional to that size.  So I don't feel
> comfortable about moving ahead with the patch without at least one
> thorough review.  Thanks.

I'll look at this in more detail after I've finishing my review of the 
TransformDict,
but my initial impression is that the original show stopper hasn't been 
overcome:
http://bugs.python.org/issue10977  

The concrete dict API is very old and is widely used in C extensions
outside the standard library.  AFAICT, there is no way to prevent that code
from bypassing your code and breaking the internal invariants of the 
ordered dict (that breakage could be silent are muck up the ordering
or could fail loudly with a segfault).

If we really want a C implementation, I think the choices boil down to:

1) living with the problem and writing defensive code so that the
ordered dictionary won't segfault when fed to existing external C code
and so that the ordered dict notices whether the parent dictionary has
different keys than those in the internal linked list.

or 2) biting the bullet and accepting an API change where ordered dicts
are no longer a subclass of real dicts.


Raymond___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] A C implementation of OrderedDict

[Eric Snow]

On Sun, Oct 20, 2013 at 9:56 PM, Raymond Hettinger
 wrote:
> I'll look at this in more detail after I've finishing my review of the
> TransformDict,
> but my initial impression is that the original show stopper hasn't been
> overcome:
> http://bugs.python.org/issue10977
>
> The concrete dict API is very old and is widely used in C extensions
> outside the standard library.  AFAICT, there is no way to prevent that code
> from bypassing your code and breaking the internal invariants of the
> ordered dict (that breakage could be silent are muck up the ordering
> or could fail loudly with a segfault).

Just to be clear, the patch I have up is meant to provide a nearly
exact duplicate of OrderedDict in C, minus the "private"
implementation details.  The current OrderedDict already suffers from
the same problems you described, since it is also a dict subclass (as
you know :-).  Those would definitely become show stoppers, as you
said, but only once we tried to actually use OrderedDict in the
interpreter in places where we currently use dict (which is something
we should mostly avoid).  Are you otherwise concerned that a C
implementation would send the wrong message to extension module
authors or cause problems for C extension modules that use
OrderedDict, beyond any they face with the pure Python implementation?

Regardless, I concur that we would need to find a solution to the
existing concrete API situation if OrderedDict were to be used in the
intepreter as a drop-in replacement for current dicts.  The same would
apply for any case of subclassing one of the fundamental types (like
str, which came up in a recent issue related to the email module).  In
the meantime, however, a C implementation of OrderedDict would offer a
performance improvement for anyone that uses it.  Furthermore, for the
couple of proposals I have cooking that make use of OrderedDict, the
concrete API problem does not factor in all that much, but having a
builtin C implementation does.  I will admit that things would be
easier for one of ideas if I could just drop OrderedDict in, but
that's a rats nest I stared at at one point and then backed away
slowly. 

Ultimately, I don't feel any urgency WRT a C implementation of
OrderedDict.  I simply wanted to make others aware of the possibility
in case it would be useful to them in 3.4.  Otherwise I am perfectly
comfortable with taking any time you feel is necessary.

>
> If we really want a C implementation, I think the choices boil down to:
>
> 1) living with the problem and writing defensive code so that the
> ordered dictionary won't segfault when fed to existing external C code
> and so that the ordered dict notices whether the parent dictionary has
> different keys than those in the internal linked list.

no thanks!  Dealing with re-entrancy was quite enough. :)

>
> or 2) biting the bullet and accepting an API change where ordered dicts
> are no longer a subclass of real dicts.

I know a few people that would argue for this regardless.

-eric
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com