Re: [Python-Dev] Creating dicts from dict subclasses
Guido van Rossum wrote:
> On 12/13/06, Walter Dörwald <[EMAIL PROTECTED]> wrote:
>> Guido van Rossum wrote:
>> > On 12/13/06, Walter Dörwald <[EMAIL PROTECTED]> wrote:
>> >> I tried to reimplement weakref.WeakValueDictionary as a subclass of
>> >> dict. The test passes except for one problem: To compare results
>> >> test_weakref.py converts a weakdict to a real dict via dict(weakdict).
>> >> This no longer works because PyDict_Merge() does a PyDict_Check()
>> on the
>> >> argument and then ignores all overwritten methods. (The old version
>> >> worked because UserDict.UserDict was used).
>> >>
>> >> The simplest solution is to replace the PyDict_Check() call with
>> >> PyDict_CheckExact(), but this might slow things down too much, because
>> >> the fallback code basically does:
>> >>
>> >>for key in iter(arg.keys()):
>> >> self[key] = arg.__getitem__(key)
>> >>
>> >> Why can't we use:
>> >>
>> >>for key in iter(arg):
>> >> self[key] = arg.__getitem__(key)
>> >>
>> >> instead?
>> >
>> > The only reason I can think of is backwards compatibility: not all
>> > "mappings" created pre-2.2 would support iteration. Maybe you could
>> > check for a tp_iter slot and if non-NULL use the latter otherwise use
>> > the original fallback?
>>
>> This doesn't seem to work. It breaks test_update() in test_dict.py which
>> does this:
>>
>> d = {}
>> class SimpleUserDict:
>> def __init__(self):
>> self.d = {1:1, 2:2, 3:3}
>> def keys(self):
>> return self.d.keys()
>> def __getitem__(self, i):
>> return self.d[i]
>> d.update(SimpleUserDict())
>> self.assertEqual(d, {1:1, 2:2, 3:3})
>>
>> This fails with
>>
>> KeyError: 0
>>
>> because SimpleUserDict doesn't implement __iter__, so it gets an
>> iterator implementation via __getitem__.
>>
>> So maybe this only makes sense for Python 3.0 where we can demand that
>> dict-like classes implement __iter__?
>
> Ah, right. But I think you should still use PyDict_CheckExact, and
> slow fallbacks be damned. (I guess you could look for iterkeys first.)
OK, here's a patch that tries iterkeys() before keys():
http://bugs.python.org/1615701
Both versions seem to be faster than Python 2.5:
class dictik:
def __init__(self, n):
self.d = dict((i, i) for i in xrange(n))
def iterkeys(self):
return iter(self.d)
def __getitem__(self, i):
return self.d[i]
class dictk:
def __init__(self, n):
self.d = dict((i, i) for i in xrange(n))
def keys(self):
return self.d.keys()
def __getitem__(self, i):
return self.d[i]
$ python2.5 -mtimeit -s'from foo import dictik, dictk; d=dictk(10)'
'dict(d)'
10 loops, best of 3: 179 msec per loop
$ ./python -mtimeit -s'from foo import dictik, dictk; d=dictk(10)'
'dict(d)'
10 loops, best of 3: 138 msec per loop
$ ./python -mtimeit -s'from foo import dictik, dictk; d=dictik(10)'
'dict(d)'
10 loops, best of 3: 123 msec per loop
Servus,
Walter
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Creating dicts from dict subclasses
Armin Rigo wrote: > Hi Walter, > > On Wed, Dec 13, 2006 at 05:57:16PM +0100, Walter D?rwald wrote: >> I tried to reimplement weakref.WeakValueDictionary as a subclass of >> dict. The test passes except for one problem: To compare results >> test_weakref.py converts a weakdict to a real dict via dict(weakdict). >> This no longer works because PyDict_Merge() does a PyDict_Check() on the >> argument and then ignores all overwritten methods. (The old version >> worked because UserDict.UserDict was used). > > This is an instance of a general problem in Python: if you subclass a > built-in type, then your overridden methods may or may not be used in > various situations. In this case you might have subtle problems with > built-in functions and statements that expect a dict and manipulate it > directly, because they will see the underlying dict structure. It is > also quite fragile: e.g. if a future version of CPython adds a new > method to dicts, then your existing code will also grow the new method > automatically - but as inherited from 'dict', which produces quite > surprizing results for the user. > >>for key in iter(arg.keys()): >> self[key] = arg.__getitem__(key) >> >> Why can't we use: >> >>for key in iter(arg): >> self[key] = arg.__getitem__(key) > > The latter would allow 'arg' to be a sequence instead of a mapping. It > may even not crash but produce nonsense instead, e.g. if 'arg' is a list > of small integers. Of course I meant: use the alternate code inside PyDict_Merge() where dict_update_common() already has decided that the argument is a mapping (which is done via PyObject_HasAttrString(arg, "keys")). > Moreover there are multiple places in the code base > that assume that mappings are "something with a 'keys' and a > '__getitem__'", so I suppose any change in that should be done > carefully. Doing a grep PyMapping_Keys `find -name '*.[ch]'` reveals the following: ./Python/ceval.c: all = PyMapping_Keys(dict); This is used for "import *" and simply iterates over the keys, so it could use iterkeys()/iter() ./Objects/object.c: result = PyMapping_Keys(locals); This is in PyObject_Dir(). It does return the keylist, so no iterkeys()/iter() here. ./Objects/descrobject.c:return PyMapping_Keys(pp->dict); This too must return a list of keys. ./Objects/dictobject.c: PyObject *keys = PyMapping_Keys(b); This is the dict constructor. ./PC/_subprocess.c: keys = PyMapping_Keys(environment); This iterates over keys() and values() to format the complete environment, so it probably could be switched to iterkeys()/iter(). ./Modules/_sre.c:keys = PyMapping_Keys(self->pattern->groupindex); This again does iteration, so could be switched. ./Modules/posixmodule.c:keys = PyMapping_Keys(env); ./Modules/posixmodule.c:keys = PyMapping_Keys(env); ./Modules/posixmodule.c:keys = PyMapping_Keys(env); Those three are for execve/spawnve/spawnvpe and do basically the same as PC/_subprocess.c, so could be switched too. Servus, Walter ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Weekly Python Patch/Bug Summary
Patch / Bug Summary
___
Patches : 414 open ( +1) / 3498 closed ( +9) / 3912 total (+10)
Bugs: 949 open ( +6) / 6376 closed (+12) / 7325 total (+18)
RFE : 247 open ( +1) / 245 closed ( +1) / 492 total ( +2)
New / Reopened Patches
__
C99 _Bool support for struct (2006-12-07)
http://python.org/sf/1610575 opened by David Remahl
BSD version of ctypes.util.find_library (2006-12-07)
http://python.org/sf/1610795 opened by Martin Kammerhofer
whitespace in `svnversion .` (2006-10-15)
CLOSED http://python.org/sf/1577756 reopened by schmaller
Enhanced tabbed pane widget (2006-12-10)
http://python.org/sf/1612746 opened by Tal Einat
encoding directive -- more examples (2006-12-11)
http://python.org/sf/1613352 opened by Jim Jewett
Write mode option for fileinput module. (2006-12-11)
http://python.org/sf/1613500 opened by Anthony Roy
POSIX capabilities support (2006-12-13)
http://python.org/sf/1615158 opened by Matt Kern
Creating dicts for dict subclasses (2006-12-14)
http://python.org/sf/1615701 opened by Walter Dörwald
BZ2File.seek() fails for large files (2006-12-14)
http://python.org/sf/1615868 opened by Lars Gustäbel
Cached globals+builtins lookup optimization (2006-12-15)
http://python.org/sf/1616125 opened by Andrea Griffini
Patches Closed
__
whitespace in `svnversion .` (2006-10-15)
http://python.org/sf/1577756 closed by gbrandl
traceback on exit if syslog handler fails to initialize (2006-12-05)
http://python.org/sf/1609407 closed by vsajip
Race condition in os.makedirs (2006-12-04)
http://python.org/sf/1608579 closed by gbrandl
Prevent race condition in os.makedirs (2005-07-17)
http://python.org/sf/1239890 closed by gbrandl
os.makedirs - robust against partial path (2005-10-05)
http://python.org/sf/1314067 closed by gbrandl
modsupport does not use va_copy when available (2003-12-11)
http://python.org/sf/858318 closed by sf-robot
acknowledge signals in non-main threads (2004-12-21)
http://python.org/sf/1088832 closed by loewis
Patch for bug 999042. (2004-12-23)
http://python.org/sf/1090482 closed by loewis
socket leak in SocketServer (2004-12-30)
http://python.org/sf/1093468 closed by loewis
mailbox.py: check that os.fsync is available before using it (2006-11-19)
http://python.org/sf/1599256 closed by akuchling
New / Reopened Bugs
___
GUI for Python 2.3, 2.4, and 2.5 is very sluggish (2006-12-06)
CLOSED http://python.org/sf/1610485 reopened by g4rlik
cgi.py multipart/form-data (2006-12-07)
http://python.org/sf/1610654 opened by Chui Tey
\b in unicode regex gives strange results (2006-12-07)
CLOSED http://python.org/sf/1611131 opened by akaihola
os.path.exists("file/") failure on Solaris 9 (2006-12-07)
http://python.org/sf/1611154 opened by Paul Eggert
can't pickle NAN's in binary mode (2006-12-08)
CLOSED http://python.org/sf/1611753 opened by Wayne Christopher
sndhdr.what() does not recognize wav file (2006-12-09)
http://python.org/sf/1611944 reopened by klankschap
sndhdr.what() does not recognize wav file (2006-12-09)
http://python.org/sf/1611944 opened by Floris van Manen
builtin compile() doc needs PyCF_DONT_IMPLY_DEDENT (2006-12-09)
http://python.org/sf/1612012 opened by Anthony Baxter
Dictionary ordering docs are too unclear of dangers (2006-12-09)
http://python.org/sf/1612113 opened by Calvin Spealman
webchecker/urllib chokes on 404 pages (2006-12-10)
CLOSED http://python.org/sf/1612729 opened by Fredrik Lundh
lambda tuple parameter bus error (2006-12-11)
CLOSED http://python.org/sf/1613059 opened by Bruce Cropley
str.split creates new string even if pattern not found (2006-12-11)
http://python.org/sf/1613130 opened by Antoine Pitrou
pydoc info for a package doesn't list all package contents (2006-12-11)
http://python.org/sf/1613479 opened by Nishkar Grover
xmlrpclib ServerProxy uses old httplib interface (2006-12-12)
http://python.org/sf/1613573 opened by Matt Brown
recv_into not documented (2006-12-11)
http://python.org/sf/1613651 opened by Eric Huss
AttributesImpl does not implement __contains__ on Linux (2006-12-13)
http://python.org/sf/1614387 opened by Jason Briggs
dict throwing inaccurate KeyError on small tuple keys (2006-12-13)
CLOSED http://python.org/sf/1614429 opened by toidinamai
python-logging compatability with Zope. (2006-12-13)
http://python.org/sf/1614460 opened by Simon Hookway
tempile.TemporaryFile differences between linux and windows (2006-12-13)
http://python.org/sf/1615275 opened by hirzel
subprocess doesn't handle SIGPIPE (2006-12-14)
http://python.org/sf/1615376 opened by Mark Diekhans
IA64/AMD64/x64 confusion
