Re: [Python-Dev] namedtuple implementation grumble
On 6/7/2014 10:46 AM, Nick Coghlan wrote: > On 7 June 2014 04:50, Chris Withers wrote: >> Curious as to what lead to that implementation approach? What does it buy >> that couldn't have been obtained by a mixin providing the functionality? > > In principle, you could get the equivalent of collections.namedtuple > through dynamically constructed classes. In practice, that's actually > easier said than done, so the fact the current implementation works > fine for almost all purposes acts as a powerful disincentive to > rewriting it. The current implementation is also *really* easy to > understand, while writing out the dynamic type creation explicitly > would likely require much deeper knowledge of the type machinery to > follow. As proof that it's harder to understand, here's an example of that dynamically creating functions and types: https://pypi.python.org/pypi/namedlist https://bitbucket.org/ericvsmith/namedlist/src/163d0d05e94f9cc0af8e269015b9ac3bf9a83826/namedlist.py?at=default#cl-155 It uses the ast module to build an __init__ (or __new__) function dynamically, without exec. Then it creates a type using that function to initialize the new type. namedlist.namedtuple passes all collections.namedtuple tests, except for those using the _source attribute (of course). namedlist.namedlist and namedlist.namedtuple both support a clunky interface to specify default values for member fields. The reasons I didn't use the collections.namedtuple exec-based approach are: - specify default values to __init__ or __new__ became very complex - 2.x and 3.x support is harder with exec Eric. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] namedtuple implementation grumble
On Sun, Jun 08, 2014 at 07:37:46PM +, [email protected] wrote: > cls = tuple(name, (_NamedTuple,), { Ugh, this should of course have been type(). David ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] namedtuple implementation grumble
On Sun, Jun 08, 2014 at 03:13:55PM -0400, Eric V. Smith wrote:
> > The current implementation is also *really* easy to understand,
> > while writing out the dynamic type creation explicitly would likely
> > require much deeper knowledge of the type machinery to follow.
> As proof that it's harder to understand, here's an example of that
> dynamically creating functions and types:
Probably I'm missing something, but there's a much simpler non-exec
approach, something like:
class _NamedTuple(...):
...
def namedtuple(name, fields):
cls = tuple(name, (_NamedTuple,), {
'_fields': fields.split()
})
for i, field_name in enumerate(cls._fields):
prop = property(functools.partial(_NamedTuple.__getitem__, i)
functools.partial(_NamedTuple.__setitem__, i))
setattr(cls, field_name, prop)
return cls
David
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] namedtuple implementation grumble
On 6/8/2014 3:37 PM, [email protected] wrote: > On Sun, Jun 08, 2014 at 03:13:55PM -0400, Eric V. Smith wrote: > >>> The current implementation is also *really* easy to understand, >>> while writing out the dynamic type creation explicitly would likely >>> require much deeper knowledge of the type machinery to follow. > >> As proof that it's harder to understand, here's an example of that >> dynamically creating functions and types: > > Probably I'm missing something, but there's a much simpler non-exec > approach, something like: > > class _NamedTuple(...): > ... > > def namedtuple(name, fields): > cls = tuple(name, (_NamedTuple,), { > '_fields': fields.split() > }) > for i, field_name in enumerate(cls._fields): > prop = property(functools.partial(_NamedTuple.__getitem__, i) > functools.partial(_NamedTuple.__setitem__, i)) > setattr(cls, field_name, prop) > return cls How would you write _Namedtuple.__new__? ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] namedtuple implementation grumble
On Sun, Jun 08, 2014 at 05:27:41PM -0400, Eric V. Smith wrote:
> How would you write _Namedtuple.__new__?
Knew something must be missing :) Obviously it's possible, but not
nearly as efficiently as reusing the argument parsing machinery as in
the original implementation.
I guess especially the kwargs implementation below would suck..
_undef = object()
class _NamedTuple(...):
def __new__(cls, *a, **kw):
if kw:
a = list(a) + ([_undef] * (len(self._fields)-len(a)))
for k, v in kw.iteritems():
i = cls._name_id_map[k]
if a[i] is not _undef:
raise TypeError(...)
a[i] = v
if _undef not in a:
return tuple.__new__(cls, a)
raise TypeError(...)
else:
if len(a) == len(self._fields):
return tuple.__new__(cls, a)
raise TypeError(...)
def namedtuple(name, fields):
fields = fields.split()
cls = type(name, (_NamedTuple,), {
'_fields': fields,
'_name_id_map': {k: i for i, k in enumerate(fields)}
})
for i, field_name in enumerate(fields):
getter = functools.partial(_NamedTuple.__getitem__, i)
setattr(cls, field_name, property(getter))
return cls
David
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] namedtuple implementation grumble
On Sat, 07 Jun 2014 10:50:16 -0400, Antoine Pitrou wrote: > Le 07/06/2014 09:25, R. David Murray a écrit : > > On Fri, 06 Jun 2014 19:50:57 +0100, Chris Withers > > wrote: > >> I've been trying to add support for explicit comparison of namedtuples > >> into testfixtures and hit a problem which lead me to read the source and > >> be sad. > >> > >> Rather than the mixin and class assembly in the function I expected to > >> find, I'm greeted by an exec of a string. > >> > >> Curious as to what lead to that implementation approach? What does it > >> buy that couldn't have been obtained by a mixin providing the > >> functionality? > >> > >> In my case, that's somewhat irrelevant, I'm looking to store a comparer > >> in a registry that would get used for all namedtuples, but I have > >> nothing to key that off, there are no shared bases other than object and > >> tuple. > >> > >> I guess I could duck-type it based on the _fields attribute but that > >> feels implicit and fragile. > >> > >> What do you guys suggest? > > > > I seem to remember a previous discussion that concluded that duck typing > > based on _fields was the way to go. (It's a public API, despite the _, > > due to name-tuple's attribute namespacing issues.) > > There could be many third-party classes with a _fields member, so that > sounds rather fragile. > There doesn't seem to be any technical reason barring the addition of a > common base class for namedtuples. For what it is worth, I found the discussion I was remembering: http://bugs.python.org/issue7796 And as someone pointed out down thread, the actual check is "inherits from tuple and has a _fields attribute". That gets you a duck type, which is generally what you want in Python. --David ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] namedtuple implementation grumble
Le 08/06/2014 18:44, R. David Murray a écrit : For what it is worth, I found the discussion I was remembering: http://bugs.python.org/issue7796 And as someone pointed out down thread, the actual check is "inherits from tuple and has a _fields attribute". That gets you a duck type, which is generally what you want in Python. I think it's a bit complicated (and not obviously discoverable) as far as duck-typing goes. Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] namedtuple implementation grumble
On Sun, Jun 08, 2014 at 03:13:55PM -0400, Eric V. Smith wrote: > On 6/7/2014 10:46 AM, Nick Coghlan wrote: > > On 7 June 2014 04:50, Chris Withers wrote: > >> Curious as to what lead to that implementation approach? What does it buy > >> that couldn't have been obtained by a mixin providing the functionality? > > > > In principle, you could get the equivalent of collections.namedtuple > > through dynamically constructed classes. In practice, that's actually > > easier said than done, so the fact the current implementation works > > fine for almost all purposes acts as a powerful disincentive to > > rewriting it. The current implementation is also *really* easy to > > understand, while writing out the dynamic type creation explicitly > > would likely require much deeper knowledge of the type machinery to > > follow. > > As proof that it's harder to understand, here's an example of that > dynamically creating functions and types: [...] I wonder how a hybrid approach would work? Use a dynamically-created class, but then construct the __new__ method using exec and inject it into the new class. As far as I can see, it's only __new__ that benefits from the exec approach. Anyone tried this yet? Is it worth an experiment? -- Steven ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] namedtuple implementation grumble
On Jun 7, 2014, at 6:25 AM, R. David Murray wrote: >> I guess I could duck-type it based on the _fields attribute but that >> feels implicit and fragile. >> >> What do you guys suggest? > > I seem to remember a previous discussion that concluded that duck typing > based on _fields was the way to go. (It's a public API, despite the _, > due to name-tuple's attribute namespacing issues.) Yes. That is the recommended approach. IIRC that was Guido's suggestion rather than creating an abstract base class for a named tuple (any tuple-like class with indexable elements that are also accessible using named attributes). Raymond ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] namedtuple implementation grumble
On 6/8/2014 7:31 PM, Steven D'Aprano wrote: > On Sun, Jun 08, 2014 at 03:13:55PM -0400, Eric V. Smith wrote: >> On 6/7/2014 10:46 AM, Nick Coghlan wrote: >>> On 7 June 2014 04:50, Chris Withers wrote: Curious as to what lead to that implementation approach? What does it buy that couldn't have been obtained by a mixin providing the functionality? >>> >>> In principle, you could get the equivalent of collections.namedtuple >>> through dynamically constructed classes. In practice, that's actually >>> easier said than done, so the fact the current implementation works >>> fine for almost all purposes acts as a powerful disincentive to >>> rewriting it. The current implementation is also *really* easy to >>> understand, while writing out the dynamic type creation explicitly >>> would likely require much deeper knowledge of the type machinery to >>> follow. >> >> As proof that it's harder to understand, here's an example of that >> dynamically creating functions and types: > [...] > > > I wonder how a hybrid approach would work? Use a dynamically-created > class, but then construct the __new__ method using exec and inject it > into the new class. As far as I can see, it's only __new__ that benefits > from the exec approach. > > Anyone tried this yet? Is it worth an experiment? I'm not sure what the benefit would be. Other than the ast manipulations for __new__, the rest of the non-exec code is easy to understand. Eric. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] namedtuple implementation grumble
On 9 Jun 2014 10:04, "Raymond Hettinger" wrote: > > > On Jun 7, 2014, at 6:25 AM, R. David Murray wrote: > >>> I guess I could duck-type it based on the _fields attribute but that >>> feels implicit and fragile. >>> >>> What do you guys suggest? >> >> >> I seem to remember a previous discussion that concluded that duck typing >> based on _fields was the way to go. (It's a public API, despite the _, >> due to name-tuple's attribute namespacing issues.) > > > Yes. That is the recommended approach. > > IIRC that was Guido's suggestion rather than creating an abstract > base class for a named tuple (any tuple-like class with indexable > elements that are also accessible using named attributes). Given the somewhat periodic recurrence of the question, might it be worth making an ABC after all, with "subclass of tuple with a _fields attribute" as its default check? "isinstance(obj, collections.NamedTupleABC)" is quite a bit more self-documenting than "isinstance(obj, tuple) and hasattr(obj, '_fields')" Cheers, Nick. > > > Raymond > > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] namedtuple implementation grumble
On Jun 8, 2014, at 6:42 PM, Nick Coghlan wrote: > >> I seem to remember a previous discussion that concluded that duck typing > >> based on _fields was the way to go. (It's a public API, despite the _, > >> due to name-tuple's attribute namespacing issues.) > > > > > > Yes. That is the recommended approach. > > > > IIRC that was Guido's suggestion rather than creating an abstract > > base class for a named tuple (any tuple-like class with indexable > > elements that are also accessible using named attributes). > > Given the somewhat periodic recurrence of the question, might it be worth > making an ABC after all, with "subclass of tuple with a _fields attribute" as > its default check? > > "isinstance(obj, collections.NamedTupleABC)" is quite a bit more > self-documenting than "isinstance(obj, tuple) and hasattr(obj, '_fields')" > The "isinstance(obj, tuple)" part isn't a requirement. The concept of a named tuple is meant to include structseq objects or user defined classes that have are "tuple-like with indexable elements that are also accessible using named attributes" (see the definition in the glossary). I could add a note to the docs saying that hasattr(obj, '_fields') is the preferred way to check for named tuples produced by the namedtuple() factory function, but it would be a waste to introduce an ABC for this. (Consider the failure of the Callable() abc leading to us deciding to reintroduce the callable() builtin function, and consider the general unwillingness to test for iterability using the Iterable abc). Another issue is that a straight abc wouldn't be sufficient. What we would really want is to check for is: 1) the presence of a _fields tuple (an abc can do this) 2) to check that all of the attribute names specified in _fields are defined (ABCMeta doesn't do this) 3) and that the type is a Sequence (ABCMeta can do this). An tricked-out ABC extension might be worth it if it provided some non-trivial mixin capabilities for implementing homegrown named tuples (not created by the factory function), but I don't think we want to go there. The problem isn't important enough to warrant throwing this much code and a new API at it (duck-typing the attributes and checking for _fields is a practical solution that works even on older pythons). Raymond ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Help with the build system and my first patch
Hello, I would like some help understanding the build system. I am currently working on an issue (http://bugs.python.org/issue21627) and plan to create some common functionality in Python/setcloexec.c and Include/setcloexec.h that is conditionally compiled in on POSIX systems and not on Windows systems. I need to extract this functionality out from _Py_set_inheritable because it needs to run in the dangerous context of right after a fork and I don't believe it can throw exceptions. How can I conditionally compile some library code for certain platforms only? Thank you, Steven Stewart-Gallus ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
