Re: [Python-Dev] Hooking into super() attribute resolution

2013-07-08 Thread Steve Dower 
The only real advantage is a simpler signature and more easily explained use 
(assuming the person you're explaining it to is familiar with metaclasses, so 
most of the hard explaining has been done).

I'm still not sure that this isn't simply a bug in super. If the superclass's 
metaclass provides a __getattr__ then it should probably use it and abandon 
it's own MRO traversal.

I still haven't thought the edge cases through, and it seems like there'd be 
some with that change, so that's where __getattribute_super__ comes in - super 
can call it without abandoning its MRO traversal.

AFAICT, the difference between that and __getlocalattribute__ is that the 
latter would be implemented on a metaclass while the former takes extra 
parameters. I think this functionality is advanced enough that requiring a 
metaclass isn't unreasonable.

(The proxy objects idea was a red herring, sorry :) )

Steve

Sent from my Windows Phone

From: Ronald Oussoren
Sent: ‎7/‎7/‎2013 12:37
To: Steve Dower
Cc: [email protected]
Subject: Re: [Python-Dev] Hooking into super() attribute resolution


On 7 Jul, 2013, at 17:17, Steve Dower  wrote:

> Could the same result be achieved by hooking the MRO that super uses and 
> returning a list of proxy objects?

What is the advantage over adding a hook to the class itself? That seems to be 
the right place to add such a hook, super already looks in the classes along 
the MRO and my proposal would add a formal interface for that instead of having 
super peek into the class __dict__. I have thought of using a custom mapping 
object for the tp_dict slot to intercept this, but that won't work because 
super assumes that tp_dict is an actual PyDictObject (and likely other parts of 
the interpreter do so as well).


> And then wouldn't you only really need a __getattribute__ that doesn't 
> recurse (__getlocalattribute__)? The end result may be conceptually simpler, 
> but you've thought through the edge cases better than I have.

__getattribute_super__ already is a kind of __getlocalattribute__, the primairy 
difference being __getattribute_super__ is a staticmethod instead of an 
instance method. To be honest I'm not sure if a staticmethod is the right 
solution, I'm having a hard time to determine if this should be a class, 
instance or static method.

Currently super(StartClass, x) basicly does (assuming x is an instance method):


def __getattribute__(self, name):
mro = type(x).mro()
idx = mro.index(StartClass)
while idx < len(mro):
   dct = mro[idx].__dict__
   try:
  result = dct[name]
  # deal with descriptors here
  return result

   except KeyError:
   continue
return object.__getattribute__(self, name)

With my proposal 'dct' would no longer be needed and 'result = dct[name]' would 
be 'mro[idx].__getattribute_super__(mro[idx], name, x, StartClass)' (I may have 
the last argument for the call to __getattribute_super__ wrong, but that's the 
idea). Given that the first argument of __get...super__ is the same as the 
object the method get getattr-ed from I guess the method should be a 
classmethod instead of an staticmethod. Changing that would be easy enough.

I'm still interested in feedback on the basic idea, I'd love to here that my 
proposal isn't necessary because there is already a way to get the behavior I'm 
looking for although that's nog going to happen ;-).

Ronald


>
> (Apologies for the HTML top-post)

I don't mind.

PS. Does anyone know if the pep editors are away (conferences, holidays, ...)? 
I could just check in my proposal in the peps repository, but as this is my 
first PEP I'd prefer to follow the documented procedure and have someone that 
knows what he's doing look at the metadata before checking in.

>
> Sent from my Windows Phone
> From: Ronald Oussoren
> Sent: ‎7/‎6/‎2013 0:47
> To: Ronald Oussoren
> Cc: [email protected] Dev
> Subject: Re: [Python-Dev] Hooking into super() attribute resolution
>
> I've updated the implementation in issue 18181 
>  while adding some tests, and have updated 
> the proposal as well.
>
> The proposal has some open issues at the moment, most important of which is 
> the actual signature for the new special method; in particular I haven't been 
> able to decide if this should be an instance-, class- or static method. It is 
> a static method in the proposal and prototype, but I'm not convinced that 
> that is the right solution.
>
> Ronald
>
>
>
>
> PEP: TODO
> Title: Hooking into super attribute resolution
> Version: $Revision$
> Last-Modified: $Date$
> Author: Ronald Oussoren 
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 12-Jun-2013
> Post-History: 2-Jul-2013, ?
>
>
> Abstract
> 
>
> In current python releases the attribute resolution of the `super class`_
> p

Re: [Python-Dev] Hooking into super() attribute resolution

2013-07-08 Thread Ronald Oussoren 

On 8 Jul, 2013, at 17:19, Steve Dower  wrote:

> The only real advantage is a simpler signature and more easily explained use 
> (assuming the person you're explaining it to is familiar with metaclasses, so 
> most of the hard explaining has been done).

The signature is as complex as it is to be able to call descr.__get__ with the 
correct arguments. I ended up with the current signature when I added 
__getattribute_super__ to object and removed the tp_dict peeking code from 
super's tp_getattro.

A way to get a simpler interface again would be a method that returns an 
attribute *without* performing calls to descr.__get__. That could then be used 
for both __getattribute__ and super.__getattribute__, instead of peeking in a 
class' dictionary. I must admit that I haven't thought about the ramifactions 
of this (both functionally and performance wise).  This might end up being 
easier to explain: both normal attribute resolution and super's resolution 
would end up using the same mechanism, with the differences being that super 
doesn't begin resolution at the start of the mro and ignores the instance 
__dict__.  The disadvantage is introducing a new way to affect attribute 
resolution (do I use "__getattribute__" or this new method?). 

The new interface would be something like:

@classmethod
def __getlocalname__(cls, object, name):
pass

Or as you mentioned later as a __getlocalname__ method on the metaclass. The 
"object" argument wouldn't be necessary to reproduce current functionality, and 
isn't necessary for my usecase as well, but a hook for attribute resolution on 
an instance that doesn't have access to that instance feels wrong.

> 
> I'm still not sure that this isn't simply a bug in super. If the superclass's 
> metaclass provides a __getattr__ then it should probably use it and abandon 
> it's own MRO traversal.

I'd have to think about this, but on first glance this would mean a change in 
the semantics that a metaclass' __getattr__ currently has.

> 
> I still haven't thought the edge cases through, and it seems like there'd be 
> some with that change, so that's where __getattribute_super__ comes in - 
> super can call it without abandoning its MRO traversal.
> 
> AFAICT, the difference between that and __getlocalattribute__ is that the 
> latter would be implemented on a metaclass while the former takes extra 
> parameters. I think this functionality is advanced enough that requiring a 
> metaclass isn't unreasonable.

I'm not necessarily oppossed to a solution that requires using a metaclass, I 
already have metaclasses with custom metaclasses in PyObjC and this wouldn't 
add that much complexity to that :-)

Ronald

___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Rough idea for adding introspection information for builtins

2013-07-08 Thread Nick Coghlan 
Since it's relevant: my recollection us that the current use of angle
brackets in inspect.Signature is just the default use of them for "there is
no canonical representation of this, but leaving them out would be
misleading" (I haven't checked if the PEP says that explicitly).

I previously forgot Larry, Guido & I discussed the appropriate use of
square brackets and the slash in the definition format at PyCon, so I now
think having the Argument Clinic PEP also cover their appropriate use in
the inspect.Signature string output is a good idea.

Cheers,
Nick.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Hooking into super() attribute resolution

2013-07-08 Thread Steve Dower 
> From: Ronald Oussoren [mailto:[email protected]]
> Sent: Monday, July 8, 2013 0858
>
> On 8 Jul, 2013, at 17:19, Steve Dower  wrote:
> 
> > The only real advantage is a simpler signature and more easily explained
> use (assuming the person you're explaining it to is familiar with metaclasses,
> so most of the hard explaining has been done).
> 
> The signature is as complex as it is to be able to call descr.__get__ with the
> correct arguments. I ended up with the current signature when I added
> __getattribute_super__ to object and removed the tp_dict peeking code
> from super's tp_getattro.
> 
> A way to get a simpler interface again would be a method that returns an
> attribute *without* performing calls to descr.__get__. That could then be
> used for both __getattribute__ and super.__getattribute__, instead of
> peeking in a class' dictionary. I must admit that I haven't thought about the
> ramifactions of this (both functionally and performance wise).  This might
> end up being easier to explain: both normal attribute resolution and super's
> resolution would end up using the same mechanism, with the differences
> being that super doesn't begin resolution at the start of the mro and ignores
> the instance __dict__.  The disadvantage is introducing a new way to affect
> attribute resolution (do I use "__getattribute__" or this new method?).
> 
> The new interface would be something like:
> 
> @classmethod
> def __getlocalname__(cls, object, name):
> pass
> 
> Or as you mentioned later as a __getlocalname__ method on the metaclass.
> The "object" argument wouldn't be necessary to reproduce current
> functionality, and isn't necessary for my usecase as well, but a hook for
> attribute resolution on an instance that doesn't have access to that instance
> feels wrong.

Except that if it's on a metaclass, the 'instance' it has access to is cls. The 
descriptor side of things is more interesting, but I see no reason why super 
can't do that itself, since it knows the actual instance to call __get__ with. 
(Presumably it already does this with the __dict__ lookup, since that won't 
call __get__ either.)

Explaining the new method is easiest if the default implementation is 
(literally):

def __getlocalname__(self, name):
try:
return self.__dict__[name]
except KeyError:
raise AttributeError(name)

which does not do any descriptor resolution (and is only a small step from 
simply replacing __dict__ with a custom object, which is basically where we 
started). The only change I've really suggested is making it an instance method 
that can be implemented on a metaclass if you want it for class members.

> >
> > I'm still not sure that this isn't simply a bug in super. If the 
> > superclass's
> metaclass provides a __getattr__ then it should probably use it and abandon
> it's own MRO traversal.
> 
> I'd have to think about this, but on first glance this would mean a change in
> the semantics that a metaclass' __getattr__ currently has.

Exactly. Probably not a great idea to change that.

> >
> > I still haven't thought the edge cases through, and it seems like there'd be
> some with that change, so that's where __getattribute_super__ comes in -
> super can call it without abandoning its MRO traversal.
> >
> > AFAICT, the difference between that and __getlocalattribute__ is that the
> latter would be implemented on a metaclass while the former takes extra
> parameters. I think this functionality is advanced enough that requiring a
> metaclass isn't unreasonable.
> 
> I'm not necessarily oppossed to a solution that requires using a metaclass, I
> already have metaclasses with custom metaclasses in PyObjC and this
> wouldn't add that much complexity to that :-)

I assumed you were - when I was working on similar sort of code they made life 
extremely easy. 

> Ronald

Steve


___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] pyfailmalloc: new debug tool to test how your applications handles MemoryError

2013-07-08 Thread Victor Stinner 
Hi,

The PEP 445 (Add new APIs to customize Python memory allocators) has
been accepted, I commited its implementation. So it's time to have fun
with this API.

I developed a small Python module (150 lines of C code) to inject
memory allocation failures:
https://pypi.python.org/pypi/pyfailmalloc

This module can be used to test your application under very low memory
condition, test how your application handles MemoryError. I ran it in
the Python test suite: Python core and stdlib do not handle
MemoryError correctly everywhere...

I opened the following issue to track all bugs found by pytracemalloc
and related commits to fix these bugs:
http://bugs.python.org/issue18408

I already fixed bugs in PyObject_GC_NewVar(), PyDict_New(),
list.pop(), type.__bases__ setter, etc. I also found a bug in
_PyMem_DebugRealloc(): a function supposed to detect bugs :-)

Have fun with this new tool on your applications and Python modules.
Tell me if you would like to help me on issue #18408, there are still
a lot of bugs to fix.

Victor
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pyfailmalloc: new debug tool to test how your applications handles MemoryError

2013-07-08 Thread Victor Stinner 
2013/7/9 Victor Stinner :
> I developed a small Python module (150 lines of C code) to inject
> memory allocation failures:
> https://pypi.python.org/pypi/pyfailmalloc

Bitbucket was down, so I was unable to give the link to its source
code. The server is back, here is the C code:
https://bitbucket.org/haypo/pyfailmalloc/src/tip/failmalloc.c

You can see that it's now easy to develop such tools thanks to the PEP 445.

Victor
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com