[Python-Dev] ctypes: is it intentional that id() is the only way to get the address of an object?
Disclaimer: I'm not a ctypes expert, so I might have this completely wrong. If so, I apologise for the noise. The id() function is documented as returning an abstract ID number. In CPython, that happens to have been implemented as the address of the object. I understand that the only way to pass the address of an object to ctypes is to use that id. Is that intentional? As I see it, there is a conflict between two facts: - that id() returns a memory address is an implementation detail; as such users should not rely on it, as the implementation could (in principle) change without notice; - but users using ctypes have no choice but to rely on id() returning the object memory address, as of it were an offical part of the API. Implementations like PyPy which emulate ctypes, while objects don't have fixed memory locations, will surely have a problem here. I don't know how PyPy solves this. Have I misunderstood something here? -- Steve ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ctypes: is it intentional that id() is the only way to get the address of an object?
On Thu, 17 Jan 2019 21:26:06 +1100 Steven D'Aprano wrote: > Disclaimer: I'm not a ctypes expert, so I might have this completely > wrong. If so, I apologise for the noise. > > The id() function is documented as returning an abstract ID number. In > CPython, that happens to have been implemented as the address of the > object. > > I understand that the only way to pass the address of an object to > ctypes is to use that id. Is that intentional? Can you explain in detail what you're doing? If you're calling a C API taking a PyObject*, it seems like you should be using ctypes.py_object as argument type specifier. Various examples can be found with Google. Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pickle5 backport updated
On Wed, 16 Jan 2019 at 20:38, Antoine Pitrou wrote: > > On Wed, 16 Jan 2019 11:25:04 +0100 > Victor Stinner wrote: > > I see that the PEP is still a draft. What's the status? > > PEP 574 is ready for pronouncement. It's waiting for an authority to > approve it (or decide on a PEP delegate, who IMHO should probably > be Nick Coghlan). Aye, and I had already taken on that role unofficially - IIRC, the only major change I requested was the one to allow buffer handling callbacks to ask the pickler to include the buffer in-line after all [1]. It was just that the BDFL-Delegate appointment process was put on hold after Guido stepped down. Cheers, Nick. [1] https://github.com/python/peps/commit/1ed3043948527893f277d6451a315a05bde123ec#diff-4576f13d94059bcccb9205ad5005f96c -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ctypes: is it intentional that id() is the only way to get the address of an object?
On 1/17/19, Steven D'Aprano wrote: > > I understand that the only way to pass the address of an object to > ctypes is to use that id. Is that intentional? It's kind of dangerous to pass an object to C without an increment of its reference count. The proper way is to use a simple pointer of type "O" (object), which is already created for you as the "py_object" type. >>> ctypes.py_object._type_ 'O' >>> ctypes.py_object.__bases__ (,) It keeps a reference in the readonly _objects attribute. For example: >>> b = bytearray(b'spam') >>> sys.getrefcount(b) 2 >>> cb = ctypes.py_object(b) >>> sys.getrefcount(b) 3 >>> cb._objects bytearray(b'spam') >>> del cb >>> sys.getrefcount(b) 2 If you need the address without relying on id(), cast to a void pointer: >>> ctypes.POINTER(ctypes.c_void_p)(cb)[0] == id(b) True Or instantiate a c_void_p from the py_object as a buffer: >>> ctypes.c_void_p.from_buffer(cb).value == id(b) True Note that ctypes.cast() doesn't work in this case. It's implemented as an FFI function that takes the object address as a void pointer. The from_param method of c_void_p doesn't support py_object: >>> ctypes.c_void_p.from_param(cb) Traceback (most recent call last): File "", line 1, in TypeError: wrong type ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ctypes: is it intentional that id() is the only way to get the address of an object?
I've heard that libraries using ctypes, cffi, or cython code of various sorts in the real world wild today does abuse the unfortunate side effect of CPython's implementation of id(). I don't have specific instances of this in mind but trust what I've heard: that it is happening. id() should never be considered to be the PyObject*. In as much as code shouldn't assume it is running on top of a specific CPython implementation. If there is a _need_ to get a pointer to a C struct handle referencing a CPython C API PyObject, we should make an explicit API for that rather than the id() hack. That way code can be explicit about its need, and code that is just doing a funky form of identity tracking without using is and is not can continue using id() without triggering regressive behavior on VMs that don't have a CPython compatible PyObject under the hood by default. [who uses id() anyways?] -gps On Thu, Jan 17, 2019 at 2:26 AM Steven D'Aprano wrote: > Disclaimer: I'm not a ctypes expert, so I might have this completely > wrong. If so, I apologise for the noise. > > The id() function is documented as returning an abstract ID number. In > CPython, that happens to have been implemented as the address of the > object. > > I understand that the only way to pass the address of an object to > ctypes is to use that id. Is that intentional? > > As I see it, there is a conflict between two facts: > > - that id() returns a memory address is an implementation detail; as > such users should not rely on it, as the implementation could (in > principle) change without notice; > > - but users using ctypes have no choice but to rely on id() returning > the object memory address, as of it were an offical part of the API. > > Implementations like PyPy which emulate ctypes, while objects don't have > fixed memory locations, will surely have a problem here. I don't know > how PyPy solves this. > > Have I misunderstood something here? > > > > -- > Steve > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/greg%40krypto.org > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ctypes: is it intentional that id() is the only way to get the address of an object?
On Fri, Jan 18, 2019 at 11:50 AM Gregory P. Smith wrote: > > I've heard that libraries using ctypes, cffi, or cython code of various sorts > in the real world wild today does abuse the unfortunate side effect of > CPython's implementation of id(). I don't have specific instances of this in > mind but trust what I've heard: that it is happening. > > id() should never be considered to be the PyObject*. In as much as code > shouldn't assume it is running on top of a specific CPython implementation. > If there is a _need_ to get a pointer to a C struct handle referencing a > CPython C API PyObject, we should make an explicit API for that rather than > the id() hack. That way code can be explicit about its need, and code that > is just doing a funky form of identity tracking without using is and is not > can continue using id() without triggering regressive behavior on VMs that > don't have a CPython compatible PyObject under the hood by default. > I would be strongly in favour of ctypes gaining a "get address of object" function, which happens (in current CPythons) to return the same value as id() does, but is specifically tied to ctypes. ChrisA ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ctypes: is it intentional that id() is the only way to get the address of an object?
On 2019-01-18 00:48, Gregory P. Smith wrote: I've heard that libraries using ctypes, cffi, or cython code of various sorts in the real world wild today does abuse the unfortunate side effect of CPython's implementation of id(). I don't have specific instances of this in mind but trust what I've heard: that it is happening. id() should never be considered to be the PyObject*. In as much as code shouldn't assume it is running on top of a specific CPython implementation. If there is a _need_ to get a pointer to a C struct handle referencing a CPython C API PyObject, we should make an explicit API for that rather than the id() hack. That way code can be explicit about its need, and code that is just doing a funky form of identity tracking without using is and is not can continue using id() without triggering regressive behavior on VMs that don't have a CPython compatible PyObject under the hood by default. [who uses id() anyways?] I use it in some of my code. If I want to cache some objects, I put them in a dict, using the id as the key. If I wanted to locate an object in a cache and didn't have id(), I'd have to do a linear search for it. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ctypes: is it intentional that id() is the only way to get the address of an object?
n Thu, Jan 17, 2019 at 4:51 PM Gregory P. Smith wrote: > > I've heard that libraries using ctypes, cffi, or cython code of various sorts > in the real world wild today does abuse the unfortunate side effect of > CPython's implementation of id(). I don't have specific instances of this in > mind but trust what I've heard: that it is happening. IME it's reasonably common with ctypes, for cases where you need to do some gross hack and there's no other option. Here's an example in jinja2: https://github.com/pallets/jinja/blob/9fe9520f2daa1df6079b188adba758d6e03d6af2/jinja2/debug.py#L350 I haven't seen it with cffi or cython. (cffi explicitly doesn't provide any way to access the CPython C API, and in cython you can just cast an object to a pointer.) > id() should never be considered to be the PyObject*. In as much as code > shouldn't assume it is running on top of a specific CPython implementation. > If there is a _need_ to get a pointer to a C struct handle referencing a > CPython C API PyObject, we should make an explicit API for that rather than > the id() hack. That way code can be explicit about its need, and code that > is just doing a funky form of identity tracking without using is and is not > can continue using id() without triggering regressive behavior on VMs that > don't have a CPython compatible PyObject under the hood by default. Using id() like this is certainly offensive to our sensibilities, but in practice I don't see how it causes much harm. If you are doing *anything* with PyObject*, then you're tying yourself to implementation details of CPython (and usually a specific version of CPython). That's not great, but at that point relying on CPython's implementation of id() is the least of your worries, and it tends to be a self-correcting problem. -n -- Nathaniel J. Smith -- https://vorpus.org ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ctypes: is it intentional that id() is the only way to get the address of an object?
For everyone who managed to reply *hours* after Eryk Sun posted the correct answer and still get it wrong, here it is again in full. As a bonus, here's a link to the place where this answer appears in the documentation: https://docs.python.org/3/library/ctypes.html#ctypes.py_object Cheers, Steve On 17Jan.2019 0550, eryk sun wrote: > On 1/17/19, Steven D'Aprano wrote: >> >> I understand that the only way to pass the address of an object to >> ctypes is to use that id. Is that intentional? > > It's kind of dangerous to pass an object to C without an increment of > its reference count. The proper way is to use a simple pointer of type > "O" (object), which is already created for you as the "py_object" > type. > > >>> ctypes.py_object._type_ > 'O' > >>> ctypes.py_object.__bases__ > (,) > > It keeps a reference in the readonly _objects attribute. For example: > > >>> b = bytearray(b'spam') > >>> sys.getrefcount(b) > 2 > >>> cb = ctypes.py_object(b) > >>> sys.getrefcount(b) > 3 > >>> cb._objects > bytearray(b'spam') > >>> del cb > >>> sys.getrefcount(b) > 2 > > If you need the address without relying on id(), cast to a void pointer: > > >>> ctypes.POINTER(ctypes.c_void_p)(cb)[0] == id(b) > True > > Or instantiate a c_void_p from the py_object as a buffer: > > >>> ctypes.c_void_p.from_buffer(cb).value == id(b) > True > > Note that ctypes.cast() doesn't work in this case. It's implemented as > an FFI function that takes the object address as a void pointer. The > from_param method of c_void_p doesn't support py_object: > > >>> ctypes.c_void_p.from_param(cb) > Traceback (most recent call last): > File "", line 1, in > TypeError: wrong type ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ctypes: is it intentional that id() is the only way to get the address of an object?
I feel like I should clarify - not everyone who posted got it wrong, and I understand there's a side discussion among those who are also interested/participants in https://discuss.python.org/t/demoting-the-is-operator-to-avoid-an-identity-crisis/86/ - but there was no of acknowledgement of Eryk Sun's correct and useful answer which I find very disappointing and a great way to discourage contributions. We can, and should, do better, at least by thanking the person for their response before running down a barely related side track. On 17Jan.2019 2209, Steve Dower wrote: > For everyone who managed to reply *hours* after Eryk Sun posted the > correct answer and still get it wrong, here it is again in full. > > As a bonus, here's a link to the place where this answer appears in the > documentation: > https://docs.python.org/3/library/ctypes.html#ctypes.py_object > > Cheers, > Steve > > On 17Jan.2019 0550, eryk sun wrote: >> On 1/17/19, Steven D'Aprano wrote: >>> >>> I understand that the only way to pass the address of an object to >>> ctypes is to use that id. Is that intentional? >> >> It's kind of dangerous to pass an object to C without an increment of >> its reference count. The proper way is to use a simple pointer of type >> "O" (object), which is already created for you as the "py_object" >> type. >> >> >>> ctypes.py_object._type_ >> 'O' >> >>> ctypes.py_object.__bases__ >> (,) >> >> It keeps a reference in the readonly _objects attribute. For example: >> >> >>> b = bytearray(b'spam') >> >>> sys.getrefcount(b) >> 2 >> >>> cb = ctypes.py_object(b) >> >>> sys.getrefcount(b) >> 3 >> >>> cb._objects >> bytearray(b'spam') >> >>> del cb >> >>> sys.getrefcount(b) >> 2 >> >> If you need the address without relying on id(), cast to a void pointer: >> >> >>> ctypes.POINTER(ctypes.c_void_p)(cb)[0] == id(b) >> True >> >> Or instantiate a c_void_p from the py_object as a buffer: >> >> >>> ctypes.c_void_p.from_buffer(cb).value == id(b) >> True >> >> Note that ctypes.cast() doesn't work in this case. It's implemented as >> an FFI function that takes the object address as a void pointer. The >> from_param method of c_void_p doesn't support py_object: >> >> >>> ctypes.c_void_p.from_param(cb) >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: wrong type ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
