Re: [Python-Dev] SystemError: new style getargs format but argument is not a tuple
On 05.01.17 06:43, Terry Reedy wrote: On 1/1/2017 6:40 PM, Serhiy Storchaka wrote: On 02.01.17 01:23, Terry Reedy wrote: There are several recent question on Stackoverflow about SystemError: new style getargs format but argument is not a tuple [snip] Resulting from using 3rd party packages. No one commenting has a clue. Is message from Victor's new calling code? Was it backported to 2.7? Could error be a result of running old 3rd party binary that needs to be recompiled? A system error "new style getargs format but argument is not a tuple" is not new. It means that PyArg_ParseTuple() is called with not a tuple as the first arguments. This is just a programmical error in third-party extension. Should the advice in the doc entry for SystemError be changed to include this case? "You should report this to the author or maintainer of your Python interpreter. Be sure to report the version of the Python interpreter (sys.version; it is also printed at the start of an interactive Python session), the exact error message (the exception’s associated value) and if possible the source of the program that triggered the error. " Yes, third-party extensions should be mentioned as a source of SystemError. But note that nobody reported this error even on CPython bugtracker. I now found questions about this error (related to OpenCV or Pillow) asked many years ago on different forums. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ctypes, memory mapped files and context manager
Hi Nick,
On Donnerstag, 5. Januar 2017 12:37:20 Nick Coghlan wrote:
> I don't know ctypes well enough myself to comment on the idea of
> offering fully deterministic cleanup, but the closest you could get to
> that without requiring a change to ctypes is to have the context
> manager introduce a layer of indirection:
>
> class _T_data(ctypes.Structure):
> _fields = [("foo", ctypes.c_uint32)]
>
> class T:
> def __init__(self, buffer):
> self.data = _T_data.from_buffer(buffer)
> def close(self):
> self.data = None
>
> @contextmanager
> def map_struct(m, n):
> m.resize(n * mmap.PAGESIZE)
> mapped = T(m)
> try:
> yield mapped
> finally:
> mapped.close()
>
> Client code would then need to consistently access the struct through
> the data attribute:
>
> with map_struct(m, 1) as a:
> a.data.foo = 1
> with map_struct(m, 2) as b:
> b.data.foo = 2
Thank you very much. Nice idea, indeed.
Here's a slightly more complex example incorporating your idea:
https://gist.github.com/frispete/97c27e24a0aae1bcaf1375e2e463d239#file-ctypes_mmap_ctx2-py
I'm not sure, if I like the resulting code more than the dreaded dels.
Real code based on this approach tends to be much more complex, and
suffers appropriately.
Anyway, your solution is working fine, and provides a choice.
Much appreciated, Nick.
@ctypes developers: with an unmap operation available, we wouldn't need
to go through these hoops, and ctypes powers would become even more
accessible for such cool use cases ;)...
For now, I'm about to resign from using a context manager at all, since
it uglifies the code in one way or another without buying much..
Cheers,
Pete
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ctypes, memory mapped files and context manager
On Thu, Jan 5, 2017 at 2:37 AM, Nick Coghlan wrote:
> On 5 January 2017 at 10:28, Hans-Peter Jansen wrote:
>> In order to get this working properly, the ctypes mapping needs a method to
>> free the mapping actively. E.g.:
>>
>> @contextmanager
>> def map_struct(m, n):
>> m.resize(n * mmap.PAGESIZE)
>> yield T.from_buffer(m)
>> T.unmap_buffer(m)
>>
>> Other attempts with weakref and the like do not work due to the nature of the
>> ctypes types.
>
> I don't know ctypes well enough myself to comment on the idea of
> offering fully deterministic cleanup, but the closest you could get to
> that without requiring a change to ctypes is to have the context
> manager introduce a layer of indirection:
I think that's the best you can do with the current state of ctypes.
from_buffer was made safer in Python 3 by ensuring it keeps a
memoryview reference in the _objects attribute (i.e.
CDataObject.b_objects). Hans-Peter's problem is a consequence of this
reference. Simply calling release() on the underlying memoryview is
unsafe. For example:
>>> b = bytearray(2**20)
>>> a = ctypes.c_char.from_buffer(b)
>>> a._objects
>>> a._objects.release()
>>> del b
>>> a.value
Segmentation fault (core dumped)
A release() method on ctypes objects could release the memoryview and
also clear the CDataObject b_ptr field. In this case, any function
that accesses b_ptr would have to be modified to raise a ValueError
for a NULL value. Currently ctypes assumes b_ptr is valid, so this
would require adding a lot of checks.
On a related note, ctypes objects aren't tracking the number of
exported views like they should. resize() should raise a BufferError
in the following example:
>>> b = (ctypes.c_char * (2**20))(255)
>>> m = memoryview(b).cast('B')
>>> m[0]
255
>>> ctypes.resize(b, 2**22)
>>> m[0]
Segmentation fault (core dumped)
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IRC logs via BotBot.me
On 01/05/2017 11:04 AM, Berker Peksağ wrote: > On Thu, Jan 5, 2017 at 2:27 AM, INADA Naoki wrote: >> Hi. >> >> IRC #python-dev channel is nice place to know what happens recently. >> But I can't log in always because I use only laptop PC. >> >> I found BotBot.me seems nice IRC log service and used by some major channels. >> https://botbot.me/ >> >> I wonder if #python-dev is logged by BotBot.me. >> >> I'm sorry if it had rejected already. > I'm using my personal VPS to stay online on Freenode. This looks like > a tooling issue on your end and it can be solved without introducing a > public logging service. > > --Berker I think it makes sense to solve this issue with a public log instead of requiring everyone to come up with their own solutions -- for the same reasons we have public mailing-list archives. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
On Wed, Oct 12, 2016 at 12:08 AM, INADA Naoki wrote: > > Now I'm sure about bytes.frombuffer() is worth enough. I would like to revive this thread (taking a liberty to shorten the subject line.) The issue of how the bytes(x) constructor should behave when given objects of various types have come up recently in issue 29159 (Regression in bytes constructor). [1] The regression was introduced in issue 27704 (bytes(x) is slow when x is bytearray) which attempted to speed-up creating bytes and bytearray from byte-like objects. I think the core problem is that the bytes(x) constructor tries to be the Jack of All Trades. Here is how it is documented in the docstring: | Construct an immutable array of bytes from: |- an iterable yielding integers in range(256) |- a text string encoded using the specified encoding |- any object implementing the buffer API. |- an integer | On the other hand, the reference manual while not having this description in the bytes section, has a similar list in the bytearray section. [3] """ The optional source parameter can be used to initialize the array in a few different ways: * If it is a string, you must also give the encoding (and optionally, errors) parameters; bytearray() then converts the string to bytes using str.encode(). * If it is an integer, the array will have that size and will be initialized with null bytes. * If it is an object conforming to the buffer interface, a read-only buffer of the object will be used to initialize the bytes array. * If it is an iterable, it must be an iterable of integers in the range 0 <= x < 256, which are used as the initial contents of the array. Without an argument, an array of size 0 is created. """ Note that the integer case is listed before buffer interface. Neither document mentions the possibility that the source type has a __bytes__ method. This ambiguity between integer-like and buffer-like sources causes a problem in the case when a 3rd party type is both integer-like and buffer-like. This is what happens with numpy arrays: >>> bytes(numpy.array([2], 'i1')) b'\x00\x00' >>> bytes(numpy.array([2, 2], 'i1')) b'\x02\x02' For better or worse, single-element numpy arrays have a working __index__ methods >>> numpy.array([2], 'i1').__index__() 2 and are interpreted as integers by the bytes(X) constructor. I propose the following: 1. For 3.6, restore and document 3.5 behavior. Recommend that 3rd party types that are both integer-like and buffer-like implement their own __bytes__ method to resolve the bytes(x) ambiguity. 2. For 3.7, I would like to see a drastically simplified bytes(x): 2.1. Accept only objects with a __bytes__ method or a sequence of ints in range(256). 2.2. Expand __bytes__ definition to accept optional encoding and errors parameters. Implement str.__bytes__(self, [encoding[, errors]]). 2.3. Implement new specialized bytes.fromsize and bytes.frombuffer constructors as per PEP 467 and Inada Naoki proposals. 2.4. Implement memoryview.__bytes__ method so that bytes(memoryview(x)) works ad before. 2.5. Implement a fast bytearray.__bytes__ method. 3. Consider promoting __bytes__ to a tp_bytes type slot. [1]: http://bugs.python.org/issue29159 [2]: http://bugs.python.org/issue27704 [3]: https://docs.python.org/3/library/functions.html#bytearray ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
On 05.01.17 22:37, Alexander Belopolsky wrote: I propose the following: 1. For 3.6, restore and document 3.5 behavior. Recommend that 3rd party types that are both integer-like and buffer-like implement their own __bytes__ method to resolve the bytes(x) ambiguity. The __bytes__ method is used only by the bytes constructor, not by the bytearray constructor. 2. For 3.7, I would like to see a drastically simplified bytes(x): 2.1. Accept only objects with a __bytes__ method or a sequence of ints in range(256). 2.2. Expand __bytes__ definition to accept optional encoding and errors parameters. Implement str.__bytes__(self, [encoding[, errors]]). I think it is better to use the encode() method if you want to encode from non-strings. 2.3. Implement new specialized bytes.fromsize and bytes.frombuffer constructors as per PEP 467 and Inada Naoki proposals. bytes.fromsize(n) is just b'\0'*n. I don't think this method is needed. bytes.frombuffer(x) is bytes(memoryview(x)) or memoryview(x).tobytes(). 2.4. Implement memoryview.__bytes__ method so that bytes(memoryview(x)) works ad before. 2.5. Implement a fast bytearray.__bytes__ method. This wouldn't help for the bytearray constructor. And wouldn't allow to avoid double copying in the constructor of bytes subclass. 3. Consider promoting __bytes__ to a tp_bytes type slot. The buffer protocol is more general than the __bytes__ method. It allows to avoid redundant memory copying in constructors of many types (bytes, bytearray, array.array, etc), not just bytes. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ctypes, memory mapped files and context manager
Hi Eryk, On Donnerstag, 5. Januar 2017 15:30:33 eryk sun wrote: > > > manager introduce a layer of indirection: > I think that's the best you can do with the current state of ctypes. > > from_buffer was made safer in Python 3 by ensuring it keeps a > memoryview reference in the _objects attribute (i.e. > CDataObject.b_objects). Hans-Peter's problem is a consequence of this > reference. Simply calling release() on the underlying memoryview is > > unsafe. For example: > >>> b = bytearray(2**20) > >>> a = ctypes.c_char.from_buffer(b) > >>> a._objects > > > > >>> a._objects.release() > >>> del b > >>> a.value > > Segmentation fault (core dumped) This is exactly, what I was after: @contextmanager def cstructmap(cstruct, mm, offset = 0): # resize the mmap (and backing file), if structure exceeds mmap size # mmap size must be aligned to mmap.PAGESIZE cssize = ctypes.sizeof(cstruct) if offset + cssize > mm.size(): newsize = align(offset + cssize, mmap.PAGESIZE) mm.resize(newsize) cmap = cstruct.from_buffer(mm, offset) try: yield cmap finally: for mv in cmap._objects.values(): mv.release() See also: https://gist.github.com/frispete/97c27e24a0aae1bcaf1375e2e463d239#file-ctypes_mmap_ctx3-py While technically possible (which is a surprise for me on its own), nothing should access the with variable after the block has finished. If that happens, a segfault is exactly was it deserves IMHO. Leaves the question, how stable this "interface" is? Accessing _objects here belongs to voodoo programming practices of course, but the magic is locally limited to just two lines of code, which is acceptable in order to get this context manager working without messing with the rest of the code. Opinions? Thanks, Pete ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ctypes, memory mapped files and context manager
On Freitag, 6. Januar 2017 00:28:37 Hans-Peter Jansen wrote: > Hi Eryk, > > This is exactly, what I was after: > > @contextmanager > def cstructmap(cstruct, mm, offset = 0): > # resize the mmap (and backing file), if structure exceeds mmap size > # mmap size must be aligned to mmap.PAGESIZE > cssize = ctypes.sizeof(cstruct) > if offset + cssize > mm.size(): > newsize = align(offset + cssize, mmap.PAGESIZE) > mm.resize(newsize) > cmap = cstruct.from_buffer(mm, offset) > try: > yield cmap > finally: > for mv in cmap._objects.values(): if isinstance(mv, memoryview): mv.release() It happens, that _objects contain other objects as well... Cheers, Pete ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
> > bytes.frombuffer(x) is bytes(memoryview(x)) or memoryview(x).tobytes(). > There is pitfall: memoryview should be closed. So b = bytes.frombuffer(x) is: with memoryview(x) as m: b = bytes(m) # or b = m.tobytes() ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
On 2017-01-05 7:11 PM, INADA Naoki wrote: bytes.frombuffer(x) is bytes(memoryview(x)) or memoryview(x).tobytes(). There is pitfall: memoryview should be closed. So b = bytes.frombuffer(x) is: with memoryview(x) as m: b = bytes(m) # or b = m.tobytes() Thinking more about this, and after looking at my own code in asyncpg and uvloop, I'm now in favor of adding bytes.frombuffer() with the proposed signature: ``bytes.frombuffer(byteslike, length=-1, offset=0)`` Inada-san is right, the memoryview should be explicitly released, but few do that. Instead, a lot of people simply rely on CPython refcounting semantics, which will cause the temporary memoryview be GCed asap. That won't work so flawlessly in PyPy and will cause hard to understand bugs. Yury ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ctypes, memory mapped files and context manager
On Thu, Jan 5, 2017 at 11:28 PM, Hans-Peter Jansen wrote: > Leaves the question, how stable this "interface" is? > Accessing _objects here belongs to voodoo programming practices of course, but > the magic is locally limited to just two lines of code, which is acceptable in > order to get this context manager working without messing with the rest of the > code. My intent was not to suggest that anyone directly use the _objects value / dict in production code. It's a private implementation detail. I was demonstrating the problem of simply releasing the buffer and the large number of checks that would be required if b_ptr is cleared. It would be simpler for a release() method to allocate new memory for the object and set the b_needsfree flag, but this may hide bugs. Operating on a released object should raise an exception. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
