[issue13072] Getting a buffer from a Unicode array uses invalid format

Stefan Krah added the comment: Good, I think this can be closed then. -- resolution: -> fixed stage: needs patch -> committed/rejected status: open -> closed type: -> behavior ___ Python tracker _

[issue13072] Getting a buffer from a Unicode array uses invalid format

Roundup Robot added the comment: New changeset 9c7515e29219 by Stefan Krah in branch 'default': Issue #13072: The array module's 'u' format code is now deprecated and http://hg.python.org/cpython/rev/9c7515e29219 -- ___ Python tracker

[issue13072] Getting a buffer from a Unicode array uses invalid format

Martin v. Löwis added the comment: Stefan, your patch array_deprecate_u.diff is fine. If you get to it, please also rephrase the clause "Python's unicode type"; not sure what the convention is to refer to Py_UNICODE now (perhaps "historical unicode type"). -- _

[issue13072] Getting a buffer from a Unicode array uses invalid format

Stefan Krah added the comment: Since actual removal is only scheduled for 4.0, I think user warnings can wait until 3.4. By then, we should have sorted out the struct format codes. In this scenario we would be sort of forced to use 'C', 'U' and 'W' as the new codes, while 'u' and 'w' would conti

[issue13072] Getting a buffer from a Unicode array uses invalid format

Antoine Pitrou added the comment: > I guess the analogy with bytes objects is that UCS-2 code points can be > handled as 16-bit integer objects. > > If we're going to do a programmatic deprecation now, that's the only > alternative typecode currently available. Do we want to recommend that? Or >

[issue13072] Getting a buffer from a Unicode array uses invalid format

Nick Coghlan added the comment: I guess the analogy with bytes objects is that UCS-2 code points can be handled as 16-bit integer objects. If we're going to do a programmatic deprecation now, that's the only alternative typecode currently available. Do we want to recommend that? Or do we want to

[issue13072] Getting a buffer from a Unicode array uses invalid format

Antoine Pitrou added the comment: > If everyone agrees on deprecating 'u', here's a patch. I think > that should be sufficient to close this issue (unless we absolutely > need deprecation warnings). I think a proper deprecation warning is preferable. --

[issue13072] Getting a buffer from a Unicode array uses invalid format

Stefan Krah added the comment: If everyone agrees on deprecating 'u', here's a patch. I think that should be sufficient to close this issue (unless we absolutely need deprecation warnings). -- Added file: http://bugs.python.org/file26892/array_deprecate_u.diff __

[issue13072] Getting a buffer from a Unicode array uses invalid format

Changes by Georg Brandl : -- priority: deferred blocker -> release blocker ___ Python tracker ___ ___ Python-bugs-list mailing list Un

[issue13072] Getting a buffer from a Unicode array uses invalid format

Martin v. Löwis added the comment: Based on the discussion in #15625, it seems that the consensus is to take no action on the format codes in this issue for 3.3, and reconsider in 3.4, to determine in what way the struct module should support Unicode. Instead, the 'u' array code will be deprec

[issue13072] Getting a buffer from a Unicode array uses invalid format

Nick Coghlan added the comment: Adding a link to #15625, which is discussing the other end of this issue (whether or not memorview should support 'u' as a typecode). -- ___ Python tracker _

[issue13072] Getting a buffer from a Unicode array uses invalid format

Martin v. Löwis added the comment: > Wide build: memoryview(array("u")).format > 'w' > > Narrow build: memoryview(array("u")).format > 'u' > > Neither of these are valid struct formats, thus they don't play > nicely with the assumptions of memoryview (or any other PEP 3118 > consumer).

[issue13072] Getting a buffer from a Unicode array uses invalid format

Nick Coghlan added the comment: There's still work to be done. The current status in 3.3 trunk is that: Wide build: >>> memoryview(array("u")).format 'w' Narrow build: >>> memoryview(array("u")).format 'u' Neither of these are valid struct formats, thus they don't play nicely with the assumpt

[issue13072] Getting a buffer from a Unicode array uses invalid format

Martin v. Löwis added the comment: Is there anything that still needs to be done on this issue? ISTM that the code is correct as it stands (i.e. Getting a buffer now only uses valid format codes) -- nosy: +loewis ___ Python tracker

[issue13072] Getting a buffer from a Unicode array uses invalid format

Georg Brandl added the comment: Deferring. -- priority: release blocker -> deferred blocker ___ Python tracker ___ ___ Python-bugs-lis

[issue13072] Getting a buffer from a Unicode array uses invalid format

Roundup Robot added the comment: New changeset 4ee4cceda047 by Victor Stinner in branch 'default': Issue #13072: Fix test_array for installation without the ctypes module http://hg.python.org/cpython/rev/4ee4cceda047 -- ___ Python tracker

[issue13072] Getting a buffer from a Unicode array uses invalid format

Antoine Pitrou added the comment: And the test fails on machines without ctypes :) -- ___ Python tracker ___ ___ Python-bugs-list mail

[issue13072] Getting a buffer from a Unicode array uses invalid format

Roundup Robot added the comment: New changeset 67a994d5657d by Victor Stinner in branch 'default': Issue #13072: Ooops, now fix test_array for Linux with 32-bit wchar_t... http://hg.python.org/cpython/rev/67a994d5657d -- ___ Python tracker

[issue13072] Getting a buffer from a Unicode array uses invalid format

Roundup Robot added the comment: New changeset e0f3406c43e4 by Victor Stinner in branch 'default': Issue #13072: Fix test_array for Windows with 16-bit wchar_t http://hg.python.org/cpython/rev/e0f3406c43e4 -- ___ Python tracker

[issue13072] Getting a buffer from a Unicode array uses invalid format

Antoine Pitrou added the comment: Someone broke the Windows buildbots. -- assignee: -> haypo stage: committed/rejected -> needs patch ___ Python tracker ___

[issue13072] Getting a buffer from a Unicode array uses invalid format

Stefan Krah added the comment: I have a patch already for the unknown format codes in memoryview. Currently fighting (as usual) with the case explosions in the tests. I think I can have a full patch by next weekend. -- ___ Python tracker

[issue13072] Getting a buffer from a Unicode array uses invalid format

Nick Coghlan added the comment: I think Victor's patch is a good solution to killing the 'u' and 'w' exports in 3.4, but we need to restore some tolerance for unknown format codes to memoryview in 3.3 regardless. -- ___ Python tracker

[issue13072] Getting a buffer from a Unicode array uses invalid format

Stefan Krah added the comment: > Did you see attached patch array_unicode_format.patch? It uses struct > format "H" or "I" depending on the size of wchar_t. I totally overlooked that. Given that memoryview can be fixed to compare buffers with unknown formats, I don't have a strong opinion on whe

[issue13072] Getting a buffer from a Unicode array uses invalid format

STINNER Victor added the comment: > memoryview is not aware of the 'u' format code, since it's not part of > the struct syntax and the PEP-3118 proposition 'u' -> UCS2, 'w' -> UCS4 > wasn't considered useful. Did you see attached patch array_unicode_format.patch? It uses struct format "H" or "I"

[issue13072] Getting a buffer from a Unicode array uses invalid format

Nick Coghlan added the comment: Perhaps if memoryview doesn't understand the format code, it can fall back on memcmp() if strcmp() indicates the format codes are the same? Otherwise we're at risk of breaking backwards compatibility with more than just array('u'). Also, if it isn't already, th

[issue13072] Getting a buffer from a Unicode array uses invalid format

Stefan Krah added the comment: Of course, if two formats *are* the same, it is possible to use memcmp(). I'll work on a patch. -- ___ Python tracker ___ _

[issue13072] Getting a buffer from a Unicode array uses invalid format

Stefan Krah added the comment: Also, it was suggested that 'u' should be deprecated: http://mail.python.org/pipermail/python-dev/2012-March/117392.html Personally, I don't have an opinion on that; I don't use the 'u' format code. Nick, could you have a look at msg167545 and see if any action

[issue13072] Getting a buffer from a Unicode array uses invalid format

Stefan Krah added the comment: STINNER Victor wrote: > Hum, this issue is a regression from Python 3.2. > > Python 3.2.3+ (3.2:243ad1a6f638+, Aug 4 2012, 01:36:41) > [GCC 4.6.3 20120306 (Red Hat 4.6.3-2)] on linux2 > >>> import array > >>> a=array.array('u', 'xyz') > >>> b=memoryview(a) > >>>

[issue13072] Getting a buffer from a Unicode array uses invalid format

Georg Brandl added the comment: Victor: the revert commit brought back "Python's Unicode character type" into the docs. This needs to be fixed to say "legacy" somewhere, as the characters in a normal Unicode string are not of that type anymore. --

[issue13072] Getting a buffer from a Unicode array uses invalid format

STINNER Victor added the comment: Hum, this issue is a regression from Python 3.2. I would like to see it fixed in Python 3.3. Example: Python 3.2.3+ (3.2:243ad1a6f638+, Aug 4 2012, 01:36:41) [GCC 4.6.3 20120306 (Red Hat 4.6.3-2)] on linux2 >>> import array >>> a=array.array('u', 'xyz') >>> b

[issue13072] Getting a buffer from a Unicode array uses invalid format

STINNER Victor added the comment: Oops, the initial issue is not solved. Attached fixes the array == memoryview issue by using a valid format for the buffer. -- resolution: fixed -> status: closed -> open Added file: http://bugs.python.org/file26704/array_unicode_format.patch

[issue13072] Getting a buffer from a Unicode array uses invalid format

Roundup Robot added the comment: New changeset 95da47ddebe0 by Victor Stinner in branch 'default': Close #13072: Restore code before the PEP 393 for the array module http://hg.python.org/cpython/rev/95da47ddebe0 -- nosy: +python-dev resolution: -> fixed stage: -> committed/rejected sta

[issue13072] Getting a buffer from a Unicode array uses invalid format

STINNER Victor added the comment: @Georg: are you ok with this change? It reverts the behaviour of Python 3.2 and avoids to have to maintain an API that nobody wants to use ('u' format using Py_UCS4, 32 bits unsigned). -- nosy: +georg.brandl ___ Pyt

[issue13072] Getting a buffer from a Unicode array uses invalid format

Stefan Krah added the comment: array_revert_pep393-2.patch looks good (checked against 7042a83f37e and all following commits that should be kept). -- ___ Python tracker ___ _

[issue13072] Getting a buffer from a Unicode array uses invalid format

Changes by Arfrever Frehtes Taifersar Arahesis : -- nosy: +Arfrever ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscri

[issue13072] Getting a buffer from a Unicode array uses invalid format

STINNER Victor added the comment: > The diff between b9558df8cc58 and default with array_revert_pep393.patch > applied is small, but I noticed that in some places you switched back to > Py_UNICODE typecode and in others not. I just copied code from Python 3.2, I forgot to update typecode type (P

[issue13072] Getting a buffer from a Unicode array uses invalid format

Stefan Krah added the comment: The diff between b9558df8cc58 and default with array_revert_pep393.patch applied is small, but I noticed that in some places you switched back to Py_UNICODE typecode and in others not. For instance, in struct arraydescr typecode is still char. I'm not sure why type

[issue13072] Getting a buffer from a Unicode array uses invalid format

STINNER Victor added the comment: Here is a patch reverting changes of the PEP 393, as suggested by Martin von Loewis. With the patch, array uses Py_UNICODE* type for the 'u' format. So array.array('u', '\u0010')[0] should return '\uDBFF' on Windows. -- keywords: +patch Added file:

[issue13072] Getting a buffer from a Unicode array uses invalid format

Stefan Krah added the comment: Is it possible without too much effort to keep the old behavior ('u' -> Py_UNICODE)? Then I'd say that should go into 3.3. The problem with the current behavior is that it's neither backwards compatible nor PEP-3118 compliant. If it is too much work to restore th

[issue13072] Getting a buffer from a Unicode array uses invalid format

STINNER Victor added the comment: Should we do something before Python 3.3 final? -- ___ Python tracker ___ ___ Python-bugs-list maili

[issue13072] Getting a buffer from a Unicode array uses invalid format

Stefan Krah added the comment: I'm not sure what to do. Martin's opinion was that the change should be reverted: http://mail.python.org/pipermail/python-dev/2012-March/117390.html -- ___ Python tracker __

[issue13072] Getting a buffer from a Unicode array uses invalid format

STINNER Victor added the comment: @Stefan: What is the status of this issue? -- ___ Python tracker ___ ___ Python-bugs-list mailing l

[issue13072] Getting a buffer from a Unicode array uses invalid format

Stefan Krah added the comment: STINNER Victor wrote: > > # Not implemented formats. Ugly, but inevitable. This is the same as > > # issue #2531: equality is also used for membership testing and must > > # return a result. > > a = array.array('u', 'xyz') > > v = memoryview(a) > > self.assertNotE

[issue13072] Getting a buffer from a Unicode array uses invalid format

STINNER Victor added the comment: > The automatic conversion of 'u' to 'I' or 'L' causes test_buffer > (PEP-3118 repo) to fail: > > > # Not implemented formats. Ugly, but inevitable. This is the same as > # issue #2531: equality is also used for membership testing and must > # return a result. >

[issue13072] Getting a buffer from a Unicode array uses invalid format

Stefan Krah added the comment: >It would be better to use a format for a Py_UCS4 string, but struct doesn't >support such type. PEP-3118 suggests for the extended struct syntax: 'c' -> ucs-1 (latin-1) encoding 'u' -> ucs-2 'w' -> ucs-4 -- ___ Pyth

[issue13072] Getting a buffer from a Unicode array uses invalid format

Changes by Stefan Krah : -- nosy: +meador.inge ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyth

[issue13072] Getting a buffer from a Unicode array uses invalid format

Stefan Krah added the comment: The automatic conversion of 'u' to 'I' or 'L' causes test_buffer (PEP-3118 repo) to fail: # Not implemented formats. Ugly, but inevitable. This is the same as # issue #2531: equality is also used for membership testing and must # return a result. a = array.array(

[issue13072] Getting a buffer from a Unicode array uses invalid format

Changes by Antoine Pitrou : -- nosy: +mark.dickinson, skrah ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: htt

[issue13072] Getting a buffer from a Unicode array uses invalid format

New submission from STINNER Victor : In Python 3.2, when you get a buffer from array.array('u'), "u" is used as buffer format. The format is supposed to be a format from the struct module, and "u" is an invalid struct format. "w" is used on wide mode. I just upgraded the array module to use th