[issue2389] Array pickling exposes internal memory representation of elements

Alexandre Vassalotti added the comment: Committed fix for 3.x in r74013 and for 2.x in r74014. -- resolution: -> fixed status: open -> closed ___ Python tracker ___

[issue2389] Array pickling exposes internal memory representation of elements

Alexandre Vassalotti added the comment: I know believe that arrays should be pickled as a list of values on Python 2.x. Doing otherwise makes it impossible to unpickle arrays coming from Python 2.x using Python 3.x, since pickle on Python 3 decodes all the strings from 2.x to Unicode. However,

[issue2389] Array pickling exposes internal memory representation of elements

Changes by Alexandre Vassalotti : Removed file: http://bugs.python.org/file11096/fix_array_pickling.patch ___ Python tracker ___ ___ Python-bug

[issue2389] Array pickling exposes internal memory representation of elements

Alexandre Vassalotti added the comment: I would like to commit my patch later this week. So if you see any issue with the patch, please speak up. -- ___ Python tracker ___ __

[issue2389] Array pickling exposes internal memory representation of elements

Alexandre Vassalotti added the comment: Here's a patch that implements the solution I described in msg85298. Please give it a good review: http://codereview.appspot.com/87072 -- Added file: http://bugs.python.org/file14369/portable_array_pickling.diff _

[issue2389] Array pickling exposes internal memory representation of elements

Changes by Collin Winter : -- nosy: +collinwinter ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.py

[issue2389] Array pickling exposes internal memory representation of elements

Alexandre Vassalotti added the comment: Ah, I just remembered the smart way I had devised some time ago to handle this issue without changing the constructor of array.array. The trick would be to add a __reduce__ method to array.array. This method would return a special constructor function, the

[issue2389] Array pickling exposes internal memory representation of elements

Martin v. Löwis <[EMAIL PROTECTED]> added the comment: I think changing the array constructor is fairly easy: just pick a set of codes that are defined to be platform-neutral (i.e. for each size two codes, one for each endianness). For example, the control characters (\0..\x1F) could be used in t

[issue2389] Array pickling exposes internal memory representation of elements

Alexandre Vassalotti <[EMAIL PROTECTED]> added the comment: I'm all in for a standardized representation of array's pickles (with width and endianness preserved). However to happen, we will either need to change array's constructor to support at least the byte-order specification (like struct) or

[issue2389] Array pickling exposes internal memory representation of elements

Guido van Rossum <[EMAIL PROTECTED]> added the comment: > Instead of sticking to network byte order, I propose to include byte > order information in the pickle (for example as '<' or '>' like struct > does), so that pickling/unpickling between the same-endianness > architectures doesn't have to

[issue2389] Array pickling exposes internal memory representation of elements

Hrvoje Nikšić <[EMAIL PROTECTED]> added the comment: I think preserving integer width is a good idea because it saves us from having to throw overflow errors when unpickling to machines with different width of C types. The cost is that pickling/unpickling the array might change the array's typec

[issue2389] Array pickling exposes internal memory representation of elements

Martin v. Löwis <[EMAIL PROTECTED]> added the comment: I like to challenge the view what "correct" behavior is here. If I pickle an array of 32-bit integer values on one system, and unpickle it as an array of 64-bit integer values on a different system, is that correct, or incorrect? IMO, correc

[issue2389] Array pickling exposes internal memory representation of elements

Hrvoje Nikšić <[EMAIL PROTECTED]> added the comment: Unfortunately dumping the internal representation of non-long arrays won't work, for several reasons. First, it breaks when porting pickles between platforms of different endianness such as Intel and SPARC. Then, it ignores the considerable w

[issue2389] Array pickling exposes internal memory representation of elements

Alexandre Vassalotti <[EMAIL PROTECTED]> added the comment: The slowdown depends of the array type. The patch makes array unpickling a few orders of magnitude slower (i.e. between 4 and 15 times slower depending of the array type). In general, pickling is about as fast as with the binary represen

[issue2389] Array pickling exposes internal memory representation of elements

Guido van Rossum <[EMAIL PROTECTED]> added the comment: Wouldn't that be lots and lots slower? I believe speed is one of the reasons why the binary representation is currently dumped. ___ Python tracker <[EMAIL PROTECTED]> _

[issue2389] Array pickling exposes internal memory representation of elements

Alexandre Vassalotti <[EMAIL PROTECTED]> added the comment: I don't see why this cannot be fixed easily. All we need to do is fix the __reduce__ method of array objects to emit a list--i.e. with array.tolist()--instead of a memory string. Since the reduce protocol is just a fancy way to store the

[issue2389] Array pickling exposes internal memory representation of elements

Hrvoje Nikšić <[EMAIL PROTECTED]> added the comment: I guess it went unnoticed due to prevalence of little-endian 32-bit machines. With 64-bit architectures becoming more and more popular, this might become a bigger issue. Raymond, why do you think fixing this bug would complicate porting to 2.

[issue2389] Array pickling exposes internal memory representation of elements

Guido van Rossum <[EMAIL PROTECTED]> added the comment: Agreed, this has been broken for a long time, and few people have noticed or complained. Let's wait. -- assignee: gvanrossum -> versions: +Python 2.7, Python 3.1 -Python 2.6, Python 3.0 ___ Pyt

[issue2389] Array pickling exposes internal memory representation of elements

Raymond Hettinger <[EMAIL PROTECTED]> added the comment: At this point, I think it better to wait until Py2.7/3.1. Changing it now would just complicate efforts to port from 2.5 to 2.6 to 3.0. Guido, do you agree? -- assignee: -> gvanrossum nosy: +rhettinger __

[issue2389] Array pickling exposes internal memory representation of elements

Benjamin Peterson <[EMAIL PROTECTED]> added the comment: Ping. -- nosy: +benjamin.peterson ___ Python tracker <[EMAIL PROTECTED]> ___ __

[issue2389] Array pickling exposes internal memory representation of elements

Changes by Jesús Cea Avión <[EMAIL PROTECTED]>: -- nosy: +jcea __ Tracker <[EMAIL PROTECTED]> __ ___ Python-bugs-list mailing list Unsubscribe:

[issue2389] Array pickling exposes internal memory representation of elements

Guido van Rossum <[EMAIL PROTECTED]> added the comment: This looks indeed wrong. Unfortunately it also looks hard to fix in a way that won't break unpickling arrays pickled by a previous Python version. We won't be able to fix this in 2.5 (it'll be a new feature) but we should try to fix this in

[issue2389] Array pickling exposes internal memory representation of elements

Hrvoje Nikšić <[EMAIL PROTECTED]> added the comment: Here is an example that directly demonstrates the bug. Pickling on x86_64: Python 2.5.1 (r251:54863, Mar 21 2008, 13:06:31) [GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2 Type "help", "copyright", "credits" or "license" for mo

[issue2389] Array pickling exposes internal memory representation of elements

New submission from Hrvoje Nikšić <[EMAIL PROTECTED]>: It would seem that pickling arrays directly exposes the underlying machine words, making the pickle non-portable to platforms with different layout of array elements. The guts of array.__reduce__ look like this: if (array->ob_size >