[issue13555] cPickle MemoryError when loading large file (while pickle works)

Changes by Jesús Cea Avión : -- nosy: +jcea ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Serhiy Storchaka added the comment: I have opened issue17299 for testing issue. -- resolution: -> fixed stage: -> committed/rejected status: open -> closed ___ Python tracker _

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Serhiy Storchaka added the comment: Thank you for the report. Standard tests do not cover pickling/unpickling to real files. -- ___ Python tracker ___ __

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Roundup Robot added the comment: New changeset f3f23ecdb1c6 by Serhiy Storchaka in branch '2.7': Issue #13555: Fix an integer overflow check. http://hg.python.org/cpython/rev/f3f23ecdb1c6 -- ___ Python tracker

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Arfrever Frehtes Taifersar Arahesis added the comment: 680959a3ae2e has caused that g-ir-scanner tool from gobject-introspection (which seems to create some pickles and next load them?) fails with MemoryError, e.g. during building of GTK+. MemoryError occurs only for a subset of pickles. I atta

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Changes by Serhiy Storchaka : -- resolution: -> fixed stage: patch review -> committed/rejected status: open -> closed ___ Python tracker ___ ___

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Roundup Robot added the comment: New changeset 680959a3ae2e by Serhiy Storchaka in branch '2.7': Issue #13555: cPickle now supports files larger than 2 GiB. http://hg.python.org/cpython/rev/680959a3ae2e -- nosy: +python-dev ___ Python tracker

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Serhiy Storchaka added the comment: Test updated too. Now it doesn't try to write a string larger than 2 GiB (it's impossible), instead writes a lot of shorter strings with total size larger than 2 GiB. -- Added file: http://bugs.python.org/file29042/pickle_overflow-4.diff ___

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Changes by Serhiy Storchaka : Removed file: http://bugs.python.org/file29020/pickle_overflow-3.diff ___ Python tracker ___ ___ Python-bugs-lis

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Serhiy Storchaka added the comment: Here is an updated patch. Fixed two bugs found by Antoine (an inappropriate format and a memory error in bigmemtest), fixed resizing of marks array and one possible integer overflow in write_other(). Used workaround to bypass limitations of cStringIO API. -

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Changes by Serhiy Storchaka : -- dependencies: +cStringIO not 64-bit safe -cStringIO.StringIO aborted when more then INT_MAX bytes written ___ Python tracker ___ ___

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Serhiy Storchaka added the comment: I have picked up it. Actually BINSTRING format supports only strings less 2GiB, so test should be changed. But there are other bugs which prohibit pickling a lot of data. For one I open issue17054. -- dependencies: +cStringIO.StringIO aborted when m

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Charles-François Natali added the comment: > Charles-François, are you going to finish this before 2.7.4 RC released? The > patch should be updated to address Antoine's comments. No. I don't have access to a 64-bit box, which makes it difficult to write (I don't get compiler warnings) and test.

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Serhiy Storchaka added the comment: Charles-François, are you going to finish this before 2.7.4 RC released? The patch should be updated to address Antoine's comments. -- ___ Python tracker ___

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Charles-François Natali added the comment: > I have a 64-bit laptop with 2 GB memory so I don't think I can do so. (though > one could use swap) AFAICT, a binary string a little longer than 1GB should be enough to reproduce the bug. Just make sure Python isn't built with '-fwrapv'. --

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Antoine Pitrou added the comment: I may tackle this but rare 2.7-only bugs are pretty low on my priorities list. -- ___ Python tracker ___ ___

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Ramchandra Apte added the comment: Bump. @neologix I have a 64-bit laptop with 2 GB memory so I don't think I can do so. (though one could use swap) -- ___ Python tracker ___ __

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Charles-François Natali added the comment: Could someone with a 64-bit box take over this one? I won't go anywhere with my 32-bit box... -- ___ Python tracker ___ ___

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Changes by Serhiy Storchaka : -- nosy: +serhiy.storchaka stage: -> needs patch ___ Python tracker ___ ___ Python-bugs-list mailing li

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Changes by Arfrever Frehtes Taifersar Arahesis : -- nosy: +Arfrever ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscri

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Antoine Pitrou added the comment: Le vendredi 06 janvier 2012 à 19:17 +, Charles-François Natali a écrit : > Charles-François Natali added the comment: > > Antoine, could you test the last version (test_pickle and if possible > with the OP testcase)? > I can't test it myself (32-bit machin

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Charles-François Natali added the comment: Antoine, could you test the last version (test_pickle and if possible with the OP testcase)? I can't test it myself (32-bit machine with 1 GB). -- ___ Python tracker

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Charles-François Natali added the comment: Here's a new version with a test (untested). Note that I'm absolutely not sure that the 'memsize' argument to bigmemtest is correct. -- Added file: http://bugs.python.org/file24079/pickle_overflow-2.diff ___

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Antoine Pitrou added the comment: > New version. Looks good to me. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsub

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Changes by Florent Xicluna : -- nosy: +flox ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Charles-François Natali added the comment: New version. -- Added file: http://bugs.python.org/file24008/pickle_overflow-1.diff ___ Python tracker ___ ___

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Antoine Pitrou added the comment: I think there's a problem here: +self->data = realloc(self->data, self->size * sizeof(PyObject *)); +if (self->data == NULL) goto nomemory; If realloc() fails, the old data pointer is lost and therefore will never get free()ed. Same for: +

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Charles-François Natali added the comment: Here's a patch which should fix this. However, I'm unable to test it. -- keywords: +patch Added file: http://bugs.python.org/file24006/pickle_overflow.diff ___ Python tracker

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Antoine Pitrou added the comment: Ah, I see. It's a bit of a pity not to be able to load files > 1GB, especially on a 64-bit build (!). Perhaps cPickle could be made partly 64-bit compatible? Or at least, indeed, do a proper anti-overflow check. -- ___

[issue13555] cPickle MemoryError when loading large file (while pickle works)

STINNER Victor added the comment: > Should we fix this (Py_ssize_t, overflow check before computation), as in > #11564? Yes. Use Py_ssize_t type for the buf_size attribute, and replace "bigger <= 0" (test if an overflow occurred) by "self->buf_size > (PY_SSIZE_T_MAX >> 1)". -- nosy:

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Charles-François Natali added the comment: > So it seems unlikely to be the explanation. Victor reproduced in on IRC, and it's indeed an overflow. The problematic code is in readline_file: """ bigger = self->buf_size << 1; if (bigger <= 0) { /* overflow */

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Antoine Pitrou added the comment: Oh, what's more interesting is that it works here (Python 2.7.1 and numpy 1.6.1 under Mageia with 8GB RAM). Looking at a pickle disassembly, the only remarkable thing is the presence of a long binary string (the raw serialization of all IEEE floats), which sh

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Antoine Pitrou added the comment: > Couldn't this be linked to #11564 (pickle not 64-bit ready)? Well, I don't know anything about numpy, but: >>> 196 * 24 4704 >>> 196 * 24 * 8 # assuming 8 bytes per float 37632 >>> 2**31 2147483648 So it seems unlikely to be the explanatio

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Changes by Charles-François Natali : -- Removed message: http://bugs.python.org/msg149321 ___ Python tracker ___ ___ Python-bugs-list

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Charles-François Natali added the comment: @Antoine Couldn't this be linked to #11564 (pickle not 64-bit ready)? Basically, an integer overflow, and malloc() would bail out when asked a ridiculous size. AFAICT this wasn't fixed in 2.7. @Philipp I'd be curious to see the last lines of $ ltrac

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Charles-François Natali added the comment: @Antoine Couldn't this be linked to #11564 (pickle not 64-bit ready)? AFAICT this wasn't fixed in 2.7. Basically, an integer overflow, and malloc() would bail out when asked a ridiculous size. @Philipp I'd be curious to see the last lines of $ ltrace

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Philipp Lies added the comment: a) it's 122GB free RAM (out of 128GB total RAM) b) when I convert the numpy array to a list it works. So seems to be a problem with cPickle and numpy at/from a certain array size c) $ /usr/bin/time -v python test_np.py Traceback (most recent call last): File

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Charles-François Natali added the comment: > my computer has 122GB free RAM 122, or 1.22? > Well, replace cPickle by pickle and it works. cPickle makes some direct call to malloc()/realloc()/free(), contrarily to pickle which uses pymalloc. This could lead to heap fragmentation. What does t

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Changes by Antoine Pitrou : -- nosy: +mark.dickinson ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mai

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Antoine Pitrou added the comment: Is there a way to reproduce that doesn't involve numpy? -- components: +Extension Modules -None nosy: +pitrou ___ Python tracker ___ __

[issue13555] cPickle MemoryError when loading large file (while pickle works)

maniram maniram added the comment: Have you checked the system monitor after all cPickle can use more memory than 1GB. -- ___ Python tracker ___ ___

[issue13555] cPickle MemoryError when loading large file (while pickle works)

Philipp Lies added the comment: Well, replace cPickle by pickle and it works. So if there is a memory allocation problem cPickle should be able to handle it, especially since it should be completely compatible to pickle. -- ___ Python tracker

[issue13555] cPickle MemoryError when loading large file (while pickle works)

maniram maniram added the comment: Maybe Ubuntu doesn't think it is safe to allocate the memory. -- nosy: +maniram.maniram ___ Python tracker ___ ___

[issue13555] cPickle MemoryError when loading large file (while pickle works)

New submission from Phillies : When I try to load a large file (>1GB) cPickle crashes with a MemoryError: $python test.py Traceback (most recent call last): File "/tmp/test.py", line 8, in A2 = cPickle.load(f2) MemoryError test.py contains following code: import numpy as np import cPickle