[issue5308] cannot marshal objects with more than 2**31 elements

Changes by Serhiy Storchaka : -- stage: commit review -> committed/rejected status: pending -> closed ___ Python tracker ___ ___ Python

[issue5308] cannot marshal objects with more than 2**31 elements

Roundup Robot added the comment: New changeset 1cf2c42af815 by Serhiy Storchaka in branch '2.7': Fix reference leaks introduced by the patch for issue #5308. http://hg.python.org/cpython/rev/1cf2c42af815 New changeset 8b99f2224c3a by Serhiy Storchaka in branch '3.3': Fix reference leaks introduc

[issue5308] cannot marshal objects with more than 2**31 elements

Changes by Serhiy Storchaka : -- resolution: -> fixed stage: needs patch -> commit review status: open -> pending versions: -Python 3.2 ___ Python tracker ___ __

[issue5308] cannot marshal objects with more than 2**31 elements

Serhiy Storchaka added the comment: Good catch. There is another resource leak for general bytes-like object (the buffer can be not released). Here is a patch. -- Added file: http://bugs.python.org/file30741/marshal_string_leak.patch ___ Python track

[issue5308] cannot marshal objects with more than 2**31 elements

Christian Heimes added the comment: The macro W_SIZE at http://hg.python.org/cpython/file/dbdb6f7f9a1a/Python/marshal.c#l130 has introduced a reference leak at http://hg.python.org/cpython/file/dbdb6f7f9a1a/Python/marshal.c#l390 . Because W_SIZE returns in an error case the reference count of

[issue5308] cannot marshal objects with more than 2**31 elements

Changes by Serhiy Storchaka : -- resolution: -> fixed stage: patch review -> committed/rejected status: open -> closed ___ Python tracker ___

[issue5308] cannot marshal objects with more than 2**31 elements

Serhiy Storchaka added the comment: > Perhaps you could add a bigmem test for this? > (assuming you have enough memory somewhere to test it) Some tests require more than 252 GiB of memory. -- ___ Python tracker ___

[issue5308] cannot marshal objects with more than 2**31 elements

Roundup Robot added the comment: New changeset 72e75ea25d00 by Serhiy Storchaka in branch '2.7': Fix tests for issue #5308. http://hg.python.org/cpython/rev/72e75ea25d00 New changeset 0407e5e5915e by Serhiy Storchaka in branch '3.3': Cleanup a test for issue #5308. http://hg.python.org/cpython/r

[issue5308] cannot marshal objects with more than 2**31 elements

Roundup Robot added the comment: New changeset 385d982ce641 by Serhiy Storchaka in branch '2.7': Issue #5308: Raise ValueError when marshalling too large object (a sequence http://hg.python.org/cpython/rev/385d982ce641 New changeset e0464fa28c85 by Serhiy Storchaka in branch '3.2': Issue #5308:

[issue5308] cannot marshal objects with more than 2**31 elements

Changes by Serhiy Storchaka : -- assignee: -> serhiy.storchaka ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue5308] cannot marshal objects with more than 2**31 elements

Mark Dickinson added the comment: Hmm, yes, I think that would also make sense. I missed those uses of int. I'll give in, though, and accept that >= 32-bit ints are a de facto standard, even if not de jure. -- ___ Python tracker

[issue5308] cannot marshal objects with more than 2**31 elements

Serhiy Storchaka added the comment: Perhaps we should change signatures of w_string() and r_string() -- replace "int" by at least "long". -- ___ Python tracker ___ __

[issue5308] cannot marshal objects with more than 2**31 elements

Mark Dickinson added the comment: > Do I have to support such a case in the code? No, I don't see any need for that: after all, you're making the code *more* portable by replacing those occurrences of INT_MAX with 0x7fff. :-) -- ___ Python trac

[issue5308] cannot marshal objects with more than 2**31 elements

Serhiy Storchaka added the comment: > No, that's not true: see Annex E of the standard, where the minimum for > INT_MAX is declared to be 32767. My fault. Do I have to support such a case in the code? -- ___ Python tracker

[issue5308] cannot marshal objects with more than 2**31 elements

Serhiy Storchaka added the comment: > Perhaps you could add a bigmem test for this? > (assuming you have enough memory somewhere to test it) I think this is possible. I now have some experience in the writing of bigmem tests. ;) -- ___ Python track

[issue5308] cannot marshal objects with more than 2**31 elements

Stefan Krah added the comment: Hmm, Mark was faster. ;) -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http:

[issue5308] cannot marshal objects with more than 2**31 elements

Stefan Krah added the comment: Theoretically int may be 16 bits (C99): 5.2.4.2.1 Sizes of integer types -- maximum value for an object of type int INT_MAX +32767 // 2**15 − 1 I agree that Python wouldn't compile on such a platform anyway. -- nosy: +skrah __

[issue5308] cannot marshal objects with more than 2**31 elements

Changes by Eric Snow : -- nosy: +eric.snow ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.or

[issue5308] cannot marshal objects with more than 2**31 elements

Mark Dickinson added the comment: > (and C standard requites at least 32-bit ints) No, that's not true: see Annex E of the standard, where the minimum for INT_MAX is declared to be 32767. -- ___ Python tracker ___

[issue5308] cannot marshal objects with more than 2**31 elements

Antoine Pitrou added the comment: Perhaps you could add a bigmem test for this? (assuming you have enough memory somewhere to test it) -- nosy: +pitrou ___ Python tracker ___

[issue5308] cannot marshal objects with more than 2**31 elements

Serhiy Storchaka added the comment: > (And as a matter of principle, > INT_MAX isn't really right here: an int might be only 16 bits long on > some strange platforms...). AFAIK Python doesn't support such platforms (and C standard requites at least 32-bit ints). However there are strange plat

[issue5308] cannot marshal objects with more than 2**31 elements

Serhiy Storchaka added the comment: And here is a patch for 2.7. -- Added file: http://bugs.python.org/file28978/marshal_overflow-2.7.patch ___ Python tracker ___

[issue5308] cannot marshal objects with more than 2**31 elements

Serhiy Storchaka added the comment: Here is a patch for 3.x which adds checks for size overflow (only on 64-bit platform). -- keywords: +patch stage: needs patch -> patch review Added file: http://bugs.python.org/file28977/marshal_overflow.patch ___

[issue5308] cannot marshal objects with more than 2**31 elements

Changes by Serhiy Storchaka : -- nosy: +serhiy.storchaka versions: +Python 3.4 ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue5308] cannot marshal objects with more than 2**31 elements

Changes by Terry J. Reedy : -- stage: -> needs patch versions: +Python 3.2, Python 3.3 -Python 3.1 ___ Python tracker ___ ___ Python-b

[issue5308] cannot marshal objects with more than 2**31 elements

Changes by Mark Dickinson : -- assignee: marketdickinson -> ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: htt

[issue5308] cannot marshal objects with more than 2**31 elements

Raymond Hettinger added the comment: Why not. Besides it ought to be fun to write the test case for this one :-) ___ Python tracker ___ ___ Py

[issue5308] cannot marshal objects with more than 2**31 elements

Mark Dickinson added the comment: It wouldn't hurt to add the overflow checks though, would it? -- assignee: -> marketdickinson priority: -> normal ___ Python tracker ___ _

[issue5308] cannot marshal objects with more than 2**31 elements

Raymond Hettinger added the comment: Given that marshal is primarily about supporting pyc files, do we care? -- nosy: +rhettinger ___ Python tracker ___ _

[issue5308] cannot marshal objects with more than 2**31 elements

New submission from Mark Dickinson : Two closely related issues in Python/marshal.c, involving writing and reading of variable-length objects (lists, strings, long integers, ...) (1) The w_object function in marshal contains many instances of code like the following: else if (PyList_CheckExac