New submission from Philipp Lies:
I just stumbled upon a very serious bug in cPickle where cPickle stores the
data passed to it only partially without a warning/error:
#creating a >8GB long random data sting
import os
import cPickle
random_string = os.urandom(int(1.1*2**33))
print len(random_string)
fout = open('test.pickle', 'wb')
cPickle.dump(random_string, fout, 2)
fout.close()
fin = open('test.pickle', 'rb')
random_string2 = cPickle.load(fin)
print len(random_string2)
print random_string == random_string2
The loaded string is significantly shorter, meaning that some of the data got
lost while storing the string. This is a serious issue. However, when I use
pickle, writing fails with
error: 'i' format requires -2147483648 <= number <= 2147483647
so I guess pickle is not able to handle large data, therefore cPickle should
either throw an error as well of pickle/cPickle should be patched to handle
larger data.
Code to reproduce error using numpy (that's how I stumbled upon it):
import numpy as np
import cPickle as pickle
A = np.random.randn(1080,1920,553)
fout = open('test.pickle', 'wb')
pickle.dump(A, fout, 2)
fout.close()
fin = open('test.pickle', 'rb')
B = pickle.load(fin)
Here, numpy detects that the amount of data is wrong and throws an error.
However, still serious because saving does not lead to an error so the user
expects that the data are safely stored.
I guess might be related to http://bugs.python.org/issue13555 which is still
open.
Python 2.7.3 on latest Ubuntu with numpy 1.6.2, 64bit architecture, 128GB RAM
----------
messages: 166906
nosy: Philipp.Lies
priority: normal
severity: normal
status: open
title: pickle/cPickle saves invalid/incomplete data
type: crash
versions: Python 2.7
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue15504>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
