New submission from Philipp Lies:
I just stumbled upon a very serious bug in cPickle where cPickle stores the
data passed to it only partially without a warning/error:
#creating a >8GB long random data sting
import os
import cPickle
random_string = os.urandom(int(1.1*2**33))
print
Philipp Lies added the comment:
a) it's 122GB free RAM (out of 128GB total RAM)
b) when I convert the numpy array to a list it works. So seems to be a problem
with cPickle and numpy at/from a certain array size
c) $ /usr/bin/time -v python test_np.py
Traceback (most recent call
Philipp Lies added the comment:
Well, replace cPickle by pickle and it works. So if there is a memory
allocation problem cPickle should be able to handle it, especially since it
should be completely compatible to pickle.
--
___
Python tracker
