Hi Peter, On Tuesday, November 17, 2015 at 3:14:57 PM UTC+1, Peter Otten wrote: > Andrea Gavana wrote: > > > Hello List, > > > > I am working with relatively humongous binary files (created via > > cPickle), and I stumbled across some unexpected (for me) performance > > differences between two approaches I use to load those files: > > > > 1. Simply use cPickle.load(fid) > > > > 2. Read the file as binary using file.read() and then use cPickle.loads on > > the resulting output > > > > In the snippet below, the MakePickle function is a dummy function that > > generates a relatively big binary file with cPickle (WARNING: around 3 GB) > > in the current directory. I am using NumPy arrays to make the file big but > > my original data structure is much more complicated, and things like HDF5 > > or databases are currently not an option - I'd like to stay with pickles. > > > > The ReadPickle function simply uses cPickle.load(fid) on the opened binary > > file, and on my PC it takes about 2.3 seconds (approach 1). > > > > The ReadPlusLoads function reads the file using file.read() and then use > > cPickle.loads on the resulting output (approach 2). On my PC, the > > file.read() process takes 15 seconds (!!!) and the cPickle.loads only 1.5 > > seconds. > > > > What baffles me is the time it takes to read the file using file.read(): > > is there any way to slurp it all in one go (somehow) into a string ready > > for cPickle.loads without that much of an overhead? > > > > Note that all of this has been done on Windows 7 64bit with Python 2.7 > > 64bit, with 16 cores and 100 GB RAM (so memory should not be a problem). > > > > Thank you in advance for all suggestions :-) . > > > > Andrea. > > > > if __name__ == '__main__': > > ReadPickle() > > ReadPlusLoads() > > Do you get roughly the same times when you execute ReadPlusLoads() before > ReadPIckle()?
Thank you for your answer. I do get similar timings when I swap the two functions, and specifically still 15 seconds to read the file via file.read() and 2.4 seconds (more or less as before) via cPickle.load(fid). I thought that the order of operations might be an issue but apparently that was not the case... Andrea. -- https://mail.python.org/mailman/listinfo/python-list
