On Tuesday, February 4, 2014 6:39:00 PM UTC+5:30, Dave Angel wrote: > Ayushi Dalmia <ayushidalmia2...@gmail.com> Wrote in message: > > > > >> getsizeof() gives you the size of the list only; to complete the picture > >> you > > >> > > >> have to add the sizes of the lines. > > >> > > >> > > >> > > >> However, why do you want to keep track of the actual memory used by > > >> > > >> variables in your script? You should instead concentrate on the algorithm, > > >> > > >> and as long as either the size of the dataset is manageable or you can > >> limit > > >> > > >> the amount of data accessed at a given time you are golden. > > > > > > As I said, I need to merge large files and I cannot afford more I/O > > operations. So in order to minimise the I/O operation I am writing in > > chunks. Also, I need to use the merged files as indexes later which should > > be loaded in the memory for fast access. Hence the concern. > > > > > > Can you please elaborate on the point of taking lines into consideration? > > > > > > > Please don't doublespace your quotes. If you must use > > googlegroups, fix its bugs before posting. > > > > There's usually no net gain in trying to 'chunk' your output to a > > text file. The python file system already knows how to do that > > for a sequential file. > > > > For list of strings just add the getsizeof for the list to the sum > > of the getsizeof of all the list items. > > > > -- > > DaveA
Hey! I need to chunk out the outputs otherwise it will give Memory Error. I need to do some postprocessing on the data read from the file too. If I donot stop before memory error, I won't be able to perform any more operations on it. -- https://mail.python.org/mailman/listinfo/python-list
