On Tue, 2011-02-22, Ben Finney wrote: > Kelson Zawack <zawack...@gis.a-star.edu.sg> writes: > >> I have a large (10gb) data file for which I want to parse each line >> into an object and then append this object to a list for sorting and >> further processing. > > What is the nature of the further processing? > > Does that further processing access the items sequentially? If so, they > don't all need to be in memory at once, and you can produce them with a > generator <URL:http://docs.python.org/glossary.html#term-generator>.
He mentioned sorting them -- you need all of them for that. If that's the *only* such use, I'd experiment with writing them as sortable text to file, and run GNU sort (the Unix utility) on the file. It seems to have a clever file-backed sort algorithm. /Jorgen -- // Jorgen Grahn <grahn@ Oo o. . . \X/ snipabacken.se> O o . -- http://mail.python.org/mailman/listinfo/python-list
