Kelson Zawack <zawack...@gis.a-star.edu.sg> writes: > I have a large (10gb) data file for which I want to parse each line > into an object and then append this object to a list for sorting and > further processing.
What is the nature of the further processing? Does that further processing access the items sequentially? If so, they don't all need to be in memory at once, and you can produce them with a generator <URL:http://docs.python.org/glossary.html#term-generator>. Note that, if you just want lines of text from a file, the file object itself is a generator for the lines of text within it. If, on the other hand, you need arbitrary access all over that large data set, you probably want a data type better suited. The standard library has the ‘array’ module for this purpose; the third-party NumPy library provides even more power. -- \ “Remember: every member of your ‘target audience’ also owns a | `\ broadcasting station. These ‘targets’ can shoot back.” —Michael | _o__) Rathbun to advertisers, news.admin.net-abuse.email | Ben Finney -- http://mail.python.org/mailman/listinfo/python-list
