Sean Davis wrote:
> This should be a relatively simple problem, but I haven't quite got
> the idea of how to go about it. I have a VERY large file that I would
> like to load a line at a time, do some manipulations on it, and then
> make it available to as a file-like object for use as input to a
> database module (psycopg2) that wants a file-like object (with read
> and readlines methods). I could write the manipulated file out to
> disk and then read it back in, but that seems wasteful. So, it seems
> like I need a buffer, a way to fill the buffer and a way to have read
> and readlines use the buffer. What I can't do is to load the ENTIRE
> file into a stringio object, as the file is much too large. Any
> suggestions?
>
The general approach would be (something like the following untested code):
def filter_lines(f):
for line in f:
if to_be_included(line):
yield line
fil = open("somefile.big.txt", "r")\
filegen = filter_lines(fil)
You can then iterate over the filegen generator, or write your own class
that makes it file-like. At least the generator manages to throw away
the unwanted content without buffering the whole file in memory.
regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------
--
http://mail.python.org/mailman/listinfo/python-list
