Jason Swails wrote:
> Hello,
>
> I have a question about a csv.writer instance. I have a utility that I
> want to write a full CSV file from lots of data, but due to performance
> (and memory) considerations, there's no way I can write the data
> sequentially. Therefore, I write the data in chunks to temporary files,
> then combine them
> all at the end. For convenience, I declare each writer instance via a
> statement like
>
> my_csv = csv.writer(open('temp.1.csv', 'wb'))
>
> so the open file object isn't bound to any explicit reference, and I don't
> know how to reference it inside the writer class (the documentation
> doesn't
> say, unless I've missed the obvious). Thus, the only way I can think of
> to make sure that all of the data is written before I start copying these
> files sequentially into the final file is to unbuffer them so the above
> command is changed to
>
> my_csv = csv.writer(open('temp.1.csv', 'wb', 0))
>
> unless, of course, I add an explicit reference to track the open file
> object
> and manually close or flush it (but I'd like to avoid it if possible). My
> question is 2-fold. Is there a way to do that directly via the CSV API,
> or is the approach I'm taking the only way without binding the open file
> object
> to another reference? Secondly, if these files are potentially very large
> (anywhere from ~1KB to 20 GB depending on the amount of data present),
> what kind of performance hit will I be looking at by disabling buffering
> on these types of files?
>
> Tips, answers, comments, and/or suggestions are all welcome.
>
> Thanks a lot!
> Jason
>
> As an afterthought, I suppose I could always subclass the csv.writer class
> and add the reference I want to that, which I may do if there's no other
> convenient solution.
A contextmanager might help:
import csv
from contextlib import contextmanager
@contextmanager
def filewriter(filename):
with open(filename, "wb") as outstream:
yield csv.writer(outstream)
if __name__ == "__main__":
with filewriter("tmp.csv") as writer:
writer.writerows([
["alpha", "beta"],
["gamma", "delta"]])
--
http://mail.python.org/mailman/listinfo/python-list
