Travis- Yes, I tried your suggestion, but found that it took longer to read a large file. Thanks for your help.
Travis E. Oliphant wrote: > oyekomova wrote: > > Thanks to everyone for their excellent suggestions. I was able to > > acheive the following results with all your suggestions. However, I am > > unable to cross file size of 6 million rows. I would appreciate any > > helpful suggestions on avoiding memory errors. None of the solutions > > posted was able to cross this limit. > > Did you try using numpy.fromfile ? > > This will not require you to allocate more memory than needed. If you > specify a count, it will also not have to re-allocate memory in blocks > as the array size grows. > > It's limitation is that it is not a very sophisticated csv reader (it > only understands a single separator (plus line-feeds are typically seen > as a separator). > > -Travis -- http://mail.python.org/mailman/listinfo/python-list
