Travis-
Yes, I tried your suggestion, but found that it took longer to read a
large file. Thanks for your help.
Travis E. Oliphant wrote:
> oyekomova wrote:
> > Thanks to everyone for their excellent suggestions. I was able to
> > acheive the following results with all your suggestions. However,
oyekomova wrote:
> Thanks to everyone for their excellent suggestions. I was able to
> acheive the following results with all your suggestions. However, I am
> unable to cross file size of 6 million rows. I would appreciate any
> helpful suggestions on avoiding memory errors. None of the solutions
Thank you so much. Your solution works! I greatly appreciate your
help.
sturlamolden wrote:
> oyekomova wrote:
>
> > Thanks for your note. I have 1Gig of RAM. Also, Matlab has no problem
> > in reading the file into memory. I am just running Istvan's code that
> > was posted earlier.
>
> You h
oyekomova wrote:
> Thanks for your note. I have 1Gig of RAM. Also, Matlab has no problem
> in reading the file into memory. I am just running Istvan's code that
> was posted earlier.
You have a CSV file of about 520 MiB, which is read into memory. Then
you have a list of list of floats, created b
oyekomova> def read_test():
oyekomova> start = time.clock()
oyekomova> reader = csv.reader( file('data.txt') )
oyekomova> data = [ map(float, row) for row in reader ]
oyekomova> data = array(data, dtype = float)
oyekomova> print 'Data size', len(dat
Thanks for your note. I have 1Gig of RAM. Also, Matlab has no problem
in reading the file into memory. I am just running Istvan's code that
was posted earlier.
import time, csv, random
from numpy import array
def make_data(rows=1E6, cols=6):
fp = open('data.txt', 'wt')
counter = range(co
oyekomova wrote:
> Thanks to everyone for their excellent suggestions. I was able to
> acheive the following results with all your suggestions. However, I am
> unable to cross file size of 6 million rows. I would appreciate any
> helpful suggestions on avoiding memory errors. None of the solutions
Thanks to everyone for their excellent suggestions. I was able to
acheive the following results with all your suggestions. However, I am
unable to cross file size of 6 million rows. I would appreciate any
helpful suggestions on avoiding memory errors. None of the solutions
posted was able to cross
Travis E. Oliphant wrote:
> If you use numpy.fromfile, you need to skip past the initial header row
> yourself. Something like this:
>
> fid = open('somename.csv')
# I think you also meant to include this line:
header = fid.readline()
> data = numpy.fromfile(fid, sep=',').reshape(-1,6)
> # fo
oyekomova wrote:
> Thanks for your help. I compared the following code in NumPy with the
> csvread in Matlab for a very large csv file. Matlab read the file in
> 577 seconds. On the other hand, this code below kept running for over 2
> hours. Can this program be made more efficient? FYI - The csv f
oyekomova wrote:
> Thanks for your help. I compared the following code in NumPy with the
> csvread in Matlab for a very large csv file. Matlab read the file in
> 577 seconds. On the other hand, this code below kept running for over 2
> hours. Can this program be made more efficient? FYI - The csv f
oyekomova wrote:
> csvread in Matlab for a very large csv file. Matlab read the file in
> 577 seconds. On the other hand, this code below kept running for over 2
> hours. Can this program be made more efficient? FYI
There must be something wrong with your setup/program. I work with
large csv fil
sturlamolden wrote:
> oyekomova wrote:
> > Thanks for your help. I compared the following code in NumPy with the
> > csvread in Matlab for a very large csv file. Matlab read the file in
> > 577 seconds. On the other hand, this code below kept running for over 2
> > hours. Can this program be made
At Wednesday 10/1/2007 16:48, oyekomova wrote:
Thanks for your help. I compared the following code in NumPy with the
csvread in Matlab for a very large csv file. Matlab read the file in
577 seconds. On the other hand, this code below kept running for over 2
hours. Can this program be made more e
oyekomova wrote:
> Thanks for your help. I compared the following code in NumPy with the
> csvread in Matlab for a very large csv file. Matlab read the file in
> 577 seconds. On the other hand, this code below kept running for over 2
> hours. Can this program be made more efficient? FYI - The csv
Thanks for your help. I compared the following code in NumPy with the
csvread in Matlab for a very large csv file. Matlab read the file in
577 seconds. On the other hand, this code below kept running for over 2
hours. Can this program be made more efficient? FYI - The csv file was
a simple 6 column
16 matches
Mail list logo
