Thanks for your note. I have 1Gig of RAM. Also, Matlab has no problem
in reading the file into memory. I am just running Istvan's code that
was posted earlier.
import time, csv, random
from numpy import array
def make_data(rows=1E6, cols=6):
fp = open('data.txt', 'wt')
counter = range(cols)
for row in xrange( int(rows) ):
vals = map(str, [ random.random() for x in counter ] )
fp.write( '%s\n' % ','.join( vals ) )
fp.close()
def read_test():
start = time.clock()
reader = csv.reader( file('data.txt') )
data = [ map(float, row) for row in reader ]
data = array(data, dtype = float)
print 'Data size', len(data)
print 'Elapsed', time.clock() - start
#make_data()
read_test()
On Jan 13, 5:47 pm, "sturlamolden" <[EMAIL PROTECTED]> wrote:
> oyekomova wrote:
> > Thanks to everyone for their excellent suggestions. I was able to
> > acheive the following results with all your suggestions. However, I am
> > unable to cross file size of 6 million rows. I would appreciate any
> > helpful suggestions on avoiding memory errors. None of the solutions
> > posted was able to cross this limit.The error message means you are running
> > out of RAM.
>
> With 6 million rows and 6 columns, the size of the data array is (only)
> 274 MiB. I have no problem allocating it on my laptop. How large is the
> csv file and how much RAM do you have?
>
> Also it helps to post the whole code you are trying to run. I don't
> care much for guesswork.
--
http://mail.python.org/mailman/listinfo/python-list
