.@web.de]
> Sent: Tuesday, August 02, 2011 4:26 PM
> To: python-list@python.org
> Subject: Re: python reading file memory cost
>
> Chris Rebert wrote:
>
>>> The running result was that read a 500M file consume almost 2GB RAM,
>>> I cannot figure it out, somebod
Subject: Re: python reading file memory cost
Chris Rebert wrote:
>> The running result was that read a 500M file consume almost 2GB RAM,
>> I cannot figure it out, somebody help!
>
> If you could store the floats themselves, rather than their string
> representations, tha
Chris Rebert wrote:
>> The running result was that read a 500M file consume almost 2GB RAM, I
>> cannot figure it out, somebody help!
>
> If you could store the floats themselves, rather than their string
> representations, that would be more space-efficient. You could then
> also use the `array`
On Mon, Aug 1, 2011 at 8:22 PM, Tony Zhang wrote:
> Thanks!
>
> Actually, I used .readline() to parse file line by line, because I need
> to find out the start position to extract data into list, and the end
> point to pause extracting, then repeat until the end of file.
> My file to read is forma
You could try forcing a garbage collection...
On Mon, Aug 1, 2011 at 8:22 PM, Tony Zhang wrote:
> Thanks!
>
> Actually, I used .readline() to parse file line by line, because I need
> to find out the start position to extract data into list, and the end
> point to pause extracting, then repeat u
Thanks!
Actually, I used .readline() to parse file line by line, because I need
to find out the start position to extract data into list, and the end
point to pause extracting, then repeat until the end of file.
My file to read is formatted like this:
blabla...useless
useless...
/sign/
data
A code snippet would work wonders in making sure you've communicated what
you really need, or at least what you have now.
But if you read the data into one big string, that'll be much more efficient
than if you read it as a list of integers or even as a list of lines.
Processing the data one chun
On 01/08/11 17:05, Tong Zhang wrote:
> Hello, everyone!
>
>
>
> I am trying to read a little big txt file (~1 GB) by python2.7, what I
> want to do is to read these data into a array, meanwhile, I monitor the
> memory cost, I found that it cost more than 6 GB RAM! So I have two
> questions:
>
