On Thu, Mar 24, 2011 at 11:03 AM, Vishal <vsapr...@gmail.com> wrote:
>
>
> On Thu, Mar 24, 2011 at 7:56 AM, Senthil Kumaran <orsent...@gmail.com>wrote:
>
>> On Thu, Mar 24, 2011 at 02:25:04AM +0530, Vishal wrote:
>> > if you could read the entire file in one go...(i.e. unless your file is
>> more
>> > than 50MB)...how about the following?
>>
>> >>> for line in reversed((open('filename').readlines()[-1:-n:-1])):
>> ... print line
>>
>> Some comments:
>>
>> > # n is the number of lines you want to read.
>> > l = open(filename).read().rsplit('\n', n+1)
>>
>> - readlines would be better.
>>
>> > # following is to keep the memory requirement low.
>> > # but this is optional, if you only want to print the lines, and then
>> end
>> > the python process.
>> > l[0] = None
>>
>> - Could not get why you are setting the first item to None.
>>
>> > gc.collect()
>>
>> This does not free anything. Where is something un-referenced for it
>> to garbage collect?
>>
>> --
>> Senthil
>> _______________________________________________
>> BangPypers mailing list
>> BangPypers@python.org
>> http://mail.python.org/mailman/listinfo/bangpypers
>>
>
> setting l[0] to None, un-references the earlier string data associated with
> that name, which is then (force) collected by the collect() call.
> I have tried it multiple times, (on my windows box) and it works perfectly.
>
> In fact, I found it to be the only way to make sure memory consumption
> stays low when I have to deal with reading data columns from files that are
> in hundreds of megabytes.
>
> Would love to know of other deterministic ways of freeing memory in Python.
>
>
> Ok, I should have explained a little bit more.
the way rsplit() is used in the my code snippet, makes the last five lines
to be sent to l[1:], where as all the other lines (from the entire file) to
be present in l[0]. That is why I thought of deleting all that memory, in
case you use this snippet as part of a larger program.
Now,
setting l[0] to None, un-references the earlier string data associated with
that name, which is then (force) collected by the collect() call.
I have tried it multiple times, (on my windows box) and it works perfectly.
In fact, I found it to be the only way to make sure memory consumption stays
low when I have to deal with reading data columns from files that are in
hundreds of megabytes.
Would love to know of other deterministic ways of freeing memory in Python.
--
Thanks and best regards,
Vishal Sapre
_______________________________________________
BangPypers mailing list
BangPypers@python.org
http://mail.python.org/mailman/listinfo/bangpypers
