On Wed, 24 Oct 2012 01:23:58 -0400, Dennis Lee Bieber wrote:
> On Tue, 23 Oct 2012 16:35:40 -0700, emile <em...@fenx.com> declaimed the
> following in gmane.comp.python.general:
>
>> On 10/23/2012 04:19 PM, David Hutto wrote:
>> > forward = [line.rstrip('\n') for line in f.readlines()]
>>
>> f.readlines() will be big(!) and have overhead... and forward results
>> in something again as big.
>>
> Well, since file objects are iterable, could one just drop the
> .readlines() ? ( ... line in f )
Yes, but the bottleneck is still that the list comprehension will run to
completion, trying to process the entire 100+ GB file in one go.
[...]
> And since the line-ends have already been stripped from forward,
> backward should just be:
>
> backward = reversed(forward)
reversed returns a lazy iterator, but it requires that forward is a non-
lazy (eager) sequence. So again you're stuck trying to read the entire
file into RAM.
--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
