Yup, I actually had log files in mind, too. I process Apache logs from an
8-way cluster every 15 minutes. There are 32,000 sites on said cluster;
that's a lot of log data!
I didn't actually think anyone would go *try* to
open("war_and_peace.txt").readlines().. I just meant it as a generalization
for "really large file." =)
-Jeff
On 7/25/07, Jay Loden <[EMAIL PROTECTED]> wrote:
Grant Edwards wrote:
> On 2007-07-25, Jeff <[EMAIL PROTECTED]> wrote:
>
>> That might be a memory problem if you are running multiple processes
>> regularly, such as on a webserver.
>
> I suppose if you did it in parallel 50 processes, you could use
> up 250MB of RAM. Still not a big deal on many servers. A
> decent OS will swap regions that aren't being used to disk, so
> it's likely not to be a problem.
Or, you might be reading from a text file dramatically larger than a 3MB
copy of War and Peace. I regularly deal with log files that are often many
times that, including some that have been well over a GB or more. Trust me,
you don't want to read in the entire file when it's a 1.5GB text file.
It's true that many times readlines() will work fine, but there are also
certainly cases where it's not acceptable for memory and performance
reasons.
-Jay
--
http://mail.python.org/mailman/listinfo/python-list
--
http://mail.python.org/mailman/listinfo/python-list
