Hi Rodrick,
On 2010-08-17 18:40, Rodrick Brown wrote:
> I have a fairly large file 1-2GB in size that I need to
> process line by line but I first need to convert the file
> to text using a 3rd party tool that prints the records
> also line by line.
>
> I've tried using Popen to do this with no luck. I'm trying
> to simulate
>
> /bin/foo myfile.dat
Is foo the 3rd-party conversion tool you've mentioned?
It would be good to have a bit more context, e. g. an actual
snippet from your code.
> And as the records are being printed do some calculations.
>
> pipe = Popen(exttool,shell=True,stdout=PIPE).stdout
>
> for data in pipe.readlines():
> print data,
>
> This operation blocks forever I'm guessing it's trying to
> process the entire file at once.
If you use `readlines` on a file object it reads the whole
file at once and returns a list of the lines (including line
end characters, by the way). What you probably want is
for line in pipe:
print line,
which reads and prints the file contents line by line.
Stefan
--
http://mail.python.org/mailman/listinfo/python-list
