redbaron wrote: > I've one big (6.9 Gb) .gz file with text inside it. > zcat bigfile.gz > /dev/null does the job in 4 minutes 50 seconds > > python code have been doing the same job for 25 minutes and still > doesn't finish =( the code is simpliest I could ever imagine: > > def main(): > fh = gzip.open(sys.argv[1]) > all(fh) > > As far as I understand most of the time it executes C code, so pythons > no overhead should be noticible. Why is it so slow?
I'm guessing here - but if gzip streams (and AFAIK it does), the commandline will simply stream to /dev/null. OTOH, python is not streaming, it will instead allocate buffers for the whole file. Which for a *zipped* 6.9Gb file might take a while. Diez -- http://mail.python.org/mailman/listinfo/python-list
