On Fri, Oct 4, 2013 at 8:33 AM, Jayakumar Bellie <jkumar.o...@gmail.com>wrote:
> > I try to run a grep command on a 30 MB file and redirect output to a tmp > folder. > Read the file in the tmp folder and display it. > > When I do this in PHP it is very fast. > But when I do it in web2py it is slow. > > This is fast for me with a sample csv of 60MB taken from http://www.briandunning.com/sample-data/350000.zip and changed the line endings from mac to unix. # coding: utf-8 import sys import os from subprocess import call def index(): response.headers['Content-Type'] = 'text/plain' TEMPFOLDER = os.path.join(request.folder, 'private') SOURCEFILE = os.path.join(TEMPFOLDER, '350000.csv') TEMPFILE = os.path.join(TEMPFOLDER, 'tempfile.tmp') print >>sys.stderr, 'grep file' try: retcode = call('grep "A" %s >%s' % (SOURCEFILE, TEMPFILE), shell=True) if retcode < 0: raise HTTP(404, "Child was terminated by signal %s" % str(-retcode)) elif retcode > 0: raise HTTP(404, "Child returned %s" % str(retcode)) except OSError as e: raise HTTP(500, "Execution failed: %r" % e) print >>sys.stderr, 'read file' return response.stream(open(TEMPFILE, 'rb'), chunk_size=10**6) The temp file in this example will have 40MB Tested with: $ curl -o /dev/null -s -w "time for first byte: %{time_starttransfer}s\ntotal time: %{time_total}s\n" http://localhost:8000/bigfile time for first byte: 1.203s total time: 1.384s Most time is spent on grep. Ricardo -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
