BTW, I've downloaded from the following places: http://stromberg.dnsalias.org/svn/bufsock/trunk/bufsock.py http://stromberg.dnsalias.org/~dstromberg/backshift/documentation/html/python2x3-pysrc.html
Are those the latest versions? ---------------------------------------- > From: carlosnepomuc...@outlook.com > To: python-list@python.org > Subject: RE: How to write fast into a file in python? > Date: Sun, 19 May 2013 08:31:08 +0300 > CC: lokeshkopp...@gmail.com > > Thanks Dan! I've never used CPython or PyPy. Will try them later. > > I think the main difference between your create_file_numbers_file_like() > and the fastwrite5.py I sent earlier is that I've used cStringIO > instead of StringIO. It took 12s less using cStringIO. > > My numbers are much greater, but I've used Python 2.7.5 instead: > > C:\src\Python>python create_file_numbers.py > time taken to write a file of size 52428800 is 39.1199457743 seconds > > time taken to write a file of size 52428800 is 14.8704800436 seconds > > time taken to write a file of size 52428800 is 23.0011990985 seconds > > > I've downloaded bufsock.py and python2x3.py. The later one was hard to remove > the source code from the web page. > > Can I use them on my projects? I'm not used to the UCI license[1]. What's the > difference to the GPL? > > > > > [1] http://stromberg.dnsalias.org/~dstromberg/UCI-license.html > > ________________________________ >> Date: Sat, 18 May 2013 12:38:30 -0700 >> Subject: Re: How to write fast into a file in python? >> From: drsali...@gmail.com >> To: lokeshkopp...@gmail.com >> CC: python-list@python.org >> >> >> With CPython 2.7.3: >> ./t >> time taken to write a file of size 52428800 is 15.86 seconds >> >> time taken to write a file of size 52428800 is 7.91 seconds >> >> time taken to write a file of size 52428800 is 9.64 seconds >> >> >> With pypy-1.9: >> ./t >> time taken to write a file of size 52428800 is 3.708232 seconds >> >> time taken to write a file of size 52428800 is 4.868304 seconds >> >> time taken to write a file of size 52428800 is 1.93612 seconds >> > >> Here's the code: >> #!/usr/local/pypy-1.9/bin/pypy >> #!/usr/bin/python >> >> import sys >> import time >> import StringIO >> >> sys.path.insert(0, '/usr/local/lib') >> import bufsock >> >> def create_file_numbers_old(filename, size): >> start = time.clock() >> >> value = 0 >> with open(filename, "w") as f: >> while f.tell() < size: >> f.write("{0}\n".format(value)) >> value += 1 >> >> end = time.clock() >> >> print "time taken to write a file of size", size, " is ", (end >> -start), "seconds \n" >> >> def create_file_numbers_bufsock(filename, intended_size): >> start = time.clock() >> >> value = 0 >> with open(filename, "w") as f: >> bs = bufsock.bufsock(f) >> actual_size = 0 >> while actual_size < intended_size: >> string = "{0}\n".format(value) >> actual_size += len(string) + 1 >> bs.write(string) >> value += 1 >> bs.flush() >> >> end = time.clock() >> >> print "time taken to write a file of size", intended_size, " is ", >> (end -start), "seconds \n" >> >> >> def create_file_numbers_file_like(filename, intended_size): >> start = time.clock() >> >> value = 0 >> with open(filename, "w") as f: >> file_like = StringIO.StringIO() >> actual_size = 0 >> while actual_size < intended_size: >> string = "{0}\n".format(value) >> actual_size += len(string) + 1 >> file_like.write(string) >> value += 1 >> file_like.seek(0) >> f.write(file_like.read()) >> >> end = time.clock() >> >> print "time taken to write a file of size", intended_size, " is ", >> (end -start), "seconds \n" >> >> create_file_numbers_old('output.txt', 50 * 2**20) >> create_file_numbers_bufsock('output2.txt', 50 * 2**20) >> create_file_numbers_file_like('output3.txt', 50 * 2**20) >> >> >> >> >> On Thu, May 16, 2013 at 9:35 PM, >> <lokeshkopp...@gmail.com<mailto:lokeshkopp...@gmail.com>> wrote: >> On Friday, May 17, 2013 8:50:26 AM UTC+5:30, >> lokesh...@gmail.com<mailto:lokesh...@gmail.com> wrote: >>> I need to write numbers into a file upto 50mb and it should be fast >>> >>> can any one help me how to do that? >>> >>> i had written the following code.. >>> >>> >> ----------------------------------------------------------------------------------------------------------- >>> >>> def create_file_numbers_old(filename, size): >>> >>> start = time.clock() >>> >>> >>> >>> value = 0 >>> >>> with open(filename, "w") as f: >>> >>> while f.tell()< size: >>> >>> f.write("{0}\n".format(value)) >>> >>> value += 1 >>> >>> >>> >>> end = time.clock() >>> >>> >>> >>> print "time taken to write a file of size", size, " is ", (end >> -start), "seconds \n" >>> >>> >> ------------------------------------------------------------------------------------------------------------------ >>> >>> it takes about 20sec i need 5 to 10 times less than that. >> size = 50mb >> -- >> http://mail.python.org/mailman/listinfo/python-list >> >> >> -- http://mail.python.org/mailman/listinfo/python-list > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
