I've got the following results on my desktop PC (Win7/Python2.7.5):
C:\src\Python>python -m timeit -cvn3 -r3 "execfile('fastwrite2.py')"
raw times: 123 126 125
3 loops, best of 3: 41 sec per loop
C:\src\Python>python -m timeit -cvn3 -r3 "execfile('fastwrite5.py')"
raw times: 34 34.3 34
3 loops, best of 3: 11.3 sec per loop
C:\src\Python>python -m timeit -cvn3 -r3 "execfile('fastwrite6.py')"
raw times: 0.4 0.447 0.391
3 loops, best of 3: 130 msec per loop
If you can just copy a preexisting file it will surely increase the speed to
the levels you need, but doing the cStringIO operations can reduce the time in
72%.
Strangely I just realised that the time it takes to complete such scripts is
the same no matter what hard drive I choose to run them. The results are the
same for an SSD (main drive) and a HDD.
I think it's very strange to take 11.3s to write 50MB (4.4MB/s) sequentially on
a SSD which is capable of 140MB/s.
Is that a Python problem? Why does it take the same time on the HDD?
### fastwrite2.py ### <<< this is your code
size = 50*1024*1024
value = 0
filename = 'fastwrite2.dat'
with open(filename, "w") as f:
while f.tell()< size:
f.write("{0}\n".format(value))
value += 1
f.close()
### fastwrite5.py ###
import cStringIO
size = 50*1024*1024
value = 0
filename = 'fastwrite5.dat'
x = 0
b = cStringIO.StringIO()
while x < size:
line = '{0}\n'.format(value)
b.write(line)
value += 1
x += len(line)+1
f = open(filename, 'w')
f.write(b.getvalue())
f.close()
b.close()
### fastwrite6.py ###
import shutil
src = 'fastwrite.dat'
dst = 'fastwrite6.dat'
shutil.copyfile(src, dst)
----------------------------------------
> Date: Fri, 17 May 2013 07:58:43 -0400
> From: da...@davea.name
> To: python-list@python.org
> Subject: Re: How to write fast into a file in python?
>
> On 05/17/2013 12:35 AM, lokeshkopp...@gmail.com wrote:
>> On Friday, May 17, 2013 8:50:26 AM UTC+5:30, lokesh...@gmail.com wrote:
>>> I need to write numbers into a file upto 50mb and it should be fast
>>>
>>> can any one help me how to do that?
>>>
>>> i had written the following code..
>>>
>>> <SNIP>
>>> value = 0
>>>
>>> with open(filename, "w") as f:
>>>
>>> while f.tell()< size:
>>>
>>> f.write("{0}\n".format(value))
>>> <SNIP more double-spaced nonsense from googlegroups>
> If you must use googlegroups, at least read this
> http://wiki.python.org/moin/GoogleGroupsPython.
>>>
>>>
>>> it takes about 20sec i need 5 to 10 times less than that.
>> size = 50mb
>>
>
> Most of the time is spent figuring out whether the file has reached its
> limit size. If you want Python to go fast, just specify the data. On
> my Linux system, it takes 11 seconds to write the first 6338888 values,
> which is just under 50mb. If I write the obvious loop, writing that
> many values takes .25 seconds.
>
> --
> DaveA
> --
> http://mail.python.org/mailman/listinfo/python-list
>
--
http://mail.python.org/mailman/listinfo/python-list
