Bugs item #1598181, was opened at 2006-11-16 22:40
Message generated for change (Comment added) made by nnorwitz
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1598181&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: Python 2.5
Status: Open
Resolution: Fixed
Priority: 5
Private: No
Submitted By: Ralf W. Grosse-Kunstleve (rwgk)
Assigned to: Peter Åstrand (astrand)
Summary: subprocess.py: O(N**2) bottleneck
Initial Comment:
subprocess.py (Python 2.5, current SVN, probably all versions) contains this
O(N**2) code:
bytes_written = os.write(self.stdin.fileno(), input[:512])
input = input[bytes_written:]
For large but reasonable "input" the second line is rate limiting. Luckily, it
is very easy to remove this bottleneck. I'll upload a simple patch. Below is a
small script that demonstrates the huge speed difference. The output on my
machine is:
creating input
0.888417959213
slow slicing input
61.1553330421
creating input
0.863168954849
fast slicing input
0.0163860321045
done
The numbers are times in seconds.
This is the source:
import time
import sys
size = 1000000
t0 = time.time()
print "creating input"
input = "\n".join([str(i) for i in xrange(size)])
print time.time()-t0
t0 = time.time()
print "slow slicing input"
n_out_slow = 0
while True:
out = input[:512]
n_out_slow += 1
input = input[512:]
if not input:
break
print time.time()-t0
t0 = time.time()
print "creating input"
input = "\n".join([str(i) for i in xrange(size)])
print time.time()-t0
t0 = time.time()
print "fast slicing input"
n_out_fast = 0
input_done = 0
while True:
out = input[input_done:input_done+512]
n_out_fast += 1
input_done += 512
if input_done >= len(input):
break
print time.time()-t0
assert n_out_fast == n_out_slow
print "done"
----------------------------------------------------------------------
>Comment By: Neal Norwitz (nnorwitz)
Date: 2007-01-16 23:00
Message:
Logged In: YES
user_id=33168
Originator: NO
Peter this is fine for 2.5.1. Please apply and update Misc/NEWS. Thanks!
----------------------------------------------------------------------
Comment By: Ralf W. Grosse-Kunstleve (rwgk)
Date: 2007-01-07 07:15
Message:
Logged In: YES
user_id=71407
Originator: YES
Thanks for the fixes!
----------------------------------------------------------------------
Comment By: Peter Åstrand (astrand)
Date: 2007-01-07 06:36
Message:
Logged In: YES
user_id=344921
Originator: NO
Fixed in trunk revision 53295. Is this a good candidate for backporting to
25-maint?
----------------------------------------------------------------------
Comment By: Mike Klaas (mklaas)
Date: 2007-01-04 10:20
Message:
Logged In: YES
user_id=1611720
Originator: NO
I reviewed the patch--the proposed fix looks good. Minor comments:
- "input_done" sounds like a flag, not a count of written bytes
- buffer() could be used to avoid the 512-byte copy created by the slice
----------------------------------------------------------------------
Comment By: Ralf W. Grosse-Kunstleve (rwgk)
Date: 2006-11-16 22:43
Message:
Logged In: YES
user_id=71407
Originator: YES
Sorry, I didn't know the tracker would destroy the indentation.
I'm uploading the demo source as a separate file.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1598181&group_id=5470
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
