Often I've written code that generates some textual output and dumps said output to a file or socket. Of course, we are all familiar with the performance impact (multiple copies of text data and O(n**2) performance) associated with the naive approach of:
string += "some other string" string += "another string" Many other programmers have faced a similar issue; cStringIO, ''.join([mydata]), map( file.write, [mydata]) are but some attempts at making this process more efficient by jamming the components to be written into a sequence. These approaches have their drawbacks. cStringIO and ''.join involve an extra copy of the string; map( file.write ... potentially makes an excessive number of os write calls. With the maturation of iterators in Python comes the consideration that cStringIO and ''.join store the entire output in memory. map(file.write, while allowing for the emission of data to a file or socket while being generated, requires a lot of os calls. Even without consideration for iterators, the elimination of the final concatenation or drastic reduction in the number of os calls (context switches hurt) could be a substantial benefit. Perusing through the posix module reveals that the posix writev call is not exposed. Writev is the POSIX answer to this problem for very similar reasons. The ability to expose a list of strings to be outputted to the hardware level would allow for the exploitation of some rather sophisticated hardware. IIRC, some operating systems (BSD I believe) have "zero copy" network code; it is conceivable that the final concatenation of output could be entirely avoided given smart enough drivers and hardware. Of course, to take advantage of this requires that writev be exposed. I have an implementation of writev. This implementation is reasonably smart, it "unrolls" only so as many iteration.next calls as necessary to fill the pointer list for the next invocation of writev. Of course, on many systems (Linux, BSD) writev's limit is 1024 items, so to accommodate users who don't want to peel off and hold in memory 1024 iteration.next()'s, you can set an optional parameter to a smaller value. I'm not sure where to take this as a "next step." It seems too small a change for a PEP. Any ideas? You can download the patch from http://deprince.net/software/writev/index.html Adam DePrince -- http://mail.python.org/mailman/listinfo/python-list
