Cross-posting to Twisted core list because this is starting to get into general non-web stuff. Future replies about this fork of the thread should go there.
On Dec 23, 2010, at 11:22 AM, exar...@twistedmatrix.com wrote: > On 22 Dec, 06:51 am, gl...@twistedmatrix.com wrote: >> >> On Dec 21, 2010, at 2:24 PM, exar...@twistedmatrix.com wrote: >>> Instead, you have to go all the way to producers/consumers, and only >>> write more data to the transport buffer when it has finished dealing >>> with what you previously gave it. >> >> While everybody should of course use producers and consumers, I feel >> like there should be a twisted core ticket for this behavior of >> transport buffering, and a twisted web ticket for this behavior of the >> request buffering. The naive implementation _could_ be much cheaper >> memory-wise; at the very least, twisted.web.static.Data ought to do the >> smart thing. > > Fixing Data sounds like a good idea. I don't know what improvement to > the transport buffering you're thinking of, though. It doesn't seem > like there is an obvious, generally correct fix. Right now, FileDescriptor.write appends its input data directly to _tempDataBuffer. So far, so good: no string mangling. So let's say we do fd.write(header); fd.write(veryBigBody). Then we start writing. FileDescriptor.doWrite comes along and notices that the dataBuffer is empty. The first thing it does in this case: ''.join(_tempDataBuffer); which copies the entire veryBigBody. FileDescriptor is kinda sorta trying to avoid this problem by maintaining an 'offset' so it doesn't need to re-copy dataBuffer; it could use a similar tactic and do write()s out of individual chunks which are greater than SEND_LIMIT directly, rather than coalescing them together. Or maybe we could wrap writev() instead of copying stuff at all? _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
