On Nov 14, 9:56 am, "Giampaolo Rodola'" <[EMAIL PROTECTED]> wrote: > Hi, > I'd like to know if there's a way to determine which is the best > buffer size to use when you have to send() and recv() some data over > the network. > I have an FTP server application which, on data channel, uses 8192 > bytes as buffer for both incoming and outgoing data. > Some time ago I received a report from a guy [1] who stated that > changing the buffers from 8192 to 4096 results in a drastical speed > improvement. > I tried to make some tests by using different buffer sizes, from 4 Kb > to 256 Kb, but I'm not sure which one to use as default in my > application since I noticed they can vary from different OSes. > Is there a recommended way to determine the best buffer size to use? > > Thanks in advance > > [1]http://groups.google.com/group/pyftpdlib/browse_thread/thread/f13a82b... > > --- Giampaolohttp://code.google.com/p/pyftpdlib/
As you stated, the answer is obviously OS/stack dependant. Regardless, I believe you'll likely find the best answer is between 16K-64K. Once you consider the various TCP stack improvements which are now available and the rapid increase of available bandwidth, you'll likely want to use the largest buffers which do not impose scalability issues for your system/application. Unless you have reason to use a smaller buffer, use 64K buffers and be done with it. This helps minimize the number of context switches and helps ensure the stack always has data to keep pumping. To look at it another way, using 64k buffers requires 1/8 the number of system calls and less time actually spent in python code. If as you say someone actually observed a performance improvement when changing from 8k buffers to 4k buffers, it likely has something to do with python's buffer allocation overhead but even that seems contrary to my expectation. The referenced article was not available to me so I was not able to follow and read. Another possibility is 4k buffers require less fragmentation and is likely to perform better on lossy connections. Is it possible he/she was testing on a high lossy connection? In short, performance wise, TCP stinks on lossy connections. -- http://mail.python.org/mailman/listinfo/python-list
