> > > > How high is too high? I have a utility that sets recv buf size > > > > to 100,000,000 and it works fine on FreeBSD and NetBSD. (Not > > > > tested yet on OpenBSD.) Necessary when the other end has buggy > > > > network code and insufficient send buf. > > > > > > Could you clarify what you mean by that? > > > > Black box sends data to BSD box using TCP. Data is generated in > > real time, the rate cannot be changed. Black box has a very small > > (way too small) send buffer. If the BSD box takes too long to > > ack, the black box's send buffer fills up and data is lost, > > and/or black box's buggy firmware screws up and data is lost. > > So I have to do everything I can to ensure that incoming packets > > do not get dropped, and that the acks get sent out as fast as > > possible. Making the TCP recv buffer very large allows the > > incoming packets to get stored and acked, even if the userland > > process reading the data doesn't get to run often enough. Even > > so, there is still the problem that other device drivers can and > > do lock out the Ethernet driver for too long. Still working on > > that problem. What we really need is true real time facilities. > > > > It is a latency problem, not a throughput problem. If the black > > box were FLOSS instead of evil closed source it should be possible > > to fix the buggy network code. > > > > A) huge recv buffer does not solve your ACK problem. > B) recv buffer is only affected by either the global > net.inet.tcp.recvspace or the setsockopt SO_RCVBUF. > C) the socketbuffers are limmited to 256kB > D) Instead of playing with knobs that don't realy do what you think they > will do you should make your userland app read faster.
It is a workaround. The way to *solve* the problem is with a true real time system. Grepping through a few log files, the userland program read 44,751,896 bytes with a single syscall. The default recv buf size of 65536 doesn't get the job done for this application. It doesn't matter how fast the userland program is if it doesn't get run often enough. I have no way to guarantee how often a userland program is run. I have to estimate, add a safety factor, and size the buffers accordingly. As far as I can tell the only remaining problem is when other device drivers lock out the Ethernet driver for too long. Nothing I do to the userland program will change that. I have to figure out what driver(s) it is, and then figure out how to fix it. At this point, problems are very rare.
