> > > > > Grepping through a few log files, the userland program read > > > > > 44,751,896 bytes > > > > > with a single syscall. The default recv buf size of 65536 doesn't > > > > > get the > > > > > job done for this application. > > OK, I'll take the bite. The following scenario might apply to your > userland app: > > If your userland program has a very big buffer, it probably takes a > while to process it once the read(2) call succeeds. In the meantime > new data arrives (which might be dropped on the floor if the sending > side does not back off quickly enough because of lack of acks): the > kernel buffer fills up to the maximum receive size, and the userland > buffer is not available because the new read call has not been issued > yet.
Again: the sending side is a "black box", I cannot change it. The data is generated in real time. I know what real time means. The sending side does not have enough buffer space, therefore it cannot just buffer the data until the BSD box gets around to servicing the Ethernet. So: (1) BSD box's Ethernet driver must run often enough, storing incoming packets in recv buffer and sending acks. (2) userland process must read data often enough so that recv buffer doesn't fill up. Unix makes no guarantee about how often a userland process runs. So the recv buffer must be big enough for the observed worst case plus a safety margin. Setting the recv buffer to 100MB has, so far, been enough. It might be possible to change the userland process so that it could get by with a smaller recv buffer. But the current method works, so I am looking at the remaining problem, which is that sometimes the Ethernet driver gets locked out for too long. I found that resetting a firewire bus would reliably lock out the Ethernet driver causing data loss. It turns out that the culprit is a few lines of text output to the console with printf(9). To isolate the effects of printf(9) from the firewire bus reset, I picked a trivial system call (chown(2)) and added some printf(9) calls. This did not interfere with Ethernet. So printf(9) interferes with Ethernet when called from the firewire driver, but not when called from a vanilla system call. We think there is a locking problem, the firewire device driver expert is working on fixing the locking. Until then, my workaround is to change the printf calls to log(9) calls. There are other events that interfere with the Ethernet driver, but they only happen rarely, so thus far I haven't been able to identify them. > If you expect us to help to solving a problem provide details, and not > conclusions. I have not asked for help on this list. Again, I haven't tried running this app on OpenBSD yet, I'm running it on FreeBSD. All I did was ask "How high is too high?", wondering if OpenBSD had some limit on recv buffer size.
