4.8-R: problem when tcp.recvspace=56K (default)
FreeBSD 4.8-R kernel.GENERIC user ppp P II - 400 128 MB RAM 56K ext. modem, 45.3K connection Something in the socket/proto/network interface area doesn't work correctly when tcp.recvspace=56K, the default value in 4.8-R. It DOES work correctly when tcp.recvspace= 16K, 32K, 48K. I see the following repeatable problem. At the same point in the ftp reception of a 218K .gz file, received data stops getting delivered to the ftp client and starts stacking up in mbufs. The ftp client reports "stalled". When the recvspace limit is reached, the entire socket receive buffer is delivered to the ftp client as fast as the client can take it. For the remainder of the file transfer there are no further ftp client stalls. Note that the stall occurs at approx. 2 * tcp.recvspace. Please note that this stall would not be perceptible on a LAN: the 56K socket receive buffer would fill up too quickly. This would be mostly harmless were it not for the extra mbufs being consumed. On a system with many TCP connections the supply of mbufs might be exhausted. Following is a log of my activities in trying to track down this problem. I haven't been able to pinpoint it. Does anyone have any idea what it might be or how to further track it down? 10/05/03 Start: tcp.recvspace=48K ftp receive 218K .gz file no ftp client stalls netstat -m 191/224/6016 mbufs in use (current/peak/max): 189 mbufs allocated to data 2 mbufs allocated to packet headers 130/146/1504 mbuf clusters in use (current/peak/max) 348 Kbytes allocated to network (7% of mb_map in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines Change tcp.recvspace=56K (default) ftp receive 218K .gz file ftp client stalls @ 113K for ~10 seconds, then jumps to 170K ftp client reports "stalled" no modem RxD stall netstat -w 2 -I tun0 shows no stall in tun0 input. netstat -m 192/464/6016 mbufs in use (current/peak/max): 190 mbufs allocated to data 2 mbufs allocated to packet headers 130/146/1504 mbuf clusters in use (current/peak/max) 408 Kbytes allocated to network (9% of mb_map in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines Consumed 464-224=240 extra mbufs, no extra mbuf clusters 408-348=60 KB additional RAM allocated to network Note: MSIZE=256 (machine/param.h) Note: No stall w/ tcp.recvspace = 16K, 32K, 48K. Tried ftp -d: echoed all commands sent to host, but didn't appear to produce any socket/proto debug output. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
tcp.recvspace=56K causes dropped packets with V.90 PPP link
FreeBSD 4.8-R kernel.GENERIC user ppp P II - 400 128 MB RAM 56K ext. modem, 45.3K connection The following is something I posted to freebsd-net about a week ago. Since then I have pinpointed the cause of the problem. It's not a bug, it's a configuration problem. >From running tcpdump on an ftp transfer I discovered that the ftp client stalls on receive due to a dropped packet. The dropped packet is not retransmitted until the socket receive buffer has filled up with packets following the dropped packet. Until the dropped packet is retransmitted the next chunk of data is not available to the ftp client. The cause of the dropped packet is a combination of a slow link (5 KB/s V.90 PPP link) and a large TCP receive window. At 56 KB the window is sufficiently large to allow the sender to more than fill the queue of some router in the path (probably the router at my ISP's POP). Keep in mind that this dropped-packet situation is 100% repeatable, with the same packet getting dropped. It's not just Internet congestion somwehere. Upon closer inspection I've also seen dropped packets with a 48 KB receive window. 32 KB doesn't give me dropped packets, though. One moral of the story is that there is such a thing as a too-large TCP receive window, and for me >= 48 KB is too large. Another moral of the story is that on a machine with both very fast and very slow network interfaces there may be no single receive window size that is optimal for all interfaces. P.S. I struck out trying to set a smaller-than-default size for the receive window of the default route (my PPP link) due to a bug in tcp_input.c that was introduced when PR 11966 was fixed. The new bug does not allow for shrinking the send/receive window sizes beneath their current value, even if that current value was NOT set by an application (i.e., it's the system-wide default value). But that's another problem. http://www.freebsd.org/cgi/query-pr.cgi?pr=11966 -- Something in the socket/proto/network interface area doesn't work correctly when tcp.recvspace=56K, the default value in 4.8-R. It DOES work correctly when tcp.recvspace= 16K, 32K, 48K. I see the following repeatable problem. At the same point in the ftp reception of a 218K .gz file, received data stops getting delivered to the ftp client and starts stacking up in mbufs. The ftp client reports "stalled". When the recvspace limit is reached, the entire socket receive buffer is delivered to the ftp client as fast as the client can take it. For the remainder of the file transfer there are no further ftp client stalls. Note that the stall occurs at approx. 2 * tcp.recvspace. Please note that this stall would not be perceptible on a LAN: the 56K socket receive buffer would fill up too quickly. This would be mostly harmless were it not for the extra mbufs being consumed. On a system with many TCP connections the supply of mbufs might be exhausted. Following is a log of my activities in trying to track down this problem. I haven't been able to pinpoint it. Does anyone have any idea what it might be or how to further track it down? 10/05/03 Start: tcp.recvspace=48K ftp receive 218K .gz file no ftp client stalls netstat -m 191/224/6016 mbufs in use (current/peak/max): 189 mbufs allocated to data 2 mbufs allocated to packet headers 130/146/1504 mbuf clusters in use (current/peak/max) 348 Kbytes allocated to network (7% of mb_map in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines Change tcp.recvspace=56K (default) ftp receive 218K .gz file ftp client stalls @ 113K for ~10 seconds, then jumps to 170K ftp client reports "stalled" no modem RxD stall netstat -w 2 -I tun0 shows no stall in tun0 input. netstat -m 192/464/6016 mbufs in use (current/peak/max): 190 mbufs allocated to data 2 mbufs allocated to packet headers 130/146/1504 mbuf clusters in use (current/peak/max) 408 Kbytes allocated to network (9% of mb_map in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines Consumed 464-224=240 extra mbufs, no extra mbuf clusters 408-348=60 KB additional RAM allocated to network Note: MSIZE=256 (machine/param.h) Note: No stall w/ tcp.recvspace = 16K, 32K, 48K. Tried ftp -d: echoed all commands sent to host, but didn't appear to produce any socket/proto debug output. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
TCP window size issues
I have a few TCP window size issues. 1. In FreeBSD 4.8-R the kernel default recv window is 56 KB. This is so large that it causes dropped packets due to queue overflow with my V.90 link (BW 5 KB/s on compressed data). 2. The 4.4BSD TCP implementation has never had the correct precedence rules for setting window sizes. App-set values should take precedence over all others. Routing table values should take precedence over kernel default values. The fix for PR 11966 alters the behavior but still doesn't implement these precedence rules. http://www.freebsd.org/cgi/query-pr.cgi?pr=11966 3. RFC 793 (TCP) says that shrinking the receive window after connection is established is "strongly discouraged". I'm currently shrinking the receive window on my default route (PPP link) from 32 KB to 4 KB with settings in /etc/ppp/ppp.conf. (Had to back out patch in PR 11966.) No problems noticed yet. To avoid shrinking the window I believe I'd have to do the following: - set kernel default send & recv windows to smallest values used on any route - modify all other routes on bootup (with script in /usr/local/etc/rc.d ?) to set desired larger window sizes Keeping the script in sync with the routing table would be a new maintenance headache. How likely am I to get into trouble by shrinking the TCP receive window? Likely enough that I should make the changes above? The patch in PR 11966 was kicking around for about a year before the PR was filed. Was this patch originally intended to solve a TCP compatibility problem caused by shrinking the TCP receive window? Thanks! ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: TCP window size issues
Actually I have just fixed it in my copy of 4.8-R. I have a document that describes the problem and my solution. I could send you that and/or a set of patches. You might want to sketch out your own solution before you look at mine, though. Also, I'm not done testing mine yet. > Carl Mascott wrote: > > > > 2. The 4.4BSD TCP implementation has never had the correct precedence > >rules for setting window sizes. App-set values should take > >precedence over all others. Routing table values should take > >precedence over kernel default values. The fix for PR 11966 > >alters the behavior but still doesn't implement these precedence > >rules. > > I'll fix this in my tcp_hostcache patch currently under review by > Sam Leffler and Ruslan Ermilov (however that is for -CURRENT). ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: TCP window size issues
Here's a case that your logic does not handle correctly. 1. Kernel default buffer size = 32 KB 2. Routing table buffer size = 48 KB 3. Application sets buffer size to 32 KB 4. tcp_mss() selects 48 KB buffer size, giving the routing table precedence over the application. I found it necessary to add a new flag with meaning "the application set my size" to struct sockbuf. I'll e-mail you a copy of my document. Note that I have backed out the patch from PR 11966. You may not want to do this. It depends upon the correct answer to Issue #3 (shrinking the receive window) in my original posting. NOTE: AFAIK, 4.4BSD through FreeBSD 4.6-R seem to have done alright without the PR 11966 patch, but if someone knows different, please speak up. Andre Oppermann wrote: > > The patch in PR11966 has already been applied. It does not fix it. > I've fixed it by redoing this check in tcp_input.c::tcp_mss(): > > if ((bufsize = rt->rt_rmx.rmx_sendpipe) == 0) > bufsize = so->sn_snd.sb_hiwat; > > to: > > if ((so->so_snd.sb_hiwat == tcp_sendspace) && rt->rt_rmx.rmx_sendpipe) > bufsize = rt->rt_rmx.rmx_sendpipe; > else > bufsize = so->so_snd.sb_hiwat; > > > The theory goes that with default buffers so_snd.sb_hiwat and > tcp_sendspace have the same size. If that is actually the case, > we use whatever we've got from the metrics. If not, the user > did some changes and we take that and ignore the metrics value. > This is not yet tested in reality... just theoretical ;-) > > Anyway, I'm interested in your solutions as well. > > Carl Mascott wrote: > > > > Actually I have just fixed it in my copy of 4.8-R. I have a document > > that describes the problem and my solution. I could send you that > > and/or a set of patches. You might want to sketch out your own > > solution before you look at mine, though. Also, I'm not done > > testing mine yet. > > > Carl Mascott wrote: > > > > > > > > 2. The 4.4BSD TCP implementation has never had the correct precedence > > > >rules for setting window sizes. App-set values should take > > > >precedence over all others. Routing table values should take > > > >precedence over kernel default values. The fix for PR 11966 > > > >alters the behavior but still doesn't implement these precedence > > > >rules. > > > > > > I'll fix this in my tcp_hostcache patch currently under review by > > > Sam Leffler and Ruslan Ermilov (however that is for -CURRENT). ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: TCP window size issues
Sorry, but you are mistaken, and I can e-mail you the tcpdump output to prove it. As I said, I'm running 4.8-R with the PR 11966 patch backed out. With this kernel, and with any stock FreeBSD 4.6-R or earlier, the TCP implementation _will_ shrink the receive window after the connection is established _if_ a routing table pipesize entry tells it to. See tcp_input.c:tcp_mss() from 4.6-R or earlier. The following statements in ppp.conf set recvpipe 4096 set sendpipe 4096 result in the following default route route to: default destination: default mask: default gateway: 10.0.0.2 interface: tun0 flags: recvpipe sendpipe ssthresh rtt,msecrttvar hopcount mtu expire 4096 4096 0 0 0 0 1500 0 This causes tcp_mss() to shrink the receive window after a connection using the default route is established, as long as the kernel default recv window size is > 4096 (slight simplification here). Lev Walkin wrote: > Carl Mascott wrote: > > I have a few TCP window size issues. > [skip] > > 3. RFC 793 (TCP) says that shrinking the receive window after > >connection is established is "strongly discouraged". I'm > >currently shrinking the receive window on my default route > >(PPP link) from 32 KB to 4 KB with settings in > >/etc/ppp/ppp.conf. (Had to back out patch in PR 11966.) > > [skip] > > >How likely am I to get into trouble by shrinking the TCP > >receive window? Likely enough that I should make the > > RFC793 has nothing to do with the defaults you're setting in ppp.conf. > RFC merely says that the TCP implementation _itself_ should not shrink > window after establishing a TCP connection. If the default dictates 4k, > the TCP implementation will use 4k and will not advertise a much smaller > window without having accepted that much data. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: TCP window size issues
> Andre Oppermann wrote: > Carl Mascott wrote: > > > > Here's a case that your logic does not handle correctly. > > > > 1. Kernel default buffer size = 32 KB > > > > 2. Routing table buffer size = 48 KB > > > > 3. Application sets buffer size to 32 KB > > > > 4. tcp_mss() selects 48 KB buffer size, giving the routing > >table precedence over the application. > > Tough luck... can't have everything. I can have everything, I do have everything, and I'm willing to share. You are free to choose whether you want a simple solution that works correctly in most cases or a slightly more complex solution that works correctly in all cases. Keep in mind that you are choosing for the entire community of FreeBSD users, not just for yourself. > > > I found it necessary to add a new flag with meaning > > "the application set my size" to struct sockbuf. > > I think you are overdoing this thing a little bit. Sendpipe and > recvpipe can only be set by root with the route command. There > is nothing changing that automagically in the kernel. > > Actually in my tcp_hostcache patch we still carry sendpipe and > recvpipe even though there ain't no way to change it anymore > (route has no longer access to it). It comes handy when we get > auto-sizing socket buffers... (hint hint!) > > I don't think there are many people setting the pipesize at all. Are you actually going to make NOPs out of the RTV_[RS]PIPE bits in the rtm_inits field of struct rt_msghdr? Unless you are, any privileged application, not just 'route', will still be able to set pipe sizes. One such application is user ppp. I _need_ this capability in user ppp, BTW, to prevent recv queue overflow with a 5 KB/s PPP link while maintaining a large default recv window on my other routes. With autotuning of pipe sizes I think the PR 11966 patch is going to have to be backed out: autotuning will sometimes want to shrink the pipes. > > > I'll e-mail you a copy of my document. Note that I have > > backed out the patch from PR 11966. You may not want to > > do this. It depends upon the correct answer to Issue #3 > > (shrinking the receive window) in my original posting. > > Got it. Still have to read it. > > > NOTE: AFAIK, 4.4BSD through FreeBSD 4.6-R seem to have done > > alright without the PR 11966 patch, but if someone knows > > different, please speak up. > > -- > Andre > > > > Andre Oppermann wrote: > > > > > > The patch in PR11966 has already been applied. It does not fix it. > > > I've fixed it by redoing this check in tcp_input.c::tcp_mss(): > > > > > > if ((bufsize = rt->rt_rmx.rmx_sendpipe) == 0) > > > bufsize = so->sn_snd.sb_hiwat; > > > > > > to: > > > > > > if ((so->so_snd.sb_hiwat == tcp_sendspace) && rt->rt_rmx.rmx_sendpipe) > > > bufsize = rt->rt_rmx.rmx_sendpipe; > > > else > > > bufsize = so->so_snd.sb_hiwat; > > > > > > > > > The theory goes that with default buffers so_snd.sb_hiwat and > > > tcp_sendspace have the same size. If that is actually the case, > > > we use whatever we've got from the metrics. If not, the user > > > did some changes and we take that and ignore the metrics value. > > > This is not yet tested in reality... just theoretical ;-) > > > > > > Anyway, I'm interested in your solutions as well. > > > > > > Carl Mascott wrote: > > > > > > > > Actually I have just fixed it in my copy of 4.8-R. I have a document > > > > that describes the problem and my solution. I could send you that > > > > and/or a set of patches. You might want to sketch out your own > > > > solution before you look at mine, though. Also, I'm not done > > > > testing mine yet. > > > > > Carl Mascott wrote: > > > > > > > > > > > > 2. The 4.4BSD TCP implementation has never had the correct precedence > > > > > >rules for setting window sizes. App-set values should take > > > > > >precedence over all others. Routing table values should take > > > > > >precedence over kernel default values. The fix for PR 11966 > > > > > >alters the behavior but still doesn't implement these precedence > > > > > >rules. > > > > > > > > > > I'll fix this in my tcp_hostcache patch currently under review by > > > > > Sam Leffler and Ruslan Ermilov (however that is for -CURRENT). > ___ > [EMAIL PROTECTED] mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "[EMAIL PROTECTED]" > ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: TCP window size issues
Ahhh, that may explain why, during FTP receive of a large file over a 5 KB/s link with a 56 KB recv window, I get one or two dropped packets at the same point every time, and then no further dropped packets for the rest of the transfer. Barney Wolff wrote: > On Fri, Oct 17, 2003 at 08:16:32PM -0400, Carl Mascott wrote: > > > > NOTE: AFAIK, 4.4BSD through FreeBSD 4.6-R seem to have done > > alright without the PR 11966 patch, but if someone knows > > different, please speak up. > > I'm not really surprised. I would have thought that a well-behaved > sender would be using congestion avoidance and limiting its transmit > window so that packet drops should be rare, even with an oversize > advertised window. What's surprising is that this sort of tweak is > ever required, other than as a minor optimization. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "[EMAIL PROTECTED]"
