Re: Sockets stuck in FIN_WAIT_1

2008-05-30 Thread Robert Blayzor
On May 30, 2008, at 3:17 PM, Doug Barton wrote: I'm not sure why, but I sense hostility on your part. I'm not sure why, since that is an odd reaction to someone who is trying to help you. If I'm wrong about that, never mind. I'm not being hostile, geez. ;) I simply asked "why not"? Plenty

Re: Sockets stuck in FIN_WAIT_1

2008-05-30 Thread Doug Barton
Robert Blayzor wrote: On May 29, 2008, at 11:07 PM, Doug Barton wrote: Hrrm, are you running ipfw ON the web server box? If so, I'd be curious as to why, and whether or not the problem goes away if you take IPFW out of the equation. If IPFW is running on another machine, never mind. Yes, I

Re: Sockets stuck in FIN_WAIT_1

2008-05-30 Thread Robert Blayzor
On May 30, 2008, at 12:43 PM, Matthew Dillon wrote: I would be very careful with any type of ruleset (IPFW or PF) which relies on keep-state. You can wind up causing legitimate connections to drop if it isn't carefully tuned. Thanks again Matt... I do agree on the firewall and k

Re: Sockets stuck in FIN_WAIT_1

2008-05-30 Thread Matthew Dillon
:Yes, IPFW is running on the box. Why not? : :-- :Robert Blayzor, BOFH :INOC, LLC :[EMAIL PROTECTED] :http://www.inoc.net/~rblayzor/ There's nothing wrong with running IPFW on the same box :-) But, I think that rule change is masking the problem rather then solving it. The keep-st

Re: Sockets stuck in FIN_WAIT_1

2008-05-30 Thread Max Laier
On Friday 30 May 2008 11:35:56 Robert Blayzor wrote: > On May 30, 2008, at 4:47 AM, David Malone wrote: > > There has been some talk about this sort of problem on the IETF TCP > > Maintainers list. I don't think any good conclusion was reached - > > whatever the solution was certainly needs to be t

Re: Sockets stuck in FIN_WAIT_1

2008-05-30 Thread Peter Jeremy
On 2008-May-30 05:35:56 -0400, Robert Blayzor <[EMAIL PROTECTED]> wrote: >A timeout value would be fine. The problem is selecting a sensible timeout - 30-60s might be reasonable for a webserver that serves static content but is a bit short for a shell session... >On a side note, I could easily fi

Re: Sockets stuck in FIN_WAIT_1

2008-05-30 Thread Robert Blayzor
On May 30, 2008, at 4:41 AM, Ian Smith wrote: Without debating your stateful alternative - either should work fine for TCP applications - this allowed inbound icmp packets for types 0,3,8,11 but no outbound icmp at all (assuming your firewall defaults to deny). Switching the ipfw rules ov

Re: Sockets stuck in FIN_WAIT_1

2008-05-30 Thread Robert Blayzor
On May 30, 2008, at 4:18 AM, Tod McQuillin wrote: This relies on tcpdrop, included as /usr/sbin/tcpdrop on FreeBSD 6.x; you may need to install it from a port on FreeBSD 4.x. Thanks, that seems like a reasonable "band aid" for now. Worked perfectly. -- Robert Blayzor, BOFH INOC, LLC [E

Re: Sockets stuck in FIN_WAIT_1

2008-05-30 Thread Robert Blayzor
On May 30, 2008, at 4:47 AM, David Malone wrote: There has been some talk about this sort of problem on the IETF TCP Maintainers list. I don't think any good conclusion was reached - whatever the solution was certainly needs to be tunable per-socket because this behaviour is perfectly valid in so

Re: Sockets stuck in FIN_WAIT_1

2008-05-30 Thread Robert Blayzor
On May 30, 2008, at 4:41 AM, Ian Smith wrote: Without debating your stateful alternative - either should work fine for TCP applications - this allowed inbound icmp packets for types 0,3,8,11 but no outbound icmp at all (assuming your firewall defaults to deny). I didn't post all the rules

Re: Sockets stuck in FIN_WAIT_1

2008-05-30 Thread Robert Blayzor
On May 29, 2008, at 11:07 PM, Doug Barton wrote: Hrrm, are you running ipfw ON the web server box? If so, I'd be curious as to why, and whether or not the problem goes away if you take IPFW out of the equation. If IPFW is running on another machine, never mind. Yes, IPFW is running on th

Re: Sockets stuck in FIN_WAIT_1

2008-05-30 Thread David Malone
On Fri, May 30, 2008 at 06:11:43PM +1000, Peter Jeremy wrote: > A real solution would require more thought. I suspect you need a > mechanism similar to the keepalive timer that starts when there is > data queued and is reset when (some) data is sent - this would catch > your situation but I'm not

Re: Sockets stuck in FIN_WAIT_1

2008-05-30 Thread Ian Smith
On Thu, 29 May 2008, Robert Blayzor wrote: > On May 29, 2008, at 8:55 PM, Matthew Dillon wrote: > >It's got to a be a bug on the client(s) in question. I can't think > >of anything else. You may have to resort to injecting a TCP RST > >packet (e.g. via a TUN device) to clear the

Re: Sockets stuck in FIN_WAIT_1

2008-05-30 Thread Tod McQuillin
On Fri, 30 May 2008, Peter Jeremy wrote: As a work-around, you could write a cronjob that scans "netstat" and temporarily creates an ipfw 'reset' rule that matches each FIN_WAIT_1 socket In the past, I've used something like this: netstat -an | grep FIN_WAIT_1 | perl -pe 's/.*\s((?:\d+\.){3}\

Re: Sockets stuck in FIN_WAIT_1

2008-05-30 Thread Peter Jeremy
On 2008-May-29 18:11:56 -0400, Robert Blayzor <[EMAIL PROTECTED]> wrote: >working. Only way to correct it seems to reboot the server... even under >RELENG_7_0 so the upgrade from 4_11 did not fix the problem. Unfortunately, your issue is a bug in the client: The server is trying to send dat

Re: Sockets stuck in FIN_WAIT_1

2008-05-29 Thread Doug Barton
Robert Blayzor wrote: On May 29, 2008, at 8:55 PM, Matthew Dillon wrote: It's got to a be a bug on the client(s) in question. I can't think of anything else. You may have to resort to injecting a TCP RST packet (e.g. via a TUN device) to clear the connections. That would be most

Re: Sockets stuck in FIN_WAIT_1

2008-05-29 Thread Robert Blayzor
On May 29, 2008, at 8:55 PM, Matthew Dillon wrote: It's got to a be a bug on the client(s) in question. I can't think of anything else. You may have to resort to injecting a TCP RST packet (e.g. via a TUN device) to clear the connections. That would be most unpleasant... and also

Re: Sockets stuck in FIN_WAIT_1

2008-05-29 Thread Matthew Dillon
:This is exactly what we're seeing, it's VERY strange. I did kill off :Apache, and all the FIN_WAIT_1's stuck around, so the kernel is in :fact sending these probe packets, every 60 seconds, which the client :responds to... (most of the time). Ach. Now that I think about it, it is sti

Re: Sockets stuck in FIN_WAIT_1

2008-05-29 Thread Robert Blayzor
On May 29, 2008, at 5:32 PM, Matthew Dillon wrote: Now, the connection is also in a half-closed state, which means that one direction is closed. I can't tell which direction that is but my guess is that 1.1.1.1 (the apache server) closed the 1.1.1.1- >2.2.2.2 direction and the 2.

Re: Sockets stuck in FIN_WAIT_1

2008-05-29 Thread Matthew Dillon
:I think we're onto something here, but for some reason it doesn't make :any sense. I have keepalives turned OFF in Apache: : :When I tcpdump this, I see something sending ack's back and forth :every 60 seconds, but what? Apache? I'm not sure why. I don't see :any timeouts in Apache for

Re: Sockets stuck in FIN_WAIT_1

2008-05-29 Thread Robert Blayzor
On May 29, 2008, at 3:30 PM, Matthew Dillon wrote: It is quite possible that the other ends of the connection are still live and that the issue could very well be a timeout setting in the server config file instead of something in the TCP stack. I think we're onto something here, b

Re: Sockets stuck in FIN_WAIT_1

2008-05-29 Thread Robert Blayzor
On May 29, 2008, at 3:30 PM, Matthew Dillon wrote: It is quite possible that the other ends of the connection are still live and that the issue could very well be a timeout setting in the server config file instead of something in the TCP stack. Good point, I didn't think about th

Re: Sockets stuck in FIN_WAIT_1

2008-05-29 Thread Matthew Dillon
:On May 29, 2008, at 3:12 PM, Matthew Dillon wrote: :>I guess nobody mentioned the obvious thing to check: Make sure :>TCP keepalive is turned on. :> :>sysctl net.inet.tcp.always_keepalive=1 : : :Thanks Matt. : :I also thought that a keepalives were not running and sessions just :st

Re: Sockets stuck in FIN_WAIT_1

2008-05-29 Thread Robert Blayzor
On May 29, 2008, at 3:12 PM, Matthew Dillon wrote: I guess nobody mentioned the obvious thing to check: Make sure TCP keepalive is turned on. sysctl net.inet.tcp.always_keepalive=1 Thanks Matt. I also thought that a keepalives were not running and sessions just stuck around forev

Re: Sockets stuck in FIN_WAIT_1

2008-05-29 Thread Matthew Dillon
I guess nobody mentioned the obvious thing to check: Make sure TCP keepalive is turned on. sysctl net.inet.tcp.always_keepalive=1 If you don't do this then dead TCP connections can build up, particularly on busy servers, due to the other end simply disappearing. Without

Re: Sockets stuck in FIN_WAIT_1

2008-05-29 Thread Doug Barton
Robert Blayzor wrote: On May 29, 2008, at 8:44 AM, Stephen Clark wrote: You know it really pains me when people blithely say just upgrade to X.X where X is the latest release. In my first message I did not say it "blithely," I actually said something like "begin looking at 7.0 ASAP." However,

Re: Sockets stuck in FIN_WAIT_1

2008-05-29 Thread Robert Blayzor
On May 29, 2008, at 8:44 AM, Stephen Clark wrote: You know it really pains me when people blithely say just upgrade to X.X where X is the latest release. It is generally just not that simple to upgrade - there are all sorts of dependencies with other software that have to be considered.

Re: Sockets stuck in FIN_WAIT_1

2008-05-29 Thread Stephen Clark
Doug Barton wrote: Mark Kirkwood wrote: Doug Barton wrote: The Apache page: http://httpd.apache.org/docs/2.3/misc/perf-tuning.html It mentions FIN_WAIT_2 not 1, so this might be a different/new problem. IIRC it actually is the same problem, but in any case you're missing the bit where 4.

Re: Sockets stuck in FIN_WAIT_1

2008-05-28 Thread Jeremy Chadwick
On Wed, May 28, 2008 at 06:13:04PM -0400, Robert Blayzor wrote: > Here is what I have on the server now: > > and loader.conf > > accf_http_load="YES" You shouldn't bother with this. Let the apache22 rc.d script handle loading it dynamically. Use apache22_http_accept_enable="yes" in rc.conf. I'v

Re: Sockets stuck in FIN_WAIT_1

2008-05-28 Thread Doug Barton
Mark Kirkwood wrote: Doug Barton wrote: The Apache page: http://httpd.apache.org/docs/2.3/misc/perf-tuning.html It mentions FIN_WAIT_2 not 1, so this might be a different/new problem. IIRC it actually is the same problem, but in any case you're missing the bit where 4.x is EOL. :) Hence m

Re: Sockets stuck in FIN_WAIT_1

2008-05-28 Thread Mark Kirkwood
Doug Barton wrote: Robert Blayzor wrote: On May 28, 2008, at 6:55 PM, Doug Barton wrote: That's a known problem with FreeBSD 4, which is now well past EOL. I would suggest moving to FreeBSD 7 ASAP. Is it? I searched and searched and never found any hits or PR's regarding this. Not sure

Re: Sockets stuck in FIN_WAIT_1

2008-05-28 Thread Doug Barton
Robert Blayzor wrote: On May 28, 2008, at 6:55 PM, Doug Barton wrote: That's a known problem with FreeBSD 4, which is now well past EOL. I would suggest moving to FreeBSD 7 ASAP. Is it? I searched and searched and never found any hits or PR's regarding this. Not sure where you looked th

Re: Sockets stuck in FIN_WAIT_1

2008-05-28 Thread Doug Barton
Robert Blayzor wrote: On May 28, 2008, at 6:43 PM, Chuck Swiger wrote: You didn't mention which version of FreeBSD you are running-- that's rather important info. Actually, I just checked, this is a 4.11 server, I thought it was running at least 6.2. That's a known problem with FreeBSD 4, w

Re: Sockets stuck in FIN_WAIT_1

2008-05-28 Thread Robert Blayzor
On May 28, 2008, at 6:55 PM, Doug Barton wrote: That's a known problem with FreeBSD 4, which is now well past EOL. I would suggest moving to FreeBSD 7 ASAP. Is it? I searched and searched and never found any hits or PR's regarding this. When was it first fixed? 5.x? 6.x? or not until

Re: Sockets stuck in FIN_WAIT_1

2008-05-28 Thread Robert Blayzor
On May 28, 2008, at 6:43 PM, Chuck Swiger wrote: You didn't mention which version of FreeBSD you are running-- that's rather important info. Actually, I just checked, this is a 4.11 server, I thought it was running at least 6.2. 00200 allow tcp from any to me 80 setup 00200 allow icmp fro

Re: Sockets stuck in FIN_WAIT_1

2008-05-28 Thread Chuck Swiger
Hi-- You didn't mention which version of FreeBSD you are running-- that's rather important info. On May 28, 2008, at 3:13 PM, Robert Blayzor wrote: ipfw: 00200 allow tcp from any to me 80 setup 00200 allow icmp from any to me icmptype 0,3,8,11 00200 deny log ip from any to me Also, surely