On May 29, 2008, at 5:32 PM, Matthew Dillon wrote:
Now, the connection is also in a half-closed state, which means
that
one direction is closed. I can't tell which direction that is
but my
guess is that 1.1.1.1 (the apache server) closed the 1.1.1.1-
>2.2.2.2
direction and the 2.2.2.2 box has a broken TCP implementation and
can't
deal with it.
This is exactly what we're seeing, it's VERY strange. I did kill off
Apache, and all the FIN_WAIT_1's stuck around, so the kernel is in
fact sending these probe packets, every 60 seconds, which the client
responds to... (most of the time).
I can suggest two things. First, the TCP connection is good but
you
still may be able to tell Apache, in the apache configuration
file, to
timeout after a certain period of time and clear the connection.
I don't think this helps since Apache sees the connection as long
gone. As far as Apache is concerned (as far as I can tell), this
connection doesn't exist. This may be proved by killing off Apache,
the connection still lives and while Apache is running, I have the max
clients connected most of the time... so I don't think the linger
around and jam up sockets to Apache. If they did, I think Apache
would spiral down quite quickly.
Secondly, it may be beneficial to identify exactly what the
client and
server were talking about which caused the client to hang with a
live
tcp connection. The only way to do that is to tcpdump EVERYTHING
going
on related to the apache srever, save it to a big-ass disk
partition
(like 500G), and then when you see a stuck connection go back
through
the tcpdump log file and locate it, grep it out, and review what
exactly
it was talking about. You'd have to tcpdump with options to tell
it to
dump the TCP data payloads.
Unfortunately it's not possible for me, not nearly enough space. This
is a VERY busy server, a spikey 20Mbps+ (8-12Mbps on average) of web
traffic almost constantly. The traffic is VERY static, just small
data files and occasional large ones (12Mb+), but the majority are
2-5k files. (it's a clamav mirror server)
It seems likely that the client is running an applet or
javascript that
receives a stream over the connection, and that applet or
javascript
program has locked up, causing the data sent from the server to
build up
and for the client's buffer space to run out, and start
advertising the
0 window.
98% of the clients are clamav (freshclam) clients on various
platforms. Using p0f most of them are various flavors of Linux, but I
can't say what OS the clients are connecting to for sure since I'd
have to look at the OS finger print of the SYN packets...
Don't get me wrong, the server keeps up well, low CPU, lots of RAM
free, lots of network available, and 99% of all HTTP connections are
completed just fine. I just see these FIN_WAIT_1 connections build up
over time until the server runs out of socket space and then things
just stop working. Only way to correct it seems to reboot the
server... even under RELENG_7_0.... so the upgrade from 4_11 did not
fix the problem.
--
Robert Blayzor, BOFH
INOC, LLC
[EMAIL PROTECTED]
http://www.inoc.net/~rblayzor/
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
