I've been noticing a problem for quite some time now on our mirror server. (I posted this issue to the devel list, but there have been no responses).
I'm noticing some buggy client behavior that seems it's from freshclam clients. Over time on our mirror we notice 1000's of connections can build up over time with clients stuck in a half-opened state. (or half- closed). As clam becomes more popular and traffic picks up on the mirrors, I notice more and more of these stuck clients. It becomes dangerous to the mirror at one point because if there are thousands of these lingering around, they can run the server out of socket space. Basically what we see happen is that when Apache closes the connection it will send the FIN to the client, sending it into FIN_WAIT_1, in which case the client should answer with a FIN+ACK, but that doesn't happen. The client will respond with an ACK and zero sized window. From that point on, the connection just stalls.. The server will try again will keep trying ack packets to the client until it gets the response it wants, but unfortunately, the client never does. It responds, but not properly, keeping the session in a half-opened state forever until they are either manually cleared on the server, or if the client just officially goes away. Normally on the server, TCP keepalives would kill the session, but since the client responds improperly, it's not considered idle. I'm guessing the it's a failure on the freshclam client somewhere. Either a network error has occurred that freshclam doesn't deal with properly or some other error (resources on the client?) has happened and the client hoses the connection. The client responding with a zero sides window size may be a good indication there is some resource bug on the client. Basically when a connection goes into FIN_WAIT_1 on the server on one of these clogged connections, this is what we see: (1.1.1.1 = server, 2.2.2.2 = client) 0:13:07.640426 IP 1.1.1.1.80 > 2.2.2.2.33379: . 4208136508:4208136509(1) ack 1471446041 win 520 <nop,nop,timestamp 3019088951 5004131> 20:13:07.736505 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5022148 3019088951> 20:14:07.702647 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019148951 5022148> 20:15:07.764920 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019208951 5022148> 20:15:07.860988 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5058183 3019208951> 20:16:07.827262 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019268951 5058183> 20:16:07.923341 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5076200 3019268951> 20:17:07.889690 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019328951 5076200> 20:17:07.984770 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5094217 3019328951> 20:18:07.952167 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019388951 5094217> 20:18:08.048249 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5112234 3019388951> 20:19:08.014715 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019448951 5112234> 20:19:08.110803 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5130252 3019448951> 20:20:08.077321 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019508951 5130252> 20:21:08.139995 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019568951 5130252> 20:21:08.236063 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5166286 3019568951> 20:22:08.202435 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019628951 5166286> 20:22:08.297499 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5184303 3019628951> 20:23:08.264631 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019688951 5184303> 20:23:08.360700 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5202321 3019688951> -- Robert Blayzor, BOFH INOC, LLC [EMAIL PROTECTED] http://www.inoc.net/~rblayzor/ _______________________________________________ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
