Re: [Clamav-users] clamav-users Digest, Vol 36, Issue 16
___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
[Clamav-users] ClamAV hanging under medium-high load
I've got a bit of a weird problem with clamav on some servers I've just put together. Basically I've got two boxes that in my tests (40 concurrent connections) can handle about 300 clam scans/sec. I've got a pretty standard clam install and also the sanesecurity rules on the boxes. We just put the boxes live as the backend to an exim cluster that handles ~60msgs/sec at peak, all of which pass through the boxes. I've set MaxThreads to 700 and the port range 1024-6. I've also tweaked the ulimits on the box setting the max number of open files/process to be 1 and the stack size to be unlimited. I've done some pthread tests indicating that aprox 3k threads can run simultaneously before kernel memory is exhausted. We're running ClamAV 0.91.2 from the DAG rpm. It all works really well, apart from several times a day when clam seems to hang. Usually there are about 10-20 entries/sec in the logs and clam is taking ~200% cpu time (out of the 800% available), however when clam hangs, it sticks at about 100% cpu for a time, nothing is written to the logs. A vast number of listeners are set up for the file transfer stage of the scanning process, but they don't seem to be receiving the file. Pretty quickly, the max number of threads are produced, and clam starts rejecting the connections on port 3310. Eventually after 10-20 seconds (sometimes up to 40 or 50), the box seems to spring back to life and clears the backlog. A SIGHUP doesn't seem to prod it back into life. Does anyone have any experiences similar to this or ideas as to how to go about debugging the problem? It didn't appear in the lab with 40 simultaneous connections, and only appears during the peak hours, so it's a bit difficult to reproduce but I'll carry on trying. Thanks, Mark -- Mark Zealey -- Systems Architect Product Development * Pipex Hosting [EMAIL PROTECTED] This mail is subject to this disclaimer: http://www.pipex.net/disclaimer.html ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] ClamAV hanging under medium-high load
On Thu, 2007-10-11 at 12:47 +0100, mark wrote: > Does anyone have any experiences similar to this or ideas as to how to go > about debugging the problem? It didn't appear in the lab with 40 > simultaneous connections, and only appears during the peak hours, so it's a > bit difficult to reproduce but I'll carry on trying. > I would probably start by checking the following: 1. Check you aren't running out of ephemeral ports. Try and monitor the number and state of your IP ports. 2. Check what files clamd is scanning, which you can do through /proc 3. Check clamd isn't doing a DB reload during this time. 4. Remove the sanesecurity sigs and see if the problem goes away. 5. strace clamd during the 'hang' to see what clamd is doing. 6. Use tcpdump to see what is happening on the network. -trog signature.asc Description: This is a digitally signed message part ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] ClamAV hanging under medium-high load
mark wrote: > I've got a bit of a weird problem with clamav on some servers I've just put > together. > > It all works really well, apart from several times a day when clam seems to > hang. > > Does anyone have any experiences similar to this or ideas as to how to go > about debugging the problem? Mark, If the experimental code is enable in your build, try setting PhishingScanURLs no This seemed to cure it on our servers (Linux & Solaris). Not sure it was quite the same problem, though. Jon Jon Armitage Systems Administrator 365 Media Group PS--Hope it's the sands you are building this for :) ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
