Török Edwin wrote: > On 2009-04-10 18:04, Jay Deiman wrote: >> I have noticed a serious problem with the latest stable branch of >> clamav, at least on FreeBSD. There seems to be a large scale memory >> leak. Starting with 0.92, I have been running a 10 machine clamd >> cluster, 9 of which running FreeBSD 6.2 RELEASE and one running 7.0 >> RELEASE. With the exception of a couple of hiccups in the past, they >> have been running pretty flawlessly, serving about 30 mail machines >> running clamav-milter. >> >> With the advent of the vulnerabilities in pre 0.95 releases, I upgraded >> to 0.95 a week ago. With the latest round of vulns., I upgraded to >> 0.95.1 yesterday. With both revisions, I'm seeing a constant escalation >> of memory usage to the point of killing the clamd process because it >> runs into the MAXDSIZ we have set of 1.5GBs of memory. Restarting the >> process just restarts the "fail counter" once again. >> > > Does it get killed by a signal, or does it exit due to 'ExitOnOOM yes' > in clamd.conf? > malloc should return NULL when out of memory, and ClamAV should be able > to handle it without crashing, if not its a bug.
Sorry, I should have been more specific. It doesn't actually crash. Here are the log messages I end up getting: ============================= Apr 8 21:53:41 mpls-clamav-04 clamd[44009]: fds_add failed Apr 8 21:53:41 mpls-clamav-04 clamd[44009]: add_fd: Memory allocation failed for command buffer ============================= At first glance, I thought I was running out of file descriptors, which was not the case. I looked at memory usage after that and noticed that I was sitting at 1.5GBs, my configured OS hard limit. > > >> In contrast, I never had any problems with 0.94.2, or the entire 0.94 >> branch for that matter. The memory usage for those versions would sit >> consistently between 100MB and 300MB of resident memory usage depending >> on load. I haven't seen anything from anyone else on the mailing list >> about this, so I'm not sure if this is something related specific to >> FreeBSD or, if it's the massive load we put on this cluster (these 10 >> machines together process approximately 65 million emails per day) that >> allows us to see this clearly, but it is quite clear to us. >> >> If there is any more info I can supply, I will do my best to do so. >> > > Usually I would say run it under valgrind, but with such a massive load > that isn't practical. > Try to get a stacktrace when clamd crashes, see "Backtrace of clamd" on > clamav.net/bugs for instructions. > > Also there is some info here on how to trace leaks on FreeBSD: > http://keramida.wordpress.com/2008/10/15/extracting-useful-info-from-freebsd-malloc-tracing/ > Unfortunately that trace only gives you the timestamp when the memory > was allocated, and not source lines. > However if you turn on LogClean and LogTime in clamd, you may be able to > match the leaks to files that were scanned at the time. > > Then you can try scanning only those files, and see if you can reproduce > the leak. Cool, there's an addition to ktrace I didn't know about. I will set this up on one of the hosts and see what I can figure out. Hopefully I will be able to report back with some good information later on today. Thanks, Jay Deiman -- Jay Deiman \033:wq! _______________________________________________ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
