Howdy-
We've recently been seeing our clamd processes run very hot (spiking
up to 85% of the CPU as reported by prtstat and top) on two different
Solaris 9 boxes. For example, here's a few lines from prtstat -L
(showing the two clamav threads who are together eating 66% of the
CPU) .
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/LWPID
18447 root 61M 60M run 0 0 0:27:51 34% clamd/14
18447 root 61M 60M cpu0 20 0 0:25:40 32% clamd/22
Both machines are running 0.88.2 clamds being fed by exim. We'll
upgrade to 88.4 shortly, but I haven't seen anything in the release
notes that mentions this issue. I've looked at all of the obvious
things I can think of (we are not swapping due to lack of memory as
far as I can tell, not out of any resources like file descriptors,
and so on) so I was hoping someone here could offer some advice on
any other avenues to pursue. I've even done things like moving the
temp directory to a tmpfs filesystem, but that hasn't helped the load
any.
I've included gdb backtraces for all threads below and also a copy of
the truss counts for the process so you can see which system calls
are being made (and which are giving errors, probably normally). I've
included our non-default config options below. I've also included an
ldd of the binary so you can see that we have it linked against
almost all standard Solaris libs (all of which I think are patched
properly, let me know if there is something I should check there).
If there is anything else I should be checking, please let me know.
Thanks for any help you can offer.
-- dNb
GDB:
(gdb) info thread
5 LWP 63 0xff0858f4 in __lwp_park () from /usr/lib/libthread.so.1
4 LWP 14 0xff330d98 in getline_from_mbox ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
3 LWP 62 0xff0858f4 in __lwp_park () from /usr/lib/libthread.so.1
2 LWP 22 0xff309c98 in cli_ac_scanbuff ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
* 1 LWP 1 0xff01c934 in _so_accept () from /usr/lib/libc.so.1
(gdb) bt
#0 0xff01c934 in _so_accept () from /usr/lib/libc.so.1
#1 0x0001a95c in acceptloop_th ()
#2 0x00018514 in localserver ()
#3 0x00017be0 in clamd ()
#4 0x00016758 in main ()
(gdb) thread 2
[Switching to thread 2 (LWP 22)]#0 0xff309c98 in cli_ac_scanbuff ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
(gdb) bt
#0 0xff309c98 in cli_ac_scanbuff ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#1 0xff30bd50 in cli_scandesc ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#2 0xff31cca8 in cli_scanraw ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#3 0xff31d864 in cli_magic_scandesc ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#4 0xff31daa4 in cli_scanfile ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#5 0xff31b134 in cli_scandir ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#6 0xff31cb90 in cli_scanmail ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#7 0xff31d4d8 in cli_magic_scandesc ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#8 0xff31daa4 in cli_scanfile ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#9 0xff31b134 in cli_scandir ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#10 0xff31cb90 in cli_scanmail ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#11 0xff31d4d8 in cli_magic_scandesc ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#12 0xff31daa4 in cli_scanfile ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#13 0xff31b134 in cli_scandir ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#14 0xff31cb90 in cli_scanmail ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#15 0xff31d4d8 in cli_magic_scandesc ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#16 0xff31daa4 in cli_scanfile ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#17 0xff31b134 in cli_scandir ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#18 0xff31cb90 in cli_scanmail ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#19 0xff31d4d8 in cli_magic_scandesc ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#20 0xff31daa4 in cli_scanfile ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#21 0xff31b134 in cli_scandir ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#22 0xff31cb90 in cli_scanmail ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#23 0xff31d4d8 in cli_magic_scandesc ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#24 0xff31daa4 in cli_scanfile ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#25 0xff31b134 in cli_scandir ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#26 0xff31cb90 in cli_scanmail ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#27 0xff31d4d8 in cli_magic_scandesc ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#28 0xff31daa4 in cli_scanfile ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#29 0xff31b134 in cli_scandir ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#30 0xff31cb90 in cli_scanmail ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#31 0xff31d4d8 in cli_magic_scandesc ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#32 0xff31daa4 in cli_scanfile ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#33 0xff31b134 in cli_scandir ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#34 0xff31cb90 in cli_scanmail ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#35 0xff31d4d8 in cli_magic_scandesc ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#36 0xff31daa4 in cli_scanfile ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#37 0xff31b134 in cli_scandir ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#38 0xff31cb90 in cli_scanmail ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#39 0xff31d4d8 in cli_magic_scandesc ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#40 0xff31da14 in cl_scandesc ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#41 0xff31db38 in cl_scanfile ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#42 0x0001b2e0 in dirscan ()
#43 0x0001b7b8 in scan ()
#44 0x000186f4 in command ()
#45 0x000197d0 in scanner_thread ()
#46 0x00019470 in thrmgr_worker ()
(gdb) thread 3
[Switching to thread 3 (LWP 62)]#0 0xff0858f4 in __lwp_park ()
from /usr/lib/libthread.so.1
(gdb) bt
#0 0xff0858f4 in __lwp_park () from /usr/lib/libthread.so.1
#1 0xff082ba4 in cond_wait_queue () from /usr/lib/libthread.so.1
#2 0xff08311c in cond_wait_common () from /usr/lib/libthread.so.1
#3 0xff0835ac in _ti_cond_timedwait () from /usr/lib/libthread.so.1
#4 0xff0835e0 in cond_timedwait () from /usr/lib/libthread.so.1
#5 0xff083620 in pthread_cond_timedwait () from /usr/lib/libthread.so.1
#6 0x000193b8 in thrmgr_worker ()
(gdb) thread 4
[Switching to thread 4 (LWP 14)]#0 0xff330d98 in getline_from_mbox ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
(gdb) bt
#0 0xff330d98 in getline_from_mbox ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#1 0xff32b364 in parseEmailFile ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#2 0xff32aa98 in cli_parse_mbox ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#3 0xff32a408 in cli_mbox ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#4 0xff31cb00 in cli_scanmail ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#5 0xff31d4d8 in cli_magic_scandesc ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#6 0xff31daa4 in cli_scanfile ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#7 0xff31b134 in cli_scandir ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#8 0xff31cb90 in cli_scanmail ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#9 0xff31d4d8 in cli_magic_scandesc ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#10 0xff31daa4 in cli_scanfile ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#11 0xff31b134 in cli_scandir ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#12 0xff31cb90 in cli_scanmail ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#13 0xff31d4d8 in cli_magic_scandesc ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#14 0xff31daa4 in cli_scanfile ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#15 0xff31b134 in cli_scandir ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#16 0xff31cb90 in cli_scanmail ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#17 0xff31d4d8 in cli_magic_scandesc ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#18 0xff31daa4 in cli_scanfile ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#19 0xff31b134 in cli_scandir ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#20 0xff31cb90 in cli_scanmail ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#21 0xff31d4d8 in cli_magic_scandesc ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#22 0xff31daa4 in cli_scanfile ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#23 0xff31b134 in cli_scandir ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#24 0xff31cb90 in cli_scanmail ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#25 0xff31d4d8 in cli_magic_scandesc ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#26 0xff31daa4 in cli_scanfile ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#27 0xff31b134 in cli_scandir ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#28 0xff31cb90 in cli_scanmail ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#29 0xff31d4d8 in cli_magic_scandesc ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#30 0xff31daa4 in cli_scanfile ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#31 0xff31b134 in cli_scandir ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#32 0xff31cb90 in cli_scanmail ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#33 0xff31d4d8 in cli_magic_scandesc ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#34 0xff31da14 in cl_scandesc ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#35 0xff31db38 in cl_scanfile ()
from /arch/daemons/packages/clamav-0.88.2/lib/libclamav.so.1
#36 0x0001b2e0 in dirscan ()
#37 0x0001b7b8 in scan ()
#38 0x000186f4 in command ()
#39 0x000197d0 in scanner_thread ()
#40 0x00019470 in thrmgr_worker ()
TRUSS:
truss -c -p 18447
^Csyscall seconds calls errors
read .601 6979
write .227 5305
open .048 1479 2
close .036 1594
unlink .047 432
time .001 119
chmod .004 172
stat .013 542 188
lseek .017 1636
getpid .005 198
fstat .006 442
access .001 49
dup .001 99
times .053 6016
ioctl .002 272 272
umask .000 10
fcntl .004 408
lwp_park .002 73 2
lwp_unpark .001 71
rmdir .012 437 172
mkdir .010 281
putmsg .000 10
poll .006 312
lstat .024 893
fchmod .000 5
sigprocmask .001 104
setustack .000 1
mmap .005 94
munmap .003 94
lwp_create .000 1
lwp_exit .000 2
lwp_continue .000 1
lwp_kill .000 1 1
llseek .004 443
door_info .000 10
door_call .000 10
lwp_schedctl .000 1
getdents64 .012 781
lstat64 .005 178 178
fstat64 .013 952
accept .011 49
recvmsg .001 49
send .002 49
-------- ------ ----
sys totals: 1.194 30654 815
usr time: 158.677
elapsed: 114.170
CONFIG:
LogFile /priv/log/clamd/clamd.log
LogFileMaxSize 20M
LogTime
LogSyslog
LogFacility LOG_MAIL
TemporaryDirectory /var/clamd
DatabaseDirectory /arch/daemons/packages/clamav-0.88.2/share/clamav
LocalSocket /etc/exim/clamd-socket
FixStaleSocket
TCPAddr 127.0.0.1
MaxConnectionQueueLength 30
StreamMaxLength 20M
MaxThreads 20
ArchiveBlockMax
LDD:
# ldd ../sbin/clamd
libclamav.so.1 => /arch/daemons/packages/
clamav-0.88.2/lib/libclamav.so.1
libz.so.1 => /usr/lib/libz.so.1
libbz2.so.1 => /usr/lib/libbz2.so.1
libgmp.so.3 => /arch/gnu/packages/gmp-4.0/lib/libgmp.so.3
libsocket.so.1 => /usr/lib/libsocket.so.1
libnsl.so.1 => /usr/lib/libnsl.so.1
libpthread.so.1 => /usr/lib/libpthread.so.1
libresolv.so.2 => /usr/lib/libresolv.so.2
libc.so.1 => /usr/lib/libc.so.1
libdl.so.1 => /usr/lib/libdl.so.1
libmp.so.2 => /usr/lib/libmp.so.2
libthread.so.1 => /usr/lib/libthread.so.1
/usr/platform/SUNW,Sun-Fire-280R/lib/libc_psr.so.1
_______________________________________________
http://lurker.clamav.net/list/clamav-users.html
