Howdy-
I just wanted to pop in and provide the latest update on our saga (clamd 0.83 just stops playing nice after running for a while) with some more interesting information like stack traces.
Last we left off I had just upped the ulimit for the clamd process from the default of 256 fds to 1024. I can't tell if this truly helped things, but the number of times a day our babysitting process restarted clamd because it couldn't connect went down considerably. We have whole stretches of days at a time with nary a restart. Things got a little worse today for no reason I can discern.
Since the time since people began to help us, I have periodically checked in on the process descriptors with pfiles and the memory on the machine and neither seemed to be even close to being pegged. We also switched from using a network socket to a local unix socket just to eliminate any funny business. You may take comfort knowing that we're going to be rev'ing everything (all dependent libraries and our MTA) on Wednesday to eliminate all of those possibilities as well.
Today I managed to catch clamd in a hung state and so I poked and prodded at it with gdb. Btw, by hung I mean that attempts to contact clamd on the local socket failed with "connection refused" from clamdmon.
I wasn't quite sure what I was looking for, so the following might be too little, two much, or the wrong info. If there was something I should have done, please let me know and I'll do it next time. Here's what I found:
info threads 2 LWP 271 0xfef1e878 in _read () from /usr/lib/libc.so.1 * 1 LWP 1 0xfee45dd4 in __lwp_park () from /usr/lib/libthread.so.1
Thread 1, presumably the thing that should be listening for new connects:
#0 0xfee45dd4 in __lwp_park () from /usr/lib/libthread.so.1 #1 0xfee430ec in cond_wait_queue () from /usr/lib/libthread.so.1 #2 0xfee438a8 in cond_wait () from /usr/lib/libthread.so.1 #3 0xfee438e4 in pthread_cond_wait () from /usr/lib/libthread.so.1 #4 0x0001864c in thrmgr_destroy () #5 0x0001a19c in acceptloop_th () #6 0x00017ac4 in localserver () #7 0x00017190 in clamd () #8 0x00015d5c in main ()
A truss confirmed that it just stayed parked like that.
Thread 2 (which was going like a busy bee, appearing to actually still be scanning based on a truss of the process):
thread 2 [Switching to thread 2 (LWP 271)]#0 0xfef1e878 in _read () from /usr/lib/libc.so.1 (gdb) where #0 0xfef1e878 in _read () from /usr/lib/libc.so.1 #1 0xfee3dd90 in read () from /usr/lib/libthread.so.1 #2 0xff30b570 in cli_scandesc () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #3 0xff319b18 in cli_magic_scandesc () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #4 0xff319f74 in cli_scanfile () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #5 0xff31794c in cli_scandir () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #6 0xff3191d4 in cli_scanmail () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #7 0xff319cc4 in cli_magic_scandesc () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #8 0xff319f74 in cli_scanfile () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #9 0xff31794c in cli_scandir () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #10 0xff3191d4 in cli_scanmail () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #11 0xff319cc4 in cli_magic_scandesc () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #12 0xff319f74 in cli_scanfile () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #13 0xff31794c in cli_scandir () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #14 0xff3191d4 in cli_scanmail () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #15 0xff319cc4 in cli_magic_scandesc () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #16 0xff319f74 in cli_scanfile () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #17 0xff31794c in cli_scandir () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #18 0xff3191d4 in cli_scanmail () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #19 0xff319cc4 in cli_magic_scandesc () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #20 0xff319f74 in cli_scanfile () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #21 0xff31794c in cli_scandir () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #22 0xff3191d4 in cli_scanmail () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #23 0xff3197e4 in cli_magic_scandesc () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #24 0xff319f74 in cli_scanfile () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #25 0xff31794c in cli_scandir () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #26 0xff3191d4 in cli_scanmail () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #27 0xff3197e4 in cli_magic_scandesc () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #28 0xff319f74 in cli_scanfile () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #29 0xff31794c in cli_scandir () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #30 0xff3191d4 in cli_scanmail () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #31 0xff319cc4 in cli_magic_scandesc () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #32 0xff319f74 in cli_scanfile () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #33 0xff31794c in cli_scandir () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #34 0xff3191d4 in cli_scanmail () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #35 0xff319cc4 in cli_magic_scandesc () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #36 0xff319f74 in cli_scanfile () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #37 0xff31794c in cli_scandir () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #38 0xff3191d4 in cli_scanmail () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #39 0xff3197e4 in cli_magic_scandesc () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #40 0xff319f74 in cli_scanfile () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #41 0xff31794c in cli_scandir () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #42 0xff3191d4 in cli_scanmail () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #43 0xff319cc4 in cli_magic_scandesc () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #44 0xff319f74 in cli_scanfile () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #45 0xff31794c in cli_scandir () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #46 0xff3191d4 in cli_scanmail () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #47 0xff3197e4 in cli_magic_scandesc () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #48 0xff319f74 in cli_scanfile () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #49 0xff31794c in cli_scandir () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #50 0xff3191d4 in cli_scanmail () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #51 0xff319cc4 in cli_magic_scandesc () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #52 0xff319f74 in cli_scanfile () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #53 0xff31794c in cli_scandir () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #54 0xff3191d4 in cli_scanmail () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #55 0xff3197e4 in cli_magic_scandesc () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #56 0xff319f74 in cli_scanfile () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #57 0xff31794c in cli_scandir () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #58 0xff3191d4 in cli_scanmail () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #59 0xff319cc4 in cli_magic_scandesc () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #60 0xff319ee4 in cl_scandesc () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #61 0xff31a008 in cl_scanfile () from /priv/daemons/packages/clamav-0.83/lib/libclamav.so.1 #62 0x0001a850 in dirscan () #63 0x0001ad20 in scan () #64 0x00017ca4 in command () #65 0x00018dc4 in scanner_thread () #66 0x00018a20 in thrmgr_worker ()
Here's a pfiles from when it was in this state:
19239: /priv/daemons/packages/clamav-0.83/sbin/clamd
Current rlimit: 1024 file descriptors
0: S_IFSOCK mode:0666 dev:275,0 ino:21256 uid:0 gid:0 size:0
O_RDWR
sockname: AF_UNIX /etc/exim/clamd-socket
1: S_IFCHR mode:0666 dev:118,8 ino:214047 uid:0 gid:3 rdev:13,2
O_WRONLY
2: S_IFCHR mode:0666 dev:118,8 ino:214047 uid:0 gid:3 rdev:13,2
O_WRONLY
3: S_IFREG mode:0644 dev:118,15 ino:50702 uid:9800 gid:1159 size:175995
O_WRONLY|O_APPEND|O_CREAT
4: S_IFCHR mode:0666 dev:118,8 ino:214043 uid:0 gid:3 rdev:21,0
O_WRONLY FD_CLOEXEC
5: S_IFCHR mode:0666 dev:118,8 ino:214047 uid:0 gid:3 rdev:13,2
O_WRONLY
6: S_IFREG mode:0600 dev:118,13 ino:20842 uid:0 gid:1 size:43799
O_RDONLY
7: S_IFSOCK mode:0666 dev:275,0 ino:51353 uid:0 gid:0 size:0
O_RDWR
sockname: AF_UNIX /etc/exim/clamd-socket
peername: AF_UNIX
8: S_IFDIR mode:0750 dev:118,13 ino:20786 uid:822 gid:822 size:2048
O_RDONLY|O_NDELAY|O_LARGEFILE FD_CLOEXEC
9: S_IFREG mode:0666 dev:118,13 ino:20793 uid:822 gid:822 size:56857
O_RDONLY
10: S_IFDIR mode:0700 dev:118,13 ino:20127 uid:0 gid:1 size:512
O_RDONLY|O_NDELAY|O_LARGEFILE FD_CLOEXEC
11: S_IFREG mode:0600 dev:118,13 ino:20833 uid:0 gid:1 size:48317
O_RDONLY
12: S_IFDIR mode:0700 dev:118,13 ino:20835 uid:0 gid:1 size:512
O_RDONLY|O_NDELAY|O_LARGEFILE FD_CLOEXEC
13: S_IFREG mode:0600 dev:118,13 ino:20836 uid:0 gid:1 size:44353
O_RDONLY
14: S_IFDIR mode:0700 dev:118,13 ino:20839 uid:0 gid:1 size:512
O_RDONLY|O_NDELAY|O_LARGEFILE FD_CLOEXEC
15: S_IFDIR mode:0700 dev:118,13 ino:14761 uid:0 gid:1 size:512
O_RDONLY|O_NDELAY|O_LARGEFILE FD_CLOEXEC
16: S_IFREG mode:0600 dev:118,13 ino:14805 uid:0 gid:1 size:42257
O_RDONLY
17: S_IFDIR mode:0700 dev:118,13 ino:14719 uid:0 gid:1 size:512
O_RDONLY|O_NDELAY|O_LARGEFILE FD_CLOEXEC
18: S_IFREG mode:0600 dev:118,13 ino:14772 uid:0 gid:1 size:40320
O_RDONLY
19: S_IFDIR mode:0700 dev:118,13 ino:14789 uid:0 gid:1 size:512
O_RDONLY|O_NDELAY|O_LARGEFILE FD_CLOEXEC
20: S_IFREG mode:0600 dev:118,13 ino:14829 uid:0 gid:1 size:38122
O_RDONLY
21: S_IFDIR mode:0700 dev:118,13 ino:14833 uid:0 gid:1 size:512
O_RDONLY|O_NDELAY|O_LARGEFILE FD_CLOEXEC
22: S_IFREG mode:0600 dev:118,13 ino:14848 uid:0 gid:1 size:35606
O_RDONLY
23: S_IFDIR mode:0700 dev:118,13 ino:14878 uid:0 gid:1 size:512
O_RDONLY|O_NDELAY|O_LARGEFILE FD_CLOEXEC
24: S_IFREG mode:0600 dev:118,13 ino:14879 uid:0 gid:1 size:31949
O_RDONLY
25: S_IFDIR mode:0700 dev:118,13 ino:14891 uid:0 gid:1 size:512
O_RDONLY|O_NDELAY|O_LARGEFILE FD_CLOEXEC
26: S_IFREG mode:0600 dev:118,13 ino:14937 uid:0 gid:1 size:31441
O_RDONLY
27: S_IFDIR mode:0700 dev:118,13 ino:14966 uid:0 gid:1 size:512
O_RDONLY|O_NDELAY|O_LARGEFILE FD_CLOEXEC
28: S_IFREG mode:0600 dev:118,13 ino:14959 uid:0 gid:1 size:29775
O_RDONLY
29: S_IFDIR mode:0700 dev:118,13 ino:14954 uid:0 gid:1 size:512
O_RDONLY|O_NDELAY|O_LARGEFILE FD_CLOEXEC
30: S_IFREG mode:0600 dev:118,13 ino:15024 uid:0 gid:1 size:25105
O_RDONLY
31: S_IFDIR mode:0700 dev:118,13 ino:18381 uid:0 gid:1 size:512
O_RDONLY|O_NDELAY|O_LARGEFILE FD_CLOEXEC
32: S_IFREG mode:0600 dev:118,13 ino:18399 uid:0 gid:1 size:21825
O_RDONLY
33: S_IFDIR mode:0700 dev:118,13 ino:18403 uid:0 gid:1 size:512
O_RDONLY|O_NDELAY|O_LARGEFILE FD_CLOEXEC
34: S_IFREG mode:0600 dev:118,13 ino:18461 uid:0 gid:1 size:19325
O_RDONLY
35: S_IFDIR mode:0700 dev:118,13 ino:18463 uid:0 gid:1 size:512
O_RDONLY|O_NDELAY|O_LARGEFILE FD_CLOEXEC
36: S_IFREG mode:0600 dev:118,13 ino:18465 uid:0 gid:1 size:18530
O_RDONLY
37: S_IFREG mode:0600 dev:118,13 ino:18471 uid:0 gid:1 size:0
O_RDWR|O_CREAT|O_EXCL
I'm not quite sure how to interpret this information. Does this mean the main thread was parked waiting for the second to complete what it was doing? Something else entirely going on?
Thanks again for any help you can offer.
-- dNb
_______________________________________________ http://lurker.clamav.net/list/clamav-users.html
