[users] Suddenly increased memory appetite
I'm running 0.24 under Red Hat 7.2 and qmail-scanner 1.14. clamscan has been working fine for weeks with a 20,000,000 byte memory limit. This morning--perhaps after the virus database was updated--I started seeing the following error: CRITICAL: Can't allocate memory (2064 bytes). I raised the limit to 25 MB, 35 MB, and finally 50 MB before it stopped complaining. Why did it suddenly get so hungry? Would 0.51 help? -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [users] cannot load Clamuko
Tomasz Kojm <[EMAIL PROTECTED]> wrote: > >You must run the daemon as root (disable "User clamav" option). Hopefully that's a temporary restriction... -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [users] Report file
Odhiambo Washington <[EMAIL PROTECTED]> wrote: > >It's a set-table value, AFAIK. In the two MLMs that I've used so far >I know it is. I don't know about what these guys user here though;-) >Looks like their own home-brewed one. It's ezmlm. Personally, I think subject prefixes are annoying and a waste of space. -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [users] Report file
[Please don't top-post. If you don't know what top-posting is, check Google.] Brian Read <[EMAIL PROTECTED]> wrote: >Not at all, [subject tags] are ideal to use for email rules to sort >into different folders. No, they suck for filtering messages into different folders for exactly the reason that this thread is taking place: they're not guaranteed to be unique. A much better field to filter upon is Return-Path. If your MUA is lame and can't filter by regular expressions--which is necessary for lists that use VERPs--then you can use Mailing-List. I use qmail extension addresses: I subscribe to each list with a unique address and file incoming messages without filtering using a .qmail file dedicated to the list's subscribed address. No filtering overhead, no unreliable heuristics, and no trash necessary in the Subject field. The eight characters devoted to "[users] " is noise that just serves to push some of the real information in the Subject off the screen. >I get between 50-100 emails a day, and need to prioritise which i >read immediatly, and which I leave until later. That's a small fraction of the number I get, but let's not start a mailbox size contest. -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [users] Report file
Brian Read <[EMAIL PROTECTED]> wrote: >At 13:30 16/10/2002, you wrote: >>[Please don't top-post. If you don't know what top-posting is, check >>Google.] > >sorry, a slip of the brain. No problem. >The RP on this message is meaningless via a vis the mail list. > >"Return-Path: <[EMAIL PROTECTED]> " That's not right. Your MTA or MUA is lying to you. The proper Return-Path should look like: Return-Path: <[EMAIL PROTECTED]> >I am still on Windoze I am afraid. Practice what you preach, Mr. www.abandonmicrosoft.co.uk. :-) -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [clamav-users] Signature updates: Where from?
Glenn Rice <[EMAIL PROTECTED]> writes: > But, the only thing that stops me putting it on our production server > is, being an O/S project how regular are the signature updates? And is > there some kind of system in place to keep up with all the new viruses > coming out? If the only thing protecting your hordes of Windows users from the wide world is a virus scanner, you might not want to rely on ClamAV at this point. Some of the commercial scanners have more complete databases and update them more frequently. If you're running ClamAV in addition to a commercial scanner, or using it to filter out junk e-mail messages, then its database is complete enough, is updated regularly enough, and clamscan is reliable enough. -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [clamav-users] Clamav and qmail - your experiences and opinions
Tomasz Kojm <[EMAIL PROTECTED]> writes: > On Wed, Mar 19, 2003 at 05:25:41PM +0100, Daniel Wiberg wrote: > > What are these modifications? Just "sed -e s/clamscan/clamdscan/ > > qmail-scanner-queue.pl"? > > exactly OK, I tried that on my RH 8.0, clamav-20030317, qmail-scanner-1.16 system and got: 19/03/2003 15:53:06:3757: --output of clamscan was: /var/spool/qmailscan/sws510481071864263757: Can't stat() the file ERROR -- (After running clamd, of course.) clamdscan run interactively seems to work OK: $ clamdscan . /home/de5/./eicar.txt: Eicar-Test-Signature FOUND /home/de5/./qmail-scanner-1.14.tgz: Eicar-Test-Signature FOUND /home/de5/./sws1: Eicar-Test-Signature FOUND /home/de5/./virus.msg: Exploit.IFrame FOUND --- SCAN SUMMARY --- Infected files: 4 Time: 113.458 sec (1 m 53 s) $ Scratch that. I just ran clamscan on the same directory and it found a more: $ clamscan . 2>&1 |grep FOUND ./.SAVE-2002-02-15: Exploit.IFrame FOUND ./clamav-0.54.tar.gz: ClamAV-Test-Signature FOUND ./eicar.txt: Eicar-Test-Signature FOUND ./qmail-scanner-1.14.tgz: Eicar-Test-Signature FOUND ./sws1: Eicar-Test-Signature FOUND ./virus.msg: Exploit.IFrame FOUND ./clamav-20030317.tar.gz: ClamAV-Test-Signature FOUND $ -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [clamav-users] Clamav and qmail - your experiences and opinions
Tomasz Kojm <[EMAIL PROTECTED]> writes: > On Wed, Mar 19, 2003 at 04:09:22PM -0500, Dave Sill wrote: > > > > OK, I tried that on my RH 8.0, clamav-20030317, qmail-scanner-1.16 > > system and got: > > > > 19/03/2003 15:53:06:3757: --output of clamscan was: > > /var/spool/qmailscan/sws510481071864263757: Can't stat() the file ERROR > > This is a permission problem. Run clamd with a proper UID and GID (check > ls -l /var/spool/qmailscan). So clamd needs to run as a user with access to the files to be scanned. That seems reasonable, except that means it needs to run as root in order to be able to scan any file...and that's not something I'm keen to do. I guess it'd be OK to run clamd as qmaild, the user that owns /var/spool/qmailscan. This is pretty important limitation to using clamd/clamdscan. Is it documented? -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [clamav-users] Clamav and qmail - your experiences and opinions
Ed Phillips <[EMAIL PROTECTED]> writes: > On Thu, 20 Mar 2003, Dave Sill wrote: > > > This is pretty important limitation to using clamd/clamdscan. Is it > > documented? > > Sure... run "man intro" on most any Unix system and read the part about > permissions, uids, etc., ... ;-) Very funny, Ed. I was referring to the fact that clamscan and clamdscan behave differently because clamd has to have access to the file, not just clamdscan. > Of course, for a process to be able to read ANY file on a Unix system the > process needs to be running with uid 0, or the files themselves need to > have proper permissions set. There's not really anything ClamAV can do to > change these simple facts. Did you think clamd would somehow be able to > bypass normal Unix file permissions? What would you like clamd to do > exactly? Well, I guess I thought clamdscan sent the file to clamd. Obviously I was wrong. > In our setup, we use sendmail + MIMEDefang + clamd. When the email > messages/attachments are broken out by MD to be scanned, they are owned by > the user that MIMEDefang runs as (in our case, "defang"). So, we just > make clamd run as "defang" so it can scan the mail files. Yes, that's exactly what I suggested in my message. The problem with that is that clamdscan then only works right for scanning files that that user has access to. If someone thinks clamdscan works just like clamscan[1], he probably won't expect clamd to silently skip files it can't access. -Dave Footnotes: [1] s/clamscan/clamdscan/ sound familiar? - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[clamav-users] clamscan --disable-cache
The clamscan man page says: --disable-cache Disable caching and cache checks for hash sums of scanned files. I've looked high and low via google, strace, looking at source code, conducting tests, and I see no sign of caching done by clamscan. Is this on the to-do list? We'd like to regularly scan systems but the overhead of scanning the same files repeatedly is a bit much. A per-system cache would be good. Even better would be a centralized, local to our network, service that clamscan could check & update. I suppose that would be pretty easy to add on top. Anyone done anything like that? -Dave ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] [EXTERNAL] Re: clamscan --disable-cache
"G.W. Haywood via clamav-users" wrote: > > In the second scan, how did clamscan manage to do what it claims to > have done in the time that it did it? OK, you could have just said that the cache is internal to each invocation of clamscan, but that helps. > For further enlightenment, on one of your systems try doing something > similar to what I did above but using 'clamdscan'. The problem with clamdscan is that it runs into permissions since it's not running as root. > Consider using a > central clamd server for all your scanning needs. How would that work? Clamd only scans files on the system on which it's running. > I doubt anyone is doing that. I'm sure it isn't necessary, as it's > already taken care of by both clamscan and clamd. Perhaps if you can > be a bit more forthcoming about your use case(s) we may be able to > help reduce scan times. One of the best ways of doing that is not to > scan so much junk so often. We've got about 3000 Linux systems that we'd like to periodically scan, primarily to ensure that they're not being used to redistribute Windows malware. We'd like to scan all of the local file systems for completeness. Any attempt to skip "junk" will potentially skip malware, and hand crafting scans for each system is not an option. Skipping multiple copies of the same file won't really help because the duplication is across systems, and because every file will be rescanned every time clamscan is run. We could do a full scan on the first run and then weekly scans of files modified in the past week. That's kludgy but may be the best we can do. -Dave ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] clamscan --disable-cache
Andrew C Aitchison via clamav-users wrote: > > No. clamD scans data passed to it by clamdscan, usually over a socket or > pipe. Ah... I missed INSTREAM in the clamd man page. Locally, though, surely SCAN/CONTSCAN/etc, are nuch more efficient. And remotely, sending the entire contents of the system over the net isn't practical at scale. > That does mean that any malware which is missed in the first run > will not be detected in subsequent runs. True. I suppose we'd want to do monthly full scans. > 3000 machines per week, gives you about 3.36 minutes for each machine to > send all its local data to the scanning machine. > Instead I would run a local, mirror, repository of the database > and use freshclam on each machine to keep its database in sync with your > mirror, then run clamd and a clamdscan cron? script on each machine. We've already got a local mirror. Is there a way to get clamd/clamdscan to work without permission problems beside running clamd as root? Does --fdpass get around that? > I would also look at on-access scanning. I tried it but got permission errors on anything not world-accessible. I suspect the overall performance hit would be too high. > Scanning files as they are used might mean more or less work > than scanning every file every week. Except full dumps are going to cause everything to be scanned. -Dave ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] clamscan --disable-cache
"G.W. Haywood via clamav-users" wrote: > > There are ways around that, even if you don't want to run clamdscan > (and clamd) as root - which I'd entirely understand. Is --fdpass one of them? And --stream? Any others? > >We've got about 3000 Linux systems that we'd like to periodically scan, > >primarily to ensure that they're not being used to redistribute > >Windows malware. > > A good use case, perhaps quite a tall order with a single clamd server > but maybe doable if you can (a) limit what needs to be scanned and (b) > define 'periodically' in terms of days (at least) and not hours. We could use multiple clamd servers. And periodically in terms of days would be OK. > >Any attempt to skip "junk" will potentially skip malware, and hand > >crafting scans for each system is not an option. > > That seems more like a management problem to me than a technical one. The nature of our environment precludes actively managing all of the Linux systems here, unfortunately. > >Skipping multiple copies of the same file won't really help because > >the duplication is across systems, and because every file will be > >rescanned every time clamscan is run. > > That's not true of clamdscan. Hmm...that's promising. I'll give it a try. > And you probably won't know what's been modified in the past week unless > you install Tripwire or something like that... mtime would be sufficient for our purposes. -Dave ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
[clamav-users] clamd cache (was Re: clamscan --disable-cache)
Dave Sill via clamav-users wrote: > > > >Skipping multiple copies of the same file won't really help because > > >the duplication is across systems, and because every file will be > > >rescanned every time clamscan is run. > > > > That's not true of clamdscan. > > Hmm...that's promising. I'll give it a try. Unfortunately, it looks like the cache is too small to help. I ran clamdscan twice on my /home (69k files) and got: # clamdscan --fdpass /home /home/de5/eicar.tar.gz: Eicar-Signature FOUND WARNING: /home/de5/.cisco/hostscan/.libcsd.ipc: Not supported file type WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached /home/s4i/eicar.tar.gz: Eicar-Signature FOUND --- SCAN SUMMARY --- Infected files: 2 Total errors: 1 Time: 1428.433 sec (23 m 48 s) # clamdscan --fdpass /home /home/de5/eicar.tar.gz: Eicar-Signature FOUND WARNING: /home/de5/.cisco/hostscan/.libcsd.ipc: Not supported file type WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached WARNING: Directory recursion limit reached /home/s4i/eicar.tar.gz: Eicar-Signature FOUND --- SCAN SUMMARY --- Infected files: 2 Total errors: 1 Time: 1355.057 sec (22 m 35 s) # But on /boot, with 342 files: # clamdscan --fdpass /boot /boot: OK --- SCAN SUMMARY --- Infected files: 0 Time: 21.186 sec (0 m 21 s) # clamdscan --fdpass /boot /boot: OK --- SCAN SUMMARY --- Infected files: 0 Time: 0.362 sec (0 m 0 s) # I don't see that the cache size is run-time configurable. Is that right? -Dave ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] clamd cache (was Re: clamscan --disable-cache)
It looks like my point was lost in the noise so I'll try to distill it. I ran clamdscan twice on my /home (69k files) and got: # clamdscan --fdpass /home ... Time: 1428.433 sec (23 m 48 s) # clamdscan --fdpass /home ... Time: 1355.057 sec (22 m 35 s) # The cache only saved a little over a minute on a 24 minute scan. But on /boot, with 342 files: # clamdscan --fdpass /boot ... Time: 21.186 sec (0 m 21 s) # clamdscan --fdpass /boot ... Time: 0.362 sec (0 m 0 s) # Here, on a much smaller scan, the cache made a huge difference. That tells me that the cache isn't large enough to significantly speed up large scans. I don't see that the cache size is run-time configurable. Is that right? -Dave ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] clamd cache (was Re: clamscan --disable-cache)
"G.W. Haywood via clamav-users" wrote: > Hi there, > > On Thu, 1 Oct 2020, Dave Sill via clamav-users wrote: > > >It looks like my point was lost in the noise ... > > Sorry, I guess it was late and I was in a hurry to get to bed. :( No worries. Thanks for your help. > >... on a much smaller scan, the cache made a huge difference. That > >tells me that the cache isn't large enough to significantly speed up > >large scans. > > It might be too soon to draw that conclusion. It's possible that the > daemon reloaded its database during your test, and I'd expect that to > cause any cached results to be discarded for obvious reasons. Fair enough. I re-ran the same scan three times after rebooting and got the following run times: 20:46 19:37 19:18 And the clamd logs show "SelfCheck: Database status OK" every 10 minutes but no DB updates. > >I don't see that the cache size is run-time configurable. Is that right? > > Correct, but I'd thought its size would be limited only by the RAM you > have free. If you look at the code in libclamav/cache.c you can see > that struct cache_set is just a few pointers, and if you only have 69k > files under your home directory I wouldn't expect storage of that many > sets of pointers to be an issue. > > I'll dig into this a bit more when I have chance if somebody doesn't > beat me to it. I only have 16 GB RAM on this system but it still shows 1 GB free. Maybe it limits itself somehow. -Dave ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] clamd cache (was Re: clamscan --disable-cache)
"G.W. Haywood via clamav-users" wrote: > > Only 4GB on my clamd server. > > $ du -sh images/ > 16G images/ > $ find ./images -type f | wc -l > 11586 > $ clamdscan images/ > ... > Time: 12547.333 sec (209 m 7 s) > ... > $ clamdscan images/ > ... > Time: 1477.782 sec (24 m 37 s) That's a nice boost. > Trying a bigger directory, this is going to take a while... > > $ find ./Personal -type f | wc -l > 144191 I'm trying some tests on a desktop system now, in case the problem is system-specific. -Dave ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] clamd cache (was Re: clamscan --disable-cache)
On the desktop system: $ find Mail -type f|wc -l 123719 # clamdscan --fdpass ~de5/Mail Time: 2137.531 sec (35 m 37 s) # clamdscan --fdpass ~de5/Mail Time: 2138.778 sec (35 m 38 s) So, still not seeing a benefit from the cache. Both of my test systems are RHEL 7, so off to try another platform. -Dave ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] clamd cache (was Re: clamscan --disable-cache)
Dave Sill via clamav-users wrote: > > Both of my test systems are RHEL 7, so off to try another platform. On Fedora 32: # find ~dave/Mail -type f|wc -l 26671 # clamdscan --fdpass ~dave/Mail Time: 932.395 sec (15 m 32 s) # clamdscan --fdpass ~dave/Mail Time: 489.627 sec (8 m 9 s) So that's an improvement. Less than I was hoping for, though. -Dave ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] clamd cache (was Re: clamscan --disable-cache)
"G.W. Haywood via clamav-users" wrote: > > Perhaps try enabling libclamav debug logging. I poked around a bit and didn't see an obvious way to do that, like a configure option or a .h file. Couldn't really tell where it would be logging. > During your scans I suspect that ClamAV may be reaching some limit(s) > which is causing caching to be disabled. The limits are mostly > tunable (in some cases perhaps care may be needed to avoid a DOS). Played around with MemoryLimit=infinite in the systemd unut file and didn't see that it helped. One thing that did help was the -m/--multiscan option to clamdscan. Depending on the number of cores, I've seen scans 2-5x faster. Interestingly, I just ran some more tests on my Fedora 32 desktop, and I see caching working great: # clamdscan --fdpass ~dave/Mail Time: 932.016 sec (15 m 32 s) # clamdscan -m --fdpass ~dave/Mail Time: 90.140 sec (1 m 30 s) # clamdscan -m --fdpass ~dave/Mail Time: 12.845 sec (0 m 12 s) # clamdscan --fdpass ~dave/Mail Time: 9.975 sec (0 m 9 s) # clamdscan -m --fdpass ~dave/Mail Time: 3.102 sec (0 m 3 s) # find ~dave/Mail -type f |wc -l 26675 So, time to test RHEL 8 and Ubuntu. Thanks, Ged! -Dave ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] [EXTERNAL] clamav scan of changed files
"Leveille, Gerald via clamav-users" wrote: > Categorization: Unclassified > Hi, > > I would like to know what would be the best way to do a virus scan of changed > or new files only. I want to run a daily scan of changed and new files during > weekdays and run a full scan on weekends. > > I did some search and was able to find a few ways of doing it but I would > also like your suggestions. I run this script from cron: #!/bin/sh export PATH=/usr/bin:$PATH find /data -type f -mtime -7 >scanfiles clamscan -f scanfiles -i rm -f scanfiles -Dave ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
