Re: [clamav-users] Can’t allocate memory error

2021-03-04 Thread Michael Kyriacou via clamav-users
Yes I use 64 bit executables. Just some more information: My I/O speed caps around 500MB/s. I’m running clamdscan across ~30 mounted file systems on each scanner. Some of these are ntfs, others are ext4. Each filesystem has its own clamd process run against it. On Thu, Mar 4, 2021 at 12:37 PM

Re: [clamav-users] Can’t allocate memory error

2021-03-04 Thread G.W. Haywood via clamav-users
Hi there, On Thu, 4 Mar 2021, Michael Kyriacou via clamav-users wrote: when I type man clamd, I don’t get any information on threads. Please accept my apologies. I meant the 'man' page for clamd.conf. I’ve tested setting MaxThreads to 100 vs 16(the amount of vcpus the scanner has), and I d

Re: [clamav-users] Can’t allocate memory error

2021-03-04 Thread Michael Kyriacou via clamav-users
I am not using on-access scanning. when I type man clamd, I don’t get any information on threads. I’ve tested setting MaxThreads to 100 vs 16(the amount of vcpus the scanner has), and I did not see a performance increase. What is the recommend setting for MaxThreads/MaxQueue for a 16 core system?

Re: [clamav-users] Can’t allocate memory error

2021-03-04 Thread G.W. Haywood via clamav-users
Hi there, On Thu, 4 Mar 2021, Michael Kyriacou via clamav-users wrote: ... This is just one example of the many times the clamd process seems to pause or hang. ... That looks to me like clamd is working as designed. It has a queueing system which manages threads doing different kinds of wor

Re: [clamav-users] Can’t allocate memory error

2021-03-04 Thread Michael Kyriacou via clamav-users
The 3 filesizes below are as follows: 70K 260M 80K This is just one example of the many times the clamd process seems to pause or hang. Thu Mar 4 09:54:03 2021 -> /imports/INPRDSS122/mnt1/Boot/et-EE/bootmgr.exe.mui: OK Thu Mar 4 09:54:03 2021 -> Closed fd 44 Thu Mar 4 09:54:03 2021 -> Finishe

Re: [clamav-users] Can’t allocate memory error

2021-03-03 Thread Micah Snyder (micasnyd) via clamav-users
: clamav-users On Behalf Of Michael Kyriacou via clamav-users Sent: Tuesday, March 2, 2021 8:36 AM To: ClamAV users ML Cc: Michael Kyriacou Subject: Re: [clamav-users] Can’t allocate memory error I am scanning large Data sets for a company. These file systems have hundreds of thousands of

Re: [clamav-users] Can’t allocate memory error

2021-03-03 Thread G.W. Haywood via clamav-users
Hi there, On Tue, 2 Mar 2021, Michael Kyriacou via clamav-users wrote: I am scanning large Data sets for a company. These file systems have hundreds of thousands of files in them. Most files are small in size, <1GB, while a few are large, >10GB. Most files are documents, archives, and executabl

Re: [clamav-users] Can’t allocate memory error

2021-03-02 Thread Michael Kyriacou via clamav-users
I am scanning large Data sets for a company. These file systems have hundreds of thousands of files in them. Most files are small in size, <1GB, while a few are large, >10GB. Most files are documents, archives, and executables. I am scanning them to detect if there are any malware. These are virtu

Re: [clamav-users] Can’t allocate memory error

2021-03-02 Thread G.W. Haywood via clamav-users
Hi there, On Tue, 2 Mar 2021, Michael Kyriacou via clamav-users wrote: On Tue, Mar 2, 2021 at 9:40 AM G.W. Haywood via clamav-users wrote: On Tue, 2 Mar 2021, Michael Kyriacou via clamav-users wrote: On Tue, Mar 2, 2021 at 4:08 AM G.W. Haywood via clamav-users wrote: On Mon, 1 Mar 2021, Micha

Re: [clamav-users] Can’t allocate memory error

2021-03-02 Thread Michael Kyriacou via clamav-users
My scanners have 16vcpus and 64 GB Ram allocated to them. (Each) I noticed that the clamd process actually began hanging on some of the scanners. This slowed the scanning by a lot. looking at the log, the only thing it says is “... sleep”. After 5-10 minutes it will continue, and then pause again.

Re: [clamav-users] Can’t allocate memory error

2021-03-02 Thread G.W. Haywood via clamav-users
Hi there, On Tue, 2 Mar 2021, Michael Kyriacou via clamav-users wrote: On Tue, Mar 2, 2021 at 4:08 AM G.W. Haywood via clamav-users wrote: On Mon, 1 Mar 2021, Michael Kyriacou via clamav-users wrote: ... clamav 103.1 on ubuntu 20.04. I am getting “can’t allocate memory errors” on very large f

Re: [clamav-users] Can’t allocate memory error

2021-03-02 Thread Arnaud Jacques
Hello Michael Le 02/03/2021 à 13:44, Michael Kyriacou via clamav-users a écrit : I was not aware of any other way to avoid scanning large files. Where can I find such solutions? As an example scan all files below 50Mb : find /your_path -type f -size -50M|parallel clamdscan -mi --fdpass --no-s

Re: [clamav-users] Can’t allocate memory error

2021-03-02 Thread Michael Kyriacou via clamav-users
I was not aware of any other way to avoid scanning large files. Where can I find such solutions? On Tue, Mar 2, 2021 at 4:08 AM G.W. Haywood via clamav-users < clamav-users@lists.clamav.net> wrote: > Hi there, > > On Mon, 1 Mar 2021, Michael Kyriacou via clamav-users wrote: > > > ... clamav 103.1

Re: [clamav-users] Can’t allocate memory error

2021-03-02 Thread G.W. Haywood via clamav-users
Hi there, On Mon, 1 Mar 2021, Michael Kyriacou via clamav-users wrote: ... clamav 103.1 on ubuntu 20.04. I am getting “can’t allocate memory errors” on very large files ( 10GB +). I thought clamdscan was supposed to skip files that are larger than what you set the maxfilesize/maxscansize to.