Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

[Reio Remma via clamav-users]

On 31/10/2019 12:04, Reio Remma wrote:

On 28/10/2019 12:55, Reio Remma via clamav-users wrote:

On 14/09/2019 17:34, G.W. Haywood via clamav-users wrote:

Hi Micah,

On Fri, 13 Sep 2019, Micah Snyder (micasnyd) wrote:


I'm sorry, Ged...


Apology accepted. :)

I'm now running the development (0.102) version of clamd, patched with
Mr. Wu's patch, alongside two version 101.4 clamd daemons (an unpatched
one, and one with the patch that I posted on Bugzilla).

The milter scans all mail with all three daemons.  On the arrival of a
message, if the database is not already being reloaded I start a fresh
reload before the scan so that, for all scans, a reload always executes
concurrently.  Nothing seems to have broken, and so far there's nothing
terribly interesting to report other than the strange failure to detect
which I sent to Joel early this week (and which I'm sure has nothing to
do with these patches). 


I've been running a patched 101.4 for a few weeks now and 
unfortunately I'm observing a memory leak from the multithreaded 
database reloads.


I'm observing clamd memory usage going up when the new database loads 
and then eventually dropping down to 1.3G again. For some reason 
"eventually" means the memory usage drops down only after clamd 
processes the next e-mail.


The problem however shows itself if clamd happens to reload its 
database 2 times if a row with no mail processed in between. 
Seemingly it will have 3 databases in memory then and the next mail 
being processed releases one of them, but the extra database will 
remain "somewhere".


All sorts of weird problems always keep popping up on due to low 
traffic on the server. :) 


Fortunately 0.102.0 with the patch from ClamAV team doesn't have that 
issue and seems to release the extra memory right away.


Happily running 0.102.0 now. 


Has anyone got the threaded reload patch working with 0.102.2?

When rebuilding my RPM with 0.102.2, I get the following error when the 
patch is being applied:


+ echo 'Patch #0 (clamd-threaded-reloading.patch):'
Patch #0 (clamd-threaded-reloading.patch):
+ /usr/bin/cat ~/rpmbuild/SOURCES/clamd-threaded-reloading.patch
+ /usr/bin/patch -p1 -b --suffix .threaded_reloading --fuzz=0
patching file clamd/clamd.c
Reversed (or previously applied) patch detected!  Assume -R? [n]

Thanks,
Reio

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] ERROR: VirusEvent: fork failed.

[G.W. Haywood via clamav-users]

Hi there,

Thanks for the excellent extra information, it makes things a lot clearer.

On Mon, 10 Feb 2020, Tom Ossman via clamav-users wrote:


... the VirusEvent script (also cleansed).


Does the script contain the first two lines as in the version which
you sent to me?  If so, remove them.  See the 'man' page for the
'file' utility and use it on your script. :)


I was not aware the the code was that new, I'll review the link ...


Maybe this will help too:

https://blog.clamav.net/2019/09/understanding-and-transitioning-to.html


The "server" instance is a t3a.small, 2 CPUs and 2 GB of memory...


I'm not sure that's be enough memory for what you're doing, I suggest
at least 4GB.  From personal experience I'd recommend Nagios or Icinga
to monitor resource usage.  Depending on your familiarity with things
like Apache it can be a steep learning curve, but once you have it
under your belt it's difficult to imagine working without something
like that when you have to look after more than about half a dozen
systems.

--

73,
Ged.

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] ClamAV® blog: ClamAV 0.102.2 security patch released

[Sergey]

On Wednesday 05 February 2020, Joel Esler (jesler) via clamav-users wrote:

> Today, we're publishing 0.102.2.
> ClamAV 0.102.2 is a security patch release to address the following issues.

Are you forgot to increase DNS version?

Tue Feb 11 16:27:01 2020 -> fc_dns_query_update_info: Software version from 
DNS: 0.102.1

-- 
Regards, Sergey

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] ERROR: VirusEvent: fork failed.

[Tom Ossman via clamav-users]

>
> > ... the VirusEvent script (also cleansed).
>
> Does the script contain the first two lines as in the version which
> you sent to me?  If so, remove them.  See the 'man' page for the
> 'file' utility and use it on your script. :)
>
> Are you refering to the comment and shebang? If so, the comment isn't
there but the shebang is and if that is one of the lines you are refering
to, correct me if I am wrong but doesn't a script need the shebang to run?



> > The "server" instance is a t3a.small, 2 CPUs and 2 GB of memory...
> I'm not sure that's be enough memory for what you're doing, I suggest
> at least 4GB.  From personal experience I'd recommend Nagios or Icinga
> to monitor resource usage.  Depending on your familiarity with things
> like Apache it can be a steep learning curve, but once you have it
> under your belt it's difficult to imagine working without something
> like that when you have to look after more than about half a dozen
> systems.
>
> I am familar with Nagios (never ran it though), I can watch the memory
usage on the server a bit closer during a test scan, but from what I have
observed in the past the memory did not look like it was being maxed out.
But spinning up a larger instance with more memory is not a big deal.


*Tom Ossman*

toss...@aspirevc.com | aspirevc.com | +1.717.468.0293

100 North Queen Street | Suite 300 | Lancaster, PA 17603

Engage with us on Twitter  | LinkedIn
 | Facebook


The information contained in this electronic message is legally privileged
and confidential information intended only for the person to whom the
message is addressed. If the reader of this message is not the intended
recipient, you are hereby notified that any dissemination, distribution, or
copy of this electronic message is strictly prohibited. If you have
received this electronic message in error, please immediately notify us by
return electronic message, and then delete this electronic message. Thank
you.




On Tue, Feb 11, 2020 at 7:01 AM G.W. Haywood via clamav-users <
clamav-users@lists.clamav.net> wrote:

> Hi there,
>
> Thanks for the excellent extra information, it makes things a lot clearer.
>
> On Mon, 10 Feb 2020, Tom Ossman via clamav-users wrote:
>
> > ... the VirusEvent script (also cleansed).
>
> Does the script contain the first two lines as in the version which
> you sent to me?  If so, remove them.  See the 'man' page for the
> 'file' utility and use it on your script. :)
>
> > I was not aware the the code was that new, I'll review the link ...
>
> Maybe this will help too:
>
> https://blog.clamav.net/2019/09/understanding-and-transitioning-to.html
>
> > The "server" instance is a t3a.small, 2 CPUs and 2 GB of memory...
>
> I'm not sure that's be enough memory for what you're doing, I suggest
> at least 4GB.  From personal experience I'd recommend Nagios or Icinga
> to monitor resource usage.  Depending on your familiarity with things
> like Apache it can be a steep learning curve, but once you have it
> under your belt it's difficult to imagine working without something
> like that when you have to look after more than about half a dozen
> systems.
>
> --
>
> 73,
> Ged.
>
> ___
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] ERROR: VirusEvent: fork failed.

[G.W. Haywood via clamav-users]

Hi there,

On Tue, 11 Feb 2020, Tom Ossman via clamav-users wrote:



... the VirusEvent script (also cleansed).


Does the script contain the first two lines as in the version which
you sent to me?  If so, remove them.  See the 'man' page for the
'file' utility and use it on your script. :)

Are you refering to the comment and shebang? If so, the comment isn't

there but the shebang is and if that is one of the lines you are refering
to, correct me if I am wrong but doesn't a script need the shebang to run?


In the script which you attached the shebang line is the third line.
The first two lines were a non-shebang line comment and a blank line.
Obviously the shebang line must be the first line in the real script.
(When I asked you to show us the script, I didn't mean for you to show
us some rough approximation to it. :)

You might want to use full path names for things like 'cat' in the
script so that it doesn't depend e.g. on environment variables which
might not be set.


... from what I have observed in the past the memory did not look like
it was being maxed out.  But spinning up a larger instance with more
memory is not a big deal.


Please let me know if you have any more information about resources.

We still don't know that this is not a fault in ClamAV itself of
course, but as it appears that at some time the sytem _was_ working,
it seems much more likely to be an issue with your configuration.

--

73,
Ged.

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[clamav-users] DB updates of (only) securiteinfo.hdb failing since last nite (Failed to load new database: Malformed database). what's up?

[PGNet Dev via clamav-users]

i'm running

 clamd -V
  ClamAV 0.102.2/25720/Mon Feb 10 03:53:41 2020

i use securiteinfo DBs.

last nite my ClamAV instance's DB update attempts (via freshclam) started 
failing for just "securiteinfo.hdb"
 

Mon Feb 10 11:23:17 2020 -> Testing database: 
'/var/lib/clamav/tmp.21cc7/clamav-1523a213ef3d31d0138315ea74e0fc47.tmp-securiteinfo.hdb'
 ...
Mon Feb 10 11:23:17 2020 -> ^[LibClamAV] cli_loadhash: Empty database 
file
Mon Feb 10 11:23:17 2020 -> ^[LibClamAV] Can't load 
/var/lib/clamav/tmp.21cc7/clamav-1523a213ef3d31d0138315ea74e0fc47.tmp-securiteinfo.hdb:
 Malformed database
Mon Feb 10 11:23:17 2020 -> !Failed to load new database: Malformed 
database
Mon Feb 10 11:23:17 2020 -> ^Database load exited with "Test failed" (8)
Mon Feb 10 11:23:17 2020 -> !Database test FAILED.
Mon Feb 10 11:23:17 2020 -> Unexpected error when attempting to update 
from custom database URL: 
https://www.securiteinfo.com/get/signatures//securiteinfo.hdb
Mon Feb 10 11:23:17 2020 -> ^fc_download_url_databases: 
fc_download_url_database failed: Test failed (8)
Mon Feb 10 11:23:17 2020 -> !Database update process failed: Test 
failed (8)
Mon Feb 10 11:23:17 2020 -> !Update failed.

curl -o securiteinfo.hdb 
https://www.securiteinfo.com/get/signatures//securiteinfo.hdb
  % Total% Received % Xferd  Average Speed   TimeTime Time  
Current
 Dload  Upload   Total   SpentLeft  
Speed
  0 00 00 0  0  0 --:--:--  0:00:01 
--:--:-- 0
ls -al securiteinfo.hdb
  -rw-r--r-- 1 root root 0 Feb 11 08:37 securiteinfo.hdb

all OTHER securiteinfo DBs seem fine.

checked from different locations; no diff - no DL.  others i've checked having 
same fails.

anyone *here* seeing the same? know what's up?

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] ERROR: VirusEvent: fork failed.

[Mickey Sola (micksola) via clamav-users]

Wanted to add a bit of insight to this convo from the dev side of things:

VirusEvent currently works by forking the existing clamd process into a new, 
short-lived process that handles execution of the user's script.

This is a legacy design choice and is problematic for a number of reasons--most 
relevant here is that you will need at minimum 2x the amount of resources clamd 
is already using to execute the VirusEvent. It was this resource drain, 
combined with the threaded nature of the old on access code, which led to us 
disabling the feature (only for on access scanning, not clamd/clamdscan).

>From what I can tell, your problem is that the fork system command is failing 
>(code path for that error requires a negative return for fork())--very likely 
>due to lack of resources on the server.

Ideally, we would fix this resource consumption issue on its own, or better, as 
part of a larger redesign of clamd, but for now--like Ged, I would also 
recommend increasing memory resources and seeing if that solves the issue.

-Mickey



On 2020-02-11 11:30:11-05:00 clamav-users wrote:

Hi there,

On Tue, 11 Feb 2020, Tom Ossman via clamav-users wrote:
>>
>>> ... the VirusEvent script (also cleansed).
>>
>> Does the script contain the first two lines as in the version which
>> you sent to me?  If so, remove them.  See the 'man' page for the
>> 'file' utility and use it on your script. :)
>>
>> Are you refering to the comment and shebang? If so, the comment isn't
> there but the shebang is and if that is one of the lines you are refering
> to, correct me if I am wrong but doesn't a script need the shebang to run?

In the script which you attached the shebang line is the third line.
The first two lines were a non-shebang line comment and a blank line.
Obviously the shebang line must be the first line in the real script.
(When I asked you to show us the script, I didn't mean for you to show
us some rough approximation to it. :)

You might want to use full path names for things like 'cat' in the
script so that it doesn't depend e.g. on environment variables which
might not be set.

> ... from what I have observed in the past the memory did not look like
> it was being maxed out.  But spinning up a larger instance with more
> memory is not a big deal.

Please let me know if you have any more information about resources.

We still don't know that this is not a fault in ClamAV itself of
course, but as it appears that at some time the sytem _was_ working,
it seems much more likely to be an issue with your configuration.

--

73,
Ged.

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml


___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[clamav-users] user list

[fritz blum via clamav-users]

please take me off from the user list!
kindregardsfritzfri_b...@yahoo.com

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[clamav-users] unsubscribe

[Christiansen, Edward - 0992 - MITLL]

unsubscribe



smime.p7s
Description: S/MIME Cryptographic Signature

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] DB updates of (only) securiteinfo.hdb failing since last nite (Failed to load new database: Malformed database). what's up?

[Arnaud Jacques]

Hello,



last nite my ClamAV instance's DB update attempts (via freshclam) started failing for 
just "securiteinfo.hdb"


This was due to a disk full on our side.
Sorry for that.
This has been resolved now.

--
Cordialement / Best regards,

Arnaud Jacques
Gérant de SecuriteInfo.com

Téléphone : +33-(0)3.44.39.76.46
E-mail : a...@securiteinfo.com
Site web : https://www.securiteinfo.com
Facebook : https://www.facebook.com/pages/SecuriteInfocom/132872523492286
Twitter : @SecuriteInfoCom

Securiteinfo.com
La Sécurité Informatique - La Sécurité des Informations.
266, rue de Villers
60123 Bonneuil en Valois

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] unsubscribe

[Joel Esler (jesler) via clamav-users]

Thank you for writing in.

Go to this URL to change user options or unsubscribe:
https://lists.clamav.net/mailman/listinfo/clamav-users

or by sending an email to clamav-users-le...@lists.clamav.net

Thanks!

On 2/11/20, 12:18 PM, "clamav-users on behalf of Christiansen, Edward - 0992 - 
MITLL"  
wrote:

unsubscribe




smime.p7s
Description: S/MIME cryptographic signature

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] user list

[Joel Esler (jesler) via clamav-users]

Thank you for writing in.

Go to this URL to change user options or unsubscribe:
https://lists.clamav.net/mailman/listinfo/clamav-users

or by sending an email to clamav-users-le...@lists.clamav.net

Thanks!

 

 

From: clamav-users  on behalf of fritz 
blum via clamav-users 
Reply-To: ClamAV users ML 
Date: Tuesday, February 11, 2020 at 12:15 PM
To: "clamav-users@lists.clamav.net" 
Cc: fritz blum 
Subject: [clamav-users] user list

 

please take me off from the user list!

 

Kindregards

Fritz

fri_b...@yahoo.com



smime.p7s
Description: S/MIME cryptographic signature

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] ClamAV® blog: ClamAV 0.102.2 security patch released

[Micah Snyder (micasnyd) via clamav-users]

Sorry about that.  It should be fixed & notifying people correctly now!

Regards,
Micah

On 2/11/20, 7:34 AM, "clamav-users on behalf of Sergey" 
 wrote:

On Wednesday 05 February 2020, Joel Esler (jesler) via clamav-users wrote:

> Today, we're publishing 0.102.2.
> ClamAV 0.102.2 is a security patch release to address the following 
issues.

Are you forgot to increase DNS version?

Tue Feb 11 16:27:01 2020 -> fc_dns_query_update_info: Software version from 
DNS: 0.102.1

-- 
Regards, Sergey

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml



___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[clamav-users] clamav from python?

[John Shelton via clamav-users]

The interface for cl_scanfile changed to accept a new struct cl_scan_options 
type from what used to be an int.
Is there any backward compatibility for this change?

Thanks,
john


___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] clamav from python?

[Scott Kitterman via clamav-users]

On Tuesday, February 11, 2020 4:23:18 PM EST John Shelton via clamav-users 
wrote:
> The interface for cl_scanfile changed to accept a new struct cl_scan_options
> type from what used to be an int. Is there any backward compatibility for
> this change?

AFAIK, there isn't.  You probably need to do something similar to:

https://salsa.debian.org/python-team/modules/python-clamav/blob/master/debian/
patches/python-clamav-add-support-for-clamav-0.101.0.patch

What Python bindings are you using?  The only libclamav bindings I'm aware of 
for python are ancient and unmaintained.  Do you know of something ~current?  
If you are using the old pyclamav module, that's the patch you need.

Scott K

P.S. I'm aware of pyclamd, but that's an interface to clamd, not libclamav.

signature.asc
Description: This is a digitally signed message part.

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] clamav from python?

[John Shelton via clamav-users]

On 2/11/20, 3:42 PM, "clamav-users on behalf of Scott Kitterman via 
clamav-users"  wrote:

What Python bindings are you using?  The only libclamav bindings I'm aware 
of 
for python are ancient and unmaintained.  Do you know of something 
~current?  

Looks like someone that used to work here rolled his own. So there is even less 
support for it that anything else out there.





___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml