Hi Micah, Thank you for your response. I have been actually trying what you suggested with the sigtool command, and when removing Windows signatures from both daily.cvd and main.cvd, we saw a memory savings of about 1 GB during the scan, from 1.5 GB to 500-600 MB. However, I still haven’t figured out a good way to rebuild the databases so that we can update them using cdiffs or some variation thereof (for example, getting the diff, removing any lines containing “Win.”, and then proceeding with the new diff).
In particular, when using `sigtool --build` I wasn’t able to find anything in the documentation/code about what’s expected for the mandatory --server parameter, or how to write the cdiffs in freshclam (or sigtool) to a file to do the aforementioned modifications to remove any references to Windows signatures. Is there anything you suggest we do to maintain a current database in this manner? Otherwise, since the majority of Windows signatures are in main.cvd, we can probably get most of the memory savings by just dropping main.cvd as you suggested, and either omitting it from the scan or just including the non-Windows signatures, and then updating the daily and bytecode databases as normal. Thank you, Mikhail From: Micah Snyder (micasnyd) <micas...@cisco.com> Sent: Monday, June 24, 2024 12:35 PM To: clamav-users@lists.clamav.net Cc: Mikhail Soumar <msou...@microsoft.com> Subject: [EXTERNAL] Re: Question on ClamAV memory usage with respect to the signature database You don't often get email from micas...@cisco.com<mailto:micas...@cisco.com>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Hi Mikhail, As you probably know, the clamav signature database is comprised of daily.cvd, main.cvd, and bytecode.cvd. Note: I say "cvd" but the file will have a "cld" extension if freshclam has updated it from an older version using our cdiff patching update mechanism. Daily.cvd is updated daily and contains the most recent threats. We infrequently migrate the most stable signatures to main.cvd. Bytecode.cvd doesn't change often but contains some more complex logic for detecting malware. One option is to drop main.cvd (or main.cld) and only scan with daily.cvd + bytecode.cvd. That won't detect as many older threats but should still detect recent threats and will reduce your memory footprint. Another idea to meet your specific request (no windows sigs) is to use "sigtool --unpack CVDFILE" to extract the signature files into the current directory from a CVD archive. You could then strip out any lines containing "Win.". I haven't actually tried this, so I don't know how much memory savings you'll realize. I'd be curious what you find if you do try it. If you look at the clamav-users mailing list archives, you may notice some folks discovered we have a linux.cvd. I DON'T recommend using it. The Linux CVD is made to be supplemental to the Secure Endpoint for Linux client, which has other detection mechanisms. Linux.cvd has barely changed in the last couple years. But it is very lightweight. I don't have any other ideas at this time. I've been pushing for quite a while now on an internal project to identify lower-value signatures so we can archive them to reduce the RAM requirements, improve load time, and improve scan time. That's been very slow going and I really don't know what else to say about it. 🤞 Regards, Micah Micah Snyder (they/them) ClamAV Development Talos Cisco Systems, Inc. ________________________________ From: clamav-users <clamav-users-boun...@lists.clamav.net<mailto:clamav-users-boun...@lists.clamav.net>> on behalf of Mikhail Soumar via clamav-users <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> Sent: Monday, June 17, 2024 9:06 PM To: clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net> <clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>> Cc: Mikhail Soumar <msou...@microsoft.com<mailto:msou...@microsoft.com>> Subject: [clamav-users] Question on ClamAV memory usage with respect to the signature database Hello, We are a team from Microsoft Azure running ClamAV on small Linux VMs, and due to business and cost reasons we cannot use larger VMs. Peak memory usage of ClamAV is between 1.2GB and 1.5GB, which is unsustainable on our VMs, and we are looking for ways to reduce this. There are some tips to reduce memory usage in the Docker section of the documentation (Docker - ClamAV Documentation<https://docs.clamav.net/manual/Installing/Docker.html#memory-ram-requirements>) although if I understand correctly the 1.2GB load is unavoidable even with the suggestions listed on this page. We have been told that one possibility is to remove all virus signatures that are Windows-specific, which would reduce the memory footprint to about 300 MB. Elsewhere on the ClamAV FAQ I see a few different ways to add signatures to the database but none about taking a subset. Would this be something you support or recommend for our use case? If not, are there alternatives we can consider to reduce the memory footprint of ClamAV well below 1.2GB? Thank you, Mikhail
_______________________________________________ Manage your clamav-users mailing list subscription / unsubscribe: https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/Cisco-Talos/clamav-documentation https://docs.clamav.net/#mailing-lists-and-chat
