Hi, Is it 4 GB? Can this size be increased ?
You can increase the maximum file size by setting the MaxFileSize option in clamd.conf. ClamAV’s option parser won’t allow you to set a maximum scan size higher than 4GB. In reality, the file size limit is 2GB. Anything larger than that will be automatically skipped and marked as “OK”. The reason for the 2GB file size limit is that in the past there were several bug reports for files larger than 2GB causing crashes. Rather than fix the parsers, the devs slipped in a 2GB file size limit to prevent crashes. I only just realized it a few weeks ago, and you can see my comments on this here: https://github.com/Cisco-Talos/clamav-devel/commit/1a3b784e1954e00b6463000a817da0c5092296cd There’s a lot of technical work to be done to safely raise that limitation, as large files of various file types types have never been tested. A large TAR, for example, may well work fine when a large ZIP might crash the program. We really have no idea. Basically it’s going to take a bunch of testing when someone goes to work on this. A lot of folks seem to be unhappy with it saying “OK” when a file hasn’t been scanned (myself included). So we have been talking about changing the output to something like the following messages when files are not scanned or are only partially scanned: * “SKIPPED (exceeded max file size)” * “INCOMPLETE (exceeded max scan size)” The exact wording is TBD. If anyone has any specific requests, I’d enjoy some help brainstorming. Is the memory or persistent storage a limit for ClamAV to scan a file ? if it is a persistent storage then can i increase the limit by attaching an external NFS ? Sorry, persistent storage is not the concern. Read somewhere the full file size is mapped to memory. Is it the case for INSTREAM command also ? Yes, INSTREAM is also limited to 4GB (or _really_ 2GB). If it is the case then even if chunking is supported then the server side must have at least 4GB of memory. Scanning a file in chunks is a waste of CPU cycles. ClamAV was designed to process a whole file all at once. Some file formats, like PDF, DMG, and ZIP* store metadata at the end of the file which is necessary to properly parse the file. Streaming scanners like the one in Snort struggle or can’t process these files. I put a * near ZIP because zips are actually pretty easy to parse in-order even if the central directory is missing. Files like DMG, on the other hand, can’t even be identified as DMG’s without reading the end of the file first, or trusting the “.dmg” file extension (which is dangerous). In short, don’t send chunks of files as separate files to be scanned; It probably won’t catch any malware that way and may print lots of warnings or errors if it gets confused about the type of the file and starts processing it with the wrong parser. Regards, -Micah Micah Snyder ClamAV Talos Cisco Systems, Inc. From: clamav-users <clamav-users-boun...@lists.clamav.net> On Behalf Of Saurav Sarkar via clamav-users Sent: Wednesday, April 7, 2021 7:39 AM To: clamav-users@lists.clamav.net Cc: Saurav Sarkar <saurav.sark...@gmail.com> Subject: [clamav-users] Scanning a large file through HTTP Hi All, We are using a HTTP enabled malware scanning service based on Clam AV. The service is made something like this https://github.com/solita/clamav-rest We have files like CAD files which can go in GBs and want to send to this malware scanning service. Is there a possibility to send the file in chunks and get it scanned in the server side in chunks. I observed that there is a INSTREAM command in clamd for this purpose and also there is a 4GB size limit. https://linux.die.net/man/8/clamd Read somewhere the full file size is mapped to memory. Is it the case for INSTREAM command also ? If it is the case then even if chunking is supported then the server side must have at least 4GB of memory. Best Regards, Saurav
_______________________________________________ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
