Vladislav, Ged: Reloading select databases is not feasible at this time, because signatures are loaded into the same structures in memory and that entire thing is recreated on reload.
Regarding the threaded reload feature ( ticket: https://bugzilla.clamav.net/show_bug.cgi?id=10979 )... The main reason the "threaded reload" patch is held back at present is primarily because the recent work and interest in the patch came at the same time that 0.102 development was in code freeze while we tested and applied bug fixes for release. Reloading in a separate thread means that the memory usage will double (going from roughly ~750MB to ~1500MB) during the reload before it frees the original signatures and drops back to ~750MB. We already have many complaints about freshclam and clamd memory usage, and this change in behavior could cause trouble for some users, so we want to provide an option to reload the traditional way. That's the second reason why the patch isn't been merged for 0.103 yet. We have to dedicate some time to code the ability to reload either way. It is absolutely on our to-do list. Regarding the increase in daily.cvd signatures causing slow load times... We identified the cause of slow load times. There are 2 loading algorithms identified that are inefficient and causing slow load times. The first has to do with signature ignore lists. This is easier to address for all users and I'm currently waiting for a larger update to daily & main that will drop load time by about 40% or so for most users. The 2nd is something harder to solve, triggered by 2 very common patterns in our signature set, used to identify functions and VBA macros. I was spending most of my time trying to figure out if we can optimize the signatures to get around the design issue, but had limited success. Alberto Wu, meanwhile, has found a way to make the loading algorithm more efficient (posted just yesterday). His patch is really promising, though I'm still testing in a very large regression test and sorting through a large number of detection changes to identify the cause. For details, please read this ticket: https://bugzilla.clamav.net/show_bug.cgi?id=12389 I'll continue to update that ticket as testing continues and I review the results. Best regards, Micah On 10/17/19, 11:08 AM, "clamav-users on behalf of Vladislav Kurz via clamav-users" <clamav-users-boun...@lists.clamav.net on behalf of clamav-users@lists.clamav.net> wrote: On 17/10/2019 16:51, G.W. Haywood via clamav-users wrote: > Hi there, > > On Thu, 17 Oct 2019, Vladislav Kurz via clamav-users wrote: > >> So the question is - what would be easier to code? >> - reloading in background thread >> - reloading limited to new files > > It is not clear to me that the latter suggestion is feasible, but... > > 1. Reload in a separate thread was first coded about six years ago. > > 2. After other, more recent discussions on this list some weeks ago, > I provided a patch for the current version of clamd, using that > original code as a basis. Yet more recently, one of the ClamAV > developers provided a similar patch. Both are freely available. > > 3. I've been running the patched code without issue for months. Oh great. Thank you Ged for writing and testing it. I apologise for not noticing that these patch is already done. Is there anything blocking this patch from being accepted ? I'm noticing clamav-reload related timeouts on more and more (mostly older or low-end) servers, which were running just fine a year or two ago. -- Best regards Vladislav Kurz _______________________________________________ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml _______________________________________________ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
