Over the last few years, Talos has invested significant amounts of time and effort into improving the infrastructure we use to automate ClamAV signature creation and testing, and especially within the last 6-9 months, this has allowed us to push out signatures for known threats much faster than we ever have before. In addition, where much of the automated coverage we could provide in the past was hash-based, we are increasingly able to create logical signatures that match on tens or hundreds of samples at a time. This increase in the breadth and depth of coverage likely plays a part in the performance degradation experienced.
I don't have an old daily.cvd handy, but looking at a directory listing of an unpacked daily.cvd from December 2018, daily.ldb is now 5 times as large as it was then (it's currently 21 MBs with 69,874 rules). This translate into a longer signature load time when running clamscan or when starting/restarting clamd, and contributes to a lesser extent to an increased file scan time. We've analyzed several sets of signatures where, when aggregated, they contribute to large slow-downs of scan times for certain file types. We've been able to deploy work-arounds for the cases that we've identified, but if you observe any files that seem especially slow to be scanned relative to their size, do let us know so we can investigate further. Also, we've spent some time investigating ways that ClamAV itself can be optimized, but haven't yet taken any concrete actions on this front (to my knowledge). -Andrew Andrew Williams Malware Research Team Cisco Talos On Tue, Jul 9, 2019 at 3:39 PM Paul Kosinski via clamav-users < clamav-users@lists.clamav.net> wrote: > I have uploaded 4 CVDs and 2 CLDs to: > > http://iment.com/paste-bin/ClamAV-Sigs/ > > The names include the dates (and times) they were downloaded. > > The reason for CVD vs CLD is that Cloudflare made running our own > "mirror" impractical. The CVD version delivered by Cloudflare's "BOS" > Anycast server was often behind the version advertised by the DNS TXT. > This caused freshclam to fail, since we triggered off the DNS TXT, so > we had to switch to using CDIFFs from *each* machine on our LAN to > update its CLDs. (Luckily there are only a few, so bandwidth was OK.) > > Note that a CLD (after unZIPping) will be much bigger than the > equivalent CVD, which might change the timings. > > It will be interesting to see the results! > > > > On Tue, 9 Jul 2019 12:05:53 +0100 > Slarty Bartfast via clamav-users <clamav-users@lists.clamav.net> wrote: > > > > On Mon, 8 Jul 2019 10:47:18 -0500 > > > "J.R. via clamav-users" <clamav-users at lists.clamav.net> wrote: > > > > > > One way you *could* get an older .cvd file is to extract it from the > > > relevant ClamAV package available on many different linux distro's. > > > Be sure to disable freshclam though (obviously). > > > > Thanks for the suggestion; I was able to get some older signatures > > from some older rpm packages e.g. https://pkgs.org/download/clamav-db > > > > However, these were mostly main.cvd and so old that comparisons > > weren't all that useful unfortunately. > > > > I don't think the main apt-based distros have included signatures in > > their packages for quite some time AFAICS. > > > > > Paul Kosinski clamav-users > > > Mon Jul 8 12:48:47 EDT 2019 > > > > > > We have a large number of old daily.cvd and daily.cld accumulated > > > over the past several years. I have kept them in case an update > > > caused a problem and I had to go back to make ClamAV work until the > > > next update. (I really should delete most of them!) > > > > > > Given some dates, I could upload a few to our Website and provide > > > URLs. > > > > Thanks for the offer, that would be great. Ideally perhaps it'd be > > useful to see daily signatures from something like: > > > > * end of Dec 2017 / start of Jan 2018 > > * end of Mar / start of Apr 2018 > > * end of Jun / start of Jul 2018 > > * end of Sep / start of Oct 2018 > > * end of Dec 2018 / start of Jan 2019 > > * end of Mar / start of Apr 2019 > > > > Any samples covering roughly that period would be useful; doesn't > > have to be these specific dates / intervals. > > > > Very much appreciate if you could share links to these, thanks again. > > _______________________________________________ > > clamav-users mailing list > clamav-users@lists.clamav.net > https://lists.clamav.net/mailman/listinfo/clamav-users > > > Help us build a comprehensive ClamAV guide: > https://github.com/vrtadmin/clamav-faq > > http://www.clamav.net/contact.html#ml >
_______________________________________________ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
