Sorry for the delay. Here are some results of using clamscan to scan a simple eicar test file using just the daily.cvd/cld samples Paul Kosinski provided, along with a couple of recent daily.cld files from this month (July 2019):
### daily.cvd.171231-0906 Known viruses: 1810339 Time: 2.797 sec (0 m 2 s) ### daily.cvd.180330-1636 Known viruses: 1887869 Time: 3.004 sec (0 m 3 s) ### daily.cvd.180630-0936 Known viruses: 1994551 Time: 3.273 sec (0 m 3 s) ### daily.cvd.181002-0118 Known viruses: 2100678 Time: 3.612 sec (0 m 3 s) ### daily.cld.190101-0146 Known viruses: 2189018 Time: 2.680 sec (0 m 2 s) ### daily.cld.190331-0431 Known viruses: 1531025 Time: 5.683 sec (0 m 5 s) ### daily.cld.190710-1218 Known viruses: 1613424 Time: 12.995 sec (0 m 12 s) ### daily.cld.190717-1740 Known viruses: 1650816 Time: 12.387 sec (0 m 12 s) FWIW this is on an AWS c5.xlarge. These tests are not particularly rigorous, but it certainly looks like the process has got a lot slower recently. The daily.cld from March seemed to make clamscan take almost twice as long as with January's signatures, and both of the files from July took as much as 4 times longer. The results seem to match reports we've had from users that the scanning process with clamscan seems to have become dramatically slower in recent months; they're typically reporting that scans which used to take around 10 seconds and now taking 30 or 40 seconds. Is this just an unavoidable cost of the advances in (logical) signature creation which provide better coverage? On Tue, 9 Jul 2019 at 20:59, Andrew Williams <awill...@sourcefire.com> wrote: > Over the last few years, Talos has invested significant amounts of time > and effort into improving the infrastructure we use to automate ClamAV > signature creation and testing, and especially within the last 6-9 months, > this has allowed us to push out signatures for known threats much faster > than we ever have before. In addition, where much of the automated > coverage we could provide in the past was hash-based, we are increasingly > able to create logical signatures that match on tens or hundreds of samples > at a time. This increase in the breadth and depth of coverage likely plays > a part in the performance degradation experienced. > > I don't have an old daily.cvd handy, but looking at a directory listing of > an unpacked daily.cvd from December 2018, daily.ldb is now 5 times as large > as it was then (it's currently 21 MBs with 69,874 rules). This translate > into a longer signature load time when running clamscan or when > starting/restarting clamd, and contributes to a lesser extent to an > increased file scan time. > > We've analyzed several sets of signatures where, when aggregated, they > contribute to large slow-downs of scan times for certain file types. We've > been able to deploy work-arounds for the cases that we've identified, but > if you observe any files that seem especially slow to be scanned relative > to their size, do let us know so we can investigate further. Also, we've > spent some time investigating ways that ClamAV itself can be optimized, but > haven't yet taken any concrete actions on this front (to my knowledge). > > -Andrew > > Andrew Williams > Malware Research Team > Cisco Talos > > On Tue, Jul 9, 2019 at 3:39 PM Paul Kosinski via clamav-users < > clamav-users@lists.clamav.net> wrote: > >> I have uploaded 4 CVDs and 2 CLDs to: >> >> http://iment.com/paste-bin/ClamAV-Sigs/ >> >> The names include the dates (and times) they were downloaded. >> >> The reason for CVD vs CLD is that Cloudflare made running our own >> "mirror" impractical. The CVD version delivered by Cloudflare's "BOS" >> Anycast server was often behind the version advertised by the DNS TXT. >> This caused freshclam to fail, since we triggered off the DNS TXT, so >> we had to switch to using CDIFFs from *each* machine on our LAN to >> update its CLDs. (Luckily there are only a few, so bandwidth was OK.) >> >> Note that a CLD (after unZIPping) will be much bigger than the >> equivalent CVD, which might change the timings. >> >> It will be interesting to see the results! >> >> >> >> On Tue, 9 Jul 2019 12:05:53 +0100 >> Slarty Bartfast via clamav-users <clamav-users@lists.clamav.net> wrote: >> >> > > On Mon, 8 Jul 2019 10:47:18 -0500 >> > > "J.R. via clamav-users" <clamav-users at lists.clamav.net> wrote: >> > > >> > > One way you *could* get an older .cvd file is to extract it from the >> > > relevant ClamAV package available on many different linux distro's. >> > > Be sure to disable freshclam though (obviously). >> > >> > Thanks for the suggestion; I was able to get some older signatures >> > from some older rpm packages e.g. https://pkgs.org/download/clamav-db >> > >> > However, these were mostly main.cvd and so old that comparisons >> > weren't all that useful unfortunately. >> > >> > I don't think the main apt-based distros have included signatures in >> > their packages for quite some time AFAICS. >> > >> > > Paul Kosinski clamav-users >> > > Mon Jul 8 12:48:47 EDT 2019 >> > > >> > > We have a large number of old daily.cvd and daily.cld accumulated >> > > over the past several years. I have kept them in case an update >> > > caused a problem and I had to go back to make ClamAV work until the >> > > next update. (I really should delete most of them!) >> > > >> > > Given some dates, I could upload a few to our Website and provide >> > > URLs. >> > >> > Thanks for the offer, that would be great. Ideally perhaps it'd be >> > useful to see daily signatures from something like: >> > >> > * end of Dec 2017 / start of Jan 2018 >> > * end of Mar / start of Apr 2018 >> > * end of Jun / start of Jul 2018 >> > * end of Sep / start of Oct 2018 >> > * end of Dec 2018 / start of Jan 2019 >> > * end of Mar / start of Apr 2019 >> > >> > Any samples covering roughly that period would be useful; doesn't >> > have to be these specific dates / intervals. >> > >> > Very much appreciate if you could share links to these, thanks again. >> >> _______________________________________________ >> >> clamav-users mailing list >> clamav-users@lists.clamav.net >> https://lists.clamav.net/mailman/listinfo/clamav-users >> >> >> Help us build a comprehensive ClamAV guide: >> https://github.com/vrtadmin/clamav-faq >> >> http://www.clamav.net/contact.html#ml >> > > _______________________________________________ > > clamav-users mailing list > clamav-users@lists.clamav.net > https://lists.clamav.net/mailman/listinfo/clamav-users > > > Help us build a comprehensive ClamAV guide: > https://github.com/vrtadmin/clamav-faq > > http://www.clamav.net/contact.html#ml >
_______________________________________________ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
