Sorry about the HTTP 500 errors : the "zip" extension had been routed to a special CGI handler for another purpose (to intercept people who were downloading some big files over and over and over). I removed that special treatment from this directory. So it should work now.
On Tue, 9 Jul 2019 23:19:50 +0100 Slarty Bartfast via clamav-users <clamav-users@lists.clamav.net> wrote: > Thank you very much for the detailed replies. > > Paul, thanks for providing the old signatures. The .zip files seem to > be throwing 500s though? > > Andrew, the details about the hashes and logical signatures make a > lot of sense. > > >From looking again at a comparison between clamscan and the daemon, > >it does > indeed look like the slower performance is almost entirely down to > loading the signatures on initial startup. This has always been the > case, but does seem to have become significantly more pronounced > recently. Andrew's explanation provides more context there. > > I'll share any results of tests we run with the older signatures, but > recognise that this may just be the cost of the "increase in the > breadth and depth of coverage". > > > Over the last few years, Talos has invested significant amounts of > > time > and > > effort into improving the infrastructure we use to automate ClamAV > > signature creation and testing, and especially within the last 6-9 > > months, this has allowed us to push out signatures for known > > threats much faster than we ever have before. In addition, where > > much of the automated coverage we could provide in the past was > > hash-based, we are increasingly able to create logical signatures > > that match on tens or hundreds of > samples > > at a time. This increase in the breadth and depth of coverage > > likely > plays > > a part in the performance degradation experienced. > > > > I don't have an old daily.cvd handy, but looking at a directory > > listing of an unpacked daily.cvd from December 2018, daily.ldb is > > now 5 times as > large > > as it was then (it's currently 21 MBs with 69,874 rules). This > > translate into a longer signature load time when running clamscan > > or when starting/restarting clamd, and contributes to a lesser > > extent to an increased file scan time. > > > > We've analyzed several sets of signatures where, when aggregated, > > they contribute to large slow-downs of scan times for certain file > > types. > We've > > been able to deploy work-arounds for the cases that we've > > identified, but if you observe any files that seem especially slow > > to be scanned relative to their size, do let us know so we can > > investigate further. Also, we've spent some time investigating > > ways that ClamAV itself can be optimized, > but > > haven't yet taken any concrete actions on this front (to my > > knowledge). > > > > -Andrew > > > > Andrew Williams > > Malware Research Team > > Cisco Talos > > > > On Tue, Jul 9, 2019 at 3:39 PM Paul Kosinski via clamav-users < > > clamav-users@lists.clamav.net> wrote: > > > > > I have uploaded 4 CVDs and 2 CLDs to: > > > > > > http://iment.com/paste-bin/ClamAV-Sigs/ > > > > > > The names include the dates (and times) they were downloaded. > > > > > > The reason for CVD vs CLD is that Cloudflare made running our own > > > "mirror" impractical. The CVD version delivered by Cloudflare's > > > "BOS" Anycast server was often behind the version advertised by > > > the DNS TXT. This caused freshclam to fail, since we triggered > > > off the DNS TXT, so we had to switch to using CDIFFs from *each* > > > machine on our LAN to update its CLDs. (Luckily there are only a > > > few, so bandwidth was OK.) > > > > > > Note that a CLD (after unZIPping) will be much bigger than the > > > equivalent CVD, which might change the timings. > > > > > > It will be interesting to see the results! > > > > > > > > > > > > On Tue, 9 Jul 2019 12:05:53 +0100 > > > Slarty Bartfast via clamav-users <clamav-users@lists.clamav.net> > > > wrote: > > > > > > > > On Mon, 8 Jul 2019 10:47:18 -0500 > > > > > "J.R. via clamav-users" <clamav-users at lists.clamav.net> > > > > > wrote: > > > > > > > > > > One way you *could* get an older .cvd file is to extract it > > > > > from the relevant ClamAV package available on many different > > > > > linux distro's. Be sure to disable freshclam though > > > > > (obviously). > > > > > > > > Thanks for the suggestion; I was able to get some older > > > > signatures from some older rpm packages e.g. > > > > https://pkgs.org/download/clamav-db > > > > > > > > However, these were mostly main.cvd and so old that comparisons > > > > weren't all that useful unfortunately. > > > > > > > > I don't think the main apt-based distros have included > > > > signatures in their packages for quite some time AFAICS. > > > > > > > > > Paul Kosinski clamav-users > > > > > Mon Jul 8 12:48:47 EDT 2019 > > > > > > > > > > We have a large number of old daily.cvd and daily.cld > > > > > accumulated over the past several years. I have kept them in > > > > > case an update caused a problem and I had to go back to make > > > > > ClamAV work until the next update. (I really should delete > > > > > most of them!) > > > > > > > > > > Given some dates, I could upload a few to our Website and > > > > > provide URLs. > > > > > > > > Thanks for the offer, that would be great. Ideally perhaps it'd > > > > be useful to see daily signatures from something like: > > > > > > > > * end of Dec 2017 / start of Jan 2018 > > > > * end of Mar / start of Apr 2018 > > > > * end of Jun / start of Jul 2018 > > > > * end of Sep / start of Oct 2018 > > > > * end of Dec 2018 / start of Jan 2019 > > > > * end of Mar / start of Apr 2019 > > > > > > > > Any samples covering roughly that period would be useful; > > > > doesn't have to be these specific dates / intervals. > > > > > > > > Very much appreciate if you could share links to these, thanks > > > > again. _______________________________________________ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
