Fantastic! I can also confirm that scan times are back to normal now - more-or-less back to what they were in early February.
The time for one of our FP test volumes which I've been referencing in this thread is back down to 3m 30s, and the total time for our *full* FP test is back down from several hours to just 47 minutes. Thank you! Mark On Thu, 18 Apr 2019 at 09:46, Al Varnell via clamav-users < clamav-users@lists.clamav.net> wrote: > Looks like all Phish.Phishing.REPHISH_ID_... signatures were dropped by > daily-25423 today. > > -Al- > > On Apr 17, 2019, at 04:02, Al Varnell <alvarn...@mac.com> wrote: > > There are still 2515 "Phish.Phishing.REPHISH_ID_...." signatures in > daily.ldb > > -Al- > > On Apr 17, 2019, at 03:36, Maarten Broekman <maarten.broek...@gmail.com> > wrote: > > Are the "Phish" REPHISH signatures still in the daily or were they removed > as well? Those were causing part of the issue. > > > --Maarten > > On Wed, Apr 17, 2019 at 5:24 AM Al Varnell via clamav-users < > clamav-users@lists.clamav.net> wrote: > >> An additional 3968 Phishtank.Phishing.PHISH_ID_??????? signatures were >> dropped by daily-25417 on 12 April, and I can't seem to locate any more. >> >> -Al- >> >> On Apr 17, 2019, at 02:01, Mark Allan via clamav-users < >> clamav-users@lists.clamav.net> wrote: >> >> Hi Micah, >> >> Sorry to pester you, but have you any update on when the remaining >> Phishtank signatures will be getting removed? It would be really great to >> get scan times properly back to normal. >> >> Best regards >> Mark >> >> On Tue, 9 Apr 2019 at 16:32, Micah Snyder (micasnyd) <micas...@cisco.com> >> wrote: >> >>> Mark, >>> >>> >>> Yes, the plan is still to remove the rest of the Phishtank signatures. >>> We wanted to get things back to relative normal and resolve the immediate >>> crisis. We’ll remove the rest of them soon. >>> >>> >>> >>> Best, >>> >>> Micah >>> >>> >>> >>> *From: *Mark Allan <markjal...@gmail.com> >>> *Date: *Tuesday, April 9, 2019 at 6:26 AM >>> *To: *"Micah Snyder (micasnyd)" <micas...@cisco.com> >>> *Cc: *ClamAV users ML <clamav-users@lists.clamav.net> >>> *Subject: *Re: [External] Re: [clamav-users] Scan very slow >>> >>> >>> >>> The scan times are definitely better than they were - in fact, they're >>> back to how they were before last week's inclusion of the Phishtank >>> signatures. They're still almost double what they used to be though, and as >>> far as I can see, there are still almost 4000 Phishtank signatures in the >>> DB: >>> >>> $ sigtool --find Phishtank | wc -l >>> >>> 3968 >>> >>> >>> >>> Can I request that those ones also be removed please? >>> >>> >>> >>> Best regards >>> >>> Mark >>> >>> >>> >>> On Sun, 7 Apr 2019 at 14:43, Micah Snyder (micasnyd) <micas...@cisco.com> >>> wrote: >>> >>> Tim, >>> >>> >>> >>> There are a couple of ways for users to drop specific categories of >>> signatures at this time. Sadly, they wouldn’t have helped this last week. >>> These include bytecode signatures, PUA (potentially unwanted applications) >>> signatures, Email.Phishing and HTML.Phishing signatures, and the >>> Safebrowsing database. >>> >>> >>> >>> If we had named the Phishtank.Phishing sigs to HTML.Phishing.Phishtank >>> or Email.Phishing.Phishtank then they could have been disabled with the >>> clamscan option `--phishing-sigs=no` (clamd.conf: `PhishingSignatures no`). >>> >>> >>> >>> Maybe a better option would be for us to create a new optional database >>> for phishing signatures. However, the names for the databases are hardcoded >>> into freshclam, so it is non-trivial to add a new database and would >>> require a few changes to ClamAV’s code. We have talked about making the >>> databases easier to add/remove in the future so users can have more >>> categories to enable/disable. In this light, it ties in well with existing >>> plans. >>> >>> >>> >>> Of note the Phishtank sigs from Friday’s daily were removed yesterday >>> and scan times should be back to normal. >>> >>> >>> >>> Regards, >>> >>> Micah >>> >>> >>> >>> *From: *Tim Hawkins <tim.hawk...@redflaggroup.com> >>> *Date: *Friday, April 5, 2019 at 6:06 PM >>> *To: *ClamAV users ML <clamav-users@lists.clamav.net>, Mark Allan < >>> markjal...@gmail.com> >>> *Cc: *"Micah Snyder (micasnyd)" <micas...@cisco.com> >>> *Subject: *Re: [External] Re: [clamav-users] Scan very slow >>> >>> >>> >>> Hi Micah >>> >>> >>> Does clamav partition the database so that signatures that are mainly >>> associated with email scanning can be dropped out for folks only needing >>> filesystems scans, none of our systems use email, and we dont make use of >>> the mailer extension. >>> >>> Having to load all the email focused signatures could as you have >>> observed impact performance. >>> >>> Sent from Nine <http://www.9folders.com/> >>> ------------------------------ >>> >>> *From:* "Micah Snyder (micasnyd) via clamav-users" < >>> clamav-users@lists.clamav.net> >>> *Sent:* Saturday, April 6, 2019 03:18 >>> *To:* ClamAV users ML; Mark Allan >>> *Cc:* Micah Snyder (micasnyd) >>> *Subject:* [External] Re: [clamav-users] Scan very slow >>> >>> >>> >>> Regarding slow scan times today (and slow scan times in general), it >>> appears that the signatures we generate based on PhishTank’s feed for >>> phishing URLs are resulting in very slow load and scan times. >>> >>> >>> >>> Today’s daily update saw 7448 new Phishtank signatures (much higher than >>> usual) coinciding with the immediate performance drop for load time and >>> scan time. One user reported that the load time today on some of his >>> slower machines was slow enough to exceed the timeout for service startup ( >>> https://bugzilla.clamav.net/show_bug.cgi?id=12317). >>> >>> >>> >>> In limited testing on my own machine I saw the following change after >>> dropping the Phishtank.Phishing signatures from daily.cvd’s daily.ldb file: >>> >>> - Database load time on my laptop went from 75.43203997612 seconds >>> down to 14.859203100204468 seconds >>> - Scan time (for an arbitrary pdf) went from 1.798 sec to 0.644 sec. >>> >>> >>> >>> After some discussion between the teams that work on ClamAV and ClamAV >>> signature content and deployment, we’ve agreed to drop PhishTank signatures >>> from the database until we can determine a way to craft Phishtank >>> signatures without incurring such a significant performance hit. >>> >>> >>> >>> The daily update tomorrow will have the change. >>> >>> >>> >>> -Micah >>> >>> >>> >>> >>> Micah Snyder >>> ClamAV Development >>> Talos >>> Cisco Systems, Inc. >>> >>> >>> >>> >>> >>> >>> >>> *From: *clamav-users <clamav-users-boun...@lists.clamav.net> on behalf >>> of "Micah Snyder (micasnyd) via clamav-users" < >>> clamav-users@lists.clamav.net> >>> *Reply-To: *ClamAV users ML <clamav-users@lists.clamav.net> >>> *Date: *Friday, April 5, 2019 at 1:08 PM >>> *To: *Mark Allan <markjal...@gmail.com>, ClamAV users ML < >>> clamav-users@lists.clamav.net> >>> *Cc: *"Micah Snyder (micasnyd)" <micas...@cisco.com> >>> *Subject: *Re: [clamav-users] Scan very slow >>> >>> >>> >>> Hi Mark, >>> >>> >>> >>> Sorry about the delay in responding. I hadn’t looked at my clamav-users >>> filter this morning. Just investigating now. Will respond when I know >>> more. >>> >>> >>> >>> -Micah >>> >>> >>> >>> *From: *Mark Allan <markjal...@gmail.com> >>> *Date: *Friday, April 5, 2019 at 9:12 AM >>> *To: *ClamAV users ML <clamav-users@lists.clamav.net>, "Micah Snyder >>> (micasnyd)" <micas...@cisco.com> >>> *Subject: *Re: [clamav-users] Scan very slow >>> >>> >>> >>> Also CC'ing Micah directly as the mailing list would appear to be >>> offline (at least lists.clamav.net isn't responding to http requests >>> anyway) >>> >>> >>> >>> It looks like scan times have gone through the roof. As Oya said, >>> they're still considerably higher than they were a couple of months ago, >>> but today's scan time is insane. >>> >>> >>> >>> Yesterday's scan using >>> >>> 0.101.2:58:25409:1554370140:1:63:48554:328 >>> >>> took 7m 3s >>> >>> >>> >>> On the same hardware, scanning the same read-only disk image, with >>> today's scan using >>> >>> 0.101.2:58:25410:1554452941:1:63:48557:328 >>> >>> the scan time has jumped to 26m 15s >>> >>> >>> >>> This is the longest it has ever taken to scan this volume (cf my >>> previous email of 25th March) >>> >>> >>> >>> Is there anything that can be excluded? >>> >>> >>> >>> Best regards >>> >>> Mark >>> >>> >>> >>> On Mon, 1 Apr 2019 at 17:11, Micah Snyder (micasnyd) via clamav-users < >>> clamav-users@lists.clamav.net> wrote: >>> >>> Thanks Oya for the update. We will continue to investigate the >>> signature performance issue. >>> >>> Regards, >>> Micah >>> >>> On 3/28/19, 9:50 AM, "clamav-users on behalf of Tsutomu Oyamada" < >>> clamav-users-boun...@lists.clamav.net on behalf of >>> oyam...@promark-inc.com> wrote: >>> >>> Hi Micah >>> >>> It seems that the scanning slow down issue of this time has been >>> solved >>> at some level with CVD Update of the other day. >>> However, there is still big discrepancy in between the current >>> condition and >>> the last condition in one month ago. >>> >>> Date Files Scan time >>> 2019/02/15 2550338 08:53:57 >>> 2019/03/15 2612792 19:22:54 >>> 2019/03/26 2634489 18:13:56 >>> 2019/03/27 2637201 18:10:05 >>> >>> We know the improvement of this time is due to the details of CVD, >>> because >>> we did not make any change on the user's system. >>> We are going to try some tuning for scanning. >>> >>> We like to know if you still have some room to make further >>> improvement >>> for this slow down issue. >>> Thank you for your help, in advance. >>> >>> Best regards, >>> Oya >>> >>> On Mon, 25 Mar 2019 15:45:02 +0000 >>> "Micah Snyder \(micasnyd\) via clamav-users" < >>> clamav-users@lists.clamav.net> wrote: >>> >>> > Hi Mark, all: >>> > >>> > I’m disappointed to hear that it is still slow for you. >>> > >>> > We found that the target-type of signatures used for >>> PhishTank.Phishing signatures were causing a significant slowdown. We >>> have dropped them as of this past Saturday ( >>> https://lists.gt.net/clamav/virusdb/75279 ) and in the last two updates >>> have been re-adding them with more specific scan target types. We’re now >>> investigating some other optimizations we can make for the next major >>> ClamAV release to improve scan times but at present we don’t have any other >>> leads for signatures that may be slowing down scans. >>> > >>> > Regards, >>> > Micah >>> > >>> > >>> > From: clamav-users <clamav-users-boun...@lists.clamav.net> on >>> behalf of Mark Allan via clamav-users <clamav-users@lists.clamav.net> >>> > Reply-To: ClamAV users ML <clamav-users@lists.clamav.net> >>> > Date: Monday, March 25, 2019 at 9:37 AM >>> > To: ClamAV users ML <clamav-users@lists.clamav.net> >>> > Cc: Mark Allan <markjal...@gmail.com> >>> > Subject: Re: [clamav-users] Scan very slow >>> > >>> > Cheers Steve, >>> > >>> > In the interest of completeness, here's the scan from today (TXT >>> from DNS: 0.101.1:58:25399:1553509741:1:63:48528:328) showing a marked >>> improvement in scan time, although at 6m 7s it's still almost twice what it >>> used to be. >>> > >>> > Mark >>> > >>> > On Mon, 25 Mar 2019 at 12:56, Steve Basford < >>> steveb_cla...@sanesecurity.com<mailto:steveb_cla...@sanesecurity.com>> >>> wrote: >>> > On 2019-03-25 10:52, Mark Allan via clamav-users wrote: >>> > > Hi all, >>> > > >>> > te. >>> > > >>> > > Hopefully this helps someone to narrow things down a bit. >>> > > >>> > > Mark >>> > > >>> > >>> > 18/3/19 10m 49s TXT from DNS: >>> > 0.101.1:58:25392:1552904941:1:63:48507:328 *** >>> > >>> > Here's the changes for the above update: >>> > >>> > https://lists.gt.net/clamav/virusdb/75154 >>> > >>> > You can also check sigs quickly per update: >>> > >>> > https://lists.gt.net/clamav/virusdb/ >>> > >>> > >>> > >>> > -- >>> > Cheers, >>> > >>> > Steve >>> > Twitter: @sanesecurity >>> > >>> > _______________________________________________ >>> > >>> > clamav-users mailing list >>> > clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net >>> > >>> > https://lists.clamav.net/mailman/listinfo/clamav-users >>> > >>> > >>> > Help us build a comprehensive ClamAV guide: >>> > https://github.com/vrtadmin/clamav-faq >>> > >>> > http://www.clamav.net/contact.html#ml >>> >>> >>> >>> _______________________________________________ >>> >>> clamav-users mailing list >>> clamav-users@lists.clamav.net >>> https://lists.clamav.net/mailman/listinfo/clamav-users >>> >>> >>> Help us build a comprehensive ClamAV guide: >>> https://github.com/vrtadmin/clamav-faq >>> >>> http://www.clamav.net/contact.html#ml >>> >>> >>> >>> _______________________________________________ >>> >>> clamav-users mailing list >>> clamav-users@lists.clamav.net >>> https://lists.clamav.net/mailman/listinfo/clamav-users >>> >>> >>> Help us build a comprehensive ClamAV guide: >>> https://github.com/vrtadmin/clamav-faq >>> >>> http://www.clamav.net/contact.html#ml >>> >>> >>> >>> *DISCLAIMER* >>> >>> The information contained in this email and any attachments are >>> confidential. It is intended solely for the individual or entity to whom >>> they are addressed. Access to this email by anyone else is unauthorized. >>> >>> If you are not the intended recipient, any disclosure, copying, >>> distribution or any action taken or omitted to be taken in reliance on it, >>> is prohibited and may be unlawful. If you have received this communication >>> in error, please notify us immediately by responding to this email and then >>> delete it from your system. >>> >>> The Red Flag Group is neither liable for the proper and complete >>> transmission of the information contained in this communication nor for any >>> delay in its receipt. >>> >>> Any advice, recommendations or opinion contained within this email or >>> its attachments are not to be construed as legal advice. >>> >>> >> _______________________________________________ >> >> clamav-users mailing list >> clamav-users@lists.clamav.net >> https://lists.clamav.net/mailman/listinfo/clamav-users >> >> >> Help us build a comprehensive ClamAV guide: >> https://github.com/vrtadmin/clamav-faq >> >> http://www.clamav.net/contact.html#ml >> >> >> >> _______________________________________________ >> >> clamav-users mailing list >> clamav-users@lists.clamav.net >> https://lists.clamav.net/mailman/listinfo/clamav-users >> >> >> Help us build a comprehensive ClamAV guide: >> https://github.com/vrtadmin/clamav-faq >> >> http://www.clamav.net/contact.html#ml >> > > > > > > _______________________________________________ > > clamav-users mailing list > clamav-users@lists.clamav.net > https://lists.clamav.net/mailman/listinfo/clamav-users > > > Help us build a comprehensive ClamAV guide: > https://github.com/vrtadmin/clamav-faq > > http://www.clamav.net/contact.html#ml >
_______________________________________________ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
