Looks like all Phish.Phishing.REPHISH_ID_... signatures were dropped by daily-25423 today.
-Al- > On Apr 17, 2019, at 04:02, Al Varnell <[email protected]> wrote: > > There are still 2515 "Phish.Phishing.REPHISH_ID_...." signatures in daily.ldb > > -Al- > >> On Apr 17, 2019, at 03:36, Maarten Broekman <[email protected] >> <mailto:[email protected]>> wrote: >> >> Are the "Phish" REPHISH signatures still in the daily or were they removed >> as well? Those were causing part of the issue. >> >> >> --Maarten >> >> On Wed, Apr 17, 2019 at 5:24 AM Al Varnell via clamav-users >> <[email protected] <mailto:[email protected]>> wrote: >> An additional 3968 Phishtank.Phishing.PHISH_ID_??????? signatures were >> dropped by daily-25417 on 12 April, and I can't seem to locate any more. >> >> -Al- >> >>> On Apr 17, 2019, at 02:01, Mark Allan via clamav-users >>> <[email protected] <mailto:[email protected]>> >>> wrote: >>> >>> Hi Micah, >>> >>> Sorry to pester you, but have you any update on when the remaining >>> Phishtank signatures will be getting removed? It would be really great to >>> get scan times properly back to normal. >>> >>> Best regards >>> Mark >>> >>> On Tue, 9 Apr 2019 at 16:32, Micah Snyder (micasnyd) <[email protected] >>> <mailto:[email protected]>> wrote: >>> Mark, >>> >>> >>> Yes, the plan is still to remove the rest of the Phishtank signatures. We >>> wanted to get things back to relative normal and resolve the immediate >>> crisis. We’ll remove the rest of them soon. >>> >>> >>> >>> Best, >>> >>> Micah >>> >>> >>> >>> From: Mark Allan <[email protected] <mailto:[email protected]>> >>> Date: Tuesday, April 9, 2019 at 6:26 AM >>> To: "Micah Snyder (micasnyd)" <[email protected] >>> <mailto:[email protected]>> >>> Cc: ClamAV users ML <[email protected] >>> <mailto:[email protected]>> >>> Subject: Re: [External] Re: [clamav-users] Scan very slow >>> >>> >>> >>> The scan times are definitely better than they were - in fact, they're back >>> to how they were before last week's inclusion of the Phishtank signatures. >>> They're still almost double what they used to be though, and as far as I >>> can see, there are still almost 4000 Phishtank signatures in the DB: >>> >>> $ sigtool --find Phishtank | wc -l >>> >>> 3968 >>> >>> >>> >>> Can I request that those ones also be removed please? >>> >>> >>> >>> Best regards >>> >>> Mark >>> >>> >>> >>> On Sun, 7 Apr 2019 at 14:43, Micah Snyder (micasnyd) <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Tim, >>> >>> >>> >>> There are a couple of ways for users to drop specific categories of >>> signatures at this time. Sadly, they wouldn’t have helped this last week. >>> These include bytecode signatures, PUA (potentially unwanted applications) >>> signatures, Email.Phishing and HTML.Phishing signatures, and the >>> Safebrowsing database. >>> >>> >>> >>> If we had named the Phishtank.Phishing sigs to HTML.Phishing.Phishtank or >>> Email.Phishing.Phishtank then they could have been disabled with the >>> clamscan option `--phishing-sigs=no` (clamd.conf: `PhishingSignatures no`). >>> >>> >>> >>> Maybe a better option would be for us to create a new optional database for >>> phishing signatures. However, the names for the databases are hardcoded >>> into freshclam, so it is non-trivial to add a new database and would >>> require a few changes to ClamAV’s code. We have talked about making the >>> databases easier to add/remove in the future so users can have more >>> categories to enable/disable. In this light, it ties in well with existing >>> plans. >>> >>> >>> >>> Of note the Phishtank sigs from Friday’s daily were removed yesterday and >>> scan times should be back to normal. >>> >>> >>> >>> Regards, >>> >>> Micah >>> >>> >>> >>> From: Tim Hawkins <[email protected] >>> <mailto:[email protected]>> >>> Date: Friday, April 5, 2019 at 6:06 PM >>> To: ClamAV users ML <[email protected] >>> <mailto:[email protected]>>, Mark Allan <[email protected] >>> <mailto:[email protected]>> >>> Cc: "Micah Snyder (micasnyd)" <[email protected] >>> <mailto:[email protected]>> >>> Subject: Re: [External] Re: [clamav-users] Scan very slow >>> >>> >>> >>> Hi Micah >>> >>> >>> Does clamav partition the database so that signatures that are mainly >>> associated with email scanning can be dropped out for folks only needing >>> filesystems scans, none of our systems use email, and we dont make use of >>> the mailer extension. >>> >>> Having to load all the email focused signatures could as you have observed >>> impact performance. >>> >>> Sent from Nine <http://www.9folders.com/> >>> From: "Micah Snyder (micasnyd) via clamav-users" >>> <[email protected] <mailto:[email protected]>> >>> Sent: Saturday, April 6, 2019 03:18 >>> To: ClamAV users ML; Mark Allan >>> Cc: Micah Snyder (micasnyd) >>> Subject: [External] Re: [clamav-users] Scan very slow >>> >>> >>> >>> Regarding slow scan times today (and slow scan times in general), it >>> appears that the signatures we generate based on PhishTank’s feed for >>> phishing URLs are resulting in very slow load and scan times. >>> >>> >>> >>> Today’s daily update saw 7448 new Phishtank signatures (much higher than >>> usual) coinciding with the immediate performance drop for load time and >>> scan time. One user reported that the load time today on some of his >>> slower machines was slow enough to exceed the timeout for service startup >>> (https://bugzilla.clamav.net/show_bug.cgi?id=12317 >>> <https://bugzilla.clamav.net/show_bug.cgi?id=12317>). >>> >>> >>> >>> In limited testing on my own machine I saw the following change after >>> dropping the Phishtank.Phishing signatures from daily.cvd’s daily.ldb file: >>> >>> Database load time on my laptop went from 75.43203997612 seconds down to >>> 14.859203100204468 seconds >>> Scan time (for an arbitrary pdf) went from 1.798 sec to 0.644 sec. >>> >>> >>> After some discussion between the teams that work on ClamAV and ClamAV >>> signature content and deployment, we’ve agreed to drop PhishTank signatures >>> from the database until we can determine a way to craft Phishtank >>> signatures without incurring such a significant performance hit. >>> >>> >>> >>> The daily update tomorrow will have the change. >>> >>> >>> >>> -Micah >>> >>> >>> >>> >>> Micah Snyder >>> ClamAV Development >>> Talos >>> Cisco Systems, Inc. >>> >>> >>> >>> >>> >>> >>> >>> From: clamav-users <[email protected] >>> <mailto:[email protected]>> on behalf of "Micah Snyder >>> (micasnyd) via clamav-users" <[email protected] >>> <mailto:[email protected]>> >>> Reply-To: ClamAV users ML <[email protected] >>> <mailto:[email protected]>> >>> Date: Friday, April 5, 2019 at 1:08 PM >>> To: Mark Allan <[email protected] <mailto:[email protected]>>, ClamAV >>> users ML <[email protected] >>> <mailto:[email protected]>> >>> Cc: "Micah Snyder (micasnyd)" <[email protected] >>> <mailto:[email protected]>> >>> Subject: Re: [clamav-users] Scan very slow >>> >>> >>> >>> Hi Mark, >>> >>> >>> >>> Sorry about the delay in responding. I hadn’t looked at my clamav-users >>> filter this morning. Just investigating now. Will respond when I know >>> more. >>> >>> >>> >>> -Micah >>> >>> >>> >>> From: Mark Allan <[email protected] <mailto:[email protected]>> >>> Date: Friday, April 5, 2019 at 9:12 AM >>> To: ClamAV users ML <[email protected] >>> <mailto:[email protected]>>, "Micah Snyder (micasnyd)" >>> <[email protected] <mailto:[email protected]>> >>> Subject: Re: [clamav-users] Scan very slow >>> >>> >>> >>> Also CC'ing Micah directly as the mailing list would appear to be offline >>> (at least lists.clamav.net <http://lists.clamav.net/> isn't responding to >>> http requests anyway) >>> >>> >>> >>> It looks like scan times have gone through the roof. As Oya said, they're >>> still considerably higher than they were a couple of months ago, but >>> today's scan time is insane. >>> >>> >>> >>> Yesterday's scan using >>> >>> 0.101.2:58:25409:1554370140:1:63:48554:328 >>> >>> took 7m 3s >>> >>> >>> >>> On the same hardware, scanning the same read-only disk image, with today's >>> scan using >>> >>> 0.101.2:58:25410:1554452941:1:63:48557:328 >>> >>> the scan time has jumped to 26m 15s >>> >>> >>> >>> This is the longest it has ever taken to scan this volume (cf my previous >>> email of 25th March) >>> >>> >>> >>> Is there anything that can be excluded? >>> >>> >>> >>> Best regards >>> >>> Mark >>> >>> >>> >>> On Mon, 1 Apr 2019 at 17:11, Micah Snyder (micasnyd) via clamav-users >>> <[email protected] <mailto:[email protected]>> >>> wrote: >>> >>> Thanks Oya for the update. We will continue to investigate the signature >>> performance issue. >>> >>> Regards, >>> Micah >>> >>> On 3/28/19, 9:50 AM, "clamav-users on behalf of Tsutomu Oyamada" >>> <[email protected] >>> <mailto:[email protected]> on behalf of >>> [email protected] <mailto:[email protected]>> wrote: >>> >>> Hi Micah >>> >>> It seems that the scanning slow down issue of this time has been solved >>> at some level with CVD Update of the other day. >>> However, there is still big discrepancy in between the current >>> condition and >>> the last condition in one month ago. >>> >>> Date Files Scan time >>> 2019/02/15 2550338 08:53:57 >>> 2019/03/15 2612792 19:22:54 >>> 2019/03/26 2634489 18:13:56 >>> 2019/03/27 2637201 18:10:05 >>> >>> We know the improvement of this time is due to the details of CVD, >>> because >>> we did not make any change on the user's system. >>> We are going to try some tuning for scanning. >>> >>> We like to know if you still have some room to make further improvement >>> for this slow down issue. >>> Thank you for your help, in advance. >>> >>> Best regards, >>> Oya >>> >>> On Mon, 25 Mar 2019 15:45:02 +0000 >>> "Micah Snyder \(micasnyd\) via clamav-users" >>> <[email protected] <mailto:[email protected]>> >>> wrote: >>> >>> > Hi Mark, all: >>> > >>> > I’m disappointed to hear that it is still slow for you. >>> > >>> > We found that the target-type of signatures used for >>> PhishTank.Phishing signatures were causing a significant slowdown. We >>> have dropped them as of this past Saturday >>> (https://lists.gt.net/clamav/virusdb/75279 >>> <https://lists.gt.net/clamav/virusdb/75279> ) and in the last two updates >>> have been re-adding them with more specific scan target types. We’re now >>> investigating some other optimizations we can make for the next major >>> ClamAV release to improve scan times but at present we don’t have any other >>> leads for signatures that may be slowing down scans. >>> > >>> > Regards, >>> > Micah >>> > >>> > >>> > From: clamav-users <[email protected] >>> <mailto:[email protected]>> on behalf of Mark Allan via >>> clamav-users <[email protected] >>> <mailto:[email protected]>> >>> > Reply-To: ClamAV users ML <[email protected] >>> <mailto:[email protected]>> >>> > Date: Monday, March 25, 2019 at 9:37 AM >>> > To: ClamAV users ML <[email protected] >>> <mailto:[email protected]>> >>> > Cc: Mark Allan <[email protected] <mailto:[email protected]>> >>> > Subject: Re: [clamav-users] Scan very slow >>> > >>> > Cheers Steve, >>> > >>> > In the interest of completeness, here's the scan from today (TXT from >>> DNS: 0.101.1:58:25399:1553509741:1:63:48528:328) showing a marked >>> improvement in scan time, although at 6m 7s it's still almost twice what it >>> used to be. >>> > >>> > Mark >>> > >>> > On Mon, 25 Mar 2019 at 12:56, Steve Basford >>> <[email protected] >>> <mailto:[email protected]><mailto:[email protected] >>> <mailto:[email protected]>>> wrote: >>> > On 2019-03-25 10:52, Mark Allan via clamav-users wrote: >>> > > Hi all, >>> > > >>> > te. >>> > > >>> > > Hopefully this helps someone to narrow things down a bit. >>> > > >>> > > Mark >>> > > >>> > >>> > 18/3/19 10m 49s TXT from DNS: >>> > 0.101.1:58:25392:1552904941:1:63:48507:328 *** >>> > >>> > Here's the changes for the above update: >>> > >>> > https://lists.gt.net/clamav/virusdb/75154 >>> <https://lists.gt.net/clamav/virusdb/75154> >>> > >>> > You can also check sigs quickly per update: >>> > >>> > https://lists.gt.net/clamav/virusdb/ >>> <https://lists.gt.net/clamav/virusdb/> >>> > >>> > >>> > >>> > -- >>> > Cheers, >>> > >>> > Steve >>> > Twitter: @sanesecurity >>> > >>> > _______________________________________________ >>> > >>> > clamav-users mailing list >>> > [email protected] >>> <mailto:[email protected]><mailto:[email protected] >>> <mailto:[email protected]>> >>> > https://lists.clamav.net/mailman/listinfo/clamav-users >>> <https://lists.clamav.net/mailman/listinfo/clamav-users> >>> > >>> > >>> > Help us build a comprehensive ClamAV guide: >>> > https://github.com/vrtadmin/clamav-faq >>> <https://github.com/vrtadmin/clamav-faq> >>> > >>> > http://www.clamav.net/contact.html#ml >>> <http://www.clamav.net/contact.html#ml> >>> >>> >>> >>> _______________________________________________ >>> >>> clamav-users mailing list >>> [email protected] <mailto:[email protected]> >>> https://lists.clamav.net/mailman/listinfo/clamav-users >>> <https://lists.clamav.net/mailman/listinfo/clamav-users> >>> >>> >>> Help us build a comprehensive ClamAV guide: >>> https://github.com/vrtadmin/clamav-faq >>> <https://github.com/vrtadmin/clamav-faq> >>> >>> http://www.clamav.net/contact.html#ml >>> <http://www.clamav.net/contact.html#ml> >>> >>> >>> >>> _______________________________________________ >>> >>> clamav-users mailing list >>> [email protected] <mailto:[email protected]> >>> https://lists.clamav.net/mailman/listinfo/clamav-users >>> <https://lists.clamav.net/mailman/listinfo/clamav-users> >>> >>> >>> Help us build a comprehensive ClamAV guide: >>> https://github.com/vrtadmin/clamav-faq >>> <https://github.com/vrtadmin/clamav-faq> >>> >>> http://www.clamav.net/contact.html#ml >>> <http://www.clamav.net/contact.html#ml> >>> >>> >>> DISCLAIMER >>> >>> The information contained in this email and any attachments are >>> confidential. It is intended solely for the individual or entity to whom >>> they are addressed. Access to this email by anyone else is unauthorized. >>> >>> If you are not the intended recipient, any disclosure, copying, >>> distribution or any action taken or omitted to be taken in reliance on it, >>> is prohibited and may be unlawful. If you have received this communication >>> in error, please notify us immediately by responding to this email and then >>> delete it from your system. >>> >>> The Red Flag Group is neither liable for the proper and complete >>> transmission of the information contained in this communication nor for any >>> delay in its receipt. >>> >>> Any advice, recommendations or opinion contained within this email or its >>> attachments are not to be construed as legal advice. >>> >>> >>> _______________________________________________ >>> >>> clamav-users mailing list >>> [email protected] <mailto:[email protected]> >>> https://lists.clamav.net/mailman/listinfo/clamav-users >>> <https://lists.clamav.net/mailman/listinfo/clamav-users> >>> >>> >>> Help us build a comprehensive ClamAV guide: >>> https://github.com/vrtadmin/clamav-faq >>> <https://github.com/vrtadmin/clamav-faq> >>> >>> http://www.clamav.net/contact.html#ml >>> <http://www.clamav.net/contact.html#ml> >> >> >> _______________________________________________ >> >> clamav-users mailing list >> [email protected] <mailto:[email protected]> >> https://lists.clamav.net/mailman/listinfo/clamav-users >> <https://lists.clamav.net/mailman/listinfo/clamav-users> >> >> >> Help us build a comprehensive ClamAV guide: >> https://github.com/vrtadmin/clamav-faq >> <https://github.com/vrtadmin/clamav-faq> >> >> http://www.clamav.net/contact.html#ml <http://www.clamav.net/contact.html#ml> >> >> >
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ clamav-users mailing list [email protected] https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
