An additional 3968 Phishtank.Phishing.PHISH_ID_??????? signatures were dropped by daily-25417 on 12 April, and I can't seem to locate any more.
-Al- > On Apr 17, 2019, at 02:01, Mark Allan via clamav-users > <clamav-users@lists.clamav.net> wrote: > > Hi Micah, > > Sorry to pester you, but have you any update on when the remaining Phishtank > signatures will be getting removed? It would be really great to get scan > times properly back to normal. > > Best regards > Mark > > On Tue, 9 Apr 2019 at 16:32, Micah Snyder (micasnyd) <micas...@cisco.com > <mailto:micas...@cisco.com>> wrote: > Mark, > > > Yes, the plan is still to remove the rest of the Phishtank signatures. We > wanted to get things back to relative normal and resolve the immediate > crisis. We’ll remove the rest of them soon. > > > > Best, > > Micah > > > > From: Mark Allan <markjal...@gmail.com <mailto:markjal...@gmail.com>> > Date: Tuesday, April 9, 2019 at 6:26 AM > To: "Micah Snyder (micasnyd)" <micas...@cisco.com <mailto:micas...@cisco.com>> > Cc: ClamAV users ML <clamav-users@lists.clamav.net > <mailto:clamav-users@lists.clamav.net>> > Subject: Re: [External] Re: [clamav-users] Scan very slow > > > > The scan times are definitely better than they were - in fact, they're back > to how they were before last week's inclusion of the Phishtank signatures. > They're still almost double what they used to be though, and as far as I can > see, there are still almost 4000 Phishtank signatures in the DB: > > $ sigtool --find Phishtank | wc -l > > 3968 > > > > Can I request that those ones also be removed please? > > > > Best regards > > Mark > > > > On Sun, 7 Apr 2019 at 14:43, Micah Snyder (micasnyd) <micas...@cisco.com > <mailto:micas...@cisco.com>> wrote: > > Tim, > > > > There are a couple of ways for users to drop specific categories of > signatures at this time. Sadly, they wouldn’t have helped this last week. > These include bytecode signatures, PUA (potentially unwanted applications) > signatures, Email.Phishing and HTML.Phishing signatures, and the Safebrowsing > database. > > > > If we had named the Phishtank.Phishing sigs to HTML.Phishing.Phishtank or > Email.Phishing.Phishtank then they could have been disabled with the clamscan > option `--phishing-sigs=no` (clamd.conf: `PhishingSignatures no`). > > > > Maybe a better option would be for us to create a new optional database for > phishing signatures. However, the names for the databases are hardcoded into > freshclam, so it is non-trivial to add a new database and would require a few > changes to ClamAV’s code. We have talked about making the databases easier to > add/remove in the future so users can have more categories to enable/disable. > In this light, it ties in well with existing plans. > > > > Of note the Phishtank sigs from Friday’s daily were removed yesterday and > scan times should be back to normal. > > > > Regards, > > Micah > > > > From: Tim Hawkins <tim.hawk...@redflaggroup.com > <mailto:tim.hawk...@redflaggroup.com>> > Date: Friday, April 5, 2019 at 6:06 PM > To: ClamAV users ML <clamav-users@lists.clamav.net > <mailto:clamav-users@lists.clamav.net>>, Mark Allan <markjal...@gmail.com > <mailto:markjal...@gmail.com>> > Cc: "Micah Snyder (micasnyd)" <micas...@cisco.com <mailto:micas...@cisco.com>> > Subject: Re: [External] Re: [clamav-users] Scan very slow > > > > Hi Micah > > > Does clamav partition the database so that signatures that are mainly > associated with email scanning can be dropped out for folks only needing > filesystems scans, none of our systems use email, and we dont make use of > the mailer extension. > > Having to load all the email focused signatures could as you have observed > impact performance. > > Sent from Nine <http://www.9folders.com/> > From: "Micah Snyder (micasnyd) via clamav-users" > <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>> > Sent: Saturday, April 6, 2019 03:18 > To: ClamAV users ML; Mark Allan > Cc: Micah Snyder (micasnyd) > Subject: [External] Re: [clamav-users] Scan very slow > > > > Regarding slow scan times today (and slow scan times in general), it appears > that the signatures we generate based on PhishTank’s feed for phishing URLs > are resulting in very slow load and scan times. > > > > Today’s daily update saw 7448 new Phishtank signatures (much higher than > usual) coinciding with the immediate performance drop for load time and scan > time. One user reported that the load time today on some of his slower > machines was slow enough to exceed the timeout for service startup > (https://bugzilla.clamav.net/show_bug.cgi?id=12317 > <https://bugzilla.clamav.net/show_bug.cgi?id=12317>). > > > > In limited testing on my own machine I saw the following change after > dropping the Phishtank.Phishing signatures from daily.cvd’s daily.ldb file: > > Database load time on my laptop went from 75.43203997612 seconds down to > 14.859203100204468 seconds > Scan time (for an arbitrary pdf) went from 1.798 sec to 0.644 sec. > > > After some discussion between the teams that work on ClamAV and ClamAV > signature content and deployment, we’ve agreed to drop PhishTank signatures > from the database until we can determine a way to craft Phishtank signatures > without incurring such a significant performance hit. > > > > The daily update tomorrow will have the change. > > > > -Micah > > > > > Micah Snyder > ClamAV Development > Talos > Cisco Systems, Inc. > > > > > > > > From: clamav-users <clamav-users-boun...@lists.clamav.net > <mailto:clamav-users-boun...@lists.clamav.net>> on behalf of "Micah Snyder > (micasnyd) via clamav-users" <clamav-users@lists.clamav.net > <mailto:clamav-users@lists.clamav.net>> > Reply-To: ClamAV users ML <clamav-users@lists.clamav.net > <mailto:clamav-users@lists.clamav.net>> > Date: Friday, April 5, 2019 at 1:08 PM > To: Mark Allan <markjal...@gmail.com <mailto:markjal...@gmail.com>>, ClamAV > users ML <clamav-users@lists.clamav.net > <mailto:clamav-users@lists.clamav.net>> > Cc: "Micah Snyder (micasnyd)" <micas...@cisco.com <mailto:micas...@cisco.com>> > Subject: Re: [clamav-users] Scan very slow > > > > Hi Mark, > > > > Sorry about the delay in responding. I hadn’t looked at my clamav-users > filter this morning. Just investigating now. Will respond when I know more. > > > > -Micah > > > > From: Mark Allan <markjal...@gmail.com <mailto:markjal...@gmail.com>> > Date: Friday, April 5, 2019 at 9:12 AM > To: ClamAV users ML <clamav-users@lists.clamav.net > <mailto:clamav-users@lists.clamav.net>>, "Micah Snyder (micasnyd)" > <micas...@cisco.com <mailto:micas...@cisco.com>> > Subject: Re: [clamav-users] Scan very slow > > > > Also CC'ing Micah directly as the mailing list would appear to be offline (at > least lists.clamav.net <http://lists.clamav.net/> isn't responding to http > requests anyway) > > > > It looks like scan times have gone through the roof. As Oya said, they're > still considerably higher than they were a couple of months ago, but today's > scan time is insane. > > > > Yesterday's scan using > > 0.101.2:58:25409:1554370140:1:63:48554:328 > > took 7m 3s > > > > On the same hardware, scanning the same read-only disk image, with today's > scan using > > 0.101.2:58:25410:1554452941:1:63:48557:328 > > the scan time has jumped to 26m 15s > > > > This is the longest it has ever taken to scan this volume (cf my previous > email of 25th March) > > > > Is there anything that can be excluded? > > > > Best regards > > Mark > > > > On Mon, 1 Apr 2019 at 17:11, Micah Snyder (micasnyd) via clamav-users > <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>> wrote: > > Thanks Oya for the update. We will continue to investigate the signature > performance issue. > > Regards, > Micah > > On 3/28/19, 9:50 AM, "clamav-users on behalf of Tsutomu Oyamada" > <clamav-users-boun...@lists.clamav.net > <mailto:clamav-users-boun...@lists.clamav.net> on behalf of > oyam...@promark-inc.com <mailto:oyam...@promark-inc.com>> wrote: > > Hi Micah > > It seems that the scanning slow down issue of this time has been solved > at some level with CVD Update of the other day. > However, there is still big discrepancy in between the current condition > and > the last condition in one month ago. > > Date Files Scan time > 2019/02/15 2550338 08:53:57 > 2019/03/15 2612792 19:22:54 > 2019/03/26 2634489 18:13:56 > 2019/03/27 2637201 18:10:05 > > We know the improvement of this time is due to the details of CVD, because > we did not make any change on the user's system. > We are going to try some tuning for scanning. > > We like to know if you still have some room to make further improvement > for this slow down issue. > Thank you for your help, in advance. > > Best regards, > Oya > > On Mon, 25 Mar 2019 15:45:02 +0000 > "Micah Snyder \(micasnyd\) via clamav-users" > <clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net>> wrote: > > > Hi Mark, all: > > > > I’m disappointed to hear that it is still slow for you. > > > > We found that the target-type of signatures used for PhishTank.Phishing > signatures were causing a significant slowdown. We have dropped them as of > this past Saturday (https://lists.gt.net/clamav/virusdb/75279 > <https://lists.gt.net/clamav/virusdb/75279> ) and in the last two updates > have been re-adding them with more specific scan target types. We’re now > investigating some other optimizations we can make for the next major ClamAV > release to improve scan times but at present we don’t have any other leads > for signatures that may be slowing down scans. > > > > Regards, > > Micah > > > > > > From: clamav-users <clamav-users-boun...@lists.clamav.net > <mailto:clamav-users-boun...@lists.clamav.net>> on behalf of Mark Allan via > clamav-users <clamav-users@lists.clamav.net > <mailto:clamav-users@lists.clamav.net>> > > Reply-To: ClamAV users ML <clamav-users@lists.clamav.net > <mailto:clamav-users@lists.clamav.net>> > > Date: Monday, March 25, 2019 at 9:37 AM > > To: ClamAV users ML <clamav-users@lists.clamav.net > <mailto:clamav-users@lists.clamav.net>> > > Cc: Mark Allan <markjal...@gmail.com <mailto:markjal...@gmail.com>> > > Subject: Re: [clamav-users] Scan very slow > > > > Cheers Steve, > > > > In the interest of completeness, here's the scan from today (TXT from > DNS: 0.101.1:58:25399:1553509741:1:63:48528:328) showing a marked improvement > in scan time, although at 6m 7s it's still almost twice what it used to be. > > > > Mark > > > > On Mon, 25 Mar 2019 at 12:56, Steve Basford > <steveb_cla...@sanesecurity.com > <mailto:steveb_cla...@sanesecurity.com><mailto:steveb_cla...@sanesecurity.com > <mailto:steveb_cla...@sanesecurity.com>>> wrote: > > On 2019-03-25 10:52, Mark Allan via clamav-users wrote: > > > Hi all, > > > > > te. > > > > > > Hopefully this helps someone to narrow things down a bit. > > > > > > Mark > > > > > > > 18/3/19 10m 49s TXT from DNS: > > 0.101.1:58:25392:1552904941:1:63:48507:328 *** > > > > Here's the changes for the above update: > > > > https://lists.gt.net/clamav/virusdb/75154 > <https://lists.gt.net/clamav/virusdb/75154> > > > > You can also check sigs quickly per update: > > > > https://lists.gt.net/clamav/virusdb/ > <https://lists.gt.net/clamav/virusdb/> > > > > > > > > -- > > Cheers, > > > > Steve > > Twitter: @sanesecurity > > > > _______________________________________________ > > > > clamav-users mailing list > > clamav-users@lists.clamav.net > <mailto:clamav-users@lists.clamav.net><mailto:clamav-users@lists.clamav.net > <mailto:clamav-users@lists.clamav.net>> > > https://lists.clamav.net/mailman/listinfo/clamav-users > <https://lists.clamav.net/mailman/listinfo/clamav-users> > > > > > > Help us build a comprehensive ClamAV guide: > > https://github.com/vrtadmin/clamav-faq > <https://github.com/vrtadmin/clamav-faq> > > > > http://www.clamav.net/contact.html#ml > <http://www.clamav.net/contact.html#ml> > > > > _______________________________________________ > > clamav-users mailing list > clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net> > https://lists.clamav.net/mailman/listinfo/clamav-users > <https://lists.clamav.net/mailman/listinfo/clamav-users> > > > Help us build a comprehensive ClamAV guide: > https://github.com/vrtadmin/clamav-faq > <https://github.com/vrtadmin/clamav-faq> > > http://www.clamav.net/contact.html#ml > <http://www.clamav.net/contact.html#ml> > > > > _______________________________________________ > > clamav-users mailing list > clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net> > https://lists.clamav.net/mailman/listinfo/clamav-users > <https://lists.clamav.net/mailman/listinfo/clamav-users> > > > Help us build a comprehensive ClamAV guide: > https://github.com/vrtadmin/clamav-faq > <https://github.com/vrtadmin/clamav-faq> > > http://www.clamav.net/contact.html#ml <http://www.clamav.net/contact.html#ml> > > > DISCLAIMER > > The information contained in this email and any attachments are confidential. > It is intended solely for the individual or entity to whom they are > addressed. Access to this email by anyone else is unauthorized. > > If you are not the intended recipient, any disclosure, copying, distribution > or any action taken or omitted to be taken in reliance on it, is prohibited > and may be unlawful. If you have received this communication in error, please > notify us immediately by responding to this email and then delete it from > your system. > > The Red Flag Group is neither liable for the proper and complete transmission > of the information contained in this communication nor for any delay in its > receipt. > > Any advice, recommendations or opinion contained within this email or its > attachments are not to be construed as legal advice. > > > _______________________________________________ > > clamav-users mailing list > clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net> > https://lists.clamav.net/mailman/listinfo/clamav-users > <https://lists.clamav.net/mailman/listinfo/clamav-users> > > > Help us build a comprehensive ClamAV guide: > https://github.com/vrtadmin/clamav-faq > <https://github.com/vrtadmin/clamav-faq> > > http://www.clamav.net/contact.html#ml <http://www.clamav.net/contact.html#ml>
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
