Are the "Phish" REPHISH signatures still in the daily or were they removed as well? Those were causing part of the issue.
--Maarten On Wed, Apr 17, 2019 at 5:24 AM Al Varnell via clamav-users < clamav-users@lists.clamav.net> wrote: > An additional 3968 Phishtank.Phishing.PHISH_ID_??????? signatures were > dropped by daily-25417 on 12 April, and I can't seem to locate any more. > > -Al- > > On Apr 17, 2019, at 02:01, Mark Allan via clamav-users < > clamav-users@lists.clamav.net> wrote: > > Hi Micah, > > Sorry to pester you, but have you any update on when the remaining > Phishtank signatures will be getting removed? It would be really great to > get scan times properly back to normal. > > Best regards > Mark > > On Tue, 9 Apr 2019 at 16:32, Micah Snyder (micasnyd) <micas...@cisco.com> > wrote: > >> Mark, >> >> >> Yes, the plan is still to remove the rest of the Phishtank signatures. >> We wanted to get things back to relative normal and resolve the immediate >> crisis. We’ll remove the rest of them soon. >> >> >> >> Best, >> >> Micah >> >> >> >> *From: *Mark Allan <markjal...@gmail.com> >> *Date: *Tuesday, April 9, 2019 at 6:26 AM >> *To: *"Micah Snyder (micasnyd)" <micas...@cisco.com> >> *Cc: *ClamAV users ML <clamav-users@lists.clamav.net> >> *Subject: *Re: [External] Re: [clamav-users] Scan very slow >> >> >> >> The scan times are definitely better than they were - in fact, they're >> back to how they were before last week's inclusion of the Phishtank >> signatures. They're still almost double what they used to be though, and as >> far as I can see, there are still almost 4000 Phishtank signatures in the >> DB: >> >> $ sigtool --find Phishtank | wc -l >> >> 3968 >> >> >> >> Can I request that those ones also be removed please? >> >> >> >> Best regards >> >> Mark >> >> >> >> On Sun, 7 Apr 2019 at 14:43, Micah Snyder (micasnyd) <micas...@cisco.com> >> wrote: >> >> Tim, >> >> >> >> There are a couple of ways for users to drop specific categories of >> signatures at this time. Sadly, they wouldn’t have helped this last week. >> These include bytecode signatures, PUA (potentially unwanted applications) >> signatures, Email.Phishing and HTML.Phishing signatures, and the >> Safebrowsing database. >> >> >> >> If we had named the Phishtank.Phishing sigs to HTML.Phishing.Phishtank or >> Email.Phishing.Phishtank then they could have been disabled with the >> clamscan option `--phishing-sigs=no` (clamd.conf: `PhishingSignatures no`). >> >> >> >> Maybe a better option would be for us to create a new optional database >> for phishing signatures. However, the names for the databases are hardcoded >> into freshclam, so it is non-trivial to add a new database and would >> require a few changes to ClamAV’s code. We have talked about making the >> databases easier to add/remove in the future so users can have more >> categories to enable/disable. In this light, it ties in well with existing >> plans. >> >> >> >> Of note the Phishtank sigs from Friday’s daily were removed yesterday and >> scan times should be back to normal. >> >> >> >> Regards, >> >> Micah >> >> >> >> *From: *Tim Hawkins <tim.hawk...@redflaggroup.com> >> *Date: *Friday, April 5, 2019 at 6:06 PM >> *To: *ClamAV users ML <clamav-users@lists.clamav.net>, Mark Allan < >> markjal...@gmail.com> >> *Cc: *"Micah Snyder (micasnyd)" <micas...@cisco.com> >> *Subject: *Re: [External] Re: [clamav-users] Scan very slow >> >> >> >> Hi Micah >> >> >> Does clamav partition the database so that signatures that are mainly >> associated with email scanning can be dropped out for folks only needing >> filesystems scans, none of our systems use email, and we dont make use of >> the mailer extension. >> >> Having to load all the email focused signatures could as you have >> observed impact performance. >> >> Sent from Nine <http://www.9folders.com/> >> ------------------------------ >> >> *From:* "Micah Snyder (micasnyd) via clamav-users" < >> clamav-users@lists.clamav.net> >> *Sent:* Saturday, April 6, 2019 03:18 >> *To:* ClamAV users ML; Mark Allan >> *Cc:* Micah Snyder (micasnyd) >> *Subject:* [External] Re: [clamav-users] Scan very slow >> >> >> >> Regarding slow scan times today (and slow scan times in general), it >> appears that the signatures we generate based on PhishTank’s feed for >> phishing URLs are resulting in very slow load and scan times. >> >> >> >> Today’s daily update saw 7448 new Phishtank signatures (much higher than >> usual) coinciding with the immediate performance drop for load time and >> scan time. One user reported that the load time today on some of his >> slower machines was slow enough to exceed the timeout for service startup ( >> https://bugzilla.clamav.net/show_bug.cgi?id=12317). >> >> >> >> In limited testing on my own machine I saw the following change after >> dropping the Phishtank.Phishing signatures from daily.cvd’s daily.ldb file: >> >> - Database load time on my laptop went from 75.43203997612 seconds >> down to 14.859203100204468 seconds >> - Scan time (for an arbitrary pdf) went from 1.798 sec to 0.644 sec. >> >> >> >> After some discussion between the teams that work on ClamAV and ClamAV >> signature content and deployment, we’ve agreed to drop PhishTank signatures >> from the database until we can determine a way to craft Phishtank >> signatures without incurring such a significant performance hit. >> >> >> >> The daily update tomorrow will have the change. >> >> >> >> -Micah >> >> >> >> >> Micah Snyder >> ClamAV Development >> Talos >> Cisco Systems, Inc. >> >> >> >> >> >> >> >> *From: *clamav-users <clamav-users-boun...@lists.clamav.net> on behalf >> of "Micah Snyder (micasnyd) via clamav-users" < >> clamav-users@lists.clamav.net> >> *Reply-To: *ClamAV users ML <clamav-users@lists.clamav.net> >> *Date: *Friday, April 5, 2019 at 1:08 PM >> *To: *Mark Allan <markjal...@gmail.com>, ClamAV users ML < >> clamav-users@lists.clamav.net> >> *Cc: *"Micah Snyder (micasnyd)" <micas...@cisco.com> >> *Subject: *Re: [clamav-users] Scan very slow >> >> >> >> Hi Mark, >> >> >> >> Sorry about the delay in responding. I hadn’t looked at my clamav-users >> filter this morning. Just investigating now. Will respond when I know >> more. >> >> >> >> -Micah >> >> >> >> *From: *Mark Allan <markjal...@gmail.com> >> *Date: *Friday, April 5, 2019 at 9:12 AM >> *To: *ClamAV users ML <clamav-users@lists.clamav.net>, "Micah Snyder >> (micasnyd)" <micas...@cisco.com> >> *Subject: *Re: [clamav-users] Scan very slow >> >> >> >> Also CC'ing Micah directly as the mailing list would appear to be offline >> (at least lists.clamav.net isn't responding to http requests anyway) >> >> >> >> It looks like scan times have gone through the roof. As Oya said, they're >> still considerably higher than they were a couple of months ago, but >> today's scan time is insane. >> >> >> >> Yesterday's scan using >> >> 0.101.2:58:25409:1554370140:1:63:48554:328 >> >> took 7m 3s >> >> >> >> On the same hardware, scanning the same read-only disk image, with >> today's scan using >> >> 0.101.2:58:25410:1554452941:1:63:48557:328 >> >> the scan time has jumped to 26m 15s >> >> >> >> This is the longest it has ever taken to scan this volume (cf my previous >> email of 25th March) >> >> >> >> Is there anything that can be excluded? >> >> >> >> Best regards >> >> Mark >> >> >> >> On Mon, 1 Apr 2019 at 17:11, Micah Snyder (micasnyd) via clamav-users < >> clamav-users@lists.clamav.net> wrote: >> >> Thanks Oya for the update. We will continue to investigate the signature >> performance issue. >> >> Regards, >> Micah >> >> On 3/28/19, 9:50 AM, "clamav-users on behalf of Tsutomu Oyamada" < >> clamav-users-boun...@lists.clamav.net on behalf of >> oyam...@promark-inc.com> wrote: >> >> Hi Micah >> >> It seems that the scanning slow down issue of this time has been >> solved >> at some level with CVD Update of the other day. >> However, there is still big discrepancy in between the current >> condition and >> the last condition in one month ago. >> >> Date Files Scan time >> 2019/02/15 2550338 08:53:57 >> 2019/03/15 2612792 19:22:54 >> 2019/03/26 2634489 18:13:56 >> 2019/03/27 2637201 18:10:05 >> >> We know the improvement of this time is due to the details of CVD, >> because >> we did not make any change on the user's system. >> We are going to try some tuning for scanning. >> >> We like to know if you still have some room to make further >> improvement >> for this slow down issue. >> Thank you for your help, in advance. >> >> Best regards, >> Oya >> >> On Mon, 25 Mar 2019 15:45:02 +0000 >> "Micah Snyder \(micasnyd\) via clamav-users" < >> clamav-users@lists.clamav.net> wrote: >> >> > Hi Mark, all: >> > >> > I’m disappointed to hear that it is still slow for you. >> > >> > We found that the target-type of signatures used for >> PhishTank.Phishing signatures were causing a significant slowdown. We >> have dropped them as of this past Saturday ( >> https://lists.gt.net/clamav/virusdb/75279 ) and in the last two updates >> have been re-adding them with more specific scan target types. We’re now >> investigating some other optimizations we can make for the next major >> ClamAV release to improve scan times but at present we don’t have any other >> leads for signatures that may be slowing down scans. >> > >> > Regards, >> > Micah >> > >> > >> > From: clamav-users <clamav-users-boun...@lists.clamav.net> on >> behalf of Mark Allan via clamav-users <clamav-users@lists.clamav.net> >> > Reply-To: ClamAV users ML <clamav-users@lists.clamav.net> >> > Date: Monday, March 25, 2019 at 9:37 AM >> > To: ClamAV users ML <clamav-users@lists.clamav.net> >> > Cc: Mark Allan <markjal...@gmail.com> >> > Subject: Re: [clamav-users] Scan very slow >> > >> > Cheers Steve, >> > >> > In the interest of completeness, here's the scan from today (TXT >> from DNS: 0.101.1:58:25399:1553509741:1:63:48528:328) showing a marked >> improvement in scan time, although at 6m 7s it's still almost twice what it >> used to be. >> > >> > Mark >> > >> > On Mon, 25 Mar 2019 at 12:56, Steve Basford < >> steveb_cla...@sanesecurity.com<mailto:steveb_cla...@sanesecurity.com>> >> wrote: >> > On 2019-03-25 10:52, Mark Allan via clamav-users wrote: >> > > Hi all, >> > > >> > te. >> > > >> > > Hopefully this helps someone to narrow things down a bit. >> > > >> > > Mark >> > > >> > >> > 18/3/19 10m 49s TXT from DNS: >> > 0.101.1:58:25392:1552904941:1:63:48507:328 *** >> > >> > Here's the changes for the above update: >> > >> > https://lists.gt.net/clamav/virusdb/75154 >> > >> > You can also check sigs quickly per update: >> > >> > https://lists.gt.net/clamav/virusdb/ >> > >> > >> > >> > -- >> > Cheers, >> > >> > Steve >> > Twitter: @sanesecurity >> > >> > _______________________________________________ >> > >> > clamav-users mailing list >> > clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net> >> > https://lists.clamav.net/mailman/listinfo/clamav-users >> > >> > >> > Help us build a comprehensive ClamAV guide: >> > https://github.com/vrtadmin/clamav-faq >> > >> > http://www.clamav.net/contact.html#ml >> >> >> >> _______________________________________________ >> >> clamav-users mailing list >> clamav-users@lists.clamav.net >> https://lists.clamav.net/mailman/listinfo/clamav-users >> >> >> Help us build a comprehensive ClamAV guide: >> https://github.com/vrtadmin/clamav-faq >> >> http://www.clamav.net/contact.html#ml >> >> >> >> _______________________________________________ >> >> clamav-users mailing list >> clamav-users@lists.clamav.net >> https://lists.clamav.net/mailman/listinfo/clamav-users >> >> >> Help us build a comprehensive ClamAV guide: >> https://github.com/vrtadmin/clamav-faq >> >> http://www.clamav.net/contact.html#ml >> >> >> >> *DISCLAIMER* >> >> The information contained in this email and any attachments are >> confidential. It is intended solely for the individual or entity to whom >> they are addressed. Access to this email by anyone else is unauthorized. >> >> If you are not the intended recipient, any disclosure, copying, >> distribution or any action taken or omitted to be taken in reliance on it, >> is prohibited and may be unlawful. If you have received this communication >> in error, please notify us immediately by responding to this email and then >> delete it from your system. >> >> The Red Flag Group is neither liable for the proper and complete >> transmission of the information contained in this communication nor for any >> delay in its receipt. >> >> Any advice, recommendations or opinion contained within this email or its >> attachments are not to be construed as legal advice. >> >> > _______________________________________________ > > clamav-users mailing list > clamav-users@lists.clamav.net > https://lists.clamav.net/mailman/listinfo/clamav-users > > > Help us build a comprehensive ClamAV guide: > https://github.com/vrtadmin/clamav-faq > > http://www.clamav.net/contact.html#ml > > > > _______________________________________________ > > clamav-users mailing list > clamav-users@lists.clamav.net > https://lists.clamav.net/mailman/listinfo/clamav-users > > > Help us build a comprehensive ClamAV guide: > https://github.com/vrtadmin/clamav-faq > > http://www.clamav.net/contact.html#ml >
_______________________________________________ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
