The function that I use only deletes files more than 10 days old so that BlockReporting still works (for sure with emails under 10 days old and most likely for those that are older).
On Tue, Sep 15, 2009 at 3:07 AM, Thomas Eckardt/eck < thomas.ecka...@thockar.com> wrote: > Sorry - I forgot to say - 1) and 2) will break the BlockReporting - > "resend" function (!!!) - because we will delete files randomly. > > Thomas > > > > > K Post <nntp.p...@gmail.com> > 15.09.2009 04:19 > Bitte antworten an > ASSP development mailing list <assp-test@lists.sourceforge.net> > > > An > ASSP development mailing list <assp-test@lists.sourceforge.net> > Kopie > > Thema > Re: [Assp-test] Antwort: Re: Antwort: Re: fixes and news in 2.0.1_RC0.4.12 > > > > > > > I wish we could just use move2numb or straight number logging, but they > want > us to review the found spam periodically, and that would just be > impossible > if we couldn't use the subjects as a start... Plus, move2numb would break > the block reporting. > Very good points on the fact that bayesian is last and that it'll never be > perfect. Most of our catches are happening before that too, but the > bayesian filters are still catching a bunch. > > We do not have a well developed bombre, nor the time to keep one updataed, > which is a big problem. This is a for a charity, and there's just not the > resources. > > As for duplicate messages not affecting the corpus, I agree, but with this > limitation of needing subject filenames, I can't think of another way of > limiting the number of messages in the file system other than randomly > deleting messages. If I'm going to do that, might as well first get rid > of > the duplicates to help keep the most diverse corpus, no? > > We don't have the luxury of another filter on the real smtp server either. > > We do get a lot of messages with the same subject. The spammers will send > the same message to 100 or more users. If it makes it to the collection > point of the code, we'll get the same message stored 100 times. Our > regexes > catch a lot, but there's plenty of times that an email with a subject like > "V__F_iGAra" will get past everything but bayesian filters. > > Sure, there's a strong possibility of the same subject having a different > body, but deleting 98 of the 100 messages leaves room for 98 other > messages > which will most likely be more diverse. > > Since we definitely need to delete in some manner (which we want to be > random) and move2numb isn't an option, my plan is to edit the maintenance > functionality if subject logging is on and move2numb is off to: > 1) First find messages with the same subject and delete all in excess of 3 > then > 2) calculate if we're over the maxfiles for each folder and then randomly > delete files based on the percent that we're over, in NO way based on file > date. > I've been doing this for 5 or so years with various 1.x installations, but > in a scheduled script, not in ASSP. > > This change will honor the MaxBayesFileAge, MaxCorrectedDays and > maxNoBayesFileAge (you might want to make MaxCorrectDays use a naming > convention more consistent like MaxCorrectedFileAge). > > Would you be intersted in this code? Doesn't it make sense for people > like > me who need to stick to subject logging? Again, I'd much prefer an > alternative from an expert like yourself, that still affords the ability > to > leave subject logging in place. > > > - it was my pleasure to help clean up the admin descriptions. I intend to > do more as soon as there's a new version with the cleanup in place. > > On Mon, Sep 14, 2009 at 1:37 PM, Thomas Eckardt/eck < > thomas.ecka...@thockar.com> wrote: > > > >What method do you recommend using to keep the number of files down? > > Simply > > >deleting by age isn't going to cut it if you want a diverse corpus is > it? > > > > The best way is: > > > > - do not use subject for filenames or use move2num > > - never delete any file by age > > - set maxfiles high enough to get a good corpus > > > > This is the way assp works for years. The new features are for special > > usage - for example: I do not use the bayes engine (only for testing the > > code). I have a very good spamdb, which is not changed for 2 years > (doing > > some small scoring) - the meen way to detect spam is done by PB and > > IP-tests (mx,ptr,helo,FBMVT......). Because some mails bypassing (going > > other ways) assp, I use a bayes engine in Lotus Domino - just for fun - > > there are 1-5 messages blocked per week. But I love to see the subject > in > > Blockreports. > > > > And keep in mind - bayesian checks should be only a small part of spam > > detection - because the simple (???) mathematics is not (could never > be) > > perfect. > > > > Setting up the bomb regexes and all PB-valence values the right way, > helps > > much more than the bayes check. It will take some time (and some work) > to > > find out the best way (values) for you. > > I think, Fritz and I (and possible some others) - we have found that > point > > - our ASSP detects 99.99% (or even more) spams. I have not seen a > blocked > > good email for over one year. > > > > And at the end you have to weight - accept that from 200 users 10 are > > getting one spam per day (having 30.000 or more connections) - or > > analysing tonns of mails and logs to get rid of the 10. > > > > > > >So what are your thoughts, Thomas, of my remove same subjects, > > > > same subject - same email => ignored by assp, only one mail is > processed > > in rebuildspamdb > > > > but > > > > same subject - different body ...... oh, the subject is only one part > of > > the header (from to msg-id ip forwarder .............) - where is the > > end???? > > Just a joke! This will help to reduce the number of mails (files) for a > > while, but not to increase the quality of the corpus (->spamdb) - what > is > > more important for you? > > If you get tonns of spams with the same subject, do not use bayes to > block > > them - use subjectRe or headerRe or blackRe or ........ . > > > > The bayesian check is one of the latest checks of assp - so try to > detect > > spams before. > > > > Thank you for your help fixing the mistakes in config descriptions. > > > > > > Thomas > > > > > > > > > > > > > > K Post <nntp.p...@gmail.com> > > 14.09.2009 16:53 > > Bitte antworten an > > ASSP development mailing list <assp-test@lists.sourceforge.net> > > > > > > An > > ASSP development mailing list <assp-test@lists.sourceforge.net> > > Kopie > > > > Thema > > Re: [Assp-test] Antwort: Re: fixes and news in 2.0.1_RC0.4.12 > > > > > > > > > > > > > > Right, BUT if we're limiting the total number of files in the directory, > > wouldn't it be better to delete these duplicates to give a more diverse > > corpus? > > > > We can look at their subject name and then delete them. This leave room > > for > > other files. > > > > Using subject logging and NOT move2numb, I guess I need some more > > clarification at this point. > > > > What method do you recommend using to keep the number of files down? > > Simply > > deleting by age isn't going to cut it if you want a diverse corpus is > it? > > > > So what are your thoughts, Thomas, of my remove same subjects, remove > > really > > old by date, then remove a percentage (selecting randomly) based on the > > overage in each folder method? > > > > > > Note: in the MaxBayesFileAge, you've got: > > Do not define this option, if you use the bayesian engine of ASSP. > > Deleting > > files because of there age, is wrong in this case!!!!! It should be > "their > > age." There's a bunch of other errors like this which I privately > emailed > > to Fritz, on request. Should I send you that email too? > > > > > > THANKS > > > > On Sun, Sep 13, 2009 at 2:00 AM, Thomas Eckardt/eck < > > thomas.ecka...@thockar.com> wrote: > > > > > >What do you think about deleting redundant corpus emails > > > >based on the subject? > > > > > > Redundant corpus emails are skipped/deleted based on there content > (md5 > > > hash). > > > > > > Thomas > > > > > > > > > > > > > > > K Post <nntp.p...@gmail.com> > > > 13.09.2009 03:28 > > > Bitte antworten an > > > ASSP development mailing list <assp-test@lists.sourceforge.net> > > > > > > > > > An > > > ASSP development mailing list <assp-test@lists.sourceforge.net> > > > Kopie > > > > > > Thema > > > Re: [Assp-test] fixes and news in 2.0.1_RC0.4.12 > > > > > > > > > > > > > > > > > > > > > This is great, and thanks SO much for adding my idea of the max days > > for > > > corrected spam. What do you think about deleting redundant corpus > > emails > > > based on the subject? > > > > > > On Sat, Sep 12, 2009 at 1:18 PM, Thomas Eckardt/eck < > > > thomas.ecka...@thockar.com> wrote: > > > > > > > Hi all, > > > > > > > > I'm back. > > > > > > > > fixed in 4.12: > > > > > > > > - for some messages the mail header was transfered two times > > > > - changing the display language was not working in any case > > > > - the hintbox in the config part of the GUI has shown wrong > > > > updated/changed values > > > > > > > > added in 4.12 > > > > > > > > MaxCorrectedDays > > > > msg008590=Max Corrected File Age > > > > msg008591=This is the number of days a error report will be kept in > > the > > > > correctednotspam and correctedspam folders. These folders are the > > > longterm > > > > memory of ASSP, therefore the default is 1000 days. > > > > > > > > changed in 4.12: > > > > > > > > - the change language part is moved to the main config form ! > > > > > > > > > > > > > > > > Thomas > > > > > > > > DISCLAIMER: > > > > ******************************************************* > > > > This email and any files transmitted with it may be confidential, > > > legally > > > > privileged and protected in law and are intended solely for the use > of > > > the > > > > > > > > individual to whom it is addressed. > > > > This email was multiple times scanned for viruses. There should be > no > > > > known virus in this email! > > > > ******************************************************* > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > > Let Crystal Reports handle the reporting - Free Crystal Reports 2008 > > > 30-Day > > > > trial. Simplify your report design, integration and deployment - and > > > focus > > > > on > > > > what you do best, core application coding. Discover what's new with > > > > Crystal Reports now. http://p.sf.net/sfu/bobj-july > > > > _______________________________________________ > > > > Assp-test mailing list > > > > Assp-test@lists.sourceforge.net > > > > https://lists.sourceforge.net/lists/listinfo/assp-test > > > > > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > Let Crystal Reports handle the reporting - Free Crystal Reports 2008 > > > 30-Day > > > trial. Simplify your report design, integration and deployment - and > > focus > > > on > > > what you do best, core application coding. Discover what's new with > > > Crystal Reports now. http://p.sf.net/sfu/bobj-july > > > _______________________________________________ > > > Assp-test mailing list > > > Assp-test@lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/assp-test > > > > > > > > > > > > > > > DISCLAIMER: > > > ******************************************************* > > > This email and any files transmitted with it may be confidential, > > legally > > > privileged and protected in law and are intended solely for the use of > > the > > > > > > individual to whom it is addressed. > > > This email was multiple times scanned for viruses. There should be no > > > known virus in this email! > > > ******************************************************* > > > > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > Let Crystal Reports handle the reporting - Free Crystal Reports 2008 > > 30-Day > > > trial. Simplify your report design, integration and deployment - and > > focus > > > on > > > what you do best, core application coding. Discover what's new with > > > Crystal Reports now. http://p.sf.net/sfu/bobj-july > > > _______________________________________________ > > > Assp-test mailing list > > > Assp-test@lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/assp-test > > > > > > > > > ------------------------------------------------------------------------------ > > Let Crystal Reports handle the reporting - Free Crystal Reports 2008 > > 30-Day > > trial. Simplify your report design, integration and deployment - and > focus > > on > > what you do best, core application coding. Discover what's new with > > Crystal Reports now. http://p.sf.net/sfu/bobj-july > > _______________________________________________ > > Assp-test mailing list > > Assp-test@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/assp-test > > > > > > > > > > DISCLAIMER: > > ******************************************************* > > This email and any files transmitted with it may be confidential, > legally > > privileged and protected in law and are intended solely for the use of > the > > > > individual to whom it is addressed. > > This email was multiple times scanned for viruses. There should be no > > known virus in this email! > > ******************************************************* > > > > > > > > ------------------------------------------------------------------------------ > > Come build with us! The BlackBerry® Developer Conference in SF, CA > > is the only developer event you need to attend this year. Jumpstart your > > developing skills, take BlackBerry mobile applications to market and > stay > > ahead of the curve. Join us from November 9-12, 2009. Register > now! > > http://p.sf.net/sfu/devconf > > _______________________________________________ > > Assp-test mailing list > > Assp-test@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/assp-test > > > > ------------------------------------------------------------------------------ > Come build with us! The BlackBerry® Developer Conference in SF, CA > is the only developer event you need to attend this year. Jumpstart your > developing skills, take BlackBerry mobile applications to market and stay > ahead of the curve. Join us from November 9-12, 2009. Register > now! > http://p.sf.net/sfu/devconf > _______________________________________________ > Assp-test mailing list > Assp-test@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/assp-test > > > > > DISCLAIMER: > ******************************************************* > This email and any files transmitted with it may be confidential, legally > privileged and protected in law and are intended solely for the use of the > > individual to whom it is addressed. > This email was multiple times scanned for viruses. There should be no > known virus in this email! > ******************************************************* > > > ------------------------------------------------------------------------------ > Come build with us! The BlackBerry® Developer Conference in SF, CA > is the only developer event you need to attend this year. Jumpstart your > developing skills, take BlackBerry mobile applications to market and stay > ahead of the curve. Join us from November 9-12, 2009. Register now! > http://p.sf.net/sfu/devconf > _______________________________________________ > Assp-test mailing list > Assp-test@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/assp-test > ------------------------------------------------------------------------------ Come build with us! The BlackBerry® Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9-12, 2009. Register now! http://p.sf.net/sfu/devconf _______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
