On Thu, Jun 25, 2015 at 09:37:44AM +0300, Henrik K wrote: > On Wed, Jun 24, 2015 at 07:37:28PM -0400, Charles Sprickman wrote: > > On Jun 22, 2015, at 5:21 PM, Marc Selig <a29508-spamassas...@sedacon.com> > > wrote: > > > > > On Mon, Jun 22, 2015 at 05:09:45PM -0400, Charles Sprickman wrote: > > > > > >> Are there any other options for filtering based on language, or any known > > >> patches/fixes for TextCat to make it a bit less aggressive when it runs > > >> across gibberish that is probably not any particular language? > > > > > > You could tinker with textcat_acceptable_score. Increasing it slightly > > > (e.g. back to the old default of 1.05) seems to reduce those wild guesses. > > > > I don?t quite follow what exactly this does, the explanation seems a bit > > circular: > > > > textcat_acceptable_score N (default: 1.05) > > "Include any language that scores at least textcat_acceptable_score in the > > returned list of languages" > > > > I?m bumping it up to see what happens, I?m also lowering > > "textcat_max_languages? to 3. How can I get more info about what this > > plugin is doing into the headers? > > The scoring is a bit vague yes.. basically 1.02 means that compared to the > "best result" (a vague ngram number) we only accept other results withing 2% > of that. If score produces more results than textcat_max_languages then > everything is ignored. > > I'm going to add some headers tags to trunk code soon, it will look something > like this: > > Jun 25 09:33:12.670 [30140] dbg: check: tagrun - tag TEXTCAT_RESULTS is now > ready, value: fi:96985(1.00) ro:112950(1.16) sv:113567(1.17) it:115650(1.19) > da:115656(1.19) fr:116506(1.20) af:117089(1.21) sr.us-ascii:117205(1.21) > sk.us-ascii:118124(1.22) en:118174(1.22) ms:118208(1.22) > hr.us-ascii:118639(1.22) id:119112(1.23) ca:119196(1.23) pt:119960(1.24) > hu:119986(1.24) sq:120081(1.24) nl:120105(1.24) es:120199(1.24) > no:120804(1.25) > > Here you see the "ngram result" and percentile (score), "fi" is a clear > winner. For sane results 1.02-1.05 score is good range. You can reduce > max_languages to 1-2 if you want even more confidence.
Committed, if anyone wants to debug things, just replace current version with this. Also added some hopefully clarifying things in the documentation. http://svn.apache.org/repos/asf/spamassassin/trunk/lib/Mail/SpamAssassin/Plugin/TextCat.pm You can add_header all Textcat _TEXTCATRESULTS_ or grep it from debug output.
