On 09/15/2017 07:26 AM, Kevin A. McGrail wrote:
On 9/15/2017 8:22 AM, Merijn van den Kroonenberg wrote:
So one of the problems with solving this is because getting help
requires
a complicated enrollment.
Maybe there are chunks of work which can be offloaded to the community
without requiring full sysadmin enrollment?
I can imagine lots of scripts are involved into running the rule
generation. Maybe some need reviewing or something? (just throwning
thing
in the air here)
I could be forest for the trees but I haven't seen the ability for
anyone to make progress with minimal access. Dave, tell me if you
agree, please?
Regards,
KAM
I hit a wall with my troubleshooting a couple of months ago. Normally I
am able to solve very detailed issues like this without much problem.
Here are the issues:
1. These are very old scripts written in disjointed pieces over a long
period of time by different people with different styles. There is very
little consistency and no proper modularity by function.
2. It takes a long time, 40+ minutes, to do a single run making
troubleshooting very slow and difficult. The scripts are mostly using
very verbose bash "set -x" output for logging which is hard to dig
through the massive output and correlate with problems. They need to
have proper debug logging after being rewritten in modular parts.
3. I have narrowed down the problem to the general area of a perl
Makefile which builds a custom garescorer.c file which does some
statistical analysis to determine the best score for rules in the
72_scores.cf. These 72_scores.cf are excluded from 50_scores.cf (static
scores) and are currently incomplete making these rules default to 1.0.
Most of theses missing rules should be much higher than 1.0 causing SA
to allow spam through on most installations that don't have an optimized
MTA in front of SA.
https://wiki.apache.org/spamassassin/InfraNotes2017#mkupdates
~/svn/trunk/build/mkupdates/mkupdate-with-scores
masses -> perl Makefile.PL && make (complete build of SA and test)
- perl hit-frequencies
- garescorer - compiles and runs it, requires
build/pga <--- THE PROBLEM IS NEAR HERE
I think the problem is somewhere behind line 127 and 128 which does many
things/steps:
http://svn.apache.org/viewvc/spamassassin/trunk/build/mkupdates/
4. Most of the ruleqa/masscheck scripts are sh or bash. I am not a perl
person. I am good at bash, Python, and a few others but not perl. I
can hack my way though reading and updating existing perl but that's
about it.
I have been thinking about this the past few weeks and maybe there is a
"band-aid" option where we could merge the current 72_scores.cf rules
that are missing with the shorter version that is being generated today
to keep the 72_scores.cf complete and get the rule updates happening again.
Incomplete version from last night:
http://svn.apache.org/viewvc/spamassassin/trunk/rulesrc/scores/72_scores.cf?revision=1808406&view=markup&sortby=date
Last good version:
http://svn.apache.org/viewvc/spamassassin/trunk/rulesrc/scores/72_scores.cf?revision=1786976&view=markup&sortby=date
Dave
--
David Jones
