Thank you John, that does help clarify things a bit. Also thanks to
Martin - I was typing this message when I received yours, but maybe
this will answer some of your questions.
I get the following results on the message whose tests I included
below:
- spamassassin -t : score 12.6 (BAYES_80)
- spamc / sudo spamd : score 12.6 (BAYES_80)
- spamc / sudo spamd -u spamd -g spamd : score 12.4 (no BAYES rule)
- spamc / sudo spamd -u spamd -g spamd -x : score 11.6 (BAYES_50)
The last version is the one I had been using, I think because it came
default with Systemd. I tested other examples where the difference was
9 vs 4. I think it's also possible to see, from what I've sent, that
not just the BAYES rules differ between the two runs, but other rules
as well - compare the original header to the one at the bottom.
X-Spam-Status: No, score=4.0 required=5.0 tests=BAYES_50,
HEADER_FROM_DIFFERENT_DOMAINS,HELO_DYNAMIC_IPADDR,HTML_MESSAGE,
MIME_QP_LONG_LINE,RDNS_DYNAMIC,T_REMOTE_IMAGE,T_SPF_HELO_TEMPERROR,
T_SPF_TEMPERROR autolearn=no autolearn_force=no version=3.4.1
Now the score is 11.6, as I noted above, but I recall that I was still
getting 4.0 when "spamassassin -t" was giving 12+.
By the way, I don't even know where the global Bayesian database is
kept, this is probably the most important part of my setup and it's
not to be found in /etc/mail/spamassassin/local.cf, it's not
documented in "man spamasssassin" or "man
Mail::SpamAssassin::Plugin::Bayes" or "man sa-learn". That's a bit
frustrating.
I had been running "sa-learn" as user "spamd" to get it to work with
the Systemd unit configuration which is installed by default, but
`locate bayes_seen` only returns results in my home directory...
Thanks again,
Frederick
On Sat, Dec 17, 2016 at 02:08:10PM -0800, John Hardin wrote:
> On Sat, 17 Dec 2016, frede...@ofb.net wrote:
>
> > Also, it seems that I should have set up a "caching nameserver". I've
> > attached the report from "spamassassin -t" (with a "URIBL_BLOCKED"
> > rule).
>
> The important part is that your MTA/SA not use your ISP or hosting
> provider's DNS sever, and the local MTA/SA DNS server not forward queries to
> an upstream DNS server. Caching results is not related to that.
>
> --
> John Hardin KA7OHZ http://www.impsec.org/~jhardin/
> jhar...@impsec.org FALaholic #11174 pgpk -a jhar...@impsec.org
> key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
> -----------------------------------------------------------------------
> "Bother," said Pooh as he struggled with /etc/sendmail.cf, "it never
> does quite what I want. I wish Christopher Robin was here."
> -- Peter da Silva in a.s.r
> -----------------------------------------------------------------------
> 8 days until Christmas
>
On Sat, Dec 17, 2016 at 11:08:01PM +0000, Martin Gregorie wrote:
> On Sat, 2016-12-17 at 13:03 -0800, frede...@ofb.net wrote:
> > I'm still investigating the problem, but I just noticed that
> > "spamassassin" gives the message a score of 12.0, while
> > "spamc"/"spamd" (which my mail setup is configured to use) still give
> > it a 4.0. So it seems that something more mundane is going on,
> > although I'm not sure what. I hope it's not that I've just done
> > something stupid again.
> >
> Two possibilities:
>
> 1) If running the message through spamassassin hits a lot of URIBL and
> DNSBL rules that the spamc/spamd run didn't AND this was done some
> time after spamc/spamd scanned the message then thats normal because
> the blacklists now know about the new spam source.
>
> 2) Are you sure, i.e. can you prove that, both spamassassin and
> spamc/spamd are using the same configuration?
>
> A way to check that would be to run the message through spamassassin
> and immediately run it through spamc/spamd, i.e:
> $ spamassasin <spam.txt >spam1.out; spamc <spam.txt >spam2.out
> $ diff spam1.out spam2.out
>
> If the lists of rule hits in the two outputs differ, then its likely
> that spamassassin and spamc are not using the same configuration.
> IOW your comparisons between spamassassin and spamc results are
> meaningless and will remain so until you've arranged things so that
> both are using the same configuration.
>
> In general, if the configuration is in the default location,
> /etc/mail/spamassassin on my systems, and both spamassassin and
> spamc are using the default configuration, they should both be
> using the same one, but if you're using glue like amavis-new to
> run spamassassin then this may not be true.
>
> FWIW my SA rule development and test setup runs on a different machine
> from the one running my live SA installation, BUT:
> - both SA instances keep their configurations in the default location
> for my OS (Fedora Linux)
> - both run SA as the spamd daemon and use spamc to pass messages to it
> - I use a set of bash scripts to maintain the production SA
> configuration by copying the complete configuration from my SA test
> and development environment to the production environment and
> immediately restart the production spamd instance.
>
> Thus, although the production system runs SA as part of my getmail MDA
> script while my development system has a manually triggered bash script
> that I use to feed selected test messages to spamc, as far as SA is
> concerned, the two environments and SA configurations are effectively
> identical.
>
>
> Martin
On Sat, Dec 17, 2016 at 01:03:48PM -0800, frede...@ofb.net wrote:
> Thanks again for the replies.
>
> I'm still investigating the problem, but I just noticed that
> "spamassassin" gives the message a score of 12.0, while
> "spamc"/"spamd" (which my mail setup is configured to use) still give
> it a 4.0. So it seems that something more mundane is going on,
> although I'm not sure what. I hope it's not that I've just done
> something stupid again.
>
> Also, it seems that I should have set up a "caching nameserver". I've
> attached the report from "spamassassin -t" (with a "URIBL_BLOCKED"
> rule).
>
> Thank you,
>
> Frederick
>
> On Sat, Dec 17, 2016 at 07:16:43PM +0000, David Jones wrote:
> >
> > >From: RW <rwmailli...@googlemail.com>
> > >Sent: Saturday, December 17, 2016 8:02 AM
> > >To: users@spamassassin.apache.org
> > >Subject: Re: recent increase in spam getting through
> >
> > >On Sat, 17 Dec 2016 13:35:16 +0000
> > >David Jones wrote:
> >
> >
> > >> That mail server IP above is on a very high number of RBLs:
> > >> http://multirbl.valli.org/lookup/173.230.94.183.html
> >
> >
> > >MultiRBL.valli.org - Results of the query 173.230.94.183
> > >multirbl.valli.org
> > >DNSBL and FCrDNS test results of the query '173.230.94.183'.
> >
> > >>
> > >> The edge MX server 104.197.242.163 must not be doing any
> > >> MTA checks of RBLs.
> >
> >
> > >As I already mentioned it's normal to get huge scores when retesting
> > >spam because most net rules are reactive. It doesn't imply anything
> > >about RBL results at the time it was received.
> >
> > When I looked at that RBL link above a few hours ago, it was listed on
> > 30 RBLs and now it says 42 so I agree with you that this is not a direct
> > indicator of receive time results. I use that link above after the receive
> > time just to get a quick idea how bad it is. When I see a mail server IP
> > with more than 10 to 12 hits, then it has been sending spam recently.
> >
> > My point was that a mail server doesn't get listed on 30 or 42 RBLs in
> > a few hours. It would have to have been sending a lot of spam for at
> > least a few days so this email would have been blocked by postscreen
> > on my servers for weeks. Looking at the senderscore.org report for
> > that IP, it has been sending spam for about 3 weeks and has a score
> > of 0 out of 100. Trustworthy mail servers should have a score in the
> > 90's.
> >
> > SA comes with a few major RBL rules that should have blocked this
> > message recently. With Postfix postscreen configured with major
> > RBLs weighted high and less reliable RBLs weighted lower, you can
> > get much better blocking at the MTA level using dozens of RBLs'
> > combined scoring. Each mail admin has to assess which RBLs
> > are considered reliable for their location and users.
> >
> > If the edge MX server just had a single zen.spamhaus.org RBL
> > configured and assuming it would be querying under the free
> > limit, then that email most likely would have been rejected before
> > SA and the OP would have never started this thread.
> Content analysis details: (12.6 points, 5.0 required)
>
> pts rule name description
> ---- ---------------------- --------------------------------------------------
> 1.2 URIBL_ABUSE_SURBL Contains an URL listed in the ABUSE SURBL
> blocklist
> [URIs: 6url.ru]
> 0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was
> blocked.
> See
>
> http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
> for more information.
> [URIs: 6url.ru]
> 0.5 RCVD_IN_SORBS_SPAM RBL: SORBS: sender is a spam source
> [173.230.94.183 listed in dnsbl.sorbs.net]
> 0.0 URIBL_DBL_ABUSE_REDIR Contains an abused redirector URL listed in the
> DBL blocklist
> [URIs: 6url.ru]
> 1.3 RCVD_IN_BL_SPAMCOP_NET RBL: Received via a relay in bl.spamcop.net
> [Blocked - see <http://www.spamcop.net/bl.shtml?173.230.94.183>]
> 0.4 RCVD_IN_XBL RBL: Received via a relay in Spamhaus XBL
> [173.230.94.183 listed in zen.spamhaus.org]
> 1.4 RCVD_IN_BRBL_LASTEXT RBL: No description available.
> [173.230.94.183 listed in bb.barracudacentral.org]
> 2.7 RCVD_IN_PSBL RBL: Received via a relay in PSBL
> [173.230.94.183 listed in psbl.surriel.com]
> 0.0 RCVD_IN_MSPIKE_L4 RBL: Bad reputation (-4)
> [173.230.94.183 listed in bl.mailspike.net]
> 0.0 HEADER_FROM_DIFFERENT_DOMAINS From and EnvelopeFrom 2nd level mail
> domains are different
> 0.0 HTML_MESSAGE BODY: HTML included in message
> 2.0 BAYES_80 BODY: Bayes spam probability is 80 to 95%
> [score: 0.9435]
> 0.0 MIME_QP_LONG_LINE RAW: Quoted-printable line longer than 76 chars
> 0.0 RCVD_IN_MSPIKE_BL Mailspike blacklisted
> 1.0 RDNS_DYNAMIC Delivered to internal network by host with
> dynamic-looking rDNS
> 2.0 HELO_DYNAMIC_IPADDR Relay HELO'd using suspicious hostname (IP addr
> 1)
> 0.0 T_REMOTE_IMAGE Message contains an external image
>
