Re: filtering locally submitted emails / tidying up the config
On 01 mai 2020, at 18:28, Wietse Venema wrote: >> >> would there be a performance/functionality penalty switching from >> smtpd_proxy_filter to smtpd_milters/non_smtpd_milters? > > It will be faster, because it avoids the need to deliver the whole > message outside of Postfix, and then to receive the whole message > again into Postfix. > > Also, the logging will be easier to follow because there is only > one queue file instead of two (one before amavisd and one after > amavisd). Thanks a lot. In mean time I've changed my all setup to use only milters (more readable config). Unfortunately I've discovered that Amavisd is unable to add headers I want as it would do as an smtp proxy filter. It will properly add: X-Virus-Scanned: my custom tag X-Crm114-Status: UNSURE ( 6.79 ) but won't add: X-Spam-Score: -3.714 X-Spam-Flag: NO X-Spam-Level: X-Spam-Status: No, score=-3.714 tagged_above=-6 required=3 tests=[ALL_TRUSTED=-1, ... It bugs me, so I've tuned amavisd-milter to use it's "server" -D delivery-care-of feature: Set AM.PDP request attribute delivery_care_of to client (default) or server. When the client method is used, then amavisd-milter is responsible for forwarding the message to the recipients. This method does not allow personalized header or body modification. When the server method is used, then amavis is responsible for forwarding the message to the recipients and may personalize the headers and the body of the messages. $forward_method variable in amavisd.conf must point to a place willing to accept the message without further checking in amazes. It negates the benefit you were writing about as amavisd-milter will drop the message on the milter interface (postfix/cleanup[26401]: 87E5316135: milter-discard: END-OF-MESSAGE from localhost[127.0.0.1]: milter triggers DISCARD action) and re-inject it in an after-queue SMTP with no filtering. No big deal for me. I'm still wondering how it can add the X-Crm114-Status header (amavisd uses spamassassin and spamassassin uses crm114, quite far fetched) but seems unable to add it's own personalized headers. patpro
Re: filtering locally submitted emails / tidying up the config
Patrick Proniewski: > It negates the benefit you were writing about as amavisd-milter > will drop the message on the milter interface (postfix/cleanup[26401]: > 87E5316135: milter-discard: END-OF-MESSAGE from localhost[127.0.0.1]: > milter triggers DISCARD action) and re-inject it in an after-queue > SMTP with no filtering. No big deal for me. Well that is broken. It should NOT return a DISCARD to Postfix then re-inject the entire message over SMTP. Instead it should just return an OK for Postfix to deliver the message (after adding the headers). Once you turn off the re-inject-over-SMTP it might actually try to add headers to the message. Wietse
Re: filtering locally submitted emails / tidying up the config
On Sat, May 02, 2020 at 11:40:52AM +0200, Patrick Proniewski wrote: > It negates the benefit you were writing about as amavisd-milter will drop the > message on the milter interface (postfix/cleanup[26401]: 87E5316135: > milter-discard: END-OF-MESSAGE from localhost[127.0.0.1]: milter triggers > DISCARD action) and re-inject it in an after-queue SMTP with no filtering. No > big deal for me. This sound like you are sending the mail into quarantine. Please check if there is an X-Envelope-From and X-Envelope-To header. If this is true, your config is not correct for your use case. But this is off-topic here. Bastian -- Violence in reality is quite different from theory. -- Spock, "The Cloud Minders", stardate 5818.4
Re: filtering locally submitted emails / tidying up the config
On 02 mai 2020, at 14:19, Wietse Venema wrote:
>
> Patrick Proniewski:
>> It negates the benefit you were writing about as amavisd-milter
>> will drop the message on the milter interface (postfix/cleanup[26401]:
>> 87E5316135: milter-discard: END-OF-MESSAGE from localhost[127.0.0.1]:
>> milter triggers DISCARD action) and re-inject it in an after-queue
>> SMTP with no filtering. No big deal for me.
>
> Well that is broken. It should NOT return a DISCARD to Postfix then
> re-inject the entire message over SMTP.
>
> Instead it should just return an OK for Postfix to deliver the
> message (after adding the headers).
>
> Once you turn off the re-inject-over-SMTP it might actually try
> to add headers to the message.
well in fact I've tried the default settings first ("client" as per
),
the milter works as expected but won't add X-Spam-* headers. I get the headers
I want if I let amavisd deliver the mail, instead of amavisd-milter.
You suggest I disable the $forward method in amavisd.conf and keep "-D server"
for amavisd-milter? I can try that.
patpro
Re: Replace null sender addresses?
RFC 7208, Sender Policy Framework (SPF) section 2.4 has a nice summary in paragraph 2: [RFC5321] allows the reverse-path to be null (see Section 4.5.5 in [RFC5321]). In this case, there is no explicit sender mailbox, and such a message can be assumed to be a notification message from the mail system itself. When the reverse-path is null, this document defines the "MAIL FROM" identity to be the mailbox composed of the local-part "postmaster" and the "HELO" identity (which might or might not have been checked separately before). I agree with the general consensus that such messages should be delivered locally or within the "family". If you're getting e.g. undeliverable messages from the wild internet and sending out postmaster notifications in response, rethink the externalities (or find someone who can) or expect a lot of your mail to be undeliverable in practice. -- Fred Morris
Re: filtering locally submitted emails / tidying up the config
On 02 mai 2020, at 15:13, Bastian Blank
wrote:
>
> On Sat, May 02, 2020 at 11:40:52AM +0200, Patrick Proniewski wrote:
>> It negates the benefit you were writing about as amavisd-milter will drop
>> the message on the milter interface (postfix/cleanup[26401]: 87E5316135:
>> milter-discard: END-OF-MESSAGE from localhost[127.0.0.1]: milter triggers
>> DISCARD action) and re-inject it in an after-queue SMTP with no filtering.
>> No big deal for me.
>
> This sound like you are sending the mail into quarantine. Please check
> if there is an X-Envelope-From and X-Envelope-To header.
>
> If this is true, your config is not correct for your use case. But this
> is off-topic here.
Nothing like this. For example, a locally generated email (mail command in the
terminal):
May 2 17:48:14 rack postfix/pickup[59232]: 43E5A164F8: uid=1001 from=
May 2 17:48:14 rack postfix/cleanup[72090]: 43E5A164F8:
message-id=<20200502154814.43e5a16...@my.host.name>
May 2 17:48:14 rack opendkim[61106]: 43E5A164F8: DKIM-Signature field added
(s=202004-abd9c69a, d=host.name)
May 2 17:48:15 rack postfix/smtpd[72472]: connect from localhost[127.0.0.1]
May 2 17:48:15 rack postfix/smtpd[72472]: 165C416500:
client=localhost[127.0.0.1], orig_queue_id=43E5A164F8,
orig_client=localhost[127.0.0.1]
May 2 17:48:15 rack postfix/cleanup[72097]: 165C416500:
message-id=<20200502154814.43e5a16...@my.host.name>
May 2 17:48:15 rack postfix/qmgr[4181]: 165C416500: from=,
size=2054, nrcpt=1 (queue active)
May 2 17:48:15 rack amavis[68109]: (68109-09) Passed CLEAN {RelayedInbound},
[127.0.0.1] -> , Queue-ID: 43E5A164F8,
Message-ID: <20200502154814.43e5a16...@my.host.name>, mail_id: La_iE098bQhw,
Hits: -3.366, size: 1283, queued_as: 165C416500,
dkim_sd=202004-abd9c69a:host.name, 703 ms
May 2 17:48:15 rack postfix/cleanup[72090]: 43E5A164F8: milter-discard:
END-OF-MESSAGE from localhost[127.0.0.1]: milter triggers DISCARD action;
from= to=
May 2 17:48:15 rack postfix/qmgr[4181]: 165C416500: removed
headers of the corresponding message:
Return-Path:
Delivered-To: pat...@host.name
Received: from my.host.name
by my.host.name with LMTP
id yCoHBj+WrV7k4QAAsbjT4g
(envelope-from )
for ; Sat, 02 May 2020 17:48:15 +0200
Received: from my.host.name (localhost [127.0.0.1])
by my.host.name (Postfix) with ESMTP id 165C416500
for ; Sat, 2 May 2020 17:48:15 +0200 (CEST)
X-Virus-Scanned: amavisd-new at host.name
X-Spam-Flag: NO
X-Spam-Score: -3.366
X-Spam-Level:
X-Spam-Status: No, score=-3.366 tagged_above=-6 required=3
tests=[ALL_TRUSTED=-1, BAYES_00=-1.9, CRM114_CHECK=-0.4668,
URIBL_BLOCKED=0.001] autolearn=disabled
X-CRM114-Status: UNSURE ( 3.89 )
Authentication-Results: my.host.name (amavisd-new);
dkim=pass (4096-bit key) header.d=host.name
Received: from unknown by my.host.name (amavisd-new, unix socket)
id La_iE098bQhw for ;
Sat, 2 May 2020 17:48:14 +0200 (CEST)
Received: from localhost (localhost [127.0.0.1])
by my.host.name (amavisd-milter) id 43E5A164F8;
Sat, 2 May 2020 17:48:14 +0200 (CEST)
(envelope-from pat...@host.name)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=host.name;
s=202004-abd9c69a; t=1588434494;
bh=g3zLYH4xKxcPrHOD18z9YfpQcnk/GaJedfustWU5uGs=;
h=To:Subject:Date:From;
b=O1NT/2XAR1YJPctZH5Ojekh4tTLcOFW1SdCg8YKIzqO1r5ObwngBeBXbGS5q7a1uc
../..
a0AcDzE3JjxalhpI4Q6Kk6qU=
To: pat...@host.name
Subject: test
Message-Id: <20200502154814.43e5a16...@my.host.name>
Date: Sat, 2 May 2020 17:48:14 +0200 (CEST)
From: Patrick Proniewski
Re: filtering locally submitted emails / tidying up the config
Patrick Proniewski wrote: > In mean time I've changed my all setup to use only > milters (more readable config). Seems reasonable to me. > Unfortunately I've discovered that Amavisd is unable to add headers > I want as it would do as an smtp proxy filter. > It will properly add: > > X-Virus-Scanned: my custom tag > X-Crm114-Status: UNSURE ( 6.79 ) Those two milters appear configured okay. Okay for CRM114 (which I like and use as well). Okay for the virus scanner. You are seeing the headers. That appears to be working okay. > but won't add: > > X-Spam-Score: -3.714 > X-Spam-Flag: NO > X-Spam-Level: > X-Spam-Status: No, score=-3.714 tagged_above=-6 required=3 > tests=[ALL_TRUSTED=-1, ... That milter configuration for SpamAssassin appears to have a problem then. If the headers are not being added. Because conceptually it is no different from the other two milters. But if that milter is not able to add those headers then something is not right about that configuration. If it were me I would keep working that part of the problem. > I'm still wondering how it can add the X-Crm114-Status header > (amavisd uses spamassassin and spamassassin uses crm114, quite far > fetched) but seems unable to add it's own personalized headers. If this were the spamassassin users list I would challenge that assertion because as far as I know SpamAssassin does not use CRM114. SA has its own Bayes engine. Instead CRM114 would be running from it's own configuration separately. I run CRM114 before SpamAssassin and then add in custom SA rules to score a final SA score using CRM114 as one of the weighted inputs, de-rating SA's internal Bayes to avoid double scoring. But since this isn't the SpamAssassin users list I will simply suggest gently that something is wrong with the milter configuration for it interfacing to Postfix. Bob
Re: filtering locally submitted emails / tidying up the config
> On 02 mai 2020, at 18:09, Bob Proulx wrote:
>
>> Unfortunately I've discovered that Amavisd is unable to add headers
>> I want as it would do as an smtp proxy filter.
>> It will properly add:
>>
>> X-Virus-Scanned: my custom tag
>> X-Crm114-Status: UNSURE ( 6.79 )
>
> Those two milters appear configured okay. Okay for CRM114 (which I
> like and use as well). Okay for the virus scanner. You are seeing
> the headers. That appears to be working okay.
>
>> but won't add:
>>
>> X-Spam-Score: -3.714
>> X-Spam-Flag: NO
>> X-Spam-Level:
>> X-Spam-Status: No, score=-3.714 tagged_above=-6 required=3
>> tests=[ALL_TRUSTED=-1, ...
>
> That milter configuration for SpamAssassin appears to have a problem
> then. If the headers are not being added. Because conceptually it is
> no different from the other two milters. But if that milter is not
> able to add those headers then something is not right about that
> configuration. If it were me I would keep working that part of the
> problem.
>
>> I'm still wondering how it can add the X-Crm114-Status header
>> (amavisd uses spamassassin and spamassassin uses crm114, quite far
>> fetched) but seems unable to add it's own personalized headers.
>
> If this were the spamassassin users list I would challenge that
> assertion because as far as I know SpamAssassin does not use CRM114.
> SA has its own Bayes engine. Instead CRM114 would be running from
> it's own configuration separately. I run CRM114 before SpamAssassin
> and then add in custom SA rules to score a final SA score using CRM114
> as one of the weighted inputs, de-rating SA's internal Bayes to avoid
> double scoring. But since this isn't the SpamAssassin users list I
> will simply suggest gently that something is wrong with the milter
> configuration for it interfacing to Postfix.
For clarity I'll share my new master.cf content (partial) here:
---
IP.ADD.RE.SS:smtp inet n - n - 1 postscreen
dnsblog unix - - n - 0 dnsblog
tlsproxy unix - - n - 0 tlsproxy
smtpd pass - - n - - smtpd
-o smtpd_client_connection_count_limit=10
-o disable_mime_output_conversion=yes
-o
smtpd_milters=unix:/var/milter-greylist/milter-greylist.sock,inet:127.0.0.1:8891,inet:localhost:8893,unix:/var/run/amavis/amavisd-milter.sock
-o non_smtpd_milters=
policyd-spf unix - n n - 0 spawn
user=nobody argv=/usr/local/bin/policyd-spf
127.0.0.1:smtp inet n - n - 5 smtpd
-o smtpd_client_connection_count_limit=10
-o disable_mime_output_conversion=yes
-o
smtpd_milters=inet:127.0.0.1:8891,unix:/var/run/amavis/amavisd-milter.sock
-o
non_smtpd_milters=inet:127.0.0.1:8891,unix:/var/run/amavis/amavisd-milter.sock
#
# After-filter SMTP server. Receive mail from the content filter
# on localhost port 10025.
#
127.0.0.1:10025 inet n - n -- smtpd
-o smtpd_authorized_xforward_hosts=127.0.0.0/8
-o smtpd_client_restrictions=
-o smtpd_helo_restrictions=
-o smtpd_sender_restrictions=
-o smtpd_recipient_restrictions=permit_mynetworks,reject
-o smtpd_data_restrictions=
-o mynetworks=127.0.0.0/8
-o receive_override_options=no_unknown_recipient_checks
-o smtpd_milters=
-o non_smtpd_milters=
---
In addition to that, my main.cf reads:
---
# grep milter main.cf
milter_default_action = accept
milter_connect_macros = j {client_name} {daemon_name} v
milter_protocol = 6
smtpd_milters = inet:127.0.0.1:8891
non_smtpd_milters = inet:127.0.0.1:8891,
unix:/var/run/amavis/amavisd-milter.sock
---
So I'm using those milters:
unix:/var/milter-greylist/milter-greylist.sock => greylisting
inet:127.0.0.1:8891=> OpenDKIM
inet:localhost:8893=> OpenDMARC
unix:/var/run/amavis/amavisd-milter.sock => Amavisd-milter
Amavisd-new runs as a daemon and handle spamassassin filtering. I've added a
crm114 perl module to spamassassin so that I don't have anything to run on the
side:
# ls -1 /usr/local/etc/mail/spamassassin/crm114*
/usr/local/etc/mail/spamassassin/crm114.cf
/usr/local/etc/mail/spamassassin/crm114.pm
X-Virus-Scanned and X-Crm114-Status are added by Amavisd-new, as every other
X-Spam-* headers.
patpro
Re: filtering locally submitted emails / tidying up the config
Patrick Proniewski:
It negates the benefit you were writing about as amavisd-milter
will drop the message on the milter interface (postfix/cleanup[26401]:
87E5316135: milter-discard: END-OF-MESSAGE from localhost[127.0.0.1]:
milter triggers DISCARD action) and re-inject it in an after-queue
SMTP with no filtering. No big deal for me.
On 02 mai 2020, at 14:19, Wietse Venema wrote:
Well that is broken. It should NOT return a DISCARD to Postfix then
re-inject the entire message over SMTP.
Instead it should just return an OK for Postfix to deliver the
message (after adding the headers).
Once you turn off the re-inject-over-SMTP it might actually try
to add headers to the message.
On 02.05.20 17:46, Patrick Proniewski wrote:
well in fact I've tried the default settings first ("client" as per
),
the milter works as expected but won't add X-Spam-* headers.
strange, were I run amavisd-milter, -D client is used and users get the
header.
Do you have $mydomains properly set up in amavis?
I get SA headers added only to incoming mail using milter and "-D client".
I get the
headers I want if I let amavisd deliver the mail, instead of
amavisd-milter.
You suggest I disable the $forward method in amavisd.conf and keep "-D server"
for amavisd-milter? I can try that.
No. The " -D server" is what causes the milter to issue the DISCARD to postfix,
and expect amavis to use $forward_method to send the mail.
I guess using "-D server" and disabling $forward_method would discard your
mail.
--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
BSE = Mad Cow Desease ... BSA = Mad Software Producents Desease
Re: filtering locally submitted emails / tidying up the config
On 02 mai 2020, at 19:25, Matus UHLAR - fantomas wrote:
>
> On 02.05.20 17:46, Patrick Proniewski wrote:
>> well in fact I've tried the default settings first ("client" as per
>> ),
>> the milter works as expected but won't add X-Spam-* headers.
>
> strange, were I run amavisd-milter, -D client is used and users get the
> header.
>
> Do you have $mydomains properly set up in amavis?
>
> I get SA headers added only to incoming mail using milter and "-D client".
OK. I'll have to do more testing and review my amavisd.conf but I'm pretty sure
my setting is good (running for years):
(amavisd.conf)
$mydomain = 'domain.tld'; # (no useful default)
@local_domains_maps = ( [ ".$mydomain", '.other.tld', '.additional.tld', etc. ]
);
thanks,
patpro
Re: filtering locally submitted emails / tidying up the config
Ok, my tests were not well done.
I've re-tested and I see that everything is indeed correct.
pure milter for amavisd-milter (no forwarding to after-queue smtp), and Amavisd
tags are present in headers. My test messages were "too clean" and would not
trigger insertion of X-Spam-* headers.
My bad.
thank you all.
patpro
> On 02 mai 2020, at 19:36, Patrick Proniewski wrote:
>
> On 02 mai 2020, at 19:25, Matus UHLAR - fantomas wrote:
>>
>> On 02.05.20 17:46, Patrick Proniewski wrote:
>>> well in fact I've tried the default settings first ("client" as per
>>> ),
>>> the milter works as expected but won't add X-Spam-* headers.
>>
>> strange, were I run amavisd-milter, -D client is used and users get the
>> header.
>>
>> Do you have $mydomains properly set up in amavis?
>>
>> I get SA headers added only to incoming mail using milter and "-D client".
>
>
> OK. I'll have to do more testing and review my amavisd.conf but I'm pretty
> sure my setting is good (running for years):
>
> (amavisd.conf)
> $mydomain = 'domain.tld'; # (no useful default)
> @local_domains_maps = ( [ ".$mydomain", '.other.tld', '.additional.tld', etc.
> ] );
>
> thanks,
> patpro
>
Re: filtering locally submitted emails / tidying up the config
Bob Proulx: > > but won't add: > > > > X-Spam-Score: -3.714 > > X-Spam-Flag: NO > > X-Spam-Level: > > X-Spam-Status: No, score=-3.714 tagged_above=-6 required=3 > > tests=[ALL_TRUSTED=-1, ... > > That milter configuration for SpamAssassin appears to have a problem > then. If the headers are not being added. Because conceptually it is This happens when he is using the smtpd process BEOFRE smtpd_proxy_filter. In order to add headers with a Milter he MUST turn off smtpd_proxy_filter of figure out how to set smtpd_milters only for smtpd process that hands after smtpd_proxy_filter. Wietse
