Cross post: Lightmeter Control Center 1.7 release
(To the list admins, this is a 3rd party software announcement, which I believe to be relevant to Postfix users, but please let me know you consider it as inappropriate use of this list) For those who don't know it yet, Lightmeter Control Center is a web based open source monitoring tool for Postfix and in version 1.7 released yesterday, we added a feature called Message Detective, which is powered by the Postfix log files, adding basic email tracing which may be familiar to you from MS Exchange or elsewhere. It aims to be used by admins for debugging disrupted mail, or by e-mail users to know the delivery status of messages they've sent or received, in order to cut the need for admins to provide 1st level support, thus saving time. More information and a video demonstrating the Message Detective are in the 1.7 release notes [1]. As Lightmeter is still in early development stage, we gladly invite you you folks to try it out and give us feedback for improvements (but please do that off-list to avoid spamming others :-)) For the curious ones, it's written in Go and shipped as a single standalone binary, with no external dependencies. The setup takes a couple of minutes, and doesn't require any external database connection (it uses SQLite). It's AGPL licensed and we are also open for contributions from the community. You can run it directly in the same server Postfix is running, or receive the logs via Logstash or Rsync. You can easily test it through the `lightmeter/controlcenter` Docker Hub image, building it from source code or using the linux-amd64 binary we make available on [2]. Or, you can try easily creating a demo instance on our website [3]. [1] https://lightmeter.io/lightmeter-1-7-email-tracing-for-mailbox-users/ [2] https://gitlab.com/lightmeter/controlcenter/-/releases [3] https://lightmeter.io/demo -- Regards, Leandro Santiago Software Craftsman at Lightmeter https://lightmeter.io OpenPGP_0xAB5F702209190A96.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: Logging - Connect Order
post...@ptld.com: > Without recompiling postfix, is there a way to get the PTR hostname > warning to come after the connect message in the logs? That would be wrong. As a matter or principle, Postfix logs a warning for a bad condition (for example client name unavailable), before doing something that is affected by that bad condition (for example logging the connect message with client name 'unknown'). This ensures that the cause is logged before the effect. As another matter of principle, Postfix logs events immediately as they happen (for example recipient rejected, or recipient delivered), because Postfix processes must be prepared to fail (crashed, killed, whatever) at any point in time. Persisting information immediately (in queue files and logs) ensures that Postfix can recover sanely and that humans have an accurate record of what happened. In all these cases, delayed logging would make the system more difficult to understand. Wietse
postscreen appears to be misinterpreting zen.spamhaus.org's error return codes
Hello, I just noticed this particular behaviour as I was trying to track down some issues as apparently my mail server was bouncing legitimate emails from a few senders (including some freebsd mailing lists and also postfix-users as I discovered afterwards). This is on the FreeBSD port of Postfix 3.6.0. Here's snippet from the maillog: May 29 04:31:47 mail-server postfix/postscreen[57886]: CONNECT from [remote-mail-server-ip]:51120 to [my-mail-server-ip]:25 May 29 04:31:47 mail-server postfix/dnsblog[59708]: addr remote-mail-server-ip listed by domain zen.spamhaus.org as 127.255.255.254 May 29 04:31:53 mail-server postfix/postscreen[57886]: DNSBL rank 2 for [remote-mail-server-ip]:51120 May 29 04:31:53 mail-server postfix/tlsproxy[60045]: CONNECT from [remote-mail-server-ip]:51120 May 29 04:31:53 mail-server postfix/tlsproxy[60045]: Anonymous TLS connection established from [remote-mail-server-ip]:51120: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256 May 29 04:31:53 mail-server postfix/postscreen[57886]: NOQUEUE: reject: RCPT from [remote-mail-server-ip]:51120: 550 5.7.1 Service unavailable; client [remote-mail-server-ip] blocked using zen.spamhaus.org; [...redacted...] Based on zen.spamhaus.org's documentation 127.255.255.25[245] are actually error codes and not indicators of allow/denylisting - in this case, their error is that I was querying via a public resolver, see link here: https://www.spamhaus.org/faq/section/DNSBL%20Usage#200 The fix/workaround in my case is relatively easy as I mostly need to update the configuration for my local DNS server. That said, I'm not sure if postscreen should treat this kind of error as a denylisted server? With best regards, Timo
Re: postscreen appears to be misinterpreting zen.spamhaus.org's error return codes
Timo Geusch: > Based on zen.spamhaus.org's documentation 127.255.255.25[245] are > actually error codes and not indicators of allow/denylisting - in this > case, their error is that I was querying via a public resolver, see link > here: https://www.spamhaus.org/faq/section/DNSBL%20Usage#200 So don't do that. > The fix/workaround in my case is relatively easy as I mostly need to > update the configuration for my local DNS server. That said, I'm not > sure if postscreen should treat this kind of error as a denylisted server? postscreen has no provider-specific parsing of DNSBL status codes. Instead, postscreen assumes that all replies are true positives. To select specific responses, see the fine postscreen_dnsbl_sites documentation. Wietse
Re: postscreen appears to be misinterpreting zen.spamhaus.org's error return codes
On 5/29/21 10:22 AM, Timo Geusch wrote: > Based on zen.spamhaus.org's documentation 127.255.255.25[245] are > actually error codes and not indicators of allow/denylisting - in this > case, their error is that I was querying via a public resolver, see link > here: https://www.spamhaus.org/faq/section/DNSBL%20Usage#200 > > The fix/workaround in my case is relatively easy as I mostly need to > update the configuration for my local DNS server. That said, I'm not > sure if postscreen should treat this kind of error as a denylisted server? I ran into the same issue. The solution in my case was to remove the DNSBL checks from Postscreen, and reconfigure rspamd to use Spamhaus's free Data Query Service. As a bonus, rspamd learns more about recognizing spam if it is performing the RBL checks itself. -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958
Re: postscreen appears to be misinterpreting zen.spamhaus.org's error return codes
On 5/29/21 11:03 AM, Wietse Venema wrote: Timo Geusch: Based on zen.spamhaus.org's documentation 127.255.255.25[245] are actually error codes and not indicators of allow/denylisting - in this case, their error is that I was querying via a public resolver, see link here: https://www.spamhaus.org/faq/section/DNSBL%20Usage#200 So don't do that. Already addressed it, however I figured it would be worth mentioning on here as it seem to be a fairly recent change at SpamHaus's end. The fix/workaround in my case is relatively easy as I mostly need to update the configuration for my local DNS server. That said, I'm not sure if postscreen should treat this kind of error as a denylisted server? postscreen has no provider-specific parsing of DNSBL status codes. Instead, postscreen assumes that all replies are true positives. To select specific responses, see the fine postscreen_dnsbl_sites documentation. Makes sense, thanks for the pointer.
Re: postscreen appears to be misinterpreting zen.spamhaus.org's error return codes
On 5/29/21 11:40 AM, Phil Stracchino wrote: On 5/29/21 10:22 AM, Timo Geusch wrote: Based on zen.spamhaus.org's documentation 127.255.255.25[245] are actually error codes and not indicators of allow/denylisting - in this case, their error is that I was querying via a public resolver, see link here: https://www.spamhaus.org/faq/section/DNSBL%20Usage#200 The fix/workaround in my case is relatively easy as I mostly need to update the configuration for my local DNS server. That said, I'm not sure if postscreen should treat this kind of error as a denylisted server? I ran into the same issue. The solution in my case was to remove the DNSBL checks from Postscreen, and reconfigure rspamd to use Spamhaus's free Data Query Service. As a bonus, rspamd learns more about recognizing spam if it is performing the RBL checks itself. Thanks, that's good to know. I'm planning to switch to rspamd when I redo the server so I'll add that to my todo list.
Re: Newbie question about transport_maps failing
Wietse Venema wrote: IL Ka: If you want to choose transport based on sender, you probably want "sender_dependent_default_transport_maps" http://www.postfix.org/postconf.5.html#sender_dependent_default_transport_maps It seems that this option doesn't support wildcards. Yes it does. Use a pcre: or regexp: table. It says The tables are searched by the envelope sender address and @domain. That's used when you DON'T specify pcre, regexp, etc. Wietse Still fails, so I'm missing something. Steps I've gone through. 1) Postfix version, out of standard Ubuntu PPA. # postconf mail_version mail_version = 3.4.13 2) Ensure pcre is available... # postconf -m | grep -e pcre pcre 3) Setup sender_dependant_default_transport file # cat /etc/postfix/sender_dependant_default_transport /.*@davidfavor.com/ : /.*@fixdeliver.com/ : /.*/ discard: 4) Compile generate hash file. # postmap sender_dependant_default_transport 5) Ingest new config. postfix reload 6) Lookups seem to be working... # postmap -q da...@davidfavor.com pcre:sender_dependant_default_transport : # postmap -q f...@foo.com pcre:sender_dependant_default_transport discard: 7) Sending message still allows random sending domains... # echo test | mailx -r some...@foo.com -s "Test Message - $(date)" da...@davidfavor.com May 29 11:29:47 net17-david-favor-smtp postfix/pickup[48008]: F25241BA2030: uid=0 from= May 29 11:29:47 net17-david-favor-smtp postfix/cleanup[48019]: F25241BA2030: message-id=<20210529162947.f25241ba2...@mta1.davidfavor.com> May 29 11:29:47 net17-david-favor-smtp postfix/qmgr[48007]: F25241BA2030: from=, size=430, nrcpt=1 (queue active) May 29 11:29:47 net17-david-favor-smtp postfix/discard[48021]: F25241BA2030: to=, relay=none, delay=0.01, delays=0.01/0/0/0, dsn=2.0.0, status=sent (davidfavor.com) May 29 11:29:47 net17-david-favor-smtp postfix/qmgr[48007]: F25241BA2030: removed
Re: postscreen appears to be misinterpreting zen.spamhaus.org's error return codes
On Saturday, 29 May 2021 16:55, Timo Geusch wrote: > On 5/29/21 11:03 AM, Wietse Venema wrote: > > > Timo Geusch: > > > > > Based on zen.spamhaus.org's documentation 127.255.255.25[245] are > > > actually error codes and not indicators of allow/denylisting - in this > > > case, their error is that I was querying via a public resolver, see link > > > here: https://www.spamhaus.org/faq/section/DNSBL%20Usage#200 > > > > So don't do that. > > Already addressed it, however I figured it would be worth mentioning on here > as it seem to be a fairly recent change at SpamHaus's end. > Not sure how "fairly recent" you are implying that change is ? Certainly not in the past year based on the main.cf on my server which hasn't been edited since May 2020 (and perhaps longer than that since I think my edit back then was unrelated).
Re: Newbie question about transport_maps failing
David Favor: [ Charset ISO-8859-1 converted... ] > Wietse Venema wrote: > > IL Ka: > >>> > If you want to choose transport based on sender, you probably want > "sender_dependent_default_transport_maps" > > > >>> http://www.postfix.org/postconf.5.html#sender_dependent_default_transport_maps > >>> > >> It seems that this option doesn't support wildcards. > > > > Yes it does. Use a pcre: or regexp: table. > > > > > >> It says > >>> The tables are searched by the envelope sender address and @domain. > > > > That's used when you DON'T specify pcre, regexp, etc. > > > > Wietse > > Still fails, so I'm missing something. Forgot to put pcre: in main.cf? > 4) Compile generate hash file. > > # postmap sender_dependant_default_transport This is fundamental: pcre is not a hash file. Wietse
Re: postscreen appears to be misinterpreting zen.spamhaus.org's error return codes
On Sat, May 29, 2021 at 11:55:02AM -0400, Timo Geusch wrote: > On 5/29/21 11:03 AM, Wietse Venema wrote: > > Timo Geusch: > > > Based on zen.spamhaus.org's documentation 127.255.255.25[245] are > > > actually error codes and not indicators of allow/denylisting - in this > > > case, their error is that I was querying via a public resolver, see link > > > here: https://www.spamhaus.org/faq/section/DNSBL%20Usage#200 > > So don't do that. > Already addressed it, however I figured it would be worth mentioning on here > as it seem to be a fairly recent change at SpamHaus's end. No, it is not a recent change. SpamHaus rejects requests via done public resolvers since a long time. Maybe you changed resolver or they recently recategorized your resolver as public. So don't do that, but always use a dedicated one. Bastian -- Is truth not truth for all? -- Natira, "For the World is Hollow and I have Touched the Sky", stardate 5476.4.
Re: Newbie question about transport_maps failing
Wietse Venema wrote: David Favor: [ Charset ISO-8859-1 converted... ] Wietse Venema wrote: IL Ka: If you want to choose transport based on sender, you probably want "sender_dependent_default_transport_maps" http://www.postfix.org/postconf.5.html#sender_dependent_default_transport_maps It seems that this option doesn't support wildcards. Yes it does. Use a pcre: or regexp: table. It says The tables are searched by the envelope sender address and @domain. That's used when you DON'T specify pcre, regexp, etc. Wietse Still fails, so I'm missing something. Forgot to put pcre: in main.cf? 4) Compile generate hash file. # postmap sender_dependant_default_transport This is fundamental: pcre is not a hash file. Wietse 1) main.cf entry # grep ^sender_dependent_default_transport_maps /etc/postfix/main.cf sender_dependent_default_transport_maps = pcre:/etc/postfix/sender_dependant_default_transport 2) Oddity... of seems to be working now + working... unexpectedly... 3) Send... swaks --silent --from f...@foo.com --to da...@davidfavor.com --server 127.0.0.1 4) Log entries... May 29 11:48:16 net17-david-favor-smtp postfix/smtpd[48778]: connect from localhost[127.0.0.1] May 29 11:48:16 net17-david-favor-smtp postfix/smtpd[48778]: C2FA51BA000C: client=localhost[127.0.0.1] May 29 11:48:16 net17-david-favor-smtp postfix/cleanup[48781]: C2FA51BA000C: message-id=<20210529114816.048777@net17-david-favor-smtp> May 29 11:48:16 net17-david-favor-smtp postfix/qmgr[48755]: C2FA51BA000C: from=, size=472, nrcpt=1 (queue active) May 29 11:48:16 net17-david-favor-smtp postfix/smtpd[48778]: disconnect from localhost[127.0.0.1] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5 May 29 11:48:16 net17-david-favor-smtp postfix/discard[48782]: C2FA51BA000C: to=, relay=none, delay=0.01, delays=0.01/0/0/0, dsn=2.0.0, status=sent (davidfavor.com) May 29 11:48:16 net17-david-favor-smtp postfix/qmgr[48755]: C2FA51BA000C: removed 5) In log entry, notice... dsn=2.0.0, status=sent (davidfavor.com) Not... dsn=2.0.0, status=sent (queued) 4) Looking - postqueue -q - queue does show empty. And, looking at my MX logs... no message is received... So this does... seem to work... 5) Last question... How do effect my logging to surface this problem. I'd have expected something like... dsn=5.1.0, status=bounced/discarded (Domain foo.com not allowed to send) Or something like this... Let me know how to setup logging to catch this type of problem. Thanks!
Re: Logging - Connect Order
On 5/28/2021 6:24 PM, post...@ptld.com wrote: Without recompiling postfix, is there a way to get the PTR hostname warning to come after the connect message in the logs? Adding to the reply from Wietse, which I have to agree with: On my Ubuntu 18 mail server, everything that postfix sends to syslog includes the PID of the process. An example: May 29 10:57:17 bilbo postfix/smtpd[8707]: disconnect from unknown[186.250.196.123] ehlo=1 auth=0/1 commands=1/2 The PID here is 8707. If you find a connect message and want to see if there is something that got logged for that connection beforehand, just search backwards for the PID. At the following link you can see all the log messages from pid 8707 in the vicinity of that one I pasted above. It includes several warnings. I have now run newaliases and postmap to get rid of two of those warnings. :) This paste will expire one month from today. https://apaste.info/m3Tq The improper pipelining warning is from xymon probing smtps. That one is annoying, and I haven't yet attempted to create a fix for it and submit it to the xymon project. Have to try and remember what I know of C programming. :) Thanks, Shawn
Re: Newbie question about transport_maps failing
On Sat, May 29, 2021 at 11:30:51AM -0500, David Favor wrote: > # cat /etc/postfix/sender_dependant_default_transport > /.*@davidfavor.com/ : > /.*@fixdeliver.com/ : > /.*/ discard: > > 7) Sending message still allows random sending domains... The transport table only selects a delivery channel, you've chosen discard as the delivery channel for messages from unexpected domains. > # echo test | mailx -r some...@foo.com -s "Test Message - $(date)" > da...@davidfavor.com > > May 29 11:29:47 net17-david-favor-smtp postfix/qmgr[48007]: F25241BA2030: > from=, size=430, nrcpt=1 (queue active) > May 29 11:29:47 net17-david-favor-smtp postfix/discard[48021]: F25241BA2030: > to=, -- >relay=none, delay=0.01, delays=0.01/0/0/0, dsn=2.0.0, status=sent > (davidfavor.com) -- The message was discarded (delivered nowhere, as requested). -- Viktor.
Re: postscreen appears to be misinterpreting zen.spamhaus.org's error return codes
According to Bastian Blank : >> Already addressed it, however I figured it would be worth mentioning on here >> as it seem to be a fairly recent change at SpamHaus's end. > >No, it is not a recent change. SpamHaus rejects requests via done >public resolvers since a long time. It is a fairly recent change, perhaps a year ago, that they return the .254 and .255 codes rather than just ignoring the request, as a hint that you need to fix your configuration. -- Regards, John Levine, jo...@taugh.com, Primary Perpetrator of "The Internet for Dummies", Please consider the environment before reading this e-mail. https://jl.ly
Re: postscreen appears to be misinterpreting zen.spamhaus.org's error return codes
On 5/29/21 2:19 PM, John Levine wrote: According to Bastian Blank : Already addressed it, however I figured it would be worth mentioning on here as it seem to be a fairly recent change at SpamHaus's end. No, it is not a recent change. SpamHaus rejects requests via done public resolvers since a long time. It is a fairly recent change, perhaps a year ago, that they return the .254 and .255 codes rather than just ignoring the request, as a hint that you need to fix your configuration. Based on this discussion, I suspect that it was a combination of one or more of the resolvers I used being (re)classified as public ones, and the change in return values that ended up making the problem visible. Fortunately once I figured out what the issue was, it was a relatively easy fix in my local resolver config without having to touch the Postfix configuration.
Re: Logging - Connect Order
On 05-29-2021 1:16 pm, Shawn Heisey wrote: Adding to the reply from Wietse, which I have to agree with: I was just assuming that a connection happens first before postfix could know if the PTR resolves or not. Otherwise how does postfix know a client hostname needs to be checked if it has never connected? So following that logic, connection happens, log connection, resolve PTR, log the warning. I did not know that postfix can first know if someone has a valid PTR before they connect. (And still don't understand how) Now for my purposes in parsing logs, it would have been more convenient for my mouse trap to rely on a new event ALWAYS starting with a connect message since those always happen where as a warning is only sometime. Since it can't be convenient, i have to look for a warning or connect for a new event then treat the connect message different based if there was a previous warning or not.
Re: Newbie question about transport_maps failing
Viktor Dukhovni wrote: On Sat, May 29, 2021 at 11:30:51AM -0500, David Favor wrote: # cat /etc/postfix/sender_dependant_default_transport /.*@davidfavor.com/ : /.*@fixdeliver.com/ : /.*/ discard: 7) Sending message still allows random sending domains... The transport table only selects a delivery channel, you've chosen discard as the delivery channel for messages from unexpected domains. # echo test | mailx -r some...@foo.com -s "Test Message - $(date)" da...@davidfavor.com May 29 11:29:47 net17-david-favor-smtp postfix/qmgr[48007]: F25241BA2030: from=, size=430, nrcpt=1 (queue active) May 29 11:29:47 net17-david-favor-smtp postfix/discard[48021]: F25241BA2030: to=, -- relay=none, delay=0.01, delays=0.01/0/0/0, dsn=2.0.0, status=sent (davidfavor.com) -- The message was discarded (delivered nowhere, as requested). Mention what to use in place "discard" to block disallowed sending domains, then log a message with DNS != 2.0.0 as the reason code. Thanks.
Re: Newbie question about transport_maps failing
David Favor: [ Charset ISO-8859-1 converted... ] > Viktor Dukhovni wrote: > > On Sat, May 29, 2021 at 11:30:51AM -0500, David Favor wrote: > > > >> # cat /etc/postfix/sender_dependant_default_transport > >> /.*@davidfavor.com/ : > >> /.*@fixdeliver.com/ : > >> /.*/ discard: > >> > >> 7) Sending message still allows random sending domains... > > > > The transport table only selects a delivery channel, you've chosen > > discard as the delivery channel for messages from unexpected domains. > > > >> # echo test | mailx -r some...@foo.com -s "Test Message - $(date)" > >> da...@davidfavor.com > >> > >> May 29 11:29:47 net17-david-favor-smtp postfix/qmgr[48007]: F25241BA2030: > >> from=, size=430, nrcpt=1 (queue active) > >> May 29 11:29:47 net17-david-favor-smtp postfix/discard[48021]: > >> F25241BA2030: to=, > > -- > >>relay=none, delay=0.01, delays=0.01/0/0/0, dsn=2.0.0, status=sent > >> (davidfavor.com) > > -- > > > > The message was discarded (delivered nowhere, as requested). > > Mention what to use in place "discard" to block disallowed sending domains, > then log a message with DNS != 2.0.0 as the reason code. Use error: instead of discard:. Wietse
Re: Newbie question about transport_maps failing
Wietse Venema wrote: David Favor: [ Charset ISO-8859-1 converted... ] Viktor Dukhovni wrote: On Sat, May 29, 2021 at 11:30:51AM -0500, David Favor wrote: # cat /etc/postfix/sender_dependant_default_transport /.*@davidfavor.com/ : /.*@fixdeliver.com/ : /.*/ discard: 7) Sending message still allows random sending domains... The transport table only selects a delivery channel, you've chosen discard as the delivery channel for messages from unexpected domains. # echo test | mailx -r some...@foo.com -s "Test Message - $(date)" da...@davidfavor.com May 29 11:29:47 net17-david-favor-smtp postfix/qmgr[48007]: F25241BA2030: from=, size=430, nrcpt=1 (queue active) May 29 11:29:47 net17-david-favor-smtp postfix/discard[48021]: F25241BA2030: to=, -- relay=none, delay=0.01, delays=0.01/0/0/0, dsn=2.0.0, status=sent (davidfavor.com) -- The message was discarded (delivered nowhere, as requested). Mention what to use in place "discard" to block disallowed sending domains, then log a message with DNS != 2.0.0 as the reason code. Use error: instead of discard:. Wietse One solution that seems to work is to add the following main.cf entry... default_transport = error:541 Delivery policy failure, unauthorized sending domain specified
