logging from scripts executed by pipe
Hello, I've to rebuild a service: messages to an address are delivered via postfix pipe to a script. This script use syslog to write it's messages. That worked well for years. Now, postfix run in a different way, supervised via "postfix start-fg" (docker) Essentially the is no syslogd. My idea was to use postlog. Pipe refuse to execute an external script as root or mail system owner. (http://www.postfix.org/pipe.8.html, user= ...) Users other then postfix/postdrop may invoke $command_directory/postlog, but postlog could not connect to the postlogd as it's not running as root or postfix/postdrop. Is there a recommended/any way to log messages from a script via postfix? Andreas
Re: ALPACA writeup
Hello again, On 2021-08-09, at 09:39 (UTC-0400), Wietse Venema had the following to say: : There was a Postfix-users thread that led to this. : https://marc.info/?t=11012289581&r=1&w=2 Uh, oh. The example conversation is noteworthy for two reasons. First, upon receiving input consisting of the three tokens "POST", "/", and "HTTP/1.0", the server should have recognised that such a token sequence does not constitute a valid SMTP command line. Second, when replying with code 502, the server suggests that in fact there exists in the standard SMTP protocol definition or the local interpretation of it an action named "POST", and the server recognised it as such, but it is not (yet) implemented. This differs semantically from "I do not recognise this token sequence as a command line at all". The semantics for 502 are made very clear in RFC 5321 [1,2], where 504 (command implemented, but one of its parameters isn't) for example, is considered a refinement of 502. The server hanging on to a connection is the expected behaviour when replying with 502, because in such a case, there was no violation of SMTP proper. The server, merely unable to fulfill the request, understood it and is giving the client the choice to try and keep going with both sides still speaking SMTP. However, upon receiving the token "POST", the server should have instead replied with code 500 (Syntax error, command unrecognized) [3], where, on the server's side, the choice to keep going is much harder to justify, since the client obviously isn't even willing to speak SMTP and it is likely that this will not change. : The current implementation is based on this contribution: : https://marc.info/?t=11013155356&r=1&w=2 I remember coming across the documentation for smtpd_forbidden_commands and wondered what that was for. Now I understand that it is a consequence of the interpretation of the example conversation. Perhaps a more suitable name would have been smtpd_forbidden_command_line_tokens , lest people start thinking "POST" is actually an SMTP command. Are there plans to revisit the SMTP command parsing and handling logic in the server in one form or another? Are people making active use of the smtpd_forbidden_commands parameter? Thanks, Mono [1] https://datatracker.ietf.org/doc/html/rfc5321#section-4.2.1 [2] https://datatracker.ietf.org/doc/html/rfc5321#section-4.2.4 [3] https://datatracker.ietf.org/doc/html/rfc5321#section-4.2.2 signature.asc Description: PGP signature
Re: EHLO argument validation
Hello, On 2021-08-09, at 16:33 (UTC+0200), Matus UHLAR - fantomas had the following to say: : I don't know where you got that address is not allowed in helo/ehlo - it is. The (current) standard says that it is for EHLO and it is not for HELO [1]. Thanks, Mono [1] https://datatracker.ietf.org/doc/html/rfc5321#section-4.1.1.1 signature.asc Description: PGP signature
Re: ALPACA writeup
On 8/10/21 7:00 AM, Mono DHS wrote: Are there plans to revisit the SMTP command parsing and handling logic in the server in one form or another? Are people making active use of the smtpd_forbidden_commands parameter? Short answer: yes Longer answer: See this shell sequence: # postconf smtpd_forbidden_commands smtpd_forbidden_commands = CONNECT GET POST # postconf mail_version mail_version = 3.4.13 On my Postfix server, this is the default value for the configuration command.
Re: ALPACA writeup
Mono DHS: > First, upon receiving input consisting of the three tokens "POST", > "/", and "HTTP/1.0", the server should have recognised that such a > token sequence does not constitute a valid SMTP command line. >From the server's point of view, the POST command is unimplemented. It could have been defined in a later RFC, like the BDAT command that was defined after Postfix was already implemented. A reasonable person would agree that the parameter syntax of an unimplemented command cannot be validated. Let's not feed this troll further. Wietse
Re: logging from scripts executed by pipe
A. Schulze: > Hello, > > I've to rebuild a service: messages to an address are delivered via postfix > pipe to a script. > This script use syslog to write it's messages. That worked well for years. > > Now, postfix run in a different way, supervised via "postfix start-fg" > (docker) > Essentially the is no syslogd. > > My idea was to use postlog. > > Pipe refuse to execute an external script as root or mail system owner. > (http://www.postfix.org/pipe.8.html, user= ...) > Users other then postfix/postdrop may invoke $command_directory/postlog, > but postlog could not connect to the postlogd as it's not running as root or > postfix/postdrop. > > Is there a recommended/any way to log messages from a script via postfix? Not at this time. Making the postlog command setgid requires a security analysis and that may require some code restructuring before this can be done without opening up a security hole. Wietse
Re: ALPACA writeup
On Tue, Aug 10, 2021 at 10:50:56AM -0400, Wietse Venema wrote:
> A reasonable person would agree that the parameter syntax of an
> unimplemented command cannot be validated.
FWIW, the pedantically standards-compliant response to an unknown
SMTP command is in fact 500, rather than 502:
https://datatracker.ietf.org/doc/html/rfc5321#section-4.2.4
The 502 response is reserved for known, but not implemented or not
enabled SMTP commands. So IMHO the patch below is technically correct,
but far from important.
--- a/src/smtpd/smtpd.c
+++ b/src/smtpd/smtpd.c
@@ -5730,7 +5730,7 @@ static void smtpd_proto(SMTPD_STATE *state)
&& (err = check_milter_reply(state, err)) != 0) {
smtpd_chat_reply(state, "%s", err);
} else
- smtpd_chat_reply(state, "502 5.5.2 Error: command not
recognized");
+ smtpd_chat_reply(state, "500 5.5.2 Error: command not
recognized");
state->error_mask |= MAIL_ERROR_PROTOCOL;
state->error_count++;
continue;
--
Viktor.
Re: ALPACA writeup
Viktor Dukhovni:
> On Tue, Aug 10, 2021 at 10:50:56AM -0400, Wietse Venema wrote:
>
> > A reasonable person would agree that the parameter syntax of an
> > unimplemented command cannot be validated.
>
> FWIW, the pedantically standards-compliant response to an unknown
> SMTP command is in fact 500, rather than 502:
>
> https://datatracker.ietf.org/doc/html/rfc5321#section-4.2.4
>
> The 502 response is reserved for known, but not implemented or not
> enabled SMTP commands. So IMHO the patch below is technically correct,
> but far from important.
>
> --- a/src/smtpd/smtpd.c
> +++ b/src/smtpd/smtpd.c
> @@ -5730,7 +5730,7 @@ static void smtpd_proto(SMTPD_STATE *state)
> && (err = check_milter_reply(state, err)) != 0) {
> smtpd_chat_reply(state, "%s", err);
> } else
> - smtpd_chat_reply(state, "502 5.5.2 Error: command not
> recognized");
> + smtpd_chat_reply(state, "500 5.5.2 Error: command not
> recognized");
> state->error_mask |= MAIL_ERROR_PROTOCOL;
> state->error_count++;
> continue;
I see. We're splitting hairs the difference between 'not implemented'
versus 'not recognized'. In that light, the patch looks correct to me.
Wietse
Re: ALPACA writeup
Hello, On 2021-08-10, at 07:39 (UTC-0700), Stephen Satchell had the following to say: : On 8/10/21 7:00 AM, Mono DHS wrote: : > Are there plans to revisit the SMTP command parsing and handling logic : > in the server in one form or another? Are people making active use of : > the smtpd_forbidden_commands parameter? : > : : Longer answer: See this shell sequence: : > # postconf smtpd_forbidden_commands : > smtpd_forbidden_commands = CONNECT GET POST : > # postconf mail_version : > mail_version = 3.4.13 Same here (same version), since it's the default. Anyone added more or other tokens, like, e.g. "EXPUNGE" and "RENAME" (both IMAP), "HAVESPACE" and "SETACTIVE" (both ManageSieve) or anything else imaginable that may or may not be thrown at the server at any point in time? An open door philosophy such as the one implied by smtpd_forbidden_commands does and can /not/ provide protection from the big bad world, even and especially in practical terms. This is simply because it starts with an "anything permitted" policy and only then selectively, more or less arbitrarily, closes the door for specific tokens that have been observed in the wild, perhaps as part of a deliberate action or just misconfiguration. Administrators of such a system are relegated to eternally playing catch-up with the outside world. In contrast, robust protocol implementations are designed according to a constructivist philosophy, starting with a clean slate, "nothing permitted" policy and only ever adding what's permitted by the protocol, and that protocol alone. Any input such a server is presented with that lies outside that protocol definition is rejected outright or dealt with in other ways according to local policy. And the right to reject non-SMTP input follows from operating an SMTP MTA on the well-known TCP port 25, while at the same time holding up the fundamental principle upon which the success of the Internet is based, namely of being liberal in what one accepts and conservative in what one sends. Cheers, Mono signature.asc Description: PGP signature
Re: logging from scripts executed by pipe
Wietse Venema: A. Schulze: Is there a recommended/any way to log messages from a script via postfix? Not at this time. Making the postlog command setgid requires a security analysis and that may require some code restructuring before this can be done without opening up a security hole. Hello Wietse, somehow I expected this "not yet supported" That's OK for me, thanks for clarification. I'll setup a separate syslogd for my application until better options are available. Andreas
Re: ALPACA writeup
Wietse Venema: > I see. We're splitting hairs the difference between 'not implemented' > versus 'not recognized'. In that light, the patch looks correct to me. Implemented :-) Wietse
Postfix 3.4.14 - Dovecot SASL - strange/buggy behavior in combination with compatibility_level
Dear Postfix experts!
Although I have found a solution/workaround for the problem mentioned here, for
what it's worth, I still wanted to make the effort to report it as a possible
bug.
Summary
===
I have a test setup on my local machine where I am running Postfix and Dovecot,
each in a separate container, for a specialized purpose (not as a public e-Mail
server).
The Postfix container is based on Debian Buster with Postfix 3.4.14 installed
from the default Debian apt repository.
The Dovecot container, reachable via hostname 'dovecot', has an open TCP port
666 for Dovecot SASL, which should be used by Postfix for SMTP Authentication.
Postfix has two smtpd processes configured in master.cf, one for SMTP (port 25,
with STARTTLS) and one for SMTPS (port 465, with implicit TLS).
In main.cf I have configured postfix to use the Dovecot SASL via
smtpd_sasl_path=inet:dovecot:666.
Also, the configuration was initially migrated from an older Postfix version
and was still missing the compatibility_level setting (so default value 0).
Now, when I test the connection using 'openssl s_client' I get two different
behaviors:
* the connection with implicit TLS to port 465 works without a problem, I can
connect incl. TLS handshake, I receive the server greeting and am able to
authenticate
* the connection with explicit TLS (STARTTLS) to port 25 has an issue: I can
connect, the initial EHLO and STARTTLS commands are sent, the TLS handshake is
done, then Postfix aborts the connection.
Solution: When I add compatibility_level=2 to main.cf the connection to port 25
is not aborted after STARTTLS and the TLS handshake and everything works fine.
Initial Configuration
=
postfinger - postfix configuration on Tue Aug 10 19:28:18 UTC 2021
version: 1.30
--System Parameters--
mail_version = 3.4.14
hostname = 3e6c12d24f52
uname = Linux 3e6c12d24f52 4.19.128-microsoft-standard #1 SMP Tue Jun 23
12:58:10 UTC 2020 x86_64 GNU/Linux
--Packaging information--
looks like this postfix comes from deb package: postfix-3.4.14-0+deb10u1
--main.cf non-default parameters--
alias_database =
alias_maps =
append_dot_mydomain = no
biff = no
debug_peer_level = 5
debug_peer_list = 0.0.0.0/0
enable_long_queue_ids = yes
local_recipient_maps =
local_transport = error:5.1.1 Mailbox unavailable
maillog_file = /dev/stdout
maximal_queue_lifetime = 100d
message_size_limit = 52428800
mydestination =
protocol_destination_recipient_limit = 1
recipient_bcc_maps = pcre:/etc/postfix/bcc_maps
smtp_host_lookup = native
smtp_tls_loglevel = 2
smtpd_banner = $myhostname ESMTP test server
smtpd_recipient_restrictions = check_recipient_access
pcre:/etc/postfix/bcc_reject
smtpd_relay_restrictions = permit_sasl_authenticated reject_unauth_destination
smtpd_sasl_auth_enable = yes
smtpd_sasl_path = inet:dovecot:666
smtpd_sasl_tls_security_options = noanonymous
smtpd_sasl_type = dovecot
smtpd_tls_auth_only = yes
smtpd_tls_cert_file = /etc/pki/tls/certs/hub.pem
smtpd_tls_ciphers = high
smtpd_tls_dh1024_param_file = ${config_directory}/dhparams2048.pem
smtpd_tls_exclude_ciphers = ADH aNULL
smtpd_tls_key_file = /etc/pki/tls/private/hub.key
smtpd_tls_loglevel = 2
smtpd_tls_mandatory_ciphers = $smtpd_tls_ciphers
smtpd_tls_mandatory_exclude_ciphers = $smtpd_tls_exclude_ciphers
smtpd_tls_mandatory_protocols = $smtpd_tls_protocols
smtpd_tls_protocols = !SSLv2 !SSLv3
smtpd_tls_security_level = may
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
tls_preempt_cipherlist = yes
transport_maps = pcre:/etc/postfix/bcc_transport_maps
proxy:pgsql:/etc/postfix/pgsql-transport_maps.cf
virtual_alias_maps = pcre:/etc/postfix/bcc_alias
virtual_mailbox_domains =
proxy:pgsql:/etc/postfix/pgsql-virtual_mailbox_domains.cf
virtual_mailbox_maps = proxy:pgsql:/etc/postfix/pgsql-virtual_mailbox_maps.cf
virtual_transport = lmtp:inet:dovecot
--master.cf--
smtp inet n - - - - smtpd
# Messages received via SMTP should be forwarded to the content filter.
-o content_filter=filter:dummy
# Automatic BCC to protocol should not happen here (only after the content
filter).
-o receive_override_options=no_address_mappings
smtps inet n - n - - smtpd
-o smtpd_tls_wrappermode=yes
# Messages received via SMTP should be forwarded to the content filter.
-o content_filter=filter:dummy
# Automatic BCC to protocol should not happen here (only after the content
filter).
-o receive_override_options=no_address_mappings
pickupunix n - n 60 1 pickup
cleanup unix n - n - 0 cleanup
qmgr unix n - n 300 1 qmgr
tlsmgrunix - - n 1000? 1 tlsmgr
rewrite unix - - n - - trivial-rewrite
bounceunix - - n - 0 bounce
defer unix - - n - 0 bounce
trace
Re: Postfix 3.4.14 - Dovecot SASL - strange/buggy behavior in combination with compatibility_level
Podskubka, Stefan:
> Aug 10 19:39:14 4097ff109d16 postfix/smtpd[112]:
> xsasl_dovecot_server_connect: Connecting
> Aug 10 19:39:14 4097ff109d16 postfix/smtpd[112]: fatal: host/service
> dovecot/666 not found: Device or resource busy
This error message is from inet_connect():
if ((parse_err = host_port(buf, &host, "localhost", &port, (char *) 0)) !=
0)
msg_fatal("%s: %s", addr, parse_err);
if ((aierr = hostname_to_sockaddr(host, port, SOCK_STREAM, &res0)) != 0)
msg_fatal("host/service %s/%s not found: %s",
host, port, MAI_STRERROR(aierr));
It means that you must have specified "inet:dovecot:666", because
host_port() was successful.
The error happened because the hostname "dovecot" did not resolve.
This may be affected by how compatibility_level is set, because
compatibility_level is processed before Postfix's chroot feature
makes some files inaccessible.
The warning could be improved by doing this:
msg_fatal("host or service %s not found: %s",
addr, MAI_STRERROR(aierr));
Wietse
AW: Postfix 3.4.14 - Dovecot SASL - strange/buggy behavior in combination with compatibility_level
Wietse Venema: > The error happened because the hostname "dovecot" did not resolve. > > This may be affected by how compatibility_level is set, because > compatibility_level is processed before Postfix's chroot feature makes some > files inaccessible. Wow, I was completely thrown off the right track by the error message, thanks for clearing the confusion up so quickly. So, to summarize: With compatibility_level=0 the chroot default is yes. My smtpd process for port 25 runs chrooted because chroot=- while the smtpd process for port 465 does not run chrooted because chroot=n (I totally overlooked this small difference between my two smtpd processes in master.cf). When the smtpd process runs chrooted it doesn't have access to /etc/hosts and /etc/resolv.conf, which prevents any host name resolution, so it cannot connect to the Dovecot SASL. With compatibility_level=2 the chroot default is no. This causes both of my smtpd processes to not run chrooted, so the hostname resolution works in both. Sorry for suspecting a buggy implementation! I'm actually a software developer myself so I should know better than to quickly jump to conclusions. Thanks Wietse! Best regards Stefan smime.p7s Description: S/MIME cryptographic signature
Re: ALPACA writeup
On Tue, Aug 10, 2021 at 02:45:06PM -0400, Wietse Venema wrote: > Wietse Venema: > > I see. We're splitting hairs the difference between 'not implemented' > > versus 'not recognized'. In that light, the patch looks correct to me. > > Implemented :-) > > Wietse That'll be a relief to non-SMTP clients that connect to port 25. :-) cheers, raf
Re: AW: Postfix 3.4.14 - Dovecot SASL - strange/buggy behavior in combination with compatibility_level
Podskubka, Stefan: > Wietse Venema: > > The error happened because the hostname "dovecot" did not resolve. > > > > This may be affected by how compatibility_level is set, because > > compatibility_level is processed before Postfix's chroot feature makes some > > files inaccessible. > > Wow, I was completely thrown off the right track by the error message, thanks > for clearing the confusion up so quickly. Indeed. Instead of "dovecot/666" it should logged dovecot:666 because that is what it was actually using. Fixed in Postfix 3.7. I don't think that change is going to break working systems. Wietse > So, to summarize: > > With compatibility_level=0 the chroot default is yes. > My smtpd process for port 25 runs chrooted because chroot=- while the smtpd > process for port 465 does not run chrooted because chroot=n (I totally > overlooked this small difference between my two smtpd processes in > master.cf). > When the smtpd process runs chrooted it doesn't have access to /etc/hosts and > /etc/resolv.conf, which prevents any host name resolution, so it cannot > connect to the Dovecot SASL. > > With compatibility_level=2 the chroot default is no. > This causes both of my smtpd processes to not run chrooted, so the hostname > resolution works in both. > > Sorry for suspecting a buggy implementation! > I'm actually a software developer myself so I should know better than to > quickly jump to conclusions. > > Thanks Wietse! > > Best regards > Stefan >
Re: ALPACA writeup
On 2021-08-10 16:07:48, Mono DHS wrote: > And the right to reject non-SMTP input follows from operating an > SMTP MTA on the well-known TCP port 25, while at the same time holding > up the fundamental principle upon which the success of the Internet is > based, namely of being liberal in what one accepts and conservative in > what one sends. Luckily, you get to decide precisely how liberal (or not) you want postfix to be in what it accepts, by overriding the default smtpd_hard_error_limit setting. If you want postfix to reject a connection immediately after the first SMTP protocol error it encounters, without the need to construct in advance a list of all possible errors, just set smtpd_hard_error_limit = 0. http://www.postfix.org/postconf.5.html#smtpd_hard_error_limit (Immediately underneath smtpd_forbidden_commands) Note: I'm not recommending this. I expect that the existing default has been arrived at after much observation and careful thought. But the option to do this is there if that's what you want. cheers, raf :)
Re: ALPACA writeup
On Wed, Aug 11, 2021 at 09:48:24AM +1000, raf wrote: > If you want postfix to reject a connection immediately > after the first SMTP protocol error it encounters, > without the need to construct in advance a list of all > possible errors, just set smtpd_hard_error_limit = 0. > > http://www.postfix.org/postconf.5.html#smtpd_hard_error_limit > (Immediately underneath smtpd_forbidden_commands) Careful there, the documentation is perhaps slightly misleading, it says "disconnects when the limit is exceeded", but it is in fact when the limit is reached. The Postfix smtpd(8) server requires the limit to be at least 1. -- Viktor.
Re: ALPACA writeup
On Tue, Aug 10, 2021 at 07:54:35PM -0400, Viktor Dukhovni wrote: > On Wed, Aug 11, 2021 at 09:48:24AM +1000, raf wrote: > > > If you want postfix to reject a connection immediately > > after the first SMTP protocol error it encounters, > > without the need to construct in advance a list of all > > possible errors, just set smtpd_hard_error_limit = 0. > > > > http://www.postfix.org/postconf.5.html#smtpd_hard_error_limit > > (Immediately underneath smtpd_forbidden_commands) > > Careful there, the documentation is perhaps slightly misleading, it says > "disconnects when the limit is exceeded", but it is in fact when the > limit is reached. The Postfix smtpd(8) server requires the limit to be > at least 1. > > -- > Viktor. Thanks. I'm gathering a list of documentation errors (There's so much (great) documentation!). I'll include that. cheers, raf
Overriding transport_maps with sender_dependent
I have a need to map some destination domains to a specific smtp nexthop, but need to override that nexthop on a sender_dependent basis. I've tried a few things and all with no luck: * transport_maps specifying the nexthop can't be overridden at all, it seems (and doesn't support sender_dependent matching), * putting the specified destination domains in relay_domains (and stopping relay to them via smtpd_relay_restrictions) but there doesn't seem to be a way to override relay_transport on a sender_dependent way * relayhost-based stuff doesn't seem to be an option as I don't want to change the delivery of mail that isn't either from one of the sender_dependent rules or to one of the specified domains. Am I missing something or is there some way to do a custom rewrite engine? Thanks, Matt
Re: Overriding transport_maps with sender_dependent
Oh, and if its possible, is it also possible to specify the original domains as "any domain with an MX of $REGEX" instead of only "any recipient domain of $REGEX"? Thanks, Matt On 8/10/21 22:34, Matt Corallo wrote: I have a need to map some destination domains to a specific smtp nexthop, but need to override that nexthop on a sender_dependent basis. I've tried a few things and all with no luck: * transport_maps specifying the nexthop can't be overridden at all, it seems (and doesn't support sender_dependent matching), * putting the specified destination domains in relay_domains (and stopping relay to them via smtpd_relay_restrictions) but there doesn't seem to be a way to override relay_transport on a sender_dependent way * relayhost-based stuff doesn't seem to be an option as I don't want to change the delivery of mail that isn't either from one of the sender_dependent rules or to one of the specified domains. Am I missing something or is there some way to do a custom rewrite engine? Thanks, Matt
Re: Overriding transport_maps with sender_dependent
On Tue, Aug 10, 2021 at 10:34:52PM -0400, Matt Corallo wrote: > I have a need to map some destination domains to a specific smtp > nexthop, but need to override that nexthop on a sender_dependent > basis. I've tried a few things and all with no luck: Sorry, that's not directly possible. Sender-dependent transport selection overrides only the *default* transport. Anything else would require looking at both the sender and recipient at the same time, and Postfix does not presently have support for multi-key tables. > Am I missing something or is there some way to do a custom rewrite > engine? It may be possible to rewrite the recipients in question to replace the domain part with the desired nexthop, so that the default transport would take care of delivery to the correct nexthop, and this case, sender-dependent overrides can kick in. You might then need to reverse the rewrites via smtp_generic_maps. Another option is a multi-instance pipeline, in which the recipient determines the downstream Postfix instance, and then in some special downstream instances the nexthop in question is the default transport, and again sender-based overrides can take place. On Tue, Aug 10, 2021 at 10:43:46PM -0400, Matt Corallo wrote: > Oh, and if its possible, is it also possible to specify the original > domains as "any domain with an MX of $REGEX" instead of only "any > recipient domain of $REGEX"? No, the queue manager has no access to MX host lookups which happen during delivery, not transport selection. -- Viktor.
