Re: Caching issues when using LDAP lookups for transports
* Viktor Dukhovni: > I strongly do not recommend using LDAP for per-user transport lookups. Shame that it does not scale, because it works. I have tried using a combination of LDAP-based virtual_alias_maps and hashed transport_maps as per your suggestion, but have not yet quite achieve the result I am looking for. Let me modify the pseudocode to describe my goal in more detail: x = ldap_lookup_recipient_record(envelope_to_address) if x.has_attribute(alpha) reject_with_code_4xx(message=value_of_attribute(alpha)) else relay_message(nexthop=value_of_attribute(beta)) When I use transport lookups, this is possible, for example by setting attributes alpha="retry:Unavailable" or beta="smtp:[somehost]". I tried to emulate this by using virtual alias lookups which return alpha="pause.domain.tld" or beta="route.domain.tld", combined with a transports hash map containing pause.domain.tld retry:Unavailable route.domain.tld smtp:[somehost] Alas, when I do it this way, Postfix accepts email for users with the alpha attribute and then stores them as deferred in the queue. I need to reject with "4xx Unavailable" instead. I feel like I am close but overlooking something. -Ralph
Re: Caching issues when using LDAP lookups for transports
A Postfix process won't look up transport_maps if the same query repeats, but when I look at the code, there is a 30-second time limit on the reusing the cached response. Is that not sufficient? Wietse
Re: Caching issues when using LDAP lookups for transports
* Wietse Venema: > A Postfix process won't look up transport_maps if the same query > repeats, but when I look at the code, there is a 30-second time > limit on the reusing the cached response. Is that not sufficient? Maybe it would help if I described the scenario in more detail. Consider an LDAP server listing every valid recipient address, and for each recipient an attribute which identifies one member of a set of "next hop" servers, i.e. the user's home server. User accounts need to be moved between home servers, which can take some time, depending on the amount of data that needs to be moved. My goal is to add an LDAP attribute which indicates that an account is undergoing maintenance. This attribute is set before maintenance starts, and while it is active, incoming email for the account should be rejected with code 4xx to avoid queueing issues. In order to keep the window for temporary message rejection as small as possible, the LDAP attribute is set immediately before maintenance starts, and is removed immediately after maintenance ends. Any caching interferes when incoming traffic volume is high, even 30 seconds matter. If messages are not rejected during maintenance, they end up in the Postfix queue. However, mail queued for next hop someserver.domain.tld will no longer be accepted by that server once maintenance ends. All mail, including the messages queued during maintenance, must only be sent to otherserver.domain.tld after maintenance finishes. The actual value of the new home server can only be determined via LDAP lookups, after maintenance finishes. My first attempt was to solve this with transport lookups, but Viktor pointed out that it does not scale well. I am now trying to solve this in a manner which does not block any given Postfix process. -Ralph
Re: Caching issues when using LDAP lookups for transports
Wietse Venema: > A Postfix process won't look up transport_maps if the same query > repeats, but when I look at the code, there is a 30-second time > limit on the reusing the cached response. Is that not sufficient? Sorry, that is the resolve client cache, not the transport_map cache. But the result is that the transport map won't be queried for a repeated resolve client request, for 30s. Finally, the transport map lookup client caches the "*" result indefinitely. Wietse
Re: Caching issues when using LDAP lookups for transports
Ralph Seichter: > In order to keep the window for temporary message rejection as small as > possible, the LDAP attribute is set immediately before maintenance > starts, and is removed immediately after maintenance ends. Any caching > interferes when incoming traffic volume is high, even 30 seconds matter. Now that you know about the 30s, how would that make a difference? The safe sequence is to 1) Stop accepting email (reply with 4xx). 2) Update LDAP 3) Wait until caches and queues have drained (al least 30s). 4) Start accepting email. Wietse > If messages are not rejected during maintenance, they end up in the > Postfix queue. However, mail queued for next hop someserver.domain.tld > will no longer be accepted by that server once maintenance ends. All > mail, including the messages queued during maintenance, must only be > sent to otherserver.domain.tld after maintenance finishes. The actual > value of the new home server can only be determined via LDAP lookups, > after maintenance finishes. > > My first attempt was to solve this with transport lookups, but Viktor > pointed out that it does not scale well. I am now trying to solve this > in a manner which does not block any given Postfix process. > > -Ralph >
Re: Caching issues when using LDAP lookups for transports
Wietse Venema: > Ralph Seichter: > > In order to keep the window for temporary message rejection as small as > > possible, the LDAP attribute is set immediately before maintenance > > starts, and is removed immediately after maintenance ends. Any caching > > interferes when incoming traffic volume is high, even 30 seconds matter. > > Now that you know about the 30s, how would that make a difference? > The safe sequence is to > > 1) Stop accepting email (reply with 4xx). > > 2) Update LDAP > > 3) Wait until caches and queues have drained (al least 30s). > > 4) Start accepting email. Actually, drain caches and queues BEFORE updating LDAP, so that LDAP is not changing while Postfix is still processing email. Wietse > > If messages are not rejected during maintenance, they end up in the > > Postfix queue. However, mail queued for next hop someserver.domain.tld > > will no longer be accepted by that server once maintenance ends. All > > mail, including the messages queued during maintenance, must only be > > sent to otherserver.domain.tld after maintenance finishes. The actual > > value of the new home server can only be determined via LDAP lookups, > > after maintenance finishes. > > > > My first attempt was to solve this with transport lookups, but Viktor > > pointed out that it does not scale well. I am now trying to solve this > > in a manner which does not block any given Postfix process. > > > > -Ralph > > >
Re: Caching issues when using LDAP lookups for transports
On Thu, Feb 18, 2021 at 02:00:11PM +0100, Ralph Seichter wrote: > > I strongly do not recommend using LDAP for per-user transport lookups. > > Shame that it does not scale, because it works. I have tried using a > combination of LDAP-based virtual_alias_maps and hashed transport_maps > as per your suggestion, but have not yet quite achieve the result I am > looking for. > > Let me modify the pseudocode to describe my goal in more detail: > > x = ldap_lookup_recipient_record(envelope_to_address) > if x.has_attribute(alpha) > reject_with_code_4xx(message=value_of_attribute(alpha)) > else > relay_message(nexthop=value_of_attribute(beta)) You should not be using the transport(5) table for SMTP access control, that's what access(5) is for. LDAP used in access(5) tables works just fine. And scales better because while there's only one queue-manager, there are many smtpd(8) processes, whose LDAP queries are concurrent, (typically via multiple instances of proxymap, which scales up on demand). > When I use transport lookups, this is possible, for example by setting > attributes alpha="retry:Unavailable" or beta="smtp:[somehost]". The mistake is using transport(5) for access control. > I tried to emulate this by using virtual alias lookups which return > alpha="pause.domain.tld" or beta="route.domain.tld", combined with a > transports hash map containing > > pause.domain.tld retry:Unavailable > route.domain.tld smtp:[somehost] Sorry, that was for actually delivering email, not for simulating access control (square peg, round hole). -- Viktor.
SASL authentication failure: Internal Error
Hello everyone, I run telnet from outside my network to a server and I receive the following message # telnet server7 25 Trying 200.x.x.x... Connected to server07. Escape character is '^]'. Connection closed by foreign host. In the postfix logs i see this: Feb 18 13:03:31 server07 postfix/smtpd[11585]: connect from mon.sever.com [200.x.x.x] Feb 18 13:03:31 server07 postfix/smtpd[11585]: warning: SASL authentication failure: Internal Error -4 in server.c near line 1757 Feb 18 13:03:31 server07 postfix/smtpd[11585]: warning: SASL authentication failure: Internal Error -4 in server.c near line 1757 Feb 18 13:03:31 server07 postfix/smtpd[11585]: warning: SASL authentication failure: Internal Error -4 in server.c near line 1757 I have never seen this error message I do not know how to solve it or where to read about it Any ideas? Regards,
Re: Caching issues when using LDAP lookups for transports
On Thu, Feb 18, 2021 at 10:56:24AM -0500, Viktor Dukhovni wrote:
> > Let me modify the pseudocode to describe my goal in more detail:
> >
> > x = ldap_lookup_recipient_record(envelope_to_address)
> > if x.has_attribute(alpha)
> > reject_with_code_4xx(message=value_of_attribute(alpha))
> > else
> > relay_message(nexthop=value_of_attribute(beta))
>
> You should not be using the transport(5) table for SMTP access control,
> that's what access(5) is for. LDAP used in access(5) tables works just
> fine. And scales better because while there's only one queue-manager,
> there are many smtpd(8) processes, whose LDAP queries are concurrent,
> (typically via multiple instances of proxymap, which scales up on
> demand).
In fact you have two potential mechanisms for this:
main.cf:
# Filter out unauthorised access before recipient checks
#
smtpd_client_restrictions =
permit_mynetworks,
reject_unauth_destination
# ... RBL lookups ...
ldap = proxy:ldap:${config_directory}/
smtpd_recipient_restrictions =
check_recipient_access ${ldap}ldap-rcpt.cf
smtpd_relay_restrictions =
permit_mynetworks,
# permit_sasl_authenticated,
reject_unauth_destination
ldap-rcpt.cf:
server = ...
...
query_filter = mail=%s
result_attribute = reject_action
This assumes that the "reject_action" is a fully formed access(5) value
starting with "REJECT" or "450" or "550". You also start with a keyword
and use a regexp "pipemap" to map the keyword to an access action.
Bottom line, use the transport(5) table for routing, and access(5) for
access control.
--
Viktor.
Re: SASL authentication failure: Internal Error
> > > Feb 18 13:03:31 server07 postfix/smtpd[11585]: warning: SASL > authentication failure: Internal Error -4 in server.c near line 1757 > Do you have cyrus sasl installed? >
Re: Caching issues when using LDAP lookups for transports
Viktor Dukhovni: > Bottom line, use the transport(5) table for routing, and access(5) for > access control. These are queried at different points in time. Is this race-condition safe, i.e. can LDAP reponses change while an email message is in flight inside Postfix? Wietse
seperate ip address for outgoing email only
ello we are a corporate email service provider only transactional / business emails. Once in a while under rare occasions due to customer's email id getting compromised our server's primary ip gets blacklisted in spam databases. So the question is Can we change the outgoing mails' ip address in POSTFIX during such rare occasions only for OUTGOING emails, while having the MX ip without any change. all our services smtp, pop and imap are in the same server. thanks rajesh
Re: seperate ip address for outgoing email only
Rajesh M: > ello > > we are a corporate email service provider only transactional / business > emails. > > Once in a while under rare occasions due to customer's email id getting > compromised our server's primary ip gets blacklisted in spam databases. > > So the question is > > Can we change the outgoing mails' ip address in POSTFIX during such rare > occasions only for OUTGOING emails, while having the MX ip without any change. > > all our services smtp, pop and imap are in the same server. Set up a dedicated SMTP client in master.cf with a unique smtp_bind_address and smtp_helo_name that matches that IP address. smtp-1.2.3.4 unix - - n - - smtp -o smtp_bind_address=1.2.3.4 -o smtp_helo_name=helo-for-1.2.3.4 Then configure sender_dependent_default_transport_maps, returning that dedicated SMTP client. Example entry: sen...@example.com smtp-1.2.3.4: (or "@example.com smtp-1.2.3.4:" for all senders in a domain). Wietse
Re: Caching issues when using LDAP lookups for transports
* Wietse Venema: > Actually, drain caches and queues BEFORE updating LDAP, so that > LDAP is not changing while Postfix is still processing email. The maintenance service and Postfix only intersect in LDAP, and moving an account between servers can happen at any time. That's why I can only rely on the LDAP query results. I have gone through many tests based on Viktor's and your suggestions, and found the following combination promising: virtual_alias_maps = ldap:/etc/postfix/virtual_alias.cf smtpd_recipient_restrictions = [... reject_* here ...] check_recipient_access ldap:/etc/postfix/recipient_access.cf The lookups of course use different result attributes with matching result data: An email address for virtual alias, and DEFER_IF_PERMIT for access while an account is undergoing maintenance. -Ralph
Re: Caching issues when using LDAP lookups for transports
On Thu, Feb 18, 2021 at 11:53:56AM -0500, Wietse Venema wrote: > Viktor Dukhovni: > > Bottom line, use the transport(5) table for routing, and access(5) for > > access control. > > These are queried at different points in time. Is this race-condition > safe, i.e. can LDAP reponses change while an email message is in > flight inside Postfix? There's no specific issue here. There's never a guarantee that an address accepted initially is still valid by the time the message lands in the active queue (possibly after being deferred). If there are access(5) rules they run in real time to reject some email that is deemed invalid at that time. Perhaps later, the accepted and rewritten via virtual(5) recipients are mapped to transports, and the delivery agent may discover that the recipient is no longer valid (not in /etc/passwd, rejected by a remote SMTP or LMTP server, ...). That's normal, some may bounce. -- Viktor.
needing to set proxy_read_maps?
Hi everyone,
We're running multi-instance postfix 3.1.15 and we want to rewrite message
headers via LDAP tables using smtp generic (so that it happens after transport
selection).
Our transport table has:
domain1.invalid affiliate:[external1.invalid]
And master.cf has:
affiliate unix - - n - - smtp
-o smtp_generic_maps=${ldap}generic-ldap.cf
smtp_generic_maps is unset in main.cf:
$ postmulti -i postfix-mta-out -x postconf smtp_generic_maps
smtp_generic_maps =
We get the following warning in our logs:
2021-02-16T20:41:17.544+00:00 server.invalid postfix-mta-out/proxymap[634976]:
warning: to approve this table for read-only access, list
proxy:ldap:/etc/postfix-mta-out/generic-ldap.cf in main.cf:proxy_read_maps
To get around this, in main.cf, we set:
proxy_read_maps = $local_recipient_maps $mydestination $virtual_alias_maps
$virtual_alias_domains $virtual_mailbox_maps $virtual_mailbox_domains
$relay_recipient_maps $relay_domains $canonical_maps $sender_canonical_maps
$recipient_canonical_maps $relocated_maps $transport_maps $mynetworks
$smtpd_sender_login_maps $sender_bcc_maps $recipient_bcc_maps
$smtp_generic_maps $lmtp_generic_maps $alias_maps $smtpd_client_restrictions
$smtpd_helo_restrictions $smtpd_sender_restrictions $smtpd_relay_restrictions
$smtpd_recipient_restrictions
$address_verify_sender_dependent_default_transport_maps
$address_verify_sender_dependent_relayhost_maps $address_verify_transport_maps
$fallback_transport_maps $lmtp_discard_lhlo_keyword_address_maps
$lmtp_pix_workaround_maps $lmtp_sasl_password_maps $lmtp_tls_policy_maps
$mailbox_command_maps $mailbox_transport_maps
$postscreen_discard_ehlo_keyword_address_maps $rbl_reply_maps
$sender_dependent_default_transport_maps $sender_dependent_relayhost_maps
$smtp_discard_ehlo_keyword_address_maps $smtp_pix_workaround_maps
$smtp_sasl_password_maps $smtp_tls_policy_maps
$smtpd_discard_ehlo_keyword_address_maps $virtual_gid_maps $virtual_uid_maps,
proxy:ldap:/etc/postfix-mta-out/generic-ldap.cf
Is this only necessary because we're setting smtp_generic_maps in master.cf
instead of main.cf?
Thanks,
-Matt
Re: needing to set proxy_read_maps?
On Thu, Feb 18, 2021 at 09:02:26PM +, Matthew Selsky wrote:
> Our transport table has:
> domain1.invalid affiliate:[external1.invalid]
>
> And master.cf has:
> affiliate unix - - n - - smtp
> -o smtp_generic_maps=${ldap}generic-ldap.cf
>
> smtp_generic_maps is unset in main.cf:
> $ postmulti -i postfix-mta-out -x postconf smtp_generic_maps
> smtp_generic_maps =
>
> We get the following warning in our logs:
> 2021-02-16T20:41:17.544+00:00 server.invalid
> postfix-mta-out/proxymap[634976]: warning: to approve this table for
> read-only access, list proxy:ldap:/etc/postfix-mta-out/generic-ldap.cf
> in main.cf:proxy_read_maps
Indeed, because your "$ldap" defininition in main.cf starts with
"proxy:ldap:", each LDAP table needs to be configured in
proxy_read_maps.
THerefore, you might want to actually "name" this table:
main.cf:
affiliate_generic_maps = ${ldap}generic-ldap.cf
proxy_read_maps = ... $affiliate_generic_maps
master.cf:
affiliate unix - - n - - smtp
-o smtp_generic_maps=$affiliate_generic_maps
> Is this only necessary because we're setting smtp_generic_maps in
> master.cf instead of main.cf?
Yes, because proxy_read_maps already tries to automatically capture all
the standard parameters that specify (lists of) tables. But master.cf
is not covered.
It would be nice to have a "+=" syntax for Postfix parameters some day,
so that one could specify "default + custom", rather than stutter the
built-in defaults.
--
Viktor.
