Re: Which user lookup wins?

[Matus UHLAR - fantomas]

@lbutlr

When postfix checks for a local user it looks at any local user (like =
/home/fred), I assume by checking /etc/passwd or similar (I have local =
users who can receive mail who are not mentioned in any /etc/postfix/* =
file, so postfix knows about them from somewhere outside of postfix=E2=80=99=
s config file) and then it also checks for virtual_mailbox_domains and =
virtual_alias_maps, yes?


On 14.03.18 20:14, Wietse Venema wrote:

The Postfix SMTP server always looks in virtual_alias_maps.


Always? isn't that a contradiction to the referenced document that indicated
only domains in virtual_alias_domains are searched for virtual aliases?

--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
WinError #9: Out of error messages.

Re: SMTP session caching

[Matus UHLAR - fantomas]

A. Schulze:

I like to ask about a documented limitation
(http://www.postfix.org/CONNECTION_CACHE_README.html#limitations)

"For this reason, the Postfix smtp(8) client always closes the
connection after completing an attempt to deliver mail over TLS."


On 07.03.18 09:07, Wietse Venema wrote:

Indeed. Postfix can migrate the TCP connection from one process to
another, but the TLS library does not support migration of live TLS
state. It supports reuse on new connections only.

Possible solutions would be:


a smtp client that able to process multiple mails in a single run is not
planned, correct?

--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Spam is for losers who can't get business any other way.

Re: Which user lookup wins?

[Wietse Venema]

Matus UHLAR - fantomas:
> >@lbutlr
> >> When postfix checks for a local user it looks at any local user (like =
> >> /home/fred), I assume by checking /etc/passwd or similar (I have local =
> >> users who can receive mail who are not mentioned in any /etc/postfix/* =
> >> file, so postfix knows about them from somewhere outside of 
> >> postfix=E2=80=99=
> >> s config file) and then it also checks for virtual_mailbox_domains and =
> >> virtual_alias_maps, yes?
> 
> On 14.03.18 20:14, Wietse Venema wrote:
> >The Postfix SMTP server always looks in virtual_alias_maps.
> 
> Always? isn't that a contradiction to the referenced document that indicated
> only domains in virtual_alias_domains are searched for virtual aliases?

Please cite the text that says 'only domains in virtual_alias_domains
are searched for virtual aliases'.

Wietse

Re: SMTP session caching

[Wietse Venema]

Matus UHLAR - fantomas:
> >A. Schulze:
> >> I like to ask about a documented limitation
> >> (http://www.postfix.org/CONNECTION_CACHE_README.html#limitations)
> >>
> >> "For this reason, the Postfix smtp(8) client always closes the
> >> connection after completing an attempt to deliver mail over TLS."
> 
> On 07.03.18 09:07, Wietse Venema wrote:
> >Indeed. Postfix can migrate the TCP connection from one process to
> >another, but the TLS library does not support migration of live TLS
> >state. It supports reuse on new connections only.
> >
> >Possible solutions would be:
> 
> a smtp client that able to process multiple mails in a single run is not
> planned, correct?

Wasn't a dedicated per-destination delivery agent one of the possible
solutions?

Wietse

Re: SMTP session caching

[Matus UHLAR - fantomas]

>A. Schulze:
>> I like to ask about a documented limitation
>> (http://www.postfix.org/CONNECTION_CACHE_README.html#limitations)
>>
>> "For this reason, the Postfix smtp(8) client always closes the
>> connection after completing an attempt to deliver mail over TLS."

On 07.03.18 09:07, Wietse Venema wrote:
>Indeed. Postfix can migrate the TCP connection from one process to
>another, but the TLS library does not support migration of live TLS
>state. It supports reuse on new connections only.
>
>Possible solutions would be:



Matus UHLAR - fantomas:

a smtp client that able to process multiple mails in a single run is not
planned, correct?


On 15.03.18 09:22, Wietse Venema wrote:

Wasn't a dedicated per-destination delivery agent one of the possible
solutions?


if you mean this one:


- For each destination, use dedicated SMTP clients that handle all
TLS sessions with that destination (no inter-process migration),
and cache TCP+TLS state in those processes. Unfortunately, that
does not scale to thousands of destinations.


... which does not scale, I was under impression that it requires site
configuration, or keeping multiple clients alive.

what I meant, is that if SMTP client connecting to destination couldn't
try to deliver multiple (all) mail directed to the destination and then
quit, the only difference would be it could deliver more than just one mail.

--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Your mouse has moved. Windows NT will now restart for changes to take
to take effect. [OK]

Re: Which user lookup wins?

[Matus UHLAR - fantomas]

>@lbutlr
>> When postfix checks for a local user it looks at any local user (like =
>> /home/fred), I assume by checking /etc/passwd or similar (I have local =
>> users who can receive mail who are not mentioned in any /etc/postfix/* =
>> file, so postfix knows about them from somewhere outside of postfix=E2=80=99=
>> s config file) and then it also checks for virtual_mailbox_domains and =
>> virtual_alias_maps, yes?

On 14.03.18 20:14, Wietse Venema wrote:
>The Postfix SMTP server always looks in virtual_alias_maps.



Matus UHLAR - fantomas:

Always? isn't that a contradiction to the referenced document that indicated
only domains in virtual_alias_domains are searched for virtual aliases?


On 15.03.18 09:20, Wietse Venema wrote:

Please cite the text that says 'only domains in virtual_alias_domains
are searched for virtual aliases'.


virtual_alias_domains and virtual_alias_maps are described in
"The virtual alias domain class." section.

* Domain names are listed in virtual_alias_domains. The default value is
$virtual_alias_maps for Postfix 1.1 compatibility.

* Valid recipient addresses are listed with the virtual_alias_maps parameter.
The Postfix SMTP server rejects invalid recipients with "User unknown in
virtual alias table". The default value is $virtual_maps for Postfix 1.1
compatibility.

That lead me to think that virtual_alias_maps does not apply to other classes. 


--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
It's now safe to throw off your computer.

Re: SMTP session caching

[Karol Augustin]

On 2018-03-15 13:58, Matus UHLAR - fantomas wrote:
>>> >A. Schulze:
>>> >> I like to ask about a documented limitation
>>> >> (http://www.postfix.org/CONNECTION_CACHE_README.html#limitations)
>>> >>
>>> >> "For this reason, the Postfix smtp(8) client always closes the
>>> >> connection after completing an attempt to deliver mail over TLS."
>>>
>>> On 07.03.18 09:07, Wietse Venema wrote:
>>> >Indeed. Postfix can migrate the TCP connection from one process to
>>> >another, but the TLS library does not support migration of live TLS
>>> >state. It supports reuse on new connections only.
>>> >
>>> >Possible solutions would be:
> 
>>Matus UHLAR - fantomas:
>>> a smtp client that able to process multiple mails in a single run is not
>>> planned, correct?
> 
> On 15.03.18 09:22, Wietse Venema wrote:
>>Wasn't a dedicated per-destination delivery agent one of the possible
>>solutions?
> 
> if you mean this one:
> 
>> - For each destination, use dedicated SMTP clients that handle all
>> TLS sessions with that destination (no inter-process migration),
>> and cache TCP+TLS state in those processes. Unfortunately, that
>> does not scale to thousands of destinations.
> 
> ... which does not scale, I was under impression that it requires site
> configuration, or keeping multiple clients alive.
> 
> what I meant, is that if SMTP client connecting to destination couldn't
> try to deliver multiple (all) mail directed to the destination and then
> quit, the only difference would be it could deliver more than just one mail.

I think what Matus is asking here is the RSET implementation in postfix
client. For example I have software that send some automated e-mail over
night using single SMTP connection (written in perl). After each e-mail
it does $smtp->reset(); and than delivers next e-mail using the same
connection. Final effect looks like that: 

disconnect from localhost[127.0.0.1]:51596 ehlo=1 mail=63 rcpt=71
data=63 rset=63 quit=1 commands=262

I believe Matus is asking if that could be implemented in postfix so it
connects to remote SMTP server and delivers one e-mail after another
issuing RSET after each one and not disconnecting.

Karol


-- 
Karol Augustin
ka...@augustin.pl
http://karolaugustin.pl/
+353 85 775 5312

Re: SMTP session caching

[Emmanuel Fusté]

Matus UHLAR - fantomas:

a smtp client that able to process multiple mails in a single run is not
planned, correct?

On 15.03.18 09:22, Wietse Venema wrote:

Wasn't a dedicated per-destination delivery agent one of the possible
solutions?

if you mean this one:


- For each destination, use dedicated SMTP clients that handle all
TLS sessions with that destination (no inter-process migration),
and cache TCP+TLS state in those processes. Unfortunately, that
does not scale to thousands of destinations.

... which does not scale, I was under impression that it requires site
configuration, or keeping multiple clients alive.

what I meant, is that if SMTP client connecting to destination couldn't
try to deliver multiple (all) mail directed to the destination and then
quit, the only difference would be it could deliver more than just one mail.

I have some problems with big local provider and their crazy (tcp) rate 
limiting rules and ugly smtp conf (only one MX, a big LB).

The problem is worse now that all connections are TLS.
(for the detail, their rate limiting is accounted at the LB level and 
they host multiple domains).
When I have big mailing running (each Monday), even with 
destination_concurrency_limit=1 for a dedicated transport, I get some 
gateways temporary blacklisted.


As I understand, by design the scheduler had no way to push "some 
destination domain emails" to "some already running delivery agents for 
that domain".
Is it a big departure from the current design, but could the protocol 
between the scheduler and the delivery agent be augmented with some 
"destination_concurrency" over-commit notification containing the domain 
for which a to be scheduled email is pending? Before closing the 
connection, the agent could check if such a mail is pending.

Yes it is overly complex.
Perhaps we should wait the landing of Linux TLS kernel app sockets which 
will perhaps simplify the connection reuse.
I hate to see Exchange be able to push to me multiple emails in a SMTP 
TLS connection and not be able to do it with Postfix. ;-)


Emmanuel.

Re: SMTP session caching

[Viktor Dukhovni]

> On Mar 15, 2018, at 10:38 AM, Karol Augustin  wrote:
> 
> I think what Matus is asking here is the RSET implementation in postfix
> client. For example I have software that send some automated e-mail over
> night using single SMTP connection (written in perl). After each e-mail
> it does $smtp->reset(); and than delivers next e-mail using the same
> connection. Final effect looks like that: 
> 
> disconnect from localhost[127.0.0.1]:51596 ehlo=1 mail=63 rcpt=71
> data=63 rset=63 quit=1 commands=262
> 
> I believe Matus is asking if that could be implemented in postfix so it
> connects to remote SMTP server and delivers one e-mail after another
> issuing RSET after each one and not disconnecting.

Yes, this much is clear, but the architectural constraints are difficult.

   * Live SSL connections (rather than cached sessions) cannot be migrated
 across processes.

   * SMTP STARTTLS does not have a STOPTLS counterpart that would allow
 the connection state to return to cleartext with TLS resumption on
 re-use.  This could be introduced, but would take many years to be
 implemented on some non-trivial fraction of servers.

   * Using TLS proxy processes for outbound TLS connections requires
 complex new code and increases the Postfix memory footprint, and
 may reduce throughput under load (when all the destination's MX
 hosts are up and accept connections quickly).

   * Caching open TLS connections in the smtp(8) delivery agent for
 reuse by scheduling repeated deliveries to the same delivery
 agent runs into complex scheduling difficulties.  The presence
 of open connections don't, and should not IMHO alter scheduler
 priority.  Messages should still leave in approximately FIFO
 order (subject to (n)qmgr's periodic preemption of messages that
 split into many envelopes).

Given the above the least bad is IMHO the proxy approach, with all
TLS processing in a pool of outbound TLS proxy processes, with the
cached connections being cleartext from the delivery agent to the
proxy.

This would be a rather non-trivial effort.  And downstream MTAs
at major providers seem to frown on connection re-use these days.
While traffic to smaller destinations is typically less
performance-critical.  So while having TLS connection caching
would be nice, it is not clear that it is a high priority given
the high cost and possibly only moderate benefit.

-- 
Viktor.

Re: SMTP session caching

[Wietse Venema]

Matus UHLAR - fantomas:
> >> >A. Schulze:
> >> >> I like to ask about a documented limitation
> >> >> (http://www.postfix.org/CONNECTION_CACHE_README.html#limitations)
> >> >>
> >> >> "For this reason, the Postfix smtp(8) client always closes the
> >> >> connection after completing an attempt to deliver mail over TLS."
> >>
> >> On 07.03.18 09:07, Wietse Venema wrote:
> >> >Indeed. Postfix can migrate the TCP connection from one process to
> >> >another, but the TLS library does not support migration of live TLS
> >> >state. It supports reuse on new connections only.
> >> >
> >> >Possible solutions would be:
> 
> >Matus UHLAR - fantomas:
> >> a smtp client that able to process multiple mails in a single run is not
> >> planned, correct?
> 
> On 15.03.18 09:22, Wietse Venema wrote:
> >Wasn't a dedicated per-destination delivery agent one of the possible
> >solutions?
> 
> if you mean this one:
> 
> > - For each destination, use dedicated SMTP clients that handle all
> > TLS sessions with that destination (no inter-process migration),
> > and cache TCP+TLS state in those processes. Unfortunately, that
> > does not scale to thousands of destinations.
> 
> ... which does not scale, I was under impression that it requires site
> configuration, or keeping multiple clients alive.

Indeed.

> what I meant, is that if SMTP client connecting to destination couldn't
> try to deliver multiple (all) mail directed to the destination and then
> quit, the only difference would be it could deliver more than just one mail.

Indeed. And there is no scalable approach for doing that.

Wietse

Re: question about envelop from.

[john]

Thanks for the help.



smtp_tls_mandatory_protocols = !SSLv2, !SSLv3, high

Where did you get the idea that "high" was a TLS protocol version?


I think this got in there by mistake, its not in my postfiix 
configuration. My guess is that I started typing before moving cursor. 
ooops!

Sorry.

John A

Re: Which user lookup wins?

[Wietse Venema]

Matus UHLAR - fantomas:
> >> >@lbutlr
> >> >> When postfix checks for a local user it looks at any local user (like =
> >> >> /home/fred), I assume by checking /etc/passwd or similar (I have local =
> >> >> users who can receive mail who are not mentioned in any /etc/postfix/* =
> >> >> file, so postfix knows about them from somewhere outside of 
> >> >> postfix=E2=80=99=
> >> >> s config file) and then it also checks for virtual_mailbox_domains and =
> >> >> virtual_alias_maps, yes?
> >>
> >> On 14.03.18 20:14, Wietse Venema wrote:
> >> >The Postfix SMTP server always looks in virtual_alias_maps.
> 
> >Matus UHLAR - fantomas:
> >> Always? isn't that a contradiction to the referenced document that 
> >> indicated
> >> only domains in virtual_alias_domains are searched for virtual aliases?
> 
> On 15.03.18 09:20, Wietse Venema wrote:
> >Please cite the text that says 'only domains in virtual_alias_domains
> >are searched for virtual aliases'.
> 
> virtual_alias_domains and virtual_alias_maps are described in
> "The virtual alias domain class." section.
> 
> * Domain names are listed in virtual_alias_domains. The default value is
> $virtual_alias_maps for Postfix 1.1 compatibility.
> 
> * Valid recipient addresses are listed with the virtual_alias_maps parameter.
> The Postfix SMTP server rejects invalid recipients with "User unknown in
> virtual alias table". The default value is $virtual_maps for Postfix 1.1
> compatibility.

That text does not exclude other virtual_alias_maps lookups.

> That lead me to think that virtual_alias_maps does not apply to other 
> classes. 
All Blacksmiths have dark skin.
All Negroes have dark skin.
All blacksmiths are negroes.

Wietse

Re: SMTP session caching

[@lbutlr]

On 2018-03-15 (09:01 MDT), Viktor Dukhovni  wrote:
> 
> Caching open TLS connections in the smtp(8) delivery agent for
> reuse by scheduling repeated deliveries to the same delivery
> agent runs into complex scheduling difficulties.  The presence
> of open connections don't, and should not IMHO alter scheduler
> priority.  Messages should still leave in approximately FIFO
> order (subject to (n)qmgr's periodic preemption of messages that
> split into many envelopes).

Isn't the issue a short-term one, where for example, a thousand mails hit the 
queue and 400 are for gmail.com and the desired behavior is to send those 400 
in one connection whilst the others get queued up via a separate process?

This could be done with a queue scheduler that pre-processes the queue when "a 
lot" of messages get queued for sending and then sorts them based on 
destination and the destinations that exceed some pre-defined threshold get 
shunted to a separate process.

That seems like it is doable, if not simple. Heck, it could even only work if 
the messages were queued in order. Something along the lines of "I'm connected 
to big.example.com, is the next message to big.example.com? It is? OK, reuse 
this connection."

This approach would do nothing about multi-destination emails, but I suspect 
that what is wanted by people in this thread doesn't involve multi-destination 
emails.

--