Re: Connection Caching Per-Destination

2020-08-04 Thread Wietse Venema
Wietse Venema: > There is a rough idea of how to enforce strict connection counts > when connection caching is turned on. But it would not help in your > case, where the number of competing domains is 100x the number of > allowed concurrent connections. Under those conditions the feature > would be

Re: Connection Caching Per-Destination

2020-08-02 Thread @lbutlr
On 02 Aug 2020, at 15:51, Wietse Venema wrote: > This would avoid the need for hundreds of transport map entries, > and would avoid the need keep adding/removing entries as cusomers host > their email at outlook, or decide to take their business elsewhere. Automatically adding the outlook domains

Re: Connection Caching Per-Destination

2020-08-02 Thread Greg Sims
> I suspect the real problem was that hundreds of domains were not > directed to the low-concurrency 'outlook' transport, and that > connection count 'overshoot' due to unused cached connections was > a red herring. Please recall that I collected 383 email domains into transport.outlook.regexp. I

Re: Connection Caching Per-Destination

2020-08-02 Thread Wietse Venema
Greg Sims: > > I suspect the real problem was that hundreds of domains were not > > directed to the low-concurrency 'outlook' transport, and that > > connection count 'overshoot' due to unused cached connections was > > a red herring. > > Please recall that I collected 383 email domains into > tra

Re: Connection Caching Per-Destination

2020-08-02 Thread Wietse Venema
Greg Sims: > I know this is likely simplistic thinking -- but how about this in master.cf: > > outlook unix - - n - - smtp > -o syslog_name=outlook > -o smtp_connection_cache_on_demand=yes > -o smtp_max_connections=8 There is no smtp_max_connections feature.

Re: Connection Caching Per-Destination

2020-08-01 Thread Greg Sims
> > I changed master.cf to 3 processes for outlook: in hopes of reducing > > MaxConnections feedback -- I can not go much smaller. > > This has been asked before: when Outlook puts you in the penalty > box and starts ratelimiting your new connections, was that because > a) you exceeded a limit for

Re: Connection Caching Per-Destination

2020-08-01 Thread Wietse Venema
Greg Sims: > > > I looked for domains that *are not* using the outlook: transport but > > > are using the outlook.com relay servers. There are 383 such domains > > > -- the vast majority are one email address per domain. These domains > > > are competing for the limited number of outlook.com conn

Re: Connection Caching Per-Destination

2020-08-01 Thread Greg Sims
> > I looked for domains that *are not* using the outlook: transport but > > are using the outlook.com relay servers. There are 383 such domains > > -- the vast majority are one email address per domain. These domains > > are competing for the limited number of outlook.com connections and > > the

Re: Connection Caching Per-Destination

2020-07-31 Thread Greg Sims
> Your real problem is however your IP reputation. If you're sending > unsolicited email, or you have relay customers sending unsolicited mail, > then your difficulties delivering it are a desirable feature of > Microsoft's email service. If you're sending email outlook.com > customers want, then

Re: Connection Caching Per-Destination

2020-07-31 Thread Greg Sims
The situation with outlook got much worse in our overnight runs. We transferred 7K subscriber emails to relays ending in outlook.com and saw the following feedback in our logs: MaxConnections: 83, Connection: 1386, RateLimited: 6392 where the following regexp is used in our log post-processor:

Re: Connection Caching Per-Destination

2020-07-31 Thread Wietse Venema
Viktor Dukhovni: > On Fri, Jul 31, 2020 at 02:16:54PM -0400, Wietse Venema wrote: > > > Logged as conn_use=xxx. By default, reuse happens only for plaintext > > connections. > > > > > >smtp_tls_connection_reuse=yes > > > > Logged as TLS handshake results plus conn_use=xxx. > > One thing we

Re: Connection Caching Per-Destination

2020-07-31 Thread Viktor Dukhovni
On Fri, Jul 31, 2020 at 02:16:54PM -0400, Wietse Venema wrote: > Logged as conn_use=xxx. By default, reuse happens only for plaintext > connections. > > > >smtp_tls_connection_reuse=yes > > Logged as TLS handshake results plus conn_use=xxx. One thing we could likely improve in TLS connectio

Re: Connection Caching Per-Destination

2020-07-31 Thread Wietse Venema
@lbutlr: > On 30 Jul 2020, at 12:53, Wietse Venema wrote: > > main.cf: > >smtp_connection_cache_on_demand=yes Logged as conn_use=xxx. By default, reuse happens only for plaintext connections. > >smtp_tls_connection_reuse=yes Logged as TLS handshake results plus conn_use=xxx. Wi

Re: Connection Caching Per-Destination

2020-07-31 Thread Viktor Dukhovni
On Fri, Jul 31, 2020 at 11:47:57AM -0600, @lbutlr wrote: > On 30 Jul 2020, at 12:53, Wietse Venema wrote: > > main.cf: > >smtp_connection_cache_on_demand=yes > >smtp_tls_connection_reuse=yes > > Do these setting show up in anyway int he logs (that is, does the log look > any different if

Re: Connection Caching Per-Destination

2020-07-31 Thread @lbutlr
On 30 Jul 2020, at 12:53, Wietse Venema wrote: > main.cf: >smtp_connection_cache_on_demand=yes >smtp_tls_connection_reuse=yes Do these setting show up in anyway int he logs (that is, does the log look any different if a TLS connection is reused or a connection is using cache_on_demand.

Re: Connection Caching Per-Destination

2020-07-31 Thread Wietse Venema
Greg Sims: > The situation with outlook got much worse in our overnight runs. We > transferred 7K subscriber emails to relays ending in outlook.com and > saw the following feedback in our logs: > > MaxConnections: 83, Connection: 1386, RateLimited: 6392 > > where the following regexp is used in

Re: Connection Caching Per-Destination

2020-07-31 Thread Viktor Dukhovni
On Fri, Jul 31, 2020 at 09:37:12AM -0700, Greg Sims wrote: > RateLimited -- "^.*The mail server .* has been temporarily rate > limited due to IP reputation.*$" There's your problem. You need a better IP reputation. > (1) smtpd_tls_security_level = none & smtp_tls_security_level = none > in mai

Re: Connection Caching Per-Destination

2020-07-31 Thread Wietse Venema
Viktor Dukhovni: > > On Jul 31, 2020, at 12:33 PM, Wietse Venema wrote: > > > >> No, you *do not* want to do that. That can increase connection > >> concurrency beyond your process limit, in the form of idle connections > >> that have a different nexthop than the one to which you're currently > >

Re: Connection Caching Per-Destination

2020-07-31 Thread Viktor Dukhovni
> On Jul 31, 2020, at 12:33 PM, Wietse Venema wrote: > >> No, you *do not* want to do that. That can increase connection >> concurrency beyond your process limit, in the form of idle connections >> that have a different nexthop than the one to which you're currently >> delivering email. > > We

Re: Connection Caching Per-Destination

2020-07-31 Thread Wietse Venema
Viktor Dukhovni: > > We would like to use Per-Destination Connection Caching to increase > > our throughput for "outlook:". > > No, you *do not* want to do that. That can increase connection > concurrency beyond your process limit, in the form of idle connections > that have a different nexthop t

Re: Connection Caching Per-Destination

2020-07-31 Thread Greg Sims
> > We would like to use Per-Destination Connection Caching to increase > > our throughput for "outlook:". > > No, you *do not* want to do that. That can increase connection > concurrency beyond your process limit, in the form of idle connections > that have a different nexthop than the one to whi

Re: Connection Caching Per-Destination

2020-07-30 Thread Viktor Dukhovni
On Thu, Jul 30, 2020 at 09:49:07PM -0700, Greg Sims wrote: > > Instead, you want to *disable* even demand connection caching. > > I updated master.cf based on your recommendation: > > outlook unix - - n - 6 smtp > -o syslog_name=outlook > -o smtp_connection_ca

Re: Connection Caching Per-Destination

2020-07-30 Thread Viktor Dukhovni
On Thu, Jul 30, 2020 at 10:58:20AM -0700, Greg Sims wrote: > We are seeing: "has exceeded the maximum number of connections" in our > logs for domains associated with outlook.com. We have a transport > named "outlook:" in transport.regexp as follows: > > # outlook.com domains > # > /@outlook(\.[

Connection Caching Per-Destination

2020-07-30 Thread Greg Sims
We are seeing: "has exceeded the maximum number of connections" in our logs for domains associated with outlook.com. We have a transport named "outlook:" in transport.regexp as follows: # outlook.com domains # /@outlook(\.[a-z]{2,3}){1,2}$/ outlook: /@hotmail(\.[a-z]{2,3}){1,2}$/ outlook: /@liv

Re: Connection Caching Per-Destination

2020-07-30 Thread Wietse Venema
Greg Sims: > We are seeing: "has exceeded the maximum number of connections" in our > logs for domains associated with outlook.com. We have a transport > named "outlook:" in transport.regexp as follows: ... > This transport is configured as follows in master.cf: > > outlook unix - -