[pfx] Active queue congestion
Hi,
I look after a SAAS site where customers can send emails to their own
domains. At times some of our customers can initiate sending of large mail
volumes - which can swamp the active queue.
>From [1]:
"The only way to reduce congestion is to either reduce the input rate or
increase the throughput. Increasing the throughput requires either
increasing the concurrency or reducing the latency of deliveries."
I thought that reducing TRANSPORT_recipient_refill_limit and
TRANSPORT_recipient_limit would prevent messages sent to the customers
domain from dominating the active queue / prevent blocking of other
customers / backup messages in the incoming queue. To test this, I setup a
new dedicated server, set default_recipient_refill_limit=50 to match the
rate at which this domain accepts messages and default_recipient_limit to
1000. I then directed the customer's traffic to that instance.
The dedicated server was only intended to be used to identify the config
required for a new transport to be implemented on the existing Postfix
instances.
>From my monitoring (Check_MK) I saw that there were still spikes where
messages were added to the active queue faster than 50/5 per second and the
active queue grew to and plateau'd at 2.
(Specifically I'm using Postfix 2.10 on AWS2 Linux - which is very like
RHEL7).
Have I misunderstood the effect of *_recipient_refill_limit and
*_recipient_limit?
Is there a more appropriate way I can enforce some fairness on the active
queue?
(I also asked about this on Serverfault [2] but have not had any useful
feedback there).
{1] https://www.postfix.org/QSHAPE_README.html
[2]
https://serverfault.com/questions/1155772/postfix-cant-limit-active-queue
TIA
Colin
--
-BEGIN GEEK CODE BLOCK-
Version: 3.1
GCM d s+:+ a+ C+++(---)$ UL+++ P+(--) L+++ E--- W+++ N++ w-- PS++(+++())
t+ 5+ X R- tv-- b++ DI++ D e+++ h
--END GEEK CODE BLOCK--
___
Postfix-users mailing list -- postfix-users@postfix.org
To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Active queue congestion
Thank you, Viktor. I am planning to look at increasing the size of the Active queue however I would need to resize to a minimum of 50x based on past events. > You can also configure a non-zero smtpd_client_message_rate_limit H, not so sure about that. The docs do advise against this for legitimate traffic - and I've yet to see anything in the documentation that describes what happens when these rates are exceeded is it a 4xx? a 5xx? Is the IP just blocked? I now know that the config settings do not do what I expected - which is unfortunate as this would have been a very simple solution. > you could use a policy service to impose rate limits per SASL login, or sender address I had not considered that as a means of load balancing across the available relays (delaying the message at the origin is very much a last resort). I will do some reading on this. C. On Thu, 7 Mar 2024 at 13:46, Viktor Dukhovni via Postfix-users < postfix-users@postfix.org> wrote: > On Thu, Mar 07, 2024 at 12:26:06PM +0000, Colin McKinnon via Postfix-users > wrote: > > > I look after a SAAS site where customers can send emails to their own > > domains. At times some of our customers can initiate sending of large > mail > > volumes - which can swamp the active queue. > > Given sufficient memory, you can substantially raise the active queue > size limit. Servers have a lot more RAM now than they did in 2001. > The default of 20,000 could easily be raised by 10x to 20 on a > server-class machine. > > If customers indeed send mail only to their own domain, the destination > concurrency limits should ensure fairness, given sufficient space in the > queue and sufficiently many delivery agent slots. > > Speaking of delivery agent slots, if you have enough network bandwidth, > you can raise the smtp(8) delivery process limit in master.cf from 100 > to 1000: > > smtp unix - - n - 1000smtp > > Not that this could require some system-dependent tuning of the open > file hard limit in whatever code starts Postfix, if the limit is not > already very generous (on a Fedora 39 system with 65GB RAM, "ulimit -Hn" > reports ~1.8 million max open files). > > > >From [1]: > > "The only way to reduce congestion is to either reduce the input rate or > > increase the throughput. Increasing the throughput requires either > > increasing the concurrency or reducing the latency of deliveries." > > I am suggesting increasing concurrency, and also increasing the queue > depth to allow your customer to send larger bursts of mail without > overflowing the queue size limit. You can also configure a non-zero > > smtpd_client_message_rate_limit > > if abuse of your resources is plausible even with the larger queue size. > If that's too crude, you could use a policy service to impose rate > limits per SASL login, or sender address, ... > > > I thought that reducing TRANSPORT_recipient_refill_limit and > > TRANSPORT_recipient_limit would prevent messages sent to the customers > > domain from dominating the active queue / prevent blocking of other > > customers / backup messages in the incoming queue. > > These controls affect deliveries of single messages with many > recipients, but have no effect on a flood of single-recipient messages. > > -- > Viktor. > ___ > Postfix-users mailing list -- postfix-users@postfix.org > To unsubscribe send an email to postfix-users-le...@postfix.org > -- -BEGIN GEEK CODE BLOCK- Version: 3.1 GCM d s+:+ a+ C+++(---)$ UL+++ P+(--) L+++ E--- W+++ N++ w-- PS++(+++()) t+ 5+ X R- tv-- b++ DI++ D e+++ h --END GEEK CODE BLOCK-- ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Dynamic transport?
Hi, In my previous question [1] Viktor Dukhovni suggested > you could use a policy service to impose rate limits per SASL login, or sender address as a means of preventing active queue congestion. This took me down a bit of a rabbit hole looking at SMTP Access Policy delegation and tcp_tables. However I have not managed to find much (ANY???!!!) information on what the server responses should look like. Although I don't yet know exactly what they look like, I have only seen mention of accepting or rejecting emails at submission using SMTP Access Policy delegation. I really want to be able to generate, on a per message basis, the behaviours that I currently have statically defined in a transport map. These are: (condition1) customtransport: (condition2) smtp:[othernode] Is this possible with either mechanism? What would the specific responses be? There is also a secondary issue: I'm trying to do load balancing here, but the load balancing would be non-deterministic. I don't want the messages to bounce around between nodes. While I could handle that based on the helo_from= in SMTP Access Policy delegation, I don't believe this information is available in tcp_tables. Assuming that the custom transport is possible via tcp_table but NOT SMTP Access Policy delegation, how can I ensure that a message only gets bumped to a peer once? TIA Colin [1] https://www.mail-archive.com/postfix-users@postfix.org/msg101676.html -- -BEGIN GEEK CODE BLOCK- Version: 3.1 GCM d s+:+ a+ C+++(---)$ UL+++ P+(--) L+++ E--- W+++ N++ w-- PS++(+++()) t+ 5+ X R- tv-- b++ DI++ D e+++ h --END GEEK CODE BLOCK-- ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Dumb question about logging
On Fri, 8 Mar 2024 at 05:52, Stephen Satchell via Postfix-users wrote: > > Assuming that one's configuration has open relay, what does a log entry > for relayed mail look like? I think you really need to look at multiple log entries to discover that, e.g. Mar 13 16:21:06 mail-4 postfix/smtpd[1867]: connect from ptr-for-192-168-0-24.mynet[192.168.0.24] Mar 13 16:21:06 mail-4 postfix/smtpd[1867]: 1A3B642F9A8: ptr-for-192-168-0-24.mynet[192.168.0.24] Mar 13 16:21:06 mail-4 postfix/smtpd[1867]: disconnect from ptr-for-192-168-0-24.mynet[192.168.0.24] Mar 13 16:21:06 mail-4 postfix/qmgr[2298]: 1A3B642F9A8: from=, size=9604, nrcpt=1 (queue active) Mar 13 16:21:07 mail-4 postfix/smtp[1386]: 1A3B642F9A8: to=, relay=rmx.otherplace.com[10.1.1.5]:25, delay=1.2, delays=0/0/0.5/0.72, dsn=2.0.0, status=sent (250 2.0.0 3wtk0e9aqf-1 Message accepted for delivery) Mar 13 16:21:07 mail-4 postfix/qmgr[2298]: 1A3B642F9A8: removed That the connection originated outside your network is in the first two log entries above. The (asserted) sender is in the fourth line. The fact it is not addressed to your domain and needs to be passed on elsewhere is in the 5th line. The host I am looking at does not implement authentication, but ISTR there would be another log entry - how exactly you reconcile this with the type of entries above is a different matter though. Maybe it's just easier to test relaying from outside your network yourself? C. ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Dynamic transport?
On Wed, 13 Mar 2024 at 17:00, Viktor Dukhovni via Postfix-users wrote: > > http://www.postfix.org/SMTPD_POLICY_README.html > Thank you Victor. "The policy server replies with any action that is allowed in a Postfix SMTPD access(5) table." is exactly what I needed. > Or deferring, with 4XX, if submission is from an MTA that can queue for > a later retry. In my case, some of the senders are applications pointing directly at the relay. > Unclear how this avoids active queue congestion. > > > (condition2) smtp:[othernode] This provides me with a mechanism for distributing the traffic across multiple relay instances (and applying warm-up). > > (condition1) customtransport: This is not directly related to the capacity issue. But it does simplify the process of distributing configuration across a cluster. Colin -- -BEGIN GEEK CODE BLOCK- Version: 3.1 GCM d s+:+ a+ C+++(---)$ UL+++ P+(--) L+++ E--- W+++ N++ w-- PS++(+++()) t+ 5+ X R- tv-- b++ DI++ D e+++ h --END GEEK CODE BLOCK-- ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] check_policy_service for customizing routing & load balancing
Hi, I want to provision load balancing for my relays. The catch is that there is already some customized routing in place based on recipient domain and large block lists. These are currently handled by a transport map. I would prefer not to implement 2 layers of relays. If this were implemented as a plugin which is told the recipient address it could determine how the mail should be routed. But I need to ensure that I don't create any loops - i.e. the routing decision is based on the recipient AND where the email came from/via. The policy server (https://www.postfix.org/SMTPD_POLICY_README.html) seems to be ideal for my requirements since I get both recipient_name and helo_name, however the documentation only covers its use in the context of 'smtpd_recipient_restrictions'. I tried provisioning using: transport_maps = check_policy_service inet:127.0.0.1:8822 hash:/etc/postfix/transport smtpd_policy_service_default_action = DUNNO smtpd_policy_service_timeout = 10s However it appears that check_policy_service is not valid here. Although `postfix check` and a restart report no errors, when I try to relay a message, it is not accepted at the relay (hostname test106) which logs this: Mar 26 13:44:35 test106 postfix/smtpd[150541]: connect from test107.southwold.net[10.0.0.107] Mar 26 13:44:47 test106 postfix/trivial-rewrite[150545]: fatal: open dictionary: expecting "type:name" form instead of "check_policy_service" Mar 26 13:44:48 test106 postfix/master[148536]: warning: process /usr/lib/postfix/sbin/trivial-rewrite pid 150545 exit status 1 Mar 26 13:44:48 test106 postfix/master[148536]: warning: /usr/lib/postfix/sbin/trivial-rewrite: bad command startup -- throttling Questions: 1) Can I use check_policy_service in other contexts than smtpd_recipient_restrictions? (I don't want to interfere with `smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination` ) 2) If so, where would be the best place to put this? TIA Colin -- -BEGIN GEEK CODE BLOCK- Version: 3.1 GCM d s+:+ a+ C+++(---)$ UL+++ P+(--) L+++ E--- W+++ N++ w-- PS++(+++()) t+ 5+ X R- tv-- b++ DI++ D e+++ h --END GEEK CODE BLOCK-- ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: check_policy_service for customizing routing & load balancing
Hi all, I found that check_policy_service works (maybe*) in `smtpd_end_of_data_restrictions =`. So I'm guessing it might work in any *_restrictions configuration. (*still having some issues getting this to work as expected, but I'll come back here if I get stuck) Colin On Tue, 26 Mar 2024 at 13:52, Colin McKinnon wrote: > > Hi, > > I want to provision load balancing for my relays. The catch is that > there is already some customized routing in place based on recipient > domain and large block lists. These are currently handled by a > transport map. > > I would prefer not to implement 2 layers of relays. If this were > implemented as a plugin which is told the recipient address it could > determine how the mail should be routed. But I need to ensure that I > don't create any loops - i.e. the routing decision is based on the > recipient AND where the email came from/via. > > The policy server (https://www.postfix.org/SMTPD_POLICY_README.html) > seems to be ideal for my requirements since I get both recipient_name > and helo_name, however the documentation only covers its use in the > context of 'smtpd_recipient_restrictions'. I tried provisioning using: > > transport_maps = check_policy_service inet:127.0.0.1:8822 > hash:/etc/postfix/transport > smtpd_policy_service_default_action = DUNNO > smtpd_policy_service_timeout = 10s > > However it appears that check_policy_service is not valid here. > Although `postfix check` and a restart report no errors, when I try to > relay a message, it is not accepted at the relay (hostname test106) > which logs this: > > Mar 26 13:44:35 test106 postfix/smtpd[150541]: connect from > test107.southwold.net[10.0.0.107] > Mar 26 13:44:47 test106 postfix/trivial-rewrite[150545]: fatal: open > dictionary: expecting "type:name" form instead of > "check_policy_service" > Mar 26 13:44:48 test106 postfix/master[148536]: warning: process > /usr/lib/postfix/sbin/trivial-rewrite pid 150545 exit status 1 > Mar 26 13:44:48 test106 postfix/master[148536]: warning: > /usr/lib/postfix/sbin/trivial-rewrite: bad command startup -- > throttling > > Questions: > > 1) Can I use check_policy_service in other contexts than > smtpd_recipient_restrictions? (I don't want to interfere with > `smtpd_relay_restrictions = permit_mynetworks > permit_sasl_authenticated defer_unauth_destination` ) > > 2) If so, where would be the best place to put this? > > TIA > > Colin > > -- > -BEGIN GEEK CODE BLOCK- > Version: 3.1 > GCM d s+:+ a+ C+++(---)$ UL+++ P+(--) L+++ E--- W+++ N++ w-- PS++(+++()) > t+ 5+ X R- tv-- b++ DI++ D e+++ h > --END GEEK CODE BLOCK-- -- -BEGIN GEEK CODE BLOCK- Version: 3.1 GCM d s+:+ a+ C+++(---)$ UL+++ P+(--) L+++ E--- W+++ N++ w-- PS++(+++()) t+ 5+ X R- tv-- b++ DI++ D e+++ h --END GEEK CODE BLOCK-- ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: check_policy_service for customizing routing & load balancing
Hi, Stuck again. Using smtpd_end_of_data_restrictions = check_policy_service inet:127.0.0.1:8822 smtpd_policy_service_default_action = DUNNO I get: Mar 26 15:49:03 test106 postfix/smtpd[163532]: warning: access table inet:127.0.0.1:8822 has entry with lookup table: smtp:[test105.southwold.net] Mar 26 15:49:03 test106 postfix/smtpd[163532]: warning: do not specify lookup tables inside SMTPD access maps Mar 26 15:49:03 test106 postfix/smtpd[163532]: warning: define a restriction class and specify its name instead. and message is rejected. So I tried this: smtpd_restriction_classes = load_balancer load_balancer = check_policy_service inet:127.0.0.1:8822 smtpd_end_of_data_restrictions = load_balancer But I get the same warnings & message is rejected again. How can I get Postfix to apply the check_policy_service? Colin On Tue, 26 Mar 2024 at 15:37, Colin McKinnon wrote: > > Hi all, > > I found that check_policy_service works (maybe*) in > `smtpd_end_of_data_restrictions =`. So I'm guessing it might work in > any *_restrictions configuration. > > (*still having some issues getting this to work as expected, but I'll > come back here if I get stuck) > > Colin > > On Tue, 26 Mar 2024 at 13:52, Colin McKinnon wrote: > > > > Hi, > > > > I want to provision load balancing for my relays. The catch is that > > there is already some customized routing in place based on recipient > > domain and large block lists. These are currently handled by a > > transport map. > > > > I would prefer not to implement 2 layers of relays. If this were > > implemented as a plugin which is told the recipient address it could > > determine how the mail should be routed. But I need to ensure that I > > don't create any loops - i.e. the routing decision is based on the > > recipient AND where the email came from/via. > > > > The policy server (https://www.postfix.org/SMTPD_POLICY_README.html) > > seems to be ideal for my requirements since I get both recipient_name > > and helo_name, however the documentation only covers its use in the > > context of 'smtpd_recipient_restrictions'. I tried provisioning using: > > > > transport_maps = check_policy_service inet:127.0.0.1:8822 > > hash:/etc/postfix/transport > > smtpd_policy_service_default_action = DUNNO > > smtpd_policy_service_timeout = 10s > > > > However it appears that check_policy_service is not valid here. > > Although `postfix check` and a restart report no errors, when I try to > > relay a message, it is not accepted at the relay (hostname test106) > > which logs this: > > > > Mar 26 13:44:35 test106 postfix/smtpd[150541]: connect from > > test107.southwold.net[10.0.0.107] > > Mar 26 13:44:47 test106 postfix/trivial-rewrite[150545]: fatal: open > > dictionary: expecting "type:name" form instead of > > "check_policy_service" > > Mar 26 13:44:48 test106 postfix/master[148536]: warning: process > > /usr/lib/postfix/sbin/trivial-rewrite pid 150545 exit status 1 > > Mar 26 13:44:48 test106 postfix/master[148536]: warning: > > /usr/lib/postfix/sbin/trivial-rewrite: bad command startup -- > > throttling > > > > Questions: > > > > 1) Can I use check_policy_service in other contexts than > > smtpd_recipient_restrictions? (I don't want to interfere with > > `smtpd_relay_restrictions = permit_mynetworks > > permit_sasl_authenticated defer_unauth_destination` ) > > > > 2) If so, where would be the best place to put this? > > > > TIA > > > > Colin > > > > -- > > -BEGIN GEEK CODE BLOCK- > > Version: 3.1 > > GCM d s+:+ a+ C+++(---)$ UL+++ P+(--) L+++ E--- W+++ N++ w-- PS++(+++()) > > t+ 5+ X R- tv-- b++ DI++ D e+++ h > > --END GEEK CODE BLOCK-- > > > > -- > -BEGIN GEEK CODE BLOCK- > Version: 3.1 > GCM d s+:+ a+ C+++(---)$ UL+++ P+(--) L+++ E--- W+++ N++ w-- PS++(+++()) > t+ 5+ X R- tv-- b++ DI++ D e+++ h > --END GEEK CODE BLOCK-- -- -BEGIN GEEK CODE BLOCK- Version: 3.1 GCM d s+:+ a+ C+++(---)$ UL+++ P+(--) L+++ E--- W+++ N++ w-- PS++(+++()) t+ 5+ X R- tv-- b++ DI++ D e+++ h --END GEEK CODE BLOCK-- ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: check_policy_service for customizing routing & load balancing
Hi Viktor, Wietse also replied to say I wasn't able to solve this with Postfix. > > What kind of "load balancing"? Why won't MX records do? For uneven > weights, you can even use SRV records: > I'm trying to setup load balancing across a cluster of relays for a SAAS application. There's several problems I was hoping to solve: 1) distributing workload 2) IP warm up when adding nodes 3) avoiding dilution effects of spreading the workload (I only send a few emails to hotmail, for example) 4) separating out very badly performing receiving MTAs 5) adapting workload distribution according to host availability and current health (load, free memory etc) Blindly distributing the traffic at the TCP level works for 1&2 but DNS is too slow for 5, while 3 & 4 need information about where the email is going. > What sort "customised routing"? > Mostly large block lists (you expect users to actually manage their data?). > That's what sender_dependent_default_transport_maps is for. > Policy server doesn't work in this context. > It isn't actually a good tool for this, since it cannot affect > fine-grained per-recipient routing in multi-recipient messages. > It can only return FILTER actions which affect all recipients, > and the intent is to support content scanning, not fine-tune > routing. > That I could have lived with. C. -- -BEGIN GEEK CODE BLOCK- Version: 3.1 GCM d s+:+ a+ C+++(---)$ UL+++ P+(--) L+++ E--- W+++ N++ w-- PS++(+++()) t+ 5+ X R- tv-- b++ DI++ D e+++ h --END GEEK CODE BLOCK-- ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
