On 24/08/2012 15:57, Simon Hobson wrote:
Fabio Sangiovanni wrote:I'm using policyd to throttle incoming mail (size over time against sasl_username) and I'm experiencing a strange delay during submission. In particular, it seems some kind of delay due to policyd execution, and it's about 1 second per message.I hit a similar issue which is documented in a thread entitled "Performance issues - restricted mail handling" from April 2011. In the end I concluded that Postfix gives the appearance of having some sort of throughput issue when calling an external policy server - I found a limit of about 2 calls/second/smtp process. Since this is per-process, it scales with the number of connections and since it won't generally be an issue for me I never took it any further. I haven't done any tests to see it it's still an issue (the boxes/software have been upgraded since then).
This could be entirely my fault ... I fixed the performance issue in v2.1 by committing a pipelining patch which fixes pipelining of requests. However I do not see this comitted to v2.0.
Please try the attached, if it works I'll commit it right away. -N
diff --git a/cbpolicyd b/cbpolicyd
index 6d1afd1..d036343 100755
--- a/cbpolicyd
+++ b/cbpolicyd
@@ -56,6 +56,8 @@ sub configure {
$cfg->{'config_file'} = "/etc/cluebringer.conf";
$server->{'timeout'} = 120;
+ $server->{'timeout_idle'} = 1015;
+ $server->{'timeout_busy'} = 115;
$server->{'background'} = "yes";
$server->{'pid_file'} = "/var/run/cbpolicyd.pid";
$server->{'log_level'} = 2;
@@ -113,6 +115,7 @@ sub configure {
'pid_file',
'user', 'group',
'timeout',
+ 'timeout_idle', 'timeout_busy',
'background',
'min_servers',
'min_spare_servers',
@@ -363,12 +366,19 @@ sub process_request {
my $log = defined($self->{'config'}{'logging'}{'modules'});
- # Found module
- my $found;
+ # How many times did we pipeline...
+ my $policyRequests = 0;
+
#
# Loop till we fill up the buffer
#
+
+ # Beginning label, we do pipelining ...
+CONN_READ:
+
+ # Found module, set to 1 if found, 0 if otherwize
+ my $found = 0;
# Buffer
my $buf = "";
@@ -400,23 +410,37 @@ sub process_request {
last if ($found);
}
+ # We need to store this cause we use it below a few times
+ my $bufLen = length($buf);
+
# Again ... too large
- if (length($buf) > 16*1024) {
+ if ($bufLen > 16*1024) {
$self->log(LOG_WARN,"[CBPOLICYD] Request too large from => Peer: ".$server->{'peeraddr'}.":".$server->{'peerport'}.", Local: ".
$server->{'sockaddr'}.":".$server->{'sockport'});
return;
}
+
+ # Setup timeout
+ my $timeout;
+ # If buffer length > 0, its a busy connection
+ if ($bufLen > 0) {
+ $timeout = $server->{'timeout_busy'};
+ # Else its idle
+ } else {
+ $timeout = $server->{'timeout_idle'};
+ }
+
# Check for timeout....
- my $n = select($fdset,undef,undef,$server->{'timeout'});
+ my $n = select($fdset,undef,undef,$timeout);
if (!$n) {
- $self->log(LOG_WARN,"[CBPOLICYD] Timeout from => Peer: ".$server->{'peeraddr'}.":".$server->{'peerport'}.", Local: ".
- $server->{'sockaddr'}.":".$server->{'sockport'});
+ $self->log(LOG_WARN,"[CBPOLICYD] Timed out after ".$timeout."s from => Peer: ".$server->{'peeraddr'}.":".
+ $server->{'peerport'}.", Local: ".$server->{'sockaddr'}.":".$server->{'sockport'});
return;
}
# Read in 8kb
- $n = sysread(STDIN,$buf,8192,length($buf));
+ $n = sysread(STDIN,$buf,8192,$bufLen);
if (!$n) {
my $reason = defined($n) ? "Client closed connection" : "sysread[$!]";
$self->log(LOG_WARN,"[CBPOLICYD] $reason => Peer: ".$server->{'peeraddr'}.":".$server->{'peerport'}.", Local: ".
@@ -431,7 +455,6 @@ sub process_request {
return;
}
-
# Set protocol handler
$server->{'_protocol_handler'} = $found;
@@ -522,6 +545,10 @@ sub process_request {
}
$self->log(LOG_DEBUG,"[CBPOLICYD] Got request, running modules...") if ($log);
+ # Increment counter
+ $policyRequests++;
+
+ $self->log(LOG_INFO,"[CBPOLICYD] Got request #$policyRequests" . ($policyRequests > 1 ? " (pipelined)" : ""));
# Loop with modules
foreach my $module ( sort { $b->{'priority'} <=> $a->{'priority'} } @{$self->{'modules'}} ) {
@@ -578,6 +605,9 @@ sub process_request {
my $response = $self->protocol_getresponse();
print($response);
+
+ # Carry on with pipelining?
+ goto CONN_READ;
}
diff --git a/cluebringer.conf b/cluebringer.conf
index 0738e59..d651c64 100644
--- a/cluebringer.conf
+++ b/cluebringer.conf
@@ -105,7 +105,10 @@ log_mail=maillog
#port=10031
# Timeout in communication with clients
-#timeout=120
+# Idle timeout in postfix defaults to 1015s (active connection)
+#timeout_idle=1015
+# Busy sockets in postfix defaults to 100s
+#timeout_busy=115
# cidr_allow/cidr_deny
# Comma, whitespace or semi-colon separated. Contains a CIDR block to
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ Users mailing list [email protected] http://lists.policyd.org/mailman/listinfo/users_lists.policyd.org
