On Fri, Jun 28, 2019 at 01:23:38PM +0200, Matus UHLAR - fantomas wrote:
> On 28.06.19 12:03, hg user wrote:
> >I did investigate a bit and was able to link spikes in emails received on
> >the external MTAs to spikes in messages queued in amavisd.
> >
> >As soon as I receive a mailing list/newsletter kind of email addressed to
> >several hundreds recipients and the message size is more than 70kb, the
> >messages starts to queue up.
> >
> >A message I was able to isolate is 94kb, has text and html part and
> >processing via command line takes about 6 seconds, I received 500+ of them.
> >Due to unsubscribe urls containing the mail address, all messages have
> >different hashes
>
> 6 seconds message scanning is quite fast. I have set up
> razor,pyzor,dcc,dnsbl timeouts to 20 seconds to improve
> acurracy (maybe marginally, but I don't care).
>
> If you receive that much mail often, you need faster computer.
> Maybe sa-compile would help you a bit, but only with CPU usage, network
> checks need time (and they are effective so don't mess them up).
If the bottleneck is network checks, one should just increase parallel
processes. But OP probably has some other problems if regex are taking so
long (suggested by the 100% CPU usage).
Here's some figures for trunk/4.0.0 with razor,pyzor,dcc,ixhash and bunch of
other stuff.
# grep TIMING-SA mail.log mail.log.1 | pcregrep -o '\d+ ms' | awk '{print $1}'
| histogram
Count: 4375
Range: 7.000 - 8551.000; Mean: 1720.447; Median: 1252.000; Stddev: 1632.929
Percentiles: 90th: 4422.000; 95th: 4589.000; 99th: 8101.000
7.000 - 15.069: 36 ##
15.069 - 31.276: 184 #########
31.276 - 63.831: 426 ####################
63.831 - 129.221: 335 ################
129.221 - 260.565: 113 #####
260.565 - 524.384: 14 #
524.384 - 1054.296: 545 #########################
1054.296 - 2118.689: 1138 #####################################################
2118.689 - 4256.650: 945 ############################################
4256.650 - 8551.000: 639 ##############################
<500ms are shortcircuits. But mostly 1-4 sec for normal stuff.
Biggest improvement comes from razor/pyzor/dcc being asynchronous in trunk,
I suggest trying it if possible. :-)
