Bulk Mailing Performance
More to satisfy my own curiosity than anything else, I'm wondering about the performance that could be squeezed out of Postfix in a bulk mailing capacity. I have a client that currently uses and ESP who have an astounding throughput of up to a million messages per hour. This brought up a discussion about high-performance MTAs and tuning and the general comments I'm hearing are that things like Postfix, Exim, Sendmail & are just not man enough for such a task and the absolute best you could expect from any of them is about 100k messages per hour. Now, I like to wipe out the fact from fiction because people like PowerMTA are looking to sell their products and it would be in their interest to neglect that any MTA (Postfix/Exim/Sendmail) could be set up in a way that would easily rival their product. Can anyone on the list tell me if it's possible to performance tune Postfix to a point where it could complete with this and possible strategies? Kind thanks
Re: Bulk Mailing Performance
Am 02.09.2012 11:43, schrieb Sam Jones: > More to satisfy my own curiosity than anything else, I'm wondering about > the performance that could be squeezed out of Postfix in a bulk mailing > capacity. > > I have a client that currently uses and ESP who have an astounding > throughput of up to a million messages per hour. This brought up a > discussion about high-performance MTAs and tuning and the general > comments I'm hearing are that things like Postfix, Exim, Sendmail & > are just not man enough for such a task and the absolute best you could > expect from any of them is about 100k messages per hour. > > Now, I like to wipe out the fact from fiction because people like > PowerMTA are looking to sell their products and it would be in their > interest to neglect that any MTA (Postfix/Exim/Sendmail) could be set up > in a way that would easily rival their product. > > Can anyone on the list tell me if it's possible to performance tune > Postfix to a point where it could complete with this and possible > strategies? > > Kind thanks > Hi look on list archives about tuning stuff also read http://www.postfix.org/TUNING_README.html at last , there are two side deliver in/out for out ,there are things you need to find out about the recipients mail servers, cause many of them have limits about get mail from the same ip adress per time periods,and/or do greylisting and other filter stuff, so its not only a question how to tune your own postfix ( which may always be possible to goal your needs, depending on hardware, filesystems network etc ) for massive bulk mails sometimes multiple instance postfix or a cascade with smtp fallback postfix servers is used for fasten deliver out mail. Also possible smtp via loadbalancer etc. At last there are a lot other recommands ( whitelisting at big mailers , use dkim and spf dmarc etc ) doing massive mail which arent related directly to the mailserver software you use, for sure postfix will never be the performance problem if setup right to your "massive" needs. In fact postfix is widly used for massive mail. -- Best Regards MfG Robert Schetterer
Re: Bulk Mailing Performance
On Sun, Sep 02, 2012 at 10:43:07AM +0100, Sam Jones wrote: > More to satisfy my own curiosity than anything else, I'm wondering about > the performance that could be squeezed out of Postfix in a bulk mailing > capacity. Running a high volume bulk email platform is not a software problem. It is a logistics problem. Enrolling on the whitelists and feedback loops of various large email providers, handling bounce-backs, jumping through rate-limit hoops, ... > I have a client that currently uses and ESP who have an astounding > throughput of up to a million messages per hour. This is not astounding, a single ~2003 Dell 1850 Postfix server was measured by me at ~300 msgs/sec of deliveries to real users with nothing but a simple MegaRAID controller (with battery cache) striping two SCSI disks. This would go another factor of 2 faster on today's commodity servers, but the real issue is finding peers who'll accept your mail at that rate. > discussion about high-performance MTAs and tuning and the general > comments I'm hearing are that things like Postfix, Exim, Sendmail & > are just not man enough for such a task and the absolute best you could > expect from any of them is about 100k messages per hour. Many bulk email services in fact use Postfix, Exim, ... The MTA software is often not the bottleneck. They split bulk deliveries over many machines (or lots of IPs on the same machine) and tune to avoid throttling by the ESP over and above all other concerns. Raw MTA performance is rarely a factor. > Now, I like to wipe out the fact from fiction because people like > PowerMTA are looking to sell their products and it would be in their > interest to neglect that any MTA (Postfix/Exim/Sendmail) could be set up > in a way that would easily rival their product. Bulk email is a logistics exercise. When you choose an bulk email delivery service, you're buying their logistics skills and their reputation with mailbox providers (Gmail, Hotmail, Yahoo, ...) > Can anyone on the list tell me if it's possible to performance tune > Postfix to a point where it could complete with this and possible > strategies? Wrong question. -- Viktor.
Re: Rejecting mail based on destination MX records
On Tue, 2012-08-28 at 15:53:28 -0500, Noel Jones wrote:
> On 8/28/2012 3:38 PM, Gábor Lénárt wrote:
> > On Tue, Aug 28, 2012 at 04:33:16PM -0400, Jon A. wrote:
> >> I'd like to immediately reject mail for all destinations with ONLY a
> >> fakemx.net record. While I could block these as I find them, I'd prefer to
> >> detect it if possible.
> >> One such:
> >>
> >> hitmail.com mail is handled by 0 mx.fakemx.net.
> ...
> Be aware the postfix built-in check_*_mx_access will match if ANY of
> the MX records match.
>
> To reject domains with ONLY fakemx MX records, you'll need to use an
> external policy service.
The OP could also query, via check_recipient_access, a spawn(8)-managed
TCP table; I do not know how well that would scale. An untested code
snippet that requires the external dnspython module is below. Please do
not use it in production; it is just to illustrate the approach.
#!/usr/local/bin/python
import os, sys, dns.resolver
# autoflush STDOUT
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
# initialize a resolver with 2s timeout
resolver = dns.resolver.Resolver()
resolver.lifetime = 2
while True:
try:
fakemx = 0
domain = raw_input().lstrip('get ').lower().rsplit('@', 1)[1]
answer = resolver.query(domain, 'MX')
for mx in answer:
if 'mx.fakemx.net' in mx.to_text(): fakemx += 1
if fakemx == len(answer):
print('200 REJECT mail not deliverable (only destination is fakemx.net)')
else:
print('200 DUNNO')
except:
print('200 DUNNO')
--
Sahil Tandon
Re: Bulk Mailing Performance
On Sun, 2012-09-02 at 15:39 +, Viktor Dukhovni wrote: > On Sun, Sep 02, 2012 at 10:43:07AM +0100, Sam Jones wrote: > > > More to satisfy my own curiosity than anything else, I'm wondering about > > the performance that could be squeezed out of Postfix in a bulk mailing > > capacity. > > Running a high volume bulk email platform is not a software problem. > It is a logistics problem. Enrolling on the whitelists and feedback > loops of various large email providers, handling bounce-backs, > jumping through rate-limit hoops, ... > > > > I have a client that currently uses and ESP who have an astounding > > throughput of up to a million messages per hour. > > This is not astounding, a single ~2003 Dell 1850 Postfix server > was measured by me at ~300 msgs/sec of deliveries to real users > with nothing but a simple MegaRAID controller (with battery cache) > striping two SCSI disks. This would go another factor of 2 faster > on today's commodity servers, but the real issue is finding peers > who'll accept your mail at that rate. > > > discussion about high-performance MTAs and tuning and the general > > comments I'm hearing are that things like Postfix, Exim, Sendmail & > > are just not man enough for such a task and the absolute best you could > > expect from any of them is about 100k messages per hour. > > Many bulk email services in fact use Postfix, Exim, ... The MTA > software is often not the bottleneck. They split bulk deliveries > over many machines (or lots of IPs on the same machine) and tune > to avoid throttling by the ESP over and above all other concerns. > Raw MTA performance is rarely a factor. > > > Now, I like to wipe out the fact from fiction because people like > > PowerMTA are looking to sell their products and it would be in their > > interest to neglect that any MTA (Postfix/Exim/Sendmail) could be set up > > in a way that would easily rival their product. > > Bulk email is a logistics exercise. When you choose an bulk email > delivery service, you're buying their logistics skills and their > reputation with mailbox providers (Gmail, Hotmail, Yahoo, ...) > > > Can anyone on the list tell me if it's possible to performance tune > > Postfix to a point where it could complete with this and possible > > strategies? > > Wrong question. > I really appreciate the post Viktor. Thought provoking and clear. I guess what I'm querying in a way is some of the sales blurb from people like PowerMTA & GreenArrow and the remarks they make about open source solutions like Postfix etc. This one in particular: "Open source Mail Transfer Agents (MTAs) often max out between 20 and 30 thousand messages per hour. GreenArrow can send 300,000 messages per hour—more than ten times as fast." If we strip this back to hypothetical and assume a perfect world without any issues like rate limiting and rejection, small emails with nomay restrictions or network issues with the recipient MX's, is the above statement plausibly true? I'm assuming - and I've not yet looked deeply at this - that there is probably a way to get Postfix to run parallel instances to improve delivery speed.
Re: Bulk Mailing Performance
Sam Jones: > I guess what I'm querying in a way is some of the sales blurb from > people like PowerMTA & GreenArrow and the remarks they make about open > source solutions like Postfix etc. This one in particular: "Open source > Mail Transfer Agents (MTAs) often max out between 20 and 30 thousand > messages per hour. That's 7 deliveries per second, a number that you might get with an MTA that makes one connection at a time (Sendmail, Exim). You get larger numbers with parallel deliveries. Those are built-into Postfix/qmail/etc. With MTAs such as Sendmail/Exim/etc. you get parallel deliveries by using parallel submission. Once you reach the file system performance limit, Postfix/qmail/etc. require parallel submission as well, with different MTAs running on top of independent file systems. As Victor says, it is not primarily an MTA problem. The technical problems are a matter of proper diagnosis and appropriate tuning. Wietse
Re: Bulk Mailing Performance
The exact same question was sent by someone calling himself "Ron White" to the exim mailing list at almost exactly the same time. Peddling one's services by soliciting comparisons with competitors is so passé . . .
Re: Bulk Mailing Performance
On Sun, 2 Sep 2012 22:46:10 +0200 Lorens Kockum wrote: > The exact same question was sent by someone calling himself > "Ron White" to the exim mailing list at almost exactly the same > time. Peddling one's services by soliciting comparisons with > competitors is so passé . . . I find it rather useful; lets me know what I should be blocking > -- John
Re: Bulk Mailing Performance
Il 02/09/2012 11:43, Sam Jones ha scritto: > More to satisfy my own curiosity than anything else, I'm wondering about > the performance that could be squeezed out of Postfix in a bulk mailing > capacity. > > I have a client that currently uses and ESP who have an astounding > throughput of up to a million messages per hour. This brought up a > discussion about high-performance MTAs and tuning and the general > comments I'm hearing are that things like Postfix, Exim, Sendmail & > are just not man enough for such a task and the absolute best you could > expect from any of them is about 100k messages per hour. > > Now, I like to wipe out the fact from fiction because people like > PowerMTA are looking to sell their products and it would be in their > interest to neglect that any MTA (Postfix/Exim/Sendmail) could be set up > in a way that would easily rival their product. > > Can anyone on the list tell me if it's possible to performance tune > Postfix to a point where it could complete with this and possible > strategies? > > Kind thanks > > In my experience, I have never had problems with performances in the past with Postfix. Though I had stability issues, but just because we reached out the limit of opened files (OS tuning). And it was during a spam attack (probably around a milion email per hour during attack). Hope this helps. Cheers, Dario "subbia" Cavallaro
Re: Bulk Mailing Performance
On Sun, Sep 02, 2012 at 07:14:35PM +0100, Sam Jones wrote: > I guess what I'm querying in a way is some of the sales blurb from > people like PowerMTA & GreenArrow and the remarks they make about open > source solutions like Postfix etc. This one in particular: "Open source > Mail Transfer Agents (MTAs) often max out between 20 and 30 thousand > messages per hour. GreenArrow can send 300,000 messages per hour?more > than ten times as fast." As I said, I measured 300 msgs/sec with Postfix on relatively modest hardware in 2003. This is not too difficult, just configure sufficient output concurrency, and provide a low latency disk (battery cache striped RAID). With RAM disk (a queue-manager bottleneck analysis, circa five years ago) Postfix yielded ~3000 msgs/sec on a dual Opteron box delivering to the discard transport. So that's your ceiling if you provide sufficient disk and network bandwidth, eventually the queue manager runs out of CPU, but this is at rates approaching 10 million messages an hour. The throughput numbers are not that interesting anymore, I go for reliability, security and flexibility. I also go for a solid architecture that degrades well under load, and that's why I really like Postfix, but this is a difficult point to make, most people are not in a position to understand why Postfix stands out in this regard. -- Viktor.
The Yahoo trickle
Hi all,
I am familiar with yahoo being difficult to send email to, as i'm sure
most all of us here are. I am now faced with needing to increase my
server's throughput to yahoo's MTA's. I was scouring the net and what
caught my attention on the postfix mailing list archives was a reply
from *Victor Duchovni*
http://tech.groups.yahoo.com/group/postfix-users/message/244843?threaded=1&p=3
I am trying to figure out how to plugin my values to get the equation
correct so I could calculate the what maximal throughput per unit of
concurrency msgs/sec? It seems like I am only sending roughly 800 -
1,000 emails an hour to yahoo. Settings for my yahoo transport are
yahoo_destination_concurrency_limit = 20
yahoo_destination_rate_delay = 1s
yahoo_destination_recipient_limit = 20
I found this cool one liner thanks to Victor
$ perl -lne '
m{\A\S+T(\S+).{5} \S+ \S+: \w+: to=<\S+\@yahoo\.com>, (?:orig_to=\S+,
)?relay=\S+, (?:conn_use=\d+, )?delay=\S+,
delays=[\d.]+/[\d.]+/([\d.]+)/([\d.])+,} or next;
$c = 0.95 * $c + 0.05 * $2;
$d = 0.95 * $d + 0.05 * $3;
if (++$i % 100 == 0) {
printf "%s %5.2f %5.2f\n", $1, $c, $d;
}' maillog
18:43:28 0.07 4.43
18:47:20 0.52 3.71
18:51:01 0.13 3.13
18:54:21 0.05 4.73
18:57:49 0.08 3.32
19:01:06 0.05 4.63
19:04:32 0.06 4.84
19:07:55 0.09 3.91
Can anyone offer any guidance on what direction I need to go?
Thanks in advance,
Joey
Re: Bulk Mailing Performance
On Sep 3, 2012, at 03:56, Viktor Dukhovni wrote: > On Sun, Sep 02, 2012 at 07:14:35PM +0100, Sam Jones wrote: > >> I guess what I'm querying in a way is some of the sales blurb from >> people like PowerMTA & GreenArrow and the remarks they make about open >> source solutions like Postfix etc. This one in particular: "Open source >> Mail Transfer Agents (MTAs) often max out between 20 and 30 thousand >> messages per hour. GreenArrow can send 300,000 messages per hour?more >> than ten times as fast." > > As I said, I measured 300 msgs/sec with Postfix on relatively modest > hardware in 2003. This is not too difficult, just configure sufficient > output concurrency, and provide a low latency disk (battery cache > striped RAID). > > With RAM disk (a queue-manager bottleneck analysis, circa five > years ago) Postfix yielded ~3000 msgs/sec on a dual Opteron box > delivering to the discard transport. So that's your ceiling if you > provide sufficient disk and network bandwidth, eventually the queue > manager runs out of CPU, but this is at rates approaching 10 million > messages an hour. > > The throughput numbers are not that interesting anymore, I go for > reliability, security and flexibility. I also go for a solid > architecture that degrades well under load, and that's why I > really like Postfix, but this is a difficult point to make, > most people are not in a position to understand why Postfix > stands out in this regard. In other words, if 'we strip this back to hypothetical and assume a perfect world without any issues', this 'GreenArrow' maxes out at 300,000 messages per hour. Postfix can send 10,8 million messages per hour, more than 35 times as fast*. Lies, damn lies, and vendor benchmarks, heh. Cya, Jona -- * Tests performed in an optimized lab environment. Operational restrictions may apply in real world environments.
Re: Bulk Mailing Performance
* Sam Jones : > I guess what I'm querying in a way is some of the sales blurb from > people like PowerMTA & GreenArrow and the remarks they make about open > source solutions like Postfix etc. This one in particular: "Open source > Mail Transfer Agents (MTAs) often max out between 20 and 30 thousand > messages per hour. GreenArrow can send 300,000 messages per hour—more > than ten times as fast." My desktop machine - the very same one I'm writing this mail on - is currently delivering 65 messages per second to a smtp-sink. On a single 2TB SATA drive that will probably suffer a heart attack if you even so much as threaten it with more than 20 IOPS/s. That's while watching a 1080p video (footage from my last holiday). With no tuning whatsoever. Granted, not a real world use case, but still, don't trust everything you read on the web. Stefan
