----- Original Message -----
From: "Victor Duchovni" <victor.ducho...@morganstanley.com>
To: <postfix-users@postfix.org>
Sent: Monday, March 23, 2009 4:29 PM
Subject: Re: Mail drop
On Mon, Mar 23, 2009 at 04:16:06PM -0400, Brandon Hilkert wrote:
Having said that, we build a huge text file (~30GB) with about 1
million
eml messages as its contents. The sender utility then parses out email
by
email and submits it to the IIS SMTP. We're trying to not have to
modify
the sender that much, so I was wondering if I could write a comparable
perl script to do the same or something like that. So maybe something
that
I could submit a text string to (containing the message and all the
headers) and have it be submitted to postfix.
The right approach is to store one copy of the message as a template,
use a pool of processes or threads to send messages in parallel, and
to use a database back-end to retrieve users for processing and mark
them done (recording the VERP id for each recipient so you can process
bounces).
I appreciate the insight. Unfortunately the process is what it is. I
don't
have any control over development. My job is to make sure the systems
work
properly. I'm trying to help as asked. The file also contains html to
allow
a user to see the contents in a browser if they choose. Point being, the
idea of redesigning the system isn't up for debate.
The application won't run any faster than the code that serially parses
the 30GB file. If this code can use a pool of SMTP sender "threads" and
can parse the file quickly enough, you could try that.
The parsing isn't a bottleneck. It currently sends over 6,000/min and we had
to actually slow it done intentionally because IIS's SMTP server was backing
up. So I don't think that's an issue. However, if we just relaying the
message to the posfix machine, the rates will not hold up either based on my
testing with smtp-source using DKIM, so that's why I'm asking if local
submission would be any faster?
If you are filling a bottle through a straw, having a wider bottle
won't help.
You need to make submission parallel. Good luck.
--
Viktor.
Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the "Reply-To" header.
To unsubscribe from the postfix-users list, visit
http://www.postfix.org/lists.html or click the link below:
<mailto:majord...@postfix.org?body=unsubscribe%20postfix-users>
If my response solves your problem, the best way to thank me is to not
send an "it worked, thanks" follow-up. If you must respond, please put
"It worked, thanks" in the "Subject" so I can delete these quickly.
