Re: Mail drop

2009-03-24 Thread Victor Duchovni
On Tue, Mar 24, 2009 at 08:54:17AM -0400, Brandon Hilkert wrote: >> MEASURE! Find out what is slowing it down. When you know what that >> is, ask the question again. > > Here's a snapshot of top during a peak testing period > > top - 08:45:55 up 4 days, 23:47, 1 user, load average: 2.75, 1.70, 0

Re: Mail drop

2009-03-24 Thread Brandon Hilkert
- Original Message - From: "Victor Duchovni" To: Sent: Monday, March 23, 2009 5:57 PM Subject: Re: Mail drop On Mon, Mar 23, 2009 at 05:54:07PM -0400, Brandon Hilkert wrote: dkim-milter is one program that you are asking to sign lots of messages in parallel. To

Re: Mail drop

2009-03-23 Thread Victor Duchovni
On Mon, Mar 23, 2009 at 07:46:16PM -0400, Brandon Hilkert wrote: >>> Tracing the system calls in the milter may help (when sending just >>> one message to reduce confusion). >> >> strace-ing a multi-threaded program, have fun. That's why only one message should be sent. There should not be too ma

Re: Mail drop

2009-03-23 Thread Brandon Hilkert
- Original Message - From: "Wietse Venema" To: Sent: Monday, March 23, 2009 7:42 PM Subject: Re: Mail drop Victor Duchovni: On Mon, Mar 23, 2009 at 07:24:50PM -0400, Wietse Venema wrote: > Victor Duchovni: > > and may be causing high disk latency. You have

Re: Mail drop

2009-03-23 Thread Wietse Venema
Victor Duchovni: > On Mon, Mar 23, 2009 at 07:24:50PM -0400, Wietse Venema wrote: > > > Victor Duchovni: > > > and may be causing high disk latency. You have to tune the milter > > > configuration > > > > There is no need for dkim-milter to touch the disk. It receives > > header and body content

Re: Mail drop

2009-03-23 Thread Victor Duchovni
On Mon, Mar 23, 2009 at 07:24:50PM -0400, Wietse Venema wrote: > Victor Duchovni: > > and may be causing high disk latency. You have to tune the milter > > configuration > > There is no need for dkim-milter to touch the disk. It receives > header and body content from Postfix via the Milter prot

Re: Mail drop

2009-03-23 Thread Wietse Venema
Brandon Hilkert: [ Charset ISO-8859-1 unsupported, converting... ] > > - Original Message - > From: "Wietse Venema" > To: "Postfix users" > Sent: Monday, March 23, 2009 5:39 PM > Subject: Re: Mail drop > > > > Brandon Hilkert: > &g

Re: Mail drop

2009-03-23 Thread Wietse Venema
Victor Duchovni: > and may be causing high disk latency. You have to tune the milter > configuration There is no need for dkim-milter to touch the disk. It receives header and body content from Postfix via the Milter protocol. I know this, because I implemented the Postfix side of the protocol.

Re: Mail drop

2009-03-23 Thread Victor Duchovni
On Mon, Mar 23, 2009 at 06:15:13PM -0400, Brandon Hilkert wrote: >> MEASURE! Find out what is slowing it down. When you know what that >> is, ask the question again. > > What are the best tools to get a feel for hardware performance and > utilitization? Top, atop, vmstat ? > > As you can probably

Re: Mail drop

2009-03-23 Thread Brandon Hilkert
- Original Message - From: "Victor Duchovni" To: Sent: Monday, March 23, 2009 5:57 PM Subject: Re: Mail drop On Mon, Mar 23, 2009 at 05:54:07PM -0400, Brandon Hilkert wrote: dkim-milter is one program that you are asking to sign lots of messages in parallel. To

Re: Mail drop

2009-03-23 Thread Victor Duchovni
On Mon, Mar 23, 2009 at 05:54:07PM -0400, Brandon Hilkert wrote: >> dkim-milter is one program that you are asking to sign lots of >> messages in parallel. To implement parallelism, dkim-milter uses >> mutiple threads in one process. To keep the threads from tripping >> over each other, dkim-milte

Re: Mail drop

2009-03-23 Thread Brandon Hilkert
- Original Message - From: "Wietse Venema" To: "Postfix users" Sent: Monday, March 23, 2009 5:39 PM Subject: Re: Mail drop Brandon Hilkert: I"m not disputing this fact. I used smtp-source with 10 connections. Without DKIM signing - 14,634 emails/min

Re: Mail drop

2009-03-23 Thread Victor Duchovni
On Mon, Mar 23, 2009 at 05:34:42PM -0400, Brandon Hilkert wrote: >> if DKIM consumes all available CPU, find a faster DKIM engine. If DKIM >> clobbers the disk capacity, consider placing the working area of the DKIM >> process in tmpfs, because neither milters nor SMTP proxies queue mail, so >> th

Re: Mail drop

2009-03-23 Thread Wietse Venema
Brandon Hilkert: > I"m not disputing this fact. I used smtp-source with 10 connections. > > Without DKIM signing - 14,634 emails/min > With DKIM signing - 4,762 emails/min > > I think we would both agree that that's a large discrepancy. Yes. > I'm using DKIM-milter. During the testing, the CPU

Re: Mail drop

2009-03-23 Thread Brandon Hilkert
- Original Message - From: "Victor Duchovni" To: Sent: Monday, March 23, 2009 5:15 PM Subject: Re: Mail drop On Mon, Mar 23, 2009 at 04:51:17PM -0400, Brandon Hilkert wrote: - Original Message - From: "Victor Duchovni" To: Sent: Monday, March 23,

Re: Mail drop

2009-03-23 Thread Victor Duchovni
On Mon, Mar 23, 2009 at 04:51:17PM -0400, Brandon Hilkert wrote: > > - Original Message - From: "Victor Duchovni" > > To: > Sent: Monday, March 23, 2009 4:43 PM > Subject: Re: Mail drop > > >> On Mon, Mar 23, 2009 at 04:32:43PM -0400, Brando

Re: Mail drop

2009-03-23 Thread Melvyn Sopacua
On Monday 23 March 2009 21:16:06 Brandon Hilkert wrote: > I appreciate the insight. Unfortunately the process is what it is. I don't > have any control over development. My job is to make sure the systems work > properly. I'm trying to help as asked. The file also contains html to allow > a user t

Re: Mail drop

2009-03-23 Thread Brandon Hilkert
- Original Message - From: "Victor Duchovni" To: Sent: Monday, March 23, 2009 4:43 PM Subject: Re: Mail drop On Mon, Mar 23, 2009 at 04:32:43PM -0400, Brandon Hilkert wrote: The application won't run any faster than the code that serially parses the 30GB file. I

Re: Mail drop

2009-03-23 Thread Victor Duchovni
On Mon, Mar 23, 2009 at 04:32:43PM -0400, Brandon Hilkert wrote: >> The application won't run any faster than the code that serially parses >> the 30GB file. If this code can use a pool of SMTP sender "threads" and >> can parse the file quickly enough, you could try that. > > The parsing isn't a b

Re: Mail drop

2009-03-23 Thread Brandon Hilkert
- Original Message - From: "Victor Duchovni" To: Sent: Monday, March 23, 2009 4:29 PM Subject: Re: Mail drop On Mon, Mar 23, 2009 at 04:16:06PM -0400, Brandon Hilkert wrote: Having said that, we build a huge text file (~30GB) with about 1 million eml messages as it

Re: Mail drop

2009-03-23 Thread Victor Duchovni
On Mon, Mar 23, 2009 at 04:16:06PM -0400, Brandon Hilkert wrote: >>> Having said that, we build a huge text file (~30GB) with about 1 million >>> eml messages as its contents. The sender utility then parses out email by >>> email and submits it to the IIS SMTP. We're trying to not have to modify >

Re: Mail drop

2009-03-23 Thread Brandon Hilkert
- Original Message - From: "Victor Duchovni" To: Sent: Monday, March 23, 2009 4:08 PM Subject: Re: Mail drop On Mon, Mar 23, 2009 at 03:58:29PM -0400, Brandon Hilkert wrote: Having said that, we build a huge text file (~30GB) with about 1 million eml messages as it

Re: Mail drop

2009-03-23 Thread Victor Duchovni
On Mon, Mar 23, 2009 at 03:58:29PM -0400, Brandon Hilkert wrote: > Having said that, we build a huge text file (~30GB) with about 1 million > eml messages as its contents. The sender utility then parses out email by > email and submits it to the IIS SMTP. We're trying to not have to modify > th

Re: Mail drop

2009-03-23 Thread Brandon Hilkert
- Original Message - From: "Victor Duchovni" To: Sent: Monday, March 23, 2009 3:37 PM Subject: Re: Mail drop On Mon, Mar 23, 2009 at 03:33:01PM -0400, Brandon Hilkert wrote: We have a large text file with contents of the eml message for each person in a list. If I fin

Re: Mail drop

2009-03-23 Thread Wietse Venema
Brandon Hilkert: > From: "Victor Duchovni" > To: > Sent: Monday, March 23, 2009 1:12 PM > Subject: Re: Mail drop > > > > On Mon, Mar 23, 2009 at 01:09:12PM -0400, Brandon Hilkert wrote: > > > >> Does postfix have the ability to send out emails

Re: Mail drop

2009-03-23 Thread Magnus Bäck
On Monday, March 23, 2009 at 20:33 CET, Brandon Hilkert wrote: > > http://www.postfix.org/OVERVIEW.html#receiving > > We have a large text file with contents of the eml message for each > person in a list. If I find a way to parse each email, is there an > easy way to inject them into the q

Re: Mail drop

2009-03-23 Thread Victor Duchovni
On Mon, Mar 23, 2009 at 03:33:01PM -0400, Brandon Hilkert wrote: > We have a large text file with contents of the eml message for each person > in a list. If I find a way to parse each email, is there an easy way to > inject them into the queue, rather than relaying, because as many have said >

Re: Mail drop

2009-03-23 Thread Brandon Hilkert
- Original Message - From: "Victor Duchovni" To: Sent: Monday, March 23, 2009 1:12 PM Subject: Re: Mail drop On Mon, Mar 23, 2009 at 01:09:12PM -0400, Brandon Hilkert wrote: Does postfix have the ability to send out emails placed in a specific folder through the file

Re: Mail drop

2009-03-23 Thread Victor Duchovni
On Mon, Mar 23, 2009 at 01:09:12PM -0400, Brandon Hilkert wrote: > Does postfix have the ability to send out emails placed in a specific > folder through the file system, or does it require a sendmail-style > command to get the sending invoked? > > If so, what folder will do this? http://www.pos