RE: FIXED -- high CPU on "lost" processes with forkserver

2008-09-26 Thread Matt Sergeant
ox processing roughly 6-12 connections per > second. > > -- > Thanks, > Ed McLain > > -Original Message- > From: Ed McLain > Sent: Tuesday, September 23, 2008 2:31 PM > To: qpsmtpd@perl.org > Subject: RE: high CPU on "lost" processes with forkserver > &

RE: FIXED -- high CPU on "lost" processes with forkserver

2008-09-26 Thread Ed McLain
box processing roughly 6-12 connections per second. -- Thanks, Ed McLain -Original Message- From: Ed McLain Sent: Tuesday, September 23, 2008 2:31 PM To: qpsmtpd@perl.org Subject: RE: high CPU on "lost" processes with forkserver Does anyone have any problems with the patch to fix t

RE: high CPU on "lost" processes with forkserver

2008-09-23 Thread Ed McLain
inal Message- From: Ed McLain Sent: Monday, September 22, 2008 10:51 AM To: Jose Luis Martinez; qpsmtpd@perl.org Subject: RE: high CPU on "lost" processes with forkserver Anything new on a fix for this bug? I seem to have quite a few connections hitting this these days. -- Thanks,

RE: high CPU on "lost" processes with forkserver

2008-09-22 Thread Ed McLain
Anything new on a fix for this bug? I seem to have quite a few connections hitting this these days. -- Thanks, Ed McLain -Original Message- From: Jose Luis Martinez [mailto:[EMAIL PROTECTED] Sent: Tuesday, April 29, 2008 11:42 AM To: qpsmtpd@perl.org Subject: Re: high CPU on "

Re: high CPU on "lost" processes with forkserver

2008-04-29 Thread Jose Luis Martinez
Peter J. Holzer escribió: On 2008-04-25 21:24:17 +0200, Jose Luis Martinez wrote: Peter J. Holzer escribió: You caught it!!! It did the trick! As I wrote previously, my guess is that both the mysql library and the tls library catch SIGPIPE but don't call the previously installed signal handler

Re: high CPU on "lost" processes with forkserver

2008-04-25 Thread Ask Bjørn Hansen
On Apr 25, 2008, at 7:19 AM, Matt Sergeant wrote: Well we could set a max session time. Or even communicate aliveness via a pipe or something. I like that ... Of course it should be abstracted somehow via the "transport" class so all this could work appropriately in each transport type

Re: high CPU on "lost" processes with forkserver

2008-04-25 Thread Peter J. Holzer
On 2008-04-25 21:24:17 +0200, Jose Luis Martinez wrote: > Peter J. Holzer escribió: > >Jose, can you try this patch? > > > >Index: lib/Qpsmtpd/TcpServer.pm > >=== > >--- lib/Qpsmtpd/TcpServer.pm(revision 876) > >+++ lib/Qpsmtpd/Tcp

Re: high CPU on "lost" processes with forkserver

2008-04-25 Thread Jose Luis Martinez
Peter J. Holzer escribió: On 2008-04-24 18:49:39 -0400, Matt Sergeant wrote: I think the core used to do something like this: Index: lib/Qpsmtpd.pm === --- lib/Qpsmtpd.pm (revision 876) +++ lib/Qpsmtpd.pm (working copy) @

Re: high CPU on "lost" processes with forkserver

2008-04-25 Thread Matt Sergeant
On 25-Apr-08, at 1:15 AM, Peter J. Holzer wrote: A better option might be to have the parent process watch for long running children and terminate them. Yup, but how long is long? If the client is trying to send a 600 MB email that will take some time ... Well we could set a max session time.

Re: forkserver and DBI (was: high CPU on "lost" processes with forkserver)

2008-04-25 Thread Steve Kemp
On Fri Apr 25, 2008 at 11:13:09 +0200, Peter J. Holzer wrote: > > I use a singleton module to gain access to the DBI instance, > > That's a good idea. I tend to use singletons pretty exclusively for DBI, project-wide config file handling, and the use of Danga's memcached (which really rock

forkserver and DBI (was: high CPU on "lost" processes with forkserver)

2008-04-25 Thread Peter J. Holzer
On 2008-04-25 09:40:01 +0100, Steve Kemp wrote: > I've seen problems with DBI under qpsmtpd, but I didn't pipe up > earlier because my symptoms are different than yours. > > I use a singleton module to gain access to the DBI instance, That's a good idea. > code almost identical to this her

Re: high CPU on "lost" processes with forkserver

2008-04-25 Thread Peter J. Holzer
On 2008-04-25 09:44:00 +0200, Jose Luis Martinez wrote: > Peter J. Holzer escribió: > >Also, explicitely setting $SIG{PIPE} = 'DEFAULT' inside of the alarm > >handler may help. > > > > Perl's default SIG_PIPE behaviour is to eat the signal (if I'm not > remembering badly). Not sure what you mea

Re: high CPU on "lost" processes with forkserver

2008-04-25 Thread Ask Bjørn Hansen
On Apr 25, 2008, at 1:40 AM, Steve Kemp wrote: Looking on perlmonks there are reports of people seeing random problems if they fork() and use DBI without special care. Yes, if you are not careful then you can end up opening a database connection in the parent process and then use it from

Re: high CPU on "lost" processes with forkserver

2008-04-25 Thread Steve Kemp
On Fri Apr 25, 2008 at 09:44:00 +0200, Jose Luis Martinez wrote: > It seems that DBI at destruction time is doing something bad... but I > can't trace the code down to what it's doing, and why only when TLS > session is active, and only (until another pattern is detected) when a > "DATA" wit

Re: high CPU on "lost" processes with forkserver

2008-04-25 Thread Jose Luis Martinez
Hanno Hecker escribió: It should be called (lib/Qpsmtpd/SMTP.pm): sub getline { my ($self, $timeout) = @_; alarm $timeout; my $line = ; # default implementation alarm 0; Didn't catch that one... didn't grep the lib files for alarm (didn't think they would be there because of possible in

Re: high CPU on "lost" processes with forkserver

2008-04-25 Thread Jose Luis Martinez
Peter J. Holzer escribió: [...] But I removed it because then alarm() features VERY heavily in the performance profiling as an expensive system call. Also I'm not sure that would help in Jose's case. The problem seems to happen either while trying to read the truncated DATA line or more like

Re: high CPU on "lost" processes with forkserver

2008-04-24 Thread Peter J. Holzer
On 2008-04-24 18:49:39 -0400, Matt Sergeant wrote: > I think the core used to do something like this: > > Index: lib/Qpsmtpd.pm > === > --- lib/Qpsmtpd.pm(revision 876) > +++ lib/Qpsmtpd.pm(working copy) > @@ -390,7 +390,10 @@

Re: high CPU on "lost" processes with forkserver

2008-04-24 Thread Matt Sergeant
On 24-Apr-08, at 5:52 PM, Brian Szymanski wrote: Ask Bjørn Hansen wrote: On Apr 24, 2008, at 11:02 AM, Charlie Brady wrote: Ask said "Yeah, this is a pretty bad bug" in March 2007, but I haven't seen anyone looking to fix it. We must be in pretty good shape when billions (or whatever) of em

Re: high CPU on "lost" processes with forkserver

2008-04-24 Thread Brian Szymanski
Ask Bjørn Hansen wrote: > > On Apr 24, 2008, at 11:02 AM, Charlie Brady wrote: > >> Ask said "Yeah, this is a pretty bad bug" in March 2007, but I >> haven't seen anyone looking to fix it. > > We must be in pretty good shape when billions (or whatever) of email > transactions are processed every da

Re: high CPU on "lost" processes with forkserver

2008-04-24 Thread Ask Bjørn Hansen
On Apr 24, 2008, at 11:02 AM, Charlie Brady wrote: Yes, it will be called if the timeout occurs while waiting to read input. But there is no timeout whenever any plugin is dispatched (and qpsmtpd spends most of its time in plugins). Ask said "Yeah, this is a pretty bad bug" in March 2007,

Re: high CPU on "lost" processes with forkserver

2008-04-24 Thread Charlie Brady
On Wed, 23 Apr 2008, Hanno Hecker wrote: On Wed, 23 Apr 2008 20:32:39 +0200 Jose Luis Martinez <[EMAIL PROTECTED]> wrote: I've been bugging aroung the Qp code to find what's going on with these processes... What was my surprise when I saw: $SIG{ALRM} = sub { print $client "421 Co

Re: high CPU on "lost" processes with forkserver

2008-04-23 Thread Hanno Hecker
On Wed, 23 Apr 2008 20:32:39 +0200 Jose Luis Martinez <[EMAIL PROTECTED]> wrote: > I've been bugging aroung the Qp code to find what's going on with these > processes... What was my surprise when I saw: > > $SIG{ALRM} = sub { > print $client "421 Connection Timed Out\n"; > ::

Re: high CPU on "lost" processes with forkserver

2008-04-23 Thread Jose Luis Martinez
Charlie Brady escribió: This is where I pop up and remind folks there's a long-standing bug that no time limits are applied to plug-in callbacks. Right now I don't know if plugin callbacks would solve this problem. I'm starting to think it is DESTROY related... I feel that plugin timeouts ar

Got the cause: high CPU on "lost" processes with forkserver

2008-04-23 Thread Jose Luis Martinez
Hi, I've been able to get the problem down to a minimum (I think). It has been a quite arduous work... And I assure you that the machines that were "hanging" the connections where doing VERY STRANGE THINGS :S The client seemed to be cutting the connection just after the DATA command (as the

Re: high CPU on "lost" processes with forkserver

2008-04-21 Thread Jose Luis Martinez
Hi, I caught the log of two of those high-CPU, hung processes. They had almost the same output. The two ended in "451 Incomplete DATA". Did a search on that: SMTP.pm:696: $self->respond(451, "Incomplete DATA"); I looked at "sub respond" code in TcpServer.pm and doesn't seem like a cand

Re: high CPU on "lost" processes with forkserver

2008-04-21 Thread Jose Luis Martinez
Jose Luis Martinez escribió: Sorry... hit Send without beeing finished :S FD 7 is the same socket as 0 and 1... I understand that these sockets are the ones that are used to talk to the client. Could this have to do with the TLS plugin. Could this have to do with the TLS plugin? (was a quest

Re: high CPU on "lost" processes with forkserver

2008-04-21 Thread Jose Luis Martinez
Jose Luis Martinez escribió: Hi, We detected high load on a server, and looks like two qpsmtpd-forkserver processes were munging up the CPU, and looks like they had been there for a long time :S It happened again! This time there were a lot more processes (10 or so)... all doing the same!

Re: high CPU on "lost" processes with forkserver

2008-04-08 Thread Jose Luis Martinez
Hanno Hecker escribió: Maybe it's related to this: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=471643 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=467298 (or any other freshclam locking db dir related bugs) Our clamd just shut down tonight, after the restart all was fine. Maybe it's ok

Re: high CPU on "lost" processes with forkserver

2008-04-07 Thread Charlie Brady
On Mon, 7 Apr 2008, Ask Bj?rn Hansen wrote: On Apr 7, 2008, at 4:45 AM, Jose Luis Martinez wrote: Running qp 0.40 with forkserver on Debian Etch. Relevant plugins are tls and qmail-queue. Any ideas on how to find the guilty plugin? Just seems like it's ignoring SIGPIPEs wh en writing. And wh

Re: high CPU on "lost" processes with forkserver

2008-04-07 Thread Hanno Hecker
On Mon, 07 Apr 2008 18:30:57 +0200 Jose Luis Martinez <[EMAIL PROTECTED]> wrote: > Ask Bjørn Hansen escribió: > > > > Strange indeed. You don't run any anti-virus plugins or spamassassin? > > Yes, clamdscan, not SVN, but Davin Carraways clamdscan posted on > 22/09/2007 to the list (Thanks Dav

Re: high CPU on "lost" processes with forkserver

2008-04-07 Thread Jose Luis Martinez
Ask Bjørn Hansen escribió: Strange indeed. You don't run any anti-virus plugins or spamassassin? Yes, clamdscan, not SVN, but Davin Carraways clamdscan posted on 22/09/2007 to the list (Thanks Davin!). BTW: would be nice if in SVN virus/clamdscan clamd_socket /var/run/clamav/clamd.ctl deny_

Re: high CPU on "lost" processes with forkserver

2008-04-07 Thread Ask Bjørn Hansen
On Apr 7, 2008, at 4:45 AM, Jose Luis Martinez wrote: Running qp 0.40 with forkserver on Debian Etch. Relevant plugins are tls and qmail-queue. Any ideas on how to find the guilty plugin? Just seems like it's ignoring SIGPIPEs wh en writing. And what could cause it? Funny thing is that the

high CPU on "lost" processes with forkserver

2008-04-07 Thread Jose Luis Martinez
Hi, We detected high load on a server, and looks like two qpsmtpd-forkserver processes were munging up the CPU, and looks like they had been there for a long time :S PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 7936 user001 25 0 21216 16m 3236 R 67 0.8 2615