I have been reporting the pop3 high load a long time ago (1 year), till now we still don't have a solid solution, I will try you patch.
I am not a experienced programmer, but from what I understand, one should use non-blocking methode with accept() and select(), but this is more for the incoming traffic as for outgoing the programmer should take care of the writes and buffer. I have the impression that the writes to the buffer is quicker thant emptying the buffer and somthing is trying to fill the buffer all the time, please correct me if I am wrong just my 2 cents Ming-Wei > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] Behalf > Of Jesse Norell > Sent: Monday, October 06, 2003 11:18 PM > To: dbmail@dbmail.org > Subject: Re: [Dbmail] 100% cpu usage with dbmail-pop3d > > > Hello, > > Please try the attached patch and see if it helps. I think it has > some, but maybe not as much as I thought originally. > > Anyone really familiar with network programming? I'm wondering if > this just needs to put in a select() loop for the writes. Does a > constant fwrite() loop end up blocking with high cpu usage when the > tcp send buffer is full, vs. using a select() loop to only write when > it wouldn't block? I can try to test that, but it'd be nice if > someone with experience there would share their thoughts/experience. > > Thanks, > Jesse > > > > ---- Original Message ---- > From: Jesse Norell <dbmail@dbmail.org> > To: dbmail@dbmail.org > Subject: Re: [Dbmail] 100% cpu usage with dbmail-pop3d > Sent: Mon, 6 Oct 2003 14:14:19 -0600 (MDT) > > > > > Hello, > > > > In very little testing, that seems to have fixed or greatly helped > > the problem. I changed it to a 4096 byte buffer, flushing every > > 2048 bytes ... I'll try to get a simple patch made that you can test > > too. (Will have to pull out a few of other patches first...) > > > > > > > > ---- Original Message ---- > > From: Jesse Norell <dbmail@dbmail.org> > > To: dbmail@dbmail.org > > Subject: Re: [Dbmail] 100% cpu usage with dbmail-pop3d > > Sent: Mon, 6 Oct 2003 12:44:48 -0600 (MDT) > > > > > > > > Ok, taking a quick look at it again, this is what I gather: > > > > > > The time is spent in db_send_message_lines when sending a > RETR reply > > > from pop3.c calling: > > > > > > fprintf ((FILE *)stream,"+OK %llu octets\r\n",((struct > message *)tmpelement->data)->msize); > > > return db_send_message_lines ((void *)stream, ((struct > message *)tmpelement->data)->realmessageid,-2, 0); > > > > > > > > > Looking in dbpgsql.c, in db_send_message_lines, we allocate a > > > 1MB buffer (ie. READ_BLOCK_SIZE from db.h * 2) and read the db > > > return data into it one char at a time, handling \n -> \r\n > > > conversion and escaping periods, and then flush that > buffer at every > > > 3000 characters, then memset the whole 1MB buffer back to null, > > > in this block: > > > > > > if (rowlength%3000==0) /* purge buffer at > every 3000 bytes */ > > > { > > > /* fprintf ((FILE *)fstream,"%s",buffer); */ > > > /* fflush ((FILE *)fstream); */ > > > > > > fwrite (buffer, sizeof(char), > strlen(buffer), (FILE *)fstream); > > > > > > /* cleanup the buffer */ > > > memset (buffer, '\0', (READ_BLOCK_SIZE*2)); > > > } > > > > > > > > > I don't know how long it takes to write to 1MB of memory, but > > > that seems a bit inefficient. I might see if I can use a smaller > > > working buffer and see if that fixes it. Any avid C coders see > > > that as an immediate alarm condition? > > > > > > jn > > > > > > > > > ---- Original Message ---- > > > From: Jesse Norell <dbmail@dbmail.org> > > > To: dbmail@dbmail.org > > > Subject: Re: [Dbmail] 100% cpu usage with dbmail-pop3d > > > Sent: Mon, 6 Oct 2003 12:11:37 -0600 (MDT) > > > > > > > > > > > Hello, > > > > > > > > Yes, that's exactly what it acts like. I looked over > the code a > > > > bit once and nothing stood out as definitely wrong, but I really > > > > don't do a lot of C network coding, so it could easily be. > > > > > > > > > > > > ---- Original Message ---- > > > > From: Bret Baptist <dbmail@dbmail.org> > > > > To: dbmail@dbmail.org > > > > Subject: Re: [Dbmail] 100% cpu usage with dbmail-pop3d > > > > Sent: Mon, 6 Oct 2003 12:48:16 -0500 > > > > > > > > > Sarge is testing. > > > > > > > > > > Well I think we can rule out the postgres libs, since > I am running MySQL. I > > > > > am thinking it must be the SQL code they are using, > instead of idling while > > > > > waiting for the client to download, it is just taking > all the cpu. > > > > > > > > > > Just silly. > > > > > > > > > > > > > > > Bret. > > > > > > > > > > > > > > > > > > > > On Monday 06 October 2003 12:36 pm, Jesse Norell wrote: > > > > > > Hello, > > > > > > > > > > > > Take comfort in the fact that you're not alone > ... but I don't > > > > > > have a solution for you. This is exactly what > we're seeing after > > > > > > having upgraded to debian unstable (is unstable now > sarge?). I > > > > > > alluded to this briefly in a reply the other day - > I think the > > > > > > problem has to either be in the linux kernel or the > postgres libs, > > > > > > not dbmail itsself, but maybe would be possible to > work around > > > > > > it if it could be nailed down. We've not had much > time to look > > > > > > into it, just have enough pop3 front ends that we > can put it off > > > > > > for a while. It has persisted through a few kernel > updates (and > > > > > > tried smp kernel, same effects), and at least one > postgres lib > > > > > > update also... > > > > > > > > > > > > > > > > > > > > > > > > ---- Original Message ---- > > > > > > From: Bret Baptist <dbmail@dbmail.org> > > > > > > To: dbmail@dbmail.org > > > > > > Subject: [Dbmail] 100% cpu usage with dbmail-pop3d > > > > > > Sent: Mon, 6 Oct 2003 12:24:21 -0500 > > > > > > > > > > > > > I recently switched over to a separate dbmail > front end server and > > > > > > > database server. When I set the new front end > server up I used debian > > > > > > > sarge. Since that time I have been having huge > problems with load being > > > > > > > caused by dbmail-pop3d. It appears to attempt to > use all the available > > > > > > > cpu while transferring messages to the client. > I had never seen this > > > > > > > before when running the front end and database on > the same server. > > > > > > > > > > > > > > Does anyone have any clues? This is where the > logs are at when it is > > > > > > > taking 100% cpu: > > > > > > > Oct 6 12:02:52 destiny dbmail/pop3d[20773]: > pop3():RETR command, > > > > > > > retrieving message > > > > > > > Oct 6 12:02:52 destiny dbmail/pop3d[20773]: > pop3(): RETR command, > > > > > > > selecting message > > > > > > > Oct 6 12:02:52 destiny dbmail/pop3d[20773]: > db_send_message_lines(): > > > > > > > request for [-2] lines > > > > > > > Oct 6 12:02:52 destiny dbmail/pop3d[20773]: > db_send_message_lines(): > > > > > > > executing query [SELECT * FROM messageblks WHERE > message_idnr=2458388 > > > > > > > ORDER BY messageblk_idnr ASC] > > > > > > > Oct 6 12:02:52 destiny dbmail/pop3d[20773]: > db_query(): executing > > > > > > > [SELECT * FROM messageblks WHERE > message_idnr=2458388 ORDER BY > > > > > > > messageblk_idnr ASC] Oct 6 12:02:52 destiny > dbmail/pop3d[20773]: > > > > > > > db_send_message_lines(): sending [-2] lines from > message [2458388] > > > > > > > Oct 6 12:02:52 destiny dbmail/pop3d[20773]: > db_send_message_lines(): > > > > > > > getting nextblock [1] > > > > > > > > > > > > > > I am running the latest CVS version available, > and still seeing this > > > > > > > issue. > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Bret Baptist > > > > > > > Systems and Technical Support Specialist > > > > > > > [EMAIL PROTECTED] > > > > > > > Internet Exposure, Inc. > > > > > > > http://www.iexposure.com > > > > > > > > > > > > > > (612)676-1946 x17 > > > > > > > Web Development-Web Marketing-ISP Services > > > > > > > ------------------------------------------ > > > > > > > > > > > > > > > > > > > > > Today is the tomorrow you worried about yesterday. > > > > > > > > > > > > > > _______________________________________________ > > > > > > > Dbmail mailing list > > > > > > > Dbmail@dbmail.org > > > > > > > https://mailman.fastxs.nl/mailman/listinfo/dbmail > > > > > > > > > > > > -- End Original Message -- > > > > > > > > > > > > > > > > > > -- > > > > > > Jesse Norell > > > > > > jesse (at) kci.net > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > Dbmail mailing list > > > > > > Dbmail@dbmail.org > > > > > > https://mailman.fastxs.nl/mailman/listinfo/dbmail > > > > > > > > > > -- > > > > > Bret Baptist > > > > > Systems and Technical Support Specialist > > > > > [EMAIL PROTECTED] > > > > > Internet Exposure, Inc. > > > > > http://www.iexposure.com > > > > > > > > > > (612)676-1946 x17 > > > > > Web Development-Web Marketing-ISP Services > > > > > ------------------------------------------ > > > > > > > > > > > > > > > Today is the tomorrow you worried about yesterday. > > > > > > > > > > _______________________________________________ > > > > > Dbmail mailing list > > > > > Dbmail@dbmail.org > > > > > https://mailman.fastxs.nl/mailman/listinfo/dbmail > > > > > > > > > -- End Original Message -- > > > > > > > > > > > > -- > > > > Jesse Norell > > > > jesse (at) kci.net > > > > > > > > > > > > _______________________________________________ > > > > Dbmail mailing list > > > > Dbmail@dbmail.org > > > > https://mailman.fastxs.nl/mailman/listinfo/dbmail > > > > > > > -- End Original Message -- > > > > > > > > > -- > > > Jesse Norell > > > jesse (at) kci.net > > > > > > > > > _______________________________________________ > > > Dbmail mailing list > > > Dbmail@dbmail.org > > > https://mailman.fastxs.nl/mailman/listinfo/dbmail > > > > > -- End Original Message -- > > > > > > -- > > Jesse Norell > > jesse (at) kci.net > > > > > > _______________________________________________ > > Dbmail mailing list > > Dbmail@dbmail.org > > https://mailman.fastxs.nl/mailman/listinfo/dbmail > > > -- End Original Message -- > > > -- > Jesse Norell > jesse (at) kci.net > >
