Hi, At 11:15 07.09.04 +0530, you wrote: >At 06/09/04 22:15 (), Erwin Hoffmann wrote: >>Hi, >> >>At 20:11 06.09.04 +0530, you wrote: >> >Dear Erwin, >> > >> >Sorry for question not really related to Vpopmail. >> > >> >It seems that I am hit by "Silly Qmail (Queue) Syndrome". >> > >> >I am using the Spamcontrol Patch v2.2.12 along with vpopmail-5.4.6, but >> >have not used the experimental "bigtodo". >> > >> >Wished to apply the bigtodo. I would like to get clarified that whether you >> >bigtodo is based on ext_todo patch or big-todo patch or both. I had not >> >initially compiled the bigtodo thinking that it is experimental. >> > >> >What do you suggest. >> >>Well. At first you have to tell why you think you are hit by the "Silly >>Qmail Syndrom". Any hints ? >> >>Second. Apart from the big-todo enhencement, my implementation of Andre >>Oppermann's performance enhancements dont work well. After investigation a >>look of time and testing I didn't find any significant performance >>improvement. >>Note: The code in SPAMCONTROL is not the ext-big-todo; however it is based >>of Andre's first suggestion to influence qmail's scheduler for mail >>processing; which was buggy by itself. >> >>Third. The best thing is to avoid bounces to non-existing accounts. >>Use my RECIPIENTS extension as part of Qmail or perhaps the "real-rcptto" >>patch. >> >>The forthcoming SPAMCONTROL version will include verion 0.42 of the >>RECIPIENTS extension; check my Qmail page (http://www.fehcom.de/qmail.html). >> >>regards. >>--eh. > >Hi Erwin, > >Thanks for nice reply. > >I am attaching "Queue Size" graph (5 Minute Average) updated Tuesday, 7 >September 2004 at 0:50 (EDT). > >You can notice between 0400 - 1000 hrs (EDT) a quite high Mail Queue. >During that time period the smtpd is running to the tune of 100/100. But >the send is running to the tune of local 3/15 remote 5/40. The "messages in >queue but not yet preprocessed" goes on increasing in wild. When the smtpd >runs to the tune of 85/100 its all okay. This has started happening on >almost every start of the week, when huge volume of genuine + virus >infected customers mails start pouring in.
Ok. Until now, you did not tell us what hardware and network connection you have. Anyway. My experience using a 2*1G PIII and fast SCSI Disks on FreeBSD show some similar behavior. >I am not using "RECIPIENTS extension", but using "badrcptto" for >whitelisting mechanism, which works very well (might be a bit slow due to >the reason that lookup is being done into txt database). Ok. Good choice. >I am also using >http://linux.voyager.hr/ucspi-tcp/tcpserver-limits-2004-07-25.diff patch to >limit concurrent connection from single IP. This helps identifying Virus >trodden computers and denying them connection (it's a boon). Good. >I also have Caching-DNS on this Server (djbdns). Excellent. >About the todo patches the comments of Dave Sill (of Qmail Handbook fame) >are interesting to note in the thread: > >"Outbound email rate slows when inbound rate is high" >http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&c2coff=1&threadm=e6c47de 7.0310091325.147cade4%40posting.google.com&rnum=2&prev=/groups%3Fq%3Dext-tod o%26hl%3Den%26lr%3D%26ie%3DUTF-8%26c2coff%3D1%26selm%3De6c47de7.0310091325.1 47cade4%2540posting.google.com%26rnum%3D2 Dave is right. No doubt. >Also one can have a look at the thread >"ext-todo and big-todo patches" >http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&c2coff=1&threadm=wx0lm56 pfo0.fsf%40sws5.ctd.ornl.gov&rnum=1&prev=/groups%3Fhl%3Den%26lr%3D%26ie%3DUT F-8%26c2coff%3D1%26q%3Ddave%2Bsill%2Bext-todo%2Band%2Bbig-todo Ok. >I tried to apply the patch >http://www.nrg4u.com/qmail/ext_todo-20030105.patch over and above >spamcontrol but it failed at: > >patch p1 < ext_todo-20030105.patch >patching file p1 >patching file EXTTODO-INFO >patching file FILES >patching file Makefile >Hunk #2 succeeded at 713 (offset 8 lines). >Hunk #4 succeeded at 818 (offset 8 lines). >Hunk #5 succeeded at 1598 (offset 87 lines). >Hunk #6 succeeded at 1585 (offset 9 lines). >Hunk #7 succeeded at 1694 (offset 87 lines). >patching file TARGETS >Hunk #1 succeeded at 405 (offset 20 lines). >patching file hier.c >Hunk #1 succeeded at 115 with fuzz 1 (offset 7 lines). >patching file install-big.c >patching file qmail-send.c >Hunk #1 FAILED at 1215. >Hunk #2 succeeded at 1527 (offset 88 lines). >Hunk #3 succeeded at 1662 (offset 20 lines). >Hunk #4 succeeded at 1797 (offset 88 lines). >1 out of 4 hunks FAILED -- saving rejects to file qmail-send.c.rej >patching file qmail-start.c >patching file qmail-todo.c Yes. I did not dare to include the ext-big-todo into SPAMCONTROL (yet). It breaks the philosphy of Qmail. >It also might be possible that my disk speed be contributing a bit to the >bottleneck. Current iostat figures are (not taken at the silly qmail >syndrome time, I would take that figure to when I am hit again). >iostat -d -x /dev/hda2 2 >Linux 2.4.18-18.7.x (slsp-da4p21) 09/07/2004 > >Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s >avgrq-sz avgqu-sz await svctm %util >/dev/hda2 0.03 2.27 1.85 0.37 14.98 21.73 7.49 10.86 >16.57 0.13 208.66 171.08 3.79 > >Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s >avgrq-sz avgqu-sz await svctm %util >/dev/hda2 0.00 0.00 2.00 0.00 16.00 0.00 8.00 0.00 >8.00 0.15 75.00 50.00 1.00 > >Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s >avgrq-sz avgqu-sz await svctm %util >/dev/hda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >0.00 0.00 0.00 0.00 0.00 > >Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s >avgrq-sz avgqu-sz await svctm %util >/dev/hda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >0.00 0.00 0.00 0.00 0.00 > >I am getting more and more convinced that ext-todo might be a possible >solution in this situation. >What do you feel ! Well. Its hard to tell and certainly a "feeling" is not enough to solve your situation. You did already a lot of diagnosis and most of your attempts are ok (as far as I can tell). As mentioned before, you have to explain a little a) what hardware you are using b) what is your average (and not mean) email volume c) what Anti-Virus and Anti-Spam tools are you using d) what is your general Qmail setup e) what is the rate of inbound and outbound traffic >From what I can see from your graph, you have typically around 400 email in the queue. Use my "newanalyse" package + qmailanaloge and you get a nice summary statistics. Some hints: - It might me worthwilhe to reduce the incoming-concurrency. Drop it to 30. Actually, the result of this is opposite from what it seems: This helps to better "arbiter" qmail-smtpd and thus improves overall performance. - Most of load originates presumably from I/0. You should try everything to reduce it. In particular, use SPAMCONTROLs badmimetype filter in the first place. However, if you use the qmail-queue extension and lets say qmail-scanner, this wont help much (for reasons I will explain in SPAMCONTROL 2.3). Use my QMVC instead. regards. --eh. PS: Very interesting discussion. Dr. Erwin Hoffmann | FEHCom | http://www.fehcom.de/ Wiener Weg 8, 50858 Cologne | T: +49 221 484 4923 | F: ...24
