Re: [SAtalk] spam assassin - tru64 unix
Todd Seeleman wrote: I'm running spamd/spamc v 2.55 on a Compaq DS20 running Tru64 UNIX v 5.1b. It processes ~ 5,500 pieces of email per day. Every week or so the system slows to a point where I must reboot. I believe I've eliminated all causes other than the spamd process. I've throttled it down to spamd -d -m 2 and this has helped so far. Does anybody have any helpful info on this? I'm about to start investigating Perl on Tru64. We have two DS20Es in a TruCluster running 5.1A. SA is processing around 80,000 messages per day. Of course it's not evenly distributed and we've had severe problems when it climbs above 22,000 per hour. The load average soars to 40 or 60, and the box just crawls. If we leave it alone it will eventually recover but mail delivery is definitely impacted. Sometimes I just shut SA down for a bit to clear out the mail queues. We've done a bunch of performance tuning with HP and it helped a little. The load average is still floating around 20 though. These boxes, in addition to running SA and our MTA also support 1300 simultaneous IMAP connections so the load isn't all SA. Last week, for unrelated reasons, I decided to temporarily put spamd on a Linux box. It's a 2.6gHz Zeon box running RedHat 7.3. To my amazement the load average on the Tru64 boxes dropped to 3-4 and the load on the Linux box never exceeds 2. So, for some reason spamd is _much_ more efficient under Linux than under Tru64. My guess is it has something to do with the Perl installation but I'm not sure yet. -- ___ Rick Beebe(203) 785-6416 Manager, Systems & Network Engineering FAX: (203) 785-3481 ITS-Med Production Systems[EMAIL PROTECTED] Yale University School of Medicine Suite 124, 100 Church Street South http://its.med.yale.edu New Haven, CT 06519 ___ --- The SF.Net email is sponsored by EclipseCon 2004 Premiere Conference on Open Tools Development and Integration See the breadth of Eclipse activity. February 3-5 in Anaheim, CA. http://www.eclipsecon.org/osdn ___ Spamassassin-talk mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/spamassassin-talk
[SAtalk] SA loads & times
I just updated my production SA environment from 2.41 to 2.60 and am running into load issues. I'm running SA on a 2-node Tru64 cluster. Each node has 4 gigs of memory and dual 667mHz Alpha CPUs. We process about 150,000 messages per day but, of course, it isn't an even distribution. We can get up to 25,000 messages an hour during the day. We're using spamd/spamc with only local tests. I'm struggling with two issues. One is the amount of memory spamd is using. I had started with evilrules, popcorn, etc but had to take them back out as spamd was sucking up too much memory. Without those, it's still considerably larger than 2.4 was. The second problem is that spamd is taking longer to process the messages. It's doing more, obviously. 2.41 processed most messages in less than a second (usually 0.4 - 0.6). 2.60 is taking 1.4 - 20 seconds. The problem here is that incoming mail starts to back up. So I try to allow more spamd processes to handle it then the machine bogs down. I've been playing with -m to try to find an optimal setting. Is anyone else running SA in a largish environment? I'm looking for any and all tuning hints as I really don't want to go back to 2.4. I am using a system-wide bayes. I'm not sure how much drag that adds either. -- ___ Rick Beebe(203) 785-6416 Manager, Systems & Network Engineering FAX: (203) 785-3481 ITS-Med Production Systems[EMAIL PROTECTED] Yale University School of Medicine Suite 124, 100 Church Street South http://its.med.yale.edu New Haven, CT 06519 ___ --- This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/ ___ Spamassassin-talk mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/spamassassin-talk
Re: [SAtalk] Re: SA loads & times
[EMAIL PROTECTED] wrote: Check for lock files. There seems to be a 20 second timeout if (in my case, it was the whitelist lockfile) gets stale. Look for minutes/days old lockfiles, and clean them out. I had steady 29+second spamd times, turned off all lookup etc, nothing changed. Snooped in the spamd users dir (I also run spamd on a remote machine with the default user 'pop3', so all bayes/whitelist info goes into that dir). Lo and behold a 15 minute old lockfile. deleted that one file and down to < 2 seconds per msg. dual AMD 1.2Ghz, 1Gb RAM, 3 UW-SCSI 9Gb HDDs RH 9.0. Thanks. I'll keep an eye on that. In my case I'm only using a global directory--there are no per-user files. No AWL and a single global bayes. I'll keep an eye out for lock files there. -- _______ Rick Beebe(203) 785-6416 Manager, Systems & Network Engineering FAX: (203) 785-3481 ITS-Med Production Systems[EMAIL PROTECTED] Yale University School of Medicine Suite 124, 100 Church Street South http://its.med.yale.edu New Haven, CT 06519 ___ --- This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/ ___ Spamassassin-talk mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/spamassassin-talk
Re: [SAtalk] SA loads & times
Rick Beebe wrote: I just updated my production SA environment from 2.41 to 2.60 and am running into load issues. I'm running SA on a 2-node Tru64 cluster. Each node has 4 gigs of memory and dual 667mHz Alpha CPUs. We process about 150,000 messages per day but, of course, it isn't an even distribution. We can get up to 25,000 messages an hour during the day. We're using spamd/spamc with only local tests. I'm struggling with two issues. One is the amount of memory spamd is using. I had started with evilrules, popcorn, etc but had to take them back out as spamd was sucking up too much memory. Without those, it's still considerably larger than 2.4 was. The second problem is that spamd is taking longer to process the messages. It's doing more, obviously. 2.41 processed most messages in less than a second (usually 0.4 - 0.6). 2.60 is taking 1.4 - 20 seconds. The problem here is that incoming mail starts to back up. So I try to allow more spamd processes to handle it then the machine bogs down. I've been playing with -m to try to find an optimal setting. Answering my own question: I discovered that much of this was caused by file-locking problems. [EMAIL PROTECTED] put me on the right trail. Basically, the copies of spamd running on each node of the cluster were sharing a single directory. There were problems with conflicting lock files as well as conflicting updates to the bayes database. I fixed it by creating separate spamd user home directories for each machine. The Bayes databases will eventually get out of sync but since both machines are handling about half the spam they should be reasonably close. I am still seeing higher load averages and I did under 2.41 but the machine isn't grinding to a halt anymore. -- _______ Rick Beebe(203) 785-6416 Manager, Systems & Network Engineering FAX: (203) 785-3481 ITS-Med Production Systems[EMAIL PROTECTED] Yale University School of Medicine Suite 124, 100 Church Street South http://its.med.yale.edu New Haven, CT 06519 ___ --- This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/ ___ Spamassassin-talk mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/spamassassin-talk
[SAtalk] "Naughty" test names
I'd like to appeal to the SA collective to change the name of the PENIS_ENLARGE tests to something a little more innocuous. Apparently some people find it offensive to see it in their email. I've received the odd complaint about the test name--usually when it shows up in a message that isn't about body part enhancement. I've been ignoring them, but apparently there is a larger client rumbling going on and now it's gotten to HR who are receiving complaints that it's sexual harrasement. All of which is ridiculous, so please don't go there. But I still have to deal with it. I asked for this once before and was told to zero out the test and put a replacement in my local.cf. I don't want to do that because then I lose the collective wisdom of this group. I've had some other thoughts on how to fix it, but a fork in the eye is against company policy. So I've taken to doing a search and replace changing it to BIGGER_BITS. It will be a minor pain to do that every time I upgrade, but it's doable. It doesn't seem like it should be a big deal to make that change globally, though. Is it? Thanks for your consideration. --Rick --- This SF.Net email sponsored by: Free pre-built ASP.NET sites including Data Reports, E-commerce, Portals, and Forums are available now. Download today and enter to win an XBOX or Visual Studio .NET. http://aspnet.click-url.com/go/psa0016ave/direct;at.asp_061203_01/01 ___ Spamassassin-talk mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/spamassassin-talk
Re: [spamassassin] Re: [SAtalk] "Naughty" test names
Don Krause wrote: How can "PENIS" be offensive? It's a medically correct term. Just offer to turn off SA for those who are so easily offended. I'm sure they'd prefer PENIS to the content of the actual mail. Yes, it's medically correct. In all the cases where someone complained, however, there was no mention of penis in the email message. They are mad at "me" for adding it. The messages were not spam and were not marked as spam. They were legitimate email messages which, for whatever reason, failed the PENIS_ENLARGE test. --Rick --- This SF.Net email sponsored by: Free pre-built ASP.NET sites including Data Reports, E-commerce, Portals, and Forums are available now. Download today and enter to win an XBOX or Visual Studio .NET. http://aspnet.click-url.com/go/psa0016ave/direct;at.asp_061203_01/01 ___ Spamassassin-talk mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/spamassassin-talk
Re: [SAtalk] compiling problems on a SGI
Dave Encisco wrote: > Hi, > > When compiling Mail-SpamAssassin-2.31 on an SGI running 6.5.15 I run > into the following error: > > cp lib/Mail/SpamAssassin/PerMsgStatus.pm blib/lib/Mail/SpamAssassin/PerMsgStatus.pm > gcc -fno-strict-aliasing -I/usr/local/include -DLANGUAGE_C -O spamd/spamc.c >\ > -o spamd/spamc -Wl,-woff,84 -L/usr/local/lib -ldb -lm -lc > In file included from spamd/spamc.c:13: > /usr/include/sysexits.h:107: warning: `EX_OK' redefined > /usr/include/unistd.h:43: warning: this is the location of the previous definition > spamd/spamc.c:50: conflicting types for `in_addr_t' > /usr/include/netinet/in.h:43: previous declaration of `in_addr_t' > *** Error code 1 (bu21) > > Has anyone had this problem? Some advice would be greatly appreciated. I had the same problem with Tru64 Unix. The quick-and-dirty fix is to comment out or delete line 50 in spamc.c, the one that says typedef unsigned long in_addr_t; --Rick --- This sf.net email is sponsored by: Jabber - The world's fastest growing real-time communications platform! Don't just IM. Build it in! http://www.jabber.com/osdn/xim ___ Spamassassin-talk mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/spamassassin-talk
Re: [SAtalk] Logging spamassassin...
> I'm currently test spamassassin (spamd/spamc) to see if we should > implement it sitewide here. Is there a way to log how many mails that > spamassassin tags as spam and how many that aren't counted as spam. If > it's possible I also want to log the score so we can have our own little > high score here :) I use the attached Perl script which grabs the numbers (including highest spam score) from my syslog files. The script accepts data from stdin. I use a shell script which basically does "spamcount.pl < mail.log" and then mails the output to me. The reason for having a separate count for each 'host' is that I'm combining logs from cluster members. --Rick #!/usr/local/bin/perl -w # Takes the file from standard input looking for spamd 'clean' lines # and counts how many were counted and how many were spam my %total; my %spam; my %prob_spam; my $highest = 0; my $host; my ($total, $spam, $prob_spam) = 0; while (<>) { next if ! /\s(\w+)\sspamd/; $host = $1; next if $host eq ''; if (/(clean message|identified spam)\s+\(([\d\-\.]+)\/([\d\-\.]+)\)/) { $total{$host}++; $spam{$host}++ if $2 >= $3; $prob_spam{$host}++ if $2 >= 5; $highest = $2 if $2 > $highest; } } print
Re: [SAtalk] spamassassin site wide
Theo Van Dinter wrote: > On Mon, Oct 14, 2002 at 01:06:26PM -0700, Seby Varghese wrote: > >>I have the SpamAssassin installed site wide and would >>like to know there is a way to configure that if the >>user is in white list don't do any other checks. We >>need this because 80% of the mail handled by our mail > > There is no current way to do this in SA. You'd want to configure > whatever you have running SA to not run SA if given conditions exist. :( It seems to me that there could be a major performance boost if SA did white/blacklist checking first and if the message met either criteria it just skipped all the rest of the tests. Is that a bad idea for some reason? --Rick --- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf ___ Spamassassin-talk mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/spamassassin-talk
Re: [SAtalk] spamd uses a lot of CPU
Scott Lambert wrote: On Wed, Oct 23, 2002 at 10:01:21AM -0400, Chris Santerre wrote: "Of course, you probably need to use a userprefs database in that case, rather than the userprefs in each user's home directory." Aside from moving the process to another system, Are you saying that seting up the users in a database is a faster process then putting configs in user directories? I haven't completed my setup for the company. I've just been testing. I was going to use mailertable for people without custom settings, and just setup users who wanted custom in regular home dirs. I've only got about 100-120 email addresses to handle. So maybe this doesn't even matter to me. I have learned to dislike NFS dependencies and user home directories on boxes that don't absolutely need them. I will take a database over those options anyday. The database also gives you one place to look for/change settings. Most of my users are lucky to know which mouse button to use at any time. Editing userprefs files would blow their minds. Eventually I will put up a web frontend so they can tweak their own settings in the database but, until then, it is not a major burden to us to add the entries for the users who need/want special settings. We've been having good luck using MySQL for the userprefs. I used the php-sa-mysql web front-end with a few minor tweaks. --Rick --- This sf.net email is sponsored by: Influence the future of Java(TM) technology. Join the Java Community Process(SM) (JCP(SM)) program now. http://ads.sourceforge.net/cgi-bin/redirect.pl?sunm0002en ___ Spamassassin-talk mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/spamassassin-talk