Having heard some good things about SnortMonster's MessageSniffer, I decided to give it a try to see if should use it in place of SpamAssassin. I downloaded the demo yesterday, which has a rule file dated December 26, 2002. SA is running version 2.43 with RBL checks off since my inbound server does the RBL blocks I want already, and Razor2 on.
I took an archive of 471 messages flagged as spam by SpamAssassin (2.43) over the last week from my server and ran sniffer on it. It took 17.36 wall clock seconds (on P3 1.3GHz, 768MB Ram, wicked fast SCSI disk running FreeBSD 4.7). Not all of these messages are directed at me. However, of those, sniffer said 50 were not spam. Examining them reveals that 49 were indeed spam, and one was not. So SA flagged more spam messages as spam, and inadvertently flagged one ligit wanted message from that set. The ligit mail was a form letter from Verisign, so it could arguably be called junk, but it was something I needed to see. Luckily they sent it via snail mail as well. Of the 50 messages flagged by SA but not sniffer for the last week, 24 were directed at me including the ligit message (as opposed to other people on the server). Next, I scanned 482 spams that snuck through SA and reached my mailbox over the last three months (ie, scored < 7.0 in SA). That took 21.89 wall clock seconds. Of those, it said 169 were not spam. Every one of those should be marked as spam, since I manully filed them to that folder after reviewing them. However, 313 of them were flagged as spam whereas SA did not flag them. This is a significant improvement of the false negatives from SA over 3 months. To be fair, let's limit the spam that slipped thru to the last 7 days as well: 16 spams reached my mailbox the last 7 days unscathed by SpamAssassin and 6 were missed by sniffer (meaning 10 were properly flagged). Of those 6, I can see that 3 of them are "iffy". Unfortunately, some porno spams made it through both SA and sniffer. So if I used sniffer, I'd see 23-10=13 more spams and one more ligit message than I would with SA in the last week. I'm not sure if it is better or worse based on such a small sample. Given that the FN rate was much lower over the 3 month archive, I'm inclined to think a larger sample could switch the results in favor of sniffer. For now, I'm sticking with SpamAssassin with amavisd. The management load of SA hasn't been significant, and I expect the managment load for MessageSniffer would be pretty low as well. However, if sniffer integrated with amavisd-new, that would be a good thing[TM], and I'd definitely give it a try in production. Hmmm.... perhaps sniffer could be a scored test within SA... How have other people's experiences been? -- =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Vivek Khera, Ph.D. Khera Communications, Inc. Internet: [EMAIL PROTECTED] Rockville, MD +1-240-453-8497 AIM: vivekkhera Y!: vivek_khera http://www.khera.org/~vivek/ ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ Spamassassin-talk mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/spamassassin-talk
