For what it's worth I have noticed that people who are familiar with
Bayesian filtering seem to have a mental block when it comes to
understanding this. People who know nothing about bayesian get it
instantly. Here's the actual formula.
card(Test_message intersect Spam diff Ham) minus card(Test_message intersect
Ham diff Spam)
On 08/17/16 09:16, Shawn Bakhtiar wrote:
On Aug 17, 2016, at 3:43 AM, Matus UHLAR - fantomas
<uh...@fantomas.sk <mailto:uh...@fantomas.sk>> wrote:
On 16.08.16 20:06, Marc Perkel wrote:
What I'm doing is looking for fingerprints in email that intersect
HAM and not in SPAM - which would be a HAM result.
If it matches SPAM and does NOT match HAM - then it's SPAM.
The magic is in the NOT matching on the other side.
so, if mail matches both hammy and spammy tokens (or token sets), you
don't
classify at all?
I guess what is confusing me (and I imagine others, as alluded to by
Matus) is the fact that you are describing a special condition
of Bayes' probability theorem. You are testing two variables (match
SPAM and match HAM) (not matching is simply the negation of matching)
thus giving you four conditions:
1) SPAM &&HAM
2) SPAM &&~HAM
3) ~SPAM &&HAM
4) ~SPAM &&~HAM
Here is a great diagram to show the four probable conditions:
https://en.wikipedia.org/wiki/Bayes%27_theorem#/media/File:Bayes%27_Theorem_2D.svg
So (if I am correct) Matus is asking what if condition 1 is true? How
are you classifying an email than? Which is often the state of most
emails, and thus why the use of Naive Bayes spam filtering, which
generates a probability based on Bayes' probability theorem and is the
conventional methodology to date. A Rose by any other name....
Condition 4 is obvious it's nothing you have ever seen so classifying
it anything other than HAM would be a huge mistake (IMHO), and fully
covered by the aforementioned theorem as the probability of SPAM would
(should) be 0. Same with Condition 3, obviously it never hits SPAM so
wether it matches HAM or not you're going to treat it as HAM anyway
same as condition 4.
That leaves condition 2. Which (if I'm not mistaken) is "... it
matches SPAM and does NOT match HAM - then it's SPAM.". Which brings
us back to Matus question, what if the email contains a single HAM
token? Two HAM tokens? This is exactly what Bayes' probability theorem
is designed for. All you are doing is defining a special condition in
which the HAM probability is ZERO.
I think that's were I need to understand a bit more about what HAM
means in this solution, does getting a hit on HAM somehow negate it
being SPAM completely? In other words if the email contains some set
of tokens that are SPAM, yet only one HAM token, that single HAM token
makes it not SPAM? If so, you have a long way to go in convincing me
that this is a good solution.
So if I say to you, "Let's get some lunch" that's ham because
spammers never say that, but normal people do. So the way to test
what "spammers never say" is to store what they do say and see if
it's NOT in the list. (Thus the infinite set)
Actually I get SPAM with that very set of tokes in it. If somehow the
HAM rating of it overrides the SPAM, I don't believe it would have a
desirable effect.
I get plenty of:
"
Hay Shawn,
Hope you have time to do some lunch, click on this link and check out
my new pictures!
Wannabe Phisher
"
Based on your example there's plenty of HAM and SPAM tokens in there,
"Click on this link" high probability of SPAM-e-ness, would it get
HAMed based on "hope you have time to do lunch". Or am I missing
something?
Similarly, there's only so many ways to misspell viagra, and good
email wouldn't have it spelled wrong.
Does that make sense?
Again, what you are saying makes sense in that it is special condition
of the probability theory, What does not make sense is why would you
not simply use the probability theory, that already encompasses that
condition?
--
Matus UHLAR - fantomas, uh...@fantomas.sk <mailto:uh...@fantomas.sk>
; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Linux - It's now safe to turn on your computer.
Linux - Teraz mozete pocitac bez obav zapnut.
--
Marc Perkel - Sales/Support
supp...@junkemailfilter.com
http://www.junkemailfilter.com
Junk Email Filter dot com
415-992-3400
