UAI members
Thank you for your many responses. You've provided at least 5 distinct answers
which I summarize below.
(Answer 5 below is clearly correct, but leads me to a new quandary.)
Answer 1: "70% chance of snow" is just a label and conceptually should be
treated as "XYZ". In other words don't be fooled by the semantics inside the
quotes.
My response: Technically correct, but intuitively unappealing. Although I
often council people on how often intuition is misleading, I just couldn't
ignore my intuition on this one.
Answer 2: The forecast "70% chance of snow is ill-defined"
My response: I agree, but in this case I was more concerned about the conflict
between math and intuition. I would be willing to accept any well-defined
forecasting statement.
Answer 3: The reference set "winter days" is the wrong reference set.
My response: I was just trying to give some justification to my subjective
prior. But this answer does point out a distinction between base rates and
subjective priors. This distinction relates to my new quandary below so please
read on.
Answer 4: The problem inherently requires more variables and cannot be treated
as a simple single evidence with two hypotheses problem.
My response: Actually I was concerned that this was the answer. As it may have
implied that using Bayes to evaluate a single evidence item was impractical for
the community of analysts I'm working with. Fortunately ...
Answer 5: The problem statement was inherently incoherent. Many of you
pointed out that if TWC predicts "70% snow" on 10% of the days that it snows
and on 1% of days that it does not snow, and a 5% base rate for snow, then the
P("70% snow" & snow) is .005 and P("70% snow" & ~snow) = .0095. So for the
days that TWC says "70% snow" it actually snows on a little over 34% of the
days. Clearly my assertion that TWC is calibrated is incoherent relative to
the rest of the problem statement. The problem was not underspecified, it was
over specified. (I hope I did the math correctly.)
My response: Thanks for pointing this out. I'm embarrassed that I didn't
notice this myself. Though this clearly solves my initial concern it leads me
to an entirely new quandary.
Consider the following revised version.
The TWC problem
1. Question: What is the chance that it will snow next Monday?
2. My subjective prior: 5%
3. Evidence: The Weather Channel (TWC) says there is a "70% chance of
snow" on Monday.
4. TWC forecasts of snow are calibrated.
Notice that I did not justify by subjective prior with a base rate.
>From P(S)=.05 and P(S|"70%") = .7 I can deduce that P("70%"|S)/P("70%"|~S) =
>44.33. So now I can "deduce" from my prior and evidence odds that P(S|"70%")
>= .7. But this seems silly. Suppose my subjective prior was 20%. Then
>P("70%"|S)/P("70%"|~S) = 9.33333 and again I can "deduce" P(S|"70%")=.7.
My latest quandary is that it seems odd that my subjective conditional
probability of the evidence should depend on my subjective prior. This may be
coherent, but is too counter intuitive for me to easily accept. It would also
suggest that when receiving a single evidence item in the form of a judgment
from a calibrated source, my posterior belief does not depend on my prior
belief. In effect, when forecasting snow, one should ignore priors and listen
to The Weather Channel.
Is this correct? If so, does this bother anyone else?
paull
From: uai-boun...@engr.orst.edu [mailto:uai-boun...@engr.orst.edu] On Behalf Of
Lehner, Paul E.
Sent: Friday, February 13, 2009 4:29 PM
To: uai@ENGR.ORST.EDU
Subject: [UAI] A perplexing problem
I was working on a set of instructions to teach simple
two-hypothesis/one-evidence Bayesian updating. I came across a problem that
perplexed me. This can't be a new problem so I'm hoping someone will clear
things up for me.
The problem
5. Question: What is the chance that it will snow next Monday?
6. My prior: 5% (because it typically snows about 5% of the days during
the winter)
7. Evidence: The Weather Channel (TWC) says there is a "70% chance of
snow" on Monday.
8. TWC forecasts of snow are calibrated.
My initial answer is to claim that this problem is underspecified. So I add
9. On winter days that it snows, TWC forecasts "70% chance of snow" about
10% of the time
10. On winter days that it does not snow, TWC forecasts "70% chance of snow"
about 1% of the time.
So now from P(S)=.05; P("70%"|S)=.10; and P("70%"|S)=.01 I apply Bayes rule and
deduce my posterior probability to be P(S|"70%") = .3448.
Now it seems particularly odd that I would conclude there is only a 34% chance
of snow when TWC says there is a 70% chance. TWC knows so much more about
weather forecasting than I do.
What am I doing wrong?
Paul E. Lehner, Ph.D.
Consulting Scientist
The MITRE Corporation
(703) 983-7968
pleh...@mitre.org<mailto:pleh...@mitre.org>
_______________________________________________
uai mailing list
uai@ENGR.ORST.EDU
https://secure.engr.oregonstate.edu/mailman/listinfo/uai
