On Mon, 23 Feb 2009, Francisco Javier Diez wrote:
> Konrad Scheffler wrote:
> > I agree this is problematic - the notion of calibration (i.e. that you can
> > say P(S|"70%") = .7) does not really make sense in the subjective Bayesian
> > framework where different individuals are working with different priors,
> > because different individuals will have different posteriors and they can't
> > all be equal to 0.7.
>
> I apologize if I have missed your point, but I think it does make sense. If
> different people have different posteriors, it means that some people will
> agree that the TWC reports are calibrated, while others will disagree.
I think this is another way of saying the same thing - if you define the
concept of calibration such that people will, depending on their priors,
disagree over whether the reports are calibrated then it is still
problematic to prescribe calibration in the problem formulation - because
this will mean different things to different people. Unless you take
"TWC is calibrated" to mean "everyone has the same prior as TWC", which I
don't think was the intention in the original question.
In my opinion the source of confusion here is the use of a subjective
Bayesian framework (i.e. one where the prior is not explicitly stated and
is assumed to be different for different people). If instead we use an
objective Bayesian framework where all priors are stated explicitly, the
difficulties disappear.
> Who is right? In the case of unrepeatable events, this question would not make
> sense, because it is not possible to determine the "true" probability, and
> therefore whether a person or a model is calibrated or not is a subjective
> opinion (of an external observer).
>
> However, in the case of repeatable events--and I acknowledge that
> repeatability is a fuzzy concept--, it does make sense to speak of an
> objective probability, which can be identified with the relative frequency.
> Subjective probabilities that agree with the objective probability (frequency)
> can be said to be correct and models that give the correct probability for
> each scenario will be considered to be calibrated.
>
> If we accept that "snow" is a repeatable event, the all the individuals should
> agree on the same probability. If it is not, P(S|"70%") may be different for
> each individual because having different priors and perhaps different
> likelihoods or even different structures in their models.
I strongly disagree with this. The ("true") relative frequency is not the
same thing as the correct posterior. One can imagine a situation where the
correct posterior (calculated from the available information) is very far
from the relative frequency which one would obtain given the opportunity
to perform exhaustive experiments.
Probabilities (in any variant of the Bayesian framework) do not describe
reality directly, they describe what we know about reality (typically in
the absence of complete information).
> Coming back to the main problem, I agree again with Peter Szolovits in making
> the distinction between likelihood and posterior probability.
>
> a) If I take the TWC forecast as the posterior probability returned by a
> calibrated model (the TWC's model), then I accept that the probability of snow
> is 70%.
>
> b) However, if I take "70% probability of snow" as a finding to be introduced
> in my model, then I should combine my prior with the likelihood ratio
> associated with this finding, and after some computation I will arrive at
> P(S|"70%") = 0.70. [Otherwise, I would be incoherent with my assumption that
> the model used by the TWC is calibrated.]
>
> Of course, if I think that the TWC's model is calibrated, I do not need to
> build a model of TWC's reports that will return as an output the same
> probability estimate that I introduce as an input.
>
> Therefore I see no contradiction in the Bayesian framework.
But this argument only considers the case where your prior is identical
to TWC's prior. If your prior were _different_ from theirs (the more
interesting case) then you would not agree that they are calibrated.
_______________________________________________
uai mailing list
uai@ENGR.ORST.EDU
https://secure.engr.oregonstate.edu/mailman/listinfo/uai
