Gad Abraham wrote:
Frank E Harrell Jr wrote:
Gad Abraham wrote:
This approach leaves much to be desired. I hope that its
practitioners start gauging it by the mean squared error of
predicted probabilities.
Is the logic here is that low MSE of predicted probabilities equals a
better calibrated model? What about discrimination? Perfect calibration
Almost. I was addressed more the wish for the use of strategies that
maximize precision while keeping bias to a minimim.
implies perfect discrimination, but I often find that you can have two
That doesn't follow. You can have perfect calibration in the large
with no discrimination.
I'm not sure I understand: if you have perfect calibration, so that you
correctly assign the probability Pr(y=1|x) to each x, doesn't it follow
that the x will also be ranked in correct order of probability, which is
what the AUC is measuring?
You can have a prediction model that assigns Pr(y=1|x) to a range of
0.45 to 0.55 such that the probabilities are perfectly accurate, but the
ROC area is 0.6.
competing models, the first with higher discrimination (AUC) and
worse calibration, and the the second the other way round. Which one
is the better model?
I judge models on the basis of both discrimination (best measured with
log likelihood measures, 2nd best AUC) and calibration. It's a
two-dimensional issue and we don't always know how to weigh the two.
For many purposes calibration is a must. In those we don't look at
discrimination until calibration-in-the-small is verified at high
resolution.
By "log likelihood measures" do you mean likelihood-ratio tests?
I mean generalized R^2, log-likelihood, or the adequacy index in my book.
Frank
--
Frank E Harrell Jr Professor and Chair School of Medicine
Department of Biostatistics Vanderbilt University
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
