Hi Frank, Thanks for your feedback! But I think we are talking about two different things.
1) Validation: The generalization performance of the classifier. See, for example, "Studies on the Validation of Internal Rating Systems" by BIS. 2) Calibration: Correct calibration of a PD rating system means that the calibrated PD estimates are accurate and conform to the observed default rates. See, for instance, An Overview and Framework for PD Backtesting and Benchmarking, by Castermans et al. Frank, you are referring the #1 and I am referring to #2. Nonetheless, I would never create a rating system if my model doesn't discriminate better than a coin toss. Regards, Pedro -----Original Message----- From: Frank E Harrell Jr [mailto:[EMAIL PROTECTED] Sent: Tuesday, October 07, 2008 11:02 AM To: Rodriguez, Pedro Cc: [EMAIL PROTECTED]; r-help@r-project.org Subject: Re: [R] How to validate model? [EMAIL PROTECTED] wrote: > Usually one validates scorecards with the ROC curve, Pietra Index, KS > test, etc. You may be interested in the WP 14 from BIS (www.bis.org). > > Regards, > > Pedro No, the validation should be done using an absolute reliability (calibration) curve. You need to verify that at all levels of predicted risk there is agreement with the true probability of failure. An ROC curve does not do that, and I doubt the others do. A resampling-corrected loess calibration curve is a good approach as implemented in the Design package's calibrate function. Frank > > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] > On Behalf Of Maithili Shiva > Sent: Tuesday, October 07, 2008 8:22 AM > To: r-help@r-project.org > Subject: [R] How to validate model? > > Hi! > > I am working on scorecard model and I have arrived at the regression > equation. I have used logistic regression using R. > > My question is how do I validate this model? I do have hold out sample > of 5000 customers. > > Please guide me. Problem is I had never used Logistic regression earlier > neither I am used to credit scoring models. > > Thanks in advance > > Maithili > > ______________________________________________ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > > ______________________________________________ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > -- Frank E Harrell Jr Professor and Chair School of Medicine Department of Biostatistics Vanderbilt University ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
