Re: [R] Custom caret metric based on prob-predictions/rankings

2012-02-10 Thread Yang Zhang
(I couldn't find answers to this question in the documentation) On Fri, Feb 10, 2012 at 11:59 AM, Yang Zhang wrote: > Sorry for not being more clear - I'm interested in accessing these > indices from within the trainControl summaryFunction, not afterward > (from the train object). > > As for the

Re: [R] Custom caret metric based on prob-predictions/rankings

2012-02-10 Thread Yang Zhang
Sorry for not being more clear - I'm interested in accessing these indices from within the trainControl summaryFunction, not afterward (from the train object). As for the weights, I'm referring to the weights argument passed into train. On Fri, Feb 10, 2012 at 5:50 AM, Max Kuhn wrote: > I think

Re: [R] Custom caret metric based on prob-predictions/rankings

2012-02-10 Thread Max Kuhn
I think you need to read the man pages and the four vignettes. A lot of your questions have answers there. If you don't specify the resampling indices, they ones generated for you are saved in the train object: > data(iris) > TrainData <- iris[,1:4] > TrainClasses <- iris[,5] > > knnFit1 <- train

Re: [R] Custom caret metric based on prob-predictions/rankings

2012-02-10 Thread Yang Zhang
Actually, is there any way to get at additional information beyond the classProbs? In particular, is there any way to find out the associated weights, or otherwise the row indices into the original model matrix corresponding to the tested instances? On Thu, Feb 9, 2012 at 4:37 PM, Yang Zhang wro

Re: [R] Custom caret metric based on prob-predictions/rankings

2012-02-09 Thread Yang Zhang
Oops, found trainControl's classProbs right after I sent! On Thu, Feb 9, 2012 at 4:30 PM, Yang Zhang wrote: > I'm dealing with classification problems, and I'm trying to specify a > custom scoring metric (recall@p, ROC, etc.) that depends on not just > the class output but the probability estimat