Thank you Jim I want to evaluate if there is a significant difference between several ML models I have just run. Evaluation metric is Mean Absolute Error (MAE). Values in brackets are MAE values of a particular model (e.g. RF) for different folds of k-fold CV. I found that 'a' performed better (in terms of MAE score) than the other models i.e. b, c, d, e.
But is there any significant difference ? I have used Wilcoxon test like below: wilcox.test(a, b, paired = T, conf.int = T, conf.level = 0.95) , then wilcox.test(a, c, paired = T, conf.int = T, conf.level = 0.95), wilcox.test(a, d, paired = T, conf.int = T, conf.level = 0.95) and so on, I compared the best model with all the other models for statistical differences. Can I do a test which compares it in just one time (and one step) and not repeating the wilcoxon test to compare the best model with all others. a= c(0.3, 0.5, ..... 0.4) b= c(0.4, 0.6,...... 0.2) .... .. f= c(0.1, 0.4, .....0.5) On Mon, Feb 28, 2022 at 11:14 PM Jim Lemon <drjimle...@gmail.com> wrote: > Hi neha, > Ouch! I've just been run over by homework. What you are probably being > asked to do is an "omnibus test", but I can't figure out what you are > actually testing. Maybe that phrase will help you in _your_ search for > the answer. > > Jim > > On Tue, Mar 1, 2022 at 8:56 AM Neha gupta <neha.bologn...@gmail.com> > wrote: > > > > Hello everyone, > > > > I have 'mean absolute error' values from different folds of CV of 6 ML > > models.. > > a= c(0.3, 0.5, ..... 0.4) > > b= c(0.4, 0.6,...... 0.2) > > .... > > .. > > f= c(0.1, 0.4, .....0.5) > > Now algorithm 'a' performed better than all the others (in terms of mean > > absolute error scores) and then I used wilcoxon signed rank test to > compare > > 'a' with all other models for significance difference. > > > > Now, I have been asked to compare all the models with all others for > > significant differences at once and not compare one with the other, how > can > > I do it, plz guide me? At this time, I just know about the Wilcoxon > signed > > rank test. > > Regards > > > > [[alternative HTML version deleted]] > > > > ______________________________________________ > > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see > > https://stat.ethz.ch/mailman/listinfo/r-help > > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > > and provide commented, minimal, self-contained, reproducible code. > [[alternative HTML version deleted]] ______________________________________________ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
