On Mon, Nov 26, 2012 at 12:58 AM, Darren Cook <[email protected]> wrote:
> > better than another with 95% of confidence. However, my adviser is asking > > me about not only the STATISTICAL significance of the results, but also > the > > PRACTICAL significance of them. I mean, if one system is, for example > only > > 1% better than another, with 99% of confidence, the result would have a > > statistical significance, but wouldn't really matter in a practical > sense. > > It all comes down to defining "practical significance", which is > subjective; but still a useful line of thought. > > I have thought of this before too. ELO is probably the right way and I have always used 200 ELO (in my mind) to define what I consider a "signficant superiority." In chess a "class" is 200 ELO wide and represents a "clear" superiority so it's almost like a standard anyway. In statistical terms a 200 ELO superiority give you about 76% chance of winning. Another way to look at this is that with 200 ELO you might not win every game, but you are quite likely to win even a short match. Of course it's arbitrary how you define a clear superiority. From another practical perspective I have noticed that it takes well over 100 ELO for most people to be able to judge that one player is better than another simply from watching it play. In fact people are so subjective and so influenced by style that they will often judge the weaker player to be the stronger one if there is something about how it plays that appeals to them. Another way to see this is to estimate how many games is required before you can measure a given superiority with close to statistical certainty, and for 200 ELO it does not take very many games. Don > I started thinking a definition could be: in any *short* series of games > (e.g. best of 5) you'd be confident it would come out ahead. That would > be my definition of clear superiority. 1% better isn't enough, as you > need hundreds of games to be sure of seeing a difference. > > When you said a "23% difference" did you mean the win-ratio is > 61.5:38.5? If so, what is the probability the stronger player will get > 3+ wins from 5 games? If over 0.9 I think you can make the case that the > strength difference is of practical significance. > > Or, yes, do it all in ELO ratings, and decide on how many ELO points > feels significant. > > Darren > > > -- > Darren Cook, Software Researcher/Developer > > http://dcook.org/work/ (About me and my work) > http://dcook.org/blogs.html (My blogs and articles) > _______________________________________________ > Computer-go mailing list > [email protected] > http://dvandva.org/cgi-bin/mailman/listinfo/computer-go >
_______________________________________________ Computer-go mailing list [email protected] http://dvandva.org/cgi-bin/mailman/listinfo/computer-go
