don't use asymptotic normality with a sample size 5, use Fisher's exact test
the p-value for the rejection of "P(alpha-Go wins a given game against Lee Sedol)<.5" might be something like 3/16 (under the "independent coin" assumption!) this is not 0.05, but still quite an impressive result :-) with 5-0 it would have been <0.05. On Wed, Mar 30, 2016 at 6:59 PM, Ryan Hayward <hayw...@ualberta.ca> wrote: > Hey Simon, > > I only now remembered: > > we actually experimented on the effect > of making 1 blunder (random move instead of learned/searched move) > in Go and Hex > > "Blunder Cost in Go and Hex" > > so this might be a starting point for your question > of measuring player strength by measuring > all move strengths... > > https://webdocs.cs.ualberta.ca/~hayward/papers/blunder.pdf > > On Wed, Mar 30, 2016 at 5:29 AM, Lucas, Simon M <s...@essex.ac.uk> wrote: > >> In my original post I put a link to >> the relevant section of the MacKay >> book that shows exactly how to calculate >> the probability of superiority >> assuming the game outcome is modelled as >> a biased coin toss: >> >> http://www.inference.phy.cam.ac.uk/itila/ >> >> >> I was making the point that for this >> >> and for other outcomes of skill-based games >> we can do so much more (and as humans we intuitively >> DO do so much more) than just look at the event >> outcome - and maybe as a community we should do that more >> routinely and more quantitatively (e.g. >> by analysing the quality of each move / action) >> >> Best wishes, >> >> Simon >> >> >> >> On 30/03/2016, 11:57, "Computer-go on behalf of djhbrown ." < >> computer-go-boun...@computer-go.org on behalf of djhbr...@gmail.com> >> wrote: >> >> >Simon wrote: "I was discussing the results with a colleague outside >> >of the Game AI area the other day when he raised >> >the question (which applies to nearly all sporting events, >> >given the small sample size involved) >> >of statistical significance - suggesting that on another week >> >the result might have been 4-1 to Lee Sedol." >> > >> >call me naive, but perhaps you could ask your colleague to calculate >> >the probability one of side winning 4 games out of 5, and then say >> >whether that is within 2 standard deviations of the norm. >> > >> >his suggestion is complete nonsense, regardless of the small sample >> >size. perhaps you could ask a statistician next time. >> > >> >-- >> >patient: "whenever i open my mouth, i get a shooting pain in my foot" >> >doctor: "fire!" >> >http://sites.google.com/site/djhbrown2/home >> >https://www.youtube.com/user/djhbrown >> >_______________________________________________ >> >Computer-go mailing list >> >Computer-go@computer-go.org >> >http://computer-go.org/mailman/listinfo/computer-go >> _______________________________________________ >> Computer-go mailing list >> Computer-go@computer-go.org >> http://computer-go.org/mailman/listinfo/computer-go >> > > > > -- > Ryan B Hayward > Professor and Director (Outreach+Diversity) > Computing Science, UAlberta > > _______________________________________________ > Computer-go mailing list > Computer-go@computer-go.org > http://computer-go.org/mailman/listinfo/computer-go > -- ========================================================= Olivier Teytaud, olivier.teyt...@inria.fr, TAO, LRI, UMR 8623(CNRS - Univ. Paris-Sud), bat 490 Univ. Paris-Sud F-91405 Orsay Cedex France http://www.slideshare.net/teytaud
_______________________________________________ Computer-go mailing list Computer-go@computer-go.org http://computer-go.org/mailman/listinfo/computer-go
