Rémi Coulom wrote: > I believe the main problem is that the Elo-rating model is wrong for > bots. The phenomenon with Mogo is probably the same as Crazy Stone: if > there are enough strong MC bots playing to shield the top MC programs > from playing against GNU, then they'll get a high rating because they > are efficient at beating other MC bots. Otherwise, they are forced to > play against GNU, and lose points. > > For instance: > http://www.lri.fr/~teytaud/cross/CS-9-17-2CPU.html > GNU 1946 22 / 27 81.48 > GnuCvs-10 1969 26 / 31 83.87 > AyaMC637_4CPU 2108 18 / 19 94.74 I don't see that. Crazy Stone beat GnuCvs-10 83.87% - based on it's rating it should be winning just slightly more, but the ratings are very noisy. This doesn't see the least bit unusual.
Similar with GNU, a little worse than it should be doing but suppose it had won just 2 more game out of 27? It would jump up to 90% from 83.87 and you could argue that it did BETTER than it should against this bot had that happened. Low sample size. If you want to pick out data points, what about AnotherGNU37 rated 1921? CrazyStone won 100% of the games, much better than it should have done against a 1946 program if you ignore the fact that only 4 games were played. I also noticed that this version was not on the December allTime rating list which means it didn't even play 200 games. I have no doubt that there might be some intransitivities between programs, but I see no evidence that is is worse than in humans or in other computer games. - Don > > > A very easy way to get over-evaluated on CGOS is to have two versions > of the same program that play each other. For instance, if I connect > CS-2CPU and CS-8CPU, they will play most of their games against each > other, ans CS-8CPU will get an incredible rating. > > Just incorporate GNU in Don's scalability study, and the rating range > will shrink a lot. > > Rémi > _______________________________________________ > computer-go mailing list > computer-go@computer-go.org > http://www.computer-go.org/mailman/listinfo/computer-go/ > _______________________________________________ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
