I guess it really depends on what the point of the test is. I'm trying to understand the performance gap between my AMAF bot(s) and Don's AMAF bots. For comparison, here's the ratings and # of simulations:
ELO 1434 - ControlBoy - 5000 simulations per move 1398 - SuperDog - 2000 simulations per move 1059 - ReadyFreddy - 256 simulations per move 763 - Dodo - 64 simulations per move <600 - all my amaf - 5000-25000 simulations per move <300 - ego110_allfirst- ??? Looking at the cross table with ReadyFreddy, running (that's doing 5% of the work that my bots are), the results are 0/14, 0/20, 0/24, and 0/10. Even with the small samples, I'm quite certain that the performance of my bot is way worse than any of Don's. I'm not particularly concerned if alternate eye method #1 is marginally better than #2 (or vice versa). I'm reasonably confident that their performance is similar and that their performance is better than my original method. I'm content for now to find out the major causes of performance gaps and then revisit what is truly the best combo when I get around to doing quality coding of features instead of quick hacks for testing. Currently, both the random move selection strategy and the game scoring strategy have come under question. On 9/20/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > > I'm going to echo Cenny's comment. Small samples like this can be very > misleading. For this kind of test, I usually give each algorithm 5000 > playouts per move and let them play 1000 games on my computer. It takes > about a day and a half. > > - Dave Hillis > > > -----Original Message----- > From: Cenny Wenner <[EMAIL PROTECTED]> > To: computer-go <computer-go@computer-go.org> > Sent: Tue, 18 Sep 2007 3:33 pm > Subject: Re: [Housebot-developers] [computer-go] ReadyFreddy on CGOS > > By the data in your upper table, the results need to uphold their mean > for 40 times as many trials before you even get a significant* > difference between #1 and #2. > > Which are the two methods you used? > > On 9/18/07, Jason House <[EMAIL PROTECTED]> wrote: > > original eye method = 407 ELO > > alt eye method #1 = 583 ELO > > alt eye method #2 = 518 ELO > > > > While both alternate methods are probably better than the original, I'm not > > convinced there's a significant difference between the two alternate > > methods. The cross-tables for both are fairly close and could be luck of > > the draw (and even which weak bots were on at the time). I put raw numbers > > below. Since I made one other change when doing the alt eye method, I > > should rerun the original with that other change as well (how I end random > > playouts and score them to allow for other eye definitions). > > > > While I think the alternate eye definitions helped, I don't think they > > accounted for more than 100-200 ELO > > > > vs ego110_allfirst > > orig= 33/46 = 71% > > #1 = 17/20 = 85% > > #2 = 16/18 = 89% > > > > vs gotraxx-1.4.2a > > orig=N/A > > #1 = 2/8 = 25% > > #2 = 3/19 = 16% > > > > > > On 9/17/07, Jason House <[EMAIL PROTECTED] > wrote: > > > > > > > > > > > > On 9/17/07, Don Dailey <[EMAIL PROTECTED]> wrote: > > > > Another way to test this, to see if this is your problem, is for ME to > > > > implement YOUR eye definition and see if/how much it hurts AnchorMan. > > > > > > > > I'm pretty much swamped with work today - but I may give this a try at > > > > some point. > > > > > > > > > > I'd be interested in seeing that. It looks like my first hack at an > > alternate eye implementation bought my AMAF version about 150 ELO (not > > tested with anything else). Of course, what I did isn't what others are > > using. I'll do another "alteye" version either today or tomorrow. It may > > be possible that some of my 150 was because I changed the lengths of the > > random playouts. > > > > > > > > > _______________________________________________ > > computer-go mailing list > > computer-go@computer-go.org > > http://www.computer-go.org/mailman/listinfo/computer-go/ > > > > > -- > Cenny Wenner > _______________________________________________ > computer-go mailing list > [EMAIL PROTECTED]://www.computer-go.org/mailman/listinfo/computer-go/ > > ------------------------------ > *Check Out the new free AIM(R) > Mail*<http://o.aolcdn.com/cdn.webmail.aol.com/mailtour/aim/en-us/index.htm>-- > Unlimited storage and industry-leading spam and email virus protection. > > _______________________________________________ > computer-go mailing list > computer-go@computer-go.org > http://www.computer-go.org/mailman/listinfo/computer-go/ >
_______________________________________________ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
