-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jason,
I noticed from several emails that you are probably doing a lot of little things differently and assuming they make no difference. For instance you still haven't tried the same exact eye-rule we are using so you can't really say with complete confidence that there is no difference. If you really want to get to the bottom of this, you should not assume anything or make approximations that you believe shouldn't make much difference even if you are right in most cases. There could be one clearly wrong thing you are doing, or it could be many little things that all make it take a hit. I am curious myself what the difference is and I'm willing to help you figure it out but we have to minimize the ambiguity. I was going to suggest the random number generator next, but for these simple bots there doesn't seem to be a great deal of sensitivity to the quality of the random number generator if it's reasonable - at least for a few games. Ogo (which is almost the same as AnchorMan) has a poor quality RNG and if you play a few hundred games you will discover lot's of repeated results. With a good quality generator there have NEVER been repeated games that I have ever seen. So it could be a minor factor. One thing you mentioned earlier that bothers me is something about when you end the random simulations. AnchorMan has a limit, but it's very conservative - a game is rarely ended early and I would say 99.9% of them get played to the bitter end. Are you cheating here? I suggest you make the program as identical to mine as you can - within reason. If you are doing little things wrong they accumulate. I learned this from computer chess. Many improvements are worth 5-20 ELO and you can't even measure them without playing thousands of games - and yet if you put a few of them together it can put your program in another class. In my first marketable chess program I worked with my partner and I obsessed every day on little tiny speedups - most of them less than 5% speedups. We found 2 or 3 of these every day for weeks it seems. But we kept finding them and when we were finished with had a program about 300% faster than we started and it had a reputation at the time as being a really fast chess program. This works with strength improvements too. So you might have one big wrong thing, but you may have several little ones. So instead of continuing to explore eye rules and experiment, as much fun as it is in it's own right, it's not a productive way to find the performance problem. Start with the eye rule every program is using with great success and if you want to find something better LATER, more power to you. If you want my help, I will try to make a very generic version of AnchorMan that doesn't have any enhancements - just the clearly stated basics which I believe will probably play around 1300-1400 ELO. By the way, Anchorman on the new server seems to be weaker than on the old server - so that might explain some of the difference. - - Don Jason House wrote: > I guess it really depends on what the point of the test is. I'm trying > to understand the performance gap between my AMAF bot(s) and Don's AMAF > bots. For comparison, here's the ratings and # of simulations: > > ELO > 1434 - ControlBoy - 5000 simulations per move > 1398 - SuperDog - 2000 simulations per move > 1059 - ReadyFreddy - 256 simulations per move > 763 - Dodo - 64 simulations per move > <600 - all my amaf - 5000-25000 simulations per move > <300 - ego110_allfirst- ??? > > Looking at the cross table with ReadyFreddy, running (that's doing 5% of > the work that my bots are), the results are 0/14, 0/20, 0/24, and 0/10. > Even with the small samples, I'm quite certain that the performance of > my bot is way worse than any of Don's. > > I'm not particularly concerned if alternate eye method #1 is marginally > better than #2 (or vice versa). I'm reasonably confident that their > performance is similar and that their performance is better than my > original method. > > I'm content for now to find out the major causes of performance gaps and > then revisit what is truly the best combo when I get around to doing > quality coding of features instead of quick hacks for testing. > Currently, both the random move selection strategy and the game scoring > strategy have come under question. > > On 9/20/07, [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>* > <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> wrote: > > I'm going to echo Cenny's comment. Small samples like this can be > very misleading. For this kind of test, I usually give each > algorithm 5000 playouts per move and let them play 1000 games on my > computer. It takes about a day and a half. > > - Dave Hillis > > > > -----Original Message----- > From: Cenny Wenner <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> > To: computer-go <computer-go@computer-go.org > <mailto:computer-go@computer-go.org>> > Sent: Tue, 18 Sep 2007 3:33 pm > Subject: Re: [Housebot-developers] [computer-go] ReadyFreddy on CGOS > > By the data in your upper table, the results need to uphold their mean > > for 40 times as many trials before you even get a significant* > difference between #1 and #2. > > Which are the two methods you used? > > On 9/18/07, Jason House < > [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> wrote: > > original eye method = 407 ELO > > alt eye method #1 = 583 ELO > > alt eye method #2 = 518 ELO > > > > While both alternate methods are probably better than the original, I'm > not > > > convinced there's a significant difference between the two alternate > > methods. The cross-tables for both are fairly close and could be luck > of > > the draw (and even which weak bots were on at the time). I put raw > numbers > > > below. Since I made one other change when doing the alt eye method, I > > should rerun the original with that other change as well (how I end > random > > playouts and score them to allow for other eye definitions). > > > > > While I think the alternate eye definitions helped, I don't think they > > accounted for more than 100-200 ELO > > > > vs ego110_allfirst > > orig= 33/46 = 71% > > #1 = 17/20 = 85% > > > #2 = 16/18 = 89% > > > > vs gotraxx-1.4.2a > > orig=N/A > > #1 = 2/8 = 25% > > #2 = 3/19 = 16% > > > > > > On 9/17/07, Jason House < > [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> > wrote: > > > > > > > > > > > > On 9/17/07, Don Dailey < > [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> wrote: > > > > Another way to test this, to see if this is your problem, is for > ME to > > > > implement YOUR eye definition and see if/how much it hurts > AnchorMan. > > > > > > > > > I'm pretty much swamped with work today - but I may give this a try > at > > > > some point. > > > > > > > > > > I'd be interested in seeing that. It looks like my first hack at an > > > alternate eye implementation bought my AMAF version about 150 ELO (not > > tested with anything else). Of course, what I did isn't what others are > > using. I'll do another "alteye" version either today or tomorrow. It > may > > > be possible that some of my 150 was because I changed the lengths of the > > random playouts. > > > > > > > > > _______________________________________________ > > computer-go mailing list > > > computer-go@computer-go.org <mailto:computer-go@computer-go.org> > > http://www.computer-go.org/mailman/listinfo/computer-go/ > > > > > -- > Cenny Wenner > _______________________________________________ > computer-go mailing list > computer-go@computer-go.org > <mailto:computer-go@computer-go.org> > http://www.computer-go.org/mailman/listinfo/computer-go/ > > ------------------------------------------------------------------------ > *Check Out the new free AIM(R) Mail* > <http://o.aolcdn.com/cdn.webmail.aol.com/mailtour/aim/en-us/index.htm> > -- Unlimited storage and industry-leading spam and email virus > protection. > > _______________________________________________ > computer-go mailing list > computer-go@computer-go.org <mailto:computer-go@computer-go.org> > http://www.computer-go.org/mailman/listinfo/computer-go/ > > > > ------------------------------------------------------------------------ > > _______________________________________________ > computer-go mailing list > computer-go@computer-go.org > http://www.computer-go.org/mailman/listinfo/computer-go/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFG8+vIDsOllbwnSikRAt9xAKDPeN2yx4kzgVwtO4Ff4/gzH9XD2gCglgFC Y5d9XLIK0AGtubZFzy0X7MI= =tID1 -----END PGP SIGNATURE----- _______________________________________________ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
