My original chart of numbers was created before it occurred to me that maybe the check whether the move has been played before is not needed. It is of course
possible that the check IS needed. I ran a new test after removing that check and the numbers don't look as good:
At 1 playouts per move, the modified version wins 47.8% of the time (±2.2%)
after 500 games.
At 2 playouts per move, the modified version wins 52.2% of the time (±2.2%)
after 500 games.
At 4 playouts per move, the modified version wins 56.3% of the time (±2.9%)
after 300 games.
At 8 playouts per move, the modified version wins 60.3% of the time (±2.8%)
after 300 games.
At 16 playouts per move, the modified version wins 58.3% of the time (±2.8%)
after 300 games.
At 32 playouts per move, the modified version wins 57.3% of the time (±2.9%)
after 300 games.
At 64 playouts per move, the modified version wins 54.3% of the time (±2.9%)
after 300 games.
At 128 playouts per move, the modified version wins 51.3% of the time (±2.9%)
after 300 games.
At 256 playouts per move, the modified version wins 57.0% of the time (±2.9%)
after 300 games.
At 512 playouts per move, the modified version wins 52.7% of the time (±2.9%)
after 300 games.
At 1024 playouts per move, the modified version wins 46.0% of the time (±2.9%)
after 300 games.
At 2048 playouts per move, the modified version wins 54.3% of the time (±2.9%)
after 300 games.
At 4096 playouts per move, the modified version wins 57.7% of the time (±2.9%)
after 300 games.
Don, did you retain the check in your version that has the weighted AMAF?
What ever you did, could you also add a version to the study that does the
opposite?
Mark Boon wrote:
What do those number mean exactly? Are those winning percentages against
your ref-bot with 1,000 playouts?
I'm not getting the same results as Michael does. I see some
improvement, but not nearly as much.
I had to make a few changes however, as I was adding up the results in a
different way than your ref-bot. So what I did was implement tracking
the AMAF statistics exactly the same way as in your ref-bot. Next I ran
a few hundred games, comparing the win/hit ratio of the newly
implemented method with my old one to make sure I got the same result.
When that checked out I made the modifications as described by Michael.
It seems rather straightforward, but after 200 games I only get a
win-rate of ~55% with 2,000 playouts instead of the 63% he has. That's a
considerable difference.
Possibly I introduced a little bug somewhere... Or I need a larger
test-sample.
I do agree with Michael that this idea is more logical and simpler. I
had implemented a different way to check which side moved at which
location first. I believe it's more efficient but it doesn't lend itself
for Michael's method. However, your way of checking seems rather costly
as it's a loop of 1/4 N^2 iterations, where N is the number of moves
(which we know is 111 in the beginning). With Michael's method this is
reduced to 1/2N iterations.
My ref-bot on CGOS is still going. It seems to be always within a few
ELO points of your Jref and Cref bots. So I'm going to stop it as I
don't see much point in populating CGOS with identical bots. It has
played over 500 games, which I think is enough empirical evidence that
the implementation is 'good enough'.
Mark
On 28-okt-08, at 10:54, Don Dailey wrote:
I have empirical evidence that mwRefbot level 4096 (the Michael Williams
enhanced reference bot at 4096 playouts) is stronger than the reference
bot at the same (and even higher levels.)
Rank Name Elo + - games score oppo. draws
1 mwRefbot-004096 2684 23 22 1529 78% 2140 0%
2 refbot-008192 2670 21 21 1532 73% 2274 0%
3 refbot-004096 2645 21 21 1528 71% 2251 0%
4 refbot-002048 2555 21 21 1530 63% 2237 0%
5 refbot-001024 2431 23 23 1529 54% 2178 0%
6 refbot-000512 2253 27 28 1531 47% 2066 0%
7 refbot-000256 2000 34 35 1529 42% 1932 0%
8 refbot-000128 1646 42 42 1529 45% 1655 0%
9 refbot-000064 1290 40 40 1529 50% 1321 0%
10 refbot-000032 970 40 40 1531 52% 1041 0%
11 refbot-000016 621 33 32 1529 55% 780 0%
12 refbot-000008 370 25 25 1529 47% 649 0%
13 refbot-000004 250 23 22 1529 43% 551 0%
14 refbot-000002 145 22 22 1529 35% 500 0%
15 refbot-000001 5 22 23 1528 22% 475 0%
16 refbot-000000 0 22 23 1531 22% 484 0%
- Don
_______________________________________________
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/
_______________________________________________
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/
_______________________________________________
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/
