Re: [computer-go] Congratulations to LeelaBot2 and to CzechBot
In message <[EMAIL PROTECTED]>, Gian-Carlo Pascutto
<[EMAIL PROTECTED]> writes
Evan Daniel wrote:
It is entirely within the power of the other bots to not lose on time.
I am not sure that is true.
LeelaBot should be perfectly capable of playing about 12 moves per
second in the default configuration.
However, it seems either KGS or kgsGtp do not (correctly) account for
connection lag,
I believe that KGS makes no attempt to account for connection lag.
Indeed, when I last looked into the issue of connection lag (about four
years ago, before CGOS existed), _no_ Go server did anything about
connection lag, but all good chess servers allowed for connection lag by
using clients which sent time-stamped packets. When people suggested to
the administrators of IGS and of KGS that they might use timestamps to
allow for lag, they replied that it would be too difficult and would
allow cheating by people hacking their clients to send false timestamps.
However this seems not to be a problem with chess servers; and chess
players are no more honest than other people.
or the interface adds lag of its own. If it is indeed connection lag
that is the problem, KGS is about 150ms from the tournament machine,
which means Leela can't actually play more than 6 or 7 moves per second
at best, even if the engine itself would move at infinite speed. I
think that in the actual game, the speed was closer to 2 moves per
second and this was not enough to avoid the time loss.
CGOS provides some lag compensation which removes most of the symptons
but does not actually solve the problem. People with fast connections
have more thinking time. If you have a connection hiccup at a bad
moment, you can still lose on time (this happened 1 or 2 times with
Leela in a few hundred games). But in general it is less of a problem
to play out games completely, and in fact Leela does exactly that on CGOS.
I also really do not see what HBotSVN has done wrong. Surely the engine
can't be at fault because it could not identify dead groups correctly
(if that is a requirement, we will all be unable to play tournaments
until about the time the game is solved). I also don't see what could
possibly be the objection against playing until the game is finished.
These rules are sensible for computer games and work fine on CGOS.
I do not like to cripple Leela ("make its time handling more
conservative"). I would like some advice on whether other people agree
that KGS(gtp) does not completely compensate for lag. If it does
compensate completely, then the error must be at my side and I will
focus on fixing whatever it is that causes the moving speed to be below
what it should be.
It used to be the case that KGS does not compensate at all for lag. I
am not aware that this has changed.
If it is indeed a KGS flaw I may add a workaround to Leela as simple as
doing time = time / 10 as soon as winrate >95% or so. There is still a
possibility of losing on time then but it should happen less.
At the time when I was playing with my chess engine on the chess
servers, those servers provided "timeseal" and "timestamp" programs
that compensated for lag. It was no problem to play at timecontrols of
game in 1 minute with such tools, without any workarounds.
PS. I sent a correction to the hardware but I see the report still has
the old information. LeelaBot was on 1 x Intel Xeon 5355 @ 2.66Ghz, so
"only" 4 CPUs.
Thank you for pointing this out, I have corrected the page.
Nick
--
Nick Wedd[EMAIL PROTECTED]
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] Congratulations to LeelaBot2 and to CzechBot
In message <[EMAIL PROTECTED]>, Michael Williams <[EMAIL PROTECTED]> writes I agree with Evan 100%. I probably would have gone berserk if I were Jason. Instead he handled it with relative grace, considering what was in that report. Ok. I am persuaded that I have acted wrongly here. I withdraw the probation on Jason's bot, and offer my apologies to him. I shall rewrite my report (and archive the old one, for the historic record). Nick Evan Daniel wrote: On Wed, May 7, 2008 at 11:45 AM, Nick Wedd <[EMAIL PROTECTED]> wrote: In message <[EMAIL PROTECTED]>, Jason House <[EMAIL PROTECTED]> writes Correction: HBotSVN was not reconfigured for speed in round 3. It was set to use two search threads in round 4, and was compiled in debug mode for the whole tournament. I apologize for the confusing PM's during the tournament about this. Thank you for explaining this, I have changed the report accordingly. What is "HBotSVN's technique"? Its technique is to refuse to admit that its dead groups are dead, and then to waste time in the resolution phase playing meaningless stones. This sometimes gives it a win on time, and is the only way that it wins games. This is annoying for the other competitors. I know it is not your intention that it behaves like this, but it is in your power to prevent it. It is not in my power to do anything about it, except by reassigning the results of games which it wins like this: this is the purpose of the probation. It is in Bill Shubert's power to change the way the server works so that if only one player sends a final_status_list, it will accept what that player says. I shall suggest it to him. The game end protocol says "To play in a tournament, programs must either implement both "kgs-genmove_cleanup" and "final_status_list dead", or they must play until all of their opponent's dead stones are removed from the board. It's OK if "play until dead stones removed" is an option, but they have to make sure that this option is turned on whenever they are going to be in a tournament, or they will do poorly in the tournament!". HouseBot (HBotSVN) handles this by playing until all of its opponent's dead stones are removed. "final_status_list dead" is not supported. It's kgsGtp (not HouseBot!) that insisists that all stones are alive. It annoys me every time I see the description that it's the bot that's behaving badly when it's really a problem with how the combination of kgsGtp and the KGS server represent this stuff. I have changed the wording of my report from "claimed they were alive" to "failed to admit that they were dead". I have done so because you have persuaded me that it is correct and what I said before was wrong. I do not expect you to be appeased by this. I consider it a bug in kgs that this perpetually gets misinterpreted by spectators. Please stop saying that my bot insists all of its stones are alive. This could be simplified by either fixing the game end protocol rules, or getting kgs fixed (kgsGtp and/or the server). In the round two game, it was HBotSVN that had 3 seconds left on the clock. Its opponent, MonteGNU, had almost a minute left (51 seconds). Thank you for pointing this out. I have corrected my mistake. The only games where HBotSVN's opponent got down to very little time left was the game against Leela. The whole probation thing has really pissed me off. Maybe one component of that is first finding out about it by reading it in the report. I have not been implementing "difficult things" for quite a while. Because stuff wasn't working, I suspended all forward progress on my bot two months ago. Since then, I've been building test harnesses, writing unit tests, and eliminating bugs. Did you know that weakbot50k and idiotbot don't actually handle the game end at all? Once both players pass, they switch to using gnu go. I didn't know that, but it seems a sensible, robust, solution. Nick I will no longer participate in these tournaments for the foreseeable future. I fail to see the problem with HBotSVN's behavior. It is playing according to the protocol as specified. Humans judging intent and reasonableness belong in human tournaments, and possibly human-computer tournaments, but most emphatically not in computer-computer tournaments. What would you do if HBotSVN implemented final_status_list dead and always returned the empty set? What if it only returned stones that were unsalvageable even in the face of opponent passes? HBotSVN seems to be requiring its opponents to demonstrate that they are actually capable of killing the groups they claim are dead. Given the skill level of some programs, and that programs are not offendable, this behavior seems at worst mildly rude, possibly deserving of derision and disrespect, and completely undeserving of any sort of sanction. It is entirely within the power of the other bots to not lose on time. If they cannot manage their time, that
Re: [computer-go] Congratulations to LeelaBot2 and to CzechBot
Thank you again Nick for running the tournament, and writing the report. I am disappointed though about your reaction towards housebot. I don't think it is fair to critisize the behaiviour of a bot that is not breaking any rules or guidlines. If you don't like the bots behaiviour, it seems to me that you should change the rules to outlaw that behaivior. You say that HBotSVN is unable to win by honest methods, which I don't think is true, and even if it were true, I don't think is something that should cause sanctions. I believe HBotSVN seemed weaker than it was in the last tournament because of (fixable) problems with the MC approach. Raw MC does not, I believe, work well at the beginning of a large board game, and it does not work well when the bot is a long way behind. If you are unfortunate, then these two conditions between then can cover the whole game. HBotSVN has a much better record in previous tournaments. This is damning it with faint praise, but I think it would easily beat my bot. I would prefer it if a bot's lack of strength did not lead to criticism from the tournament official. I find that I am not, at present, interested in making my bot as strong as it can be. I respect those that have this goal, but to do that, I'd have to introduce what I think of as hacks (they would not be hacks if my short term goal were bot strength). Things that I know as a human, but that my bot can't deduce itself. These hacks would hide faults in the program. My goal at the moment is more to understand the solution space of certain sub-problems. I program my bot as a hobby, for my own interest. I enjoy these tournaments, even when I don't participate. However, I don't enjoy them enough to induce me to introduce (to my eyes) hacks into my program, so if my bot being too weak, or too stubborn is likely to earn me a rebuke, I won't be entering again. I think that, even if KGS does not handle lag fairly, it is not HBotSVN's fault that LeelaBot lost on time. My preference is for a clean set of tournament rules, and for judgments to be made according to their rules, and with no moral overtones. I can understand Jason's annoyance. The tone of your report implied that his bot used dishonest methods, and you applied a sanction arbitrarily when no rule was broken, or even a warning given. My ideal outcome would be for you to remove the probation, and, if you feel it is necessary, to introduce rules to forbid the behaviour you don't like. I suspect that, as programmers, we would tend to respond better to explicit rules than to implicitly understood (or not understood) 'moral' considerations. I am sorry for the critical tone of this message. I can see that your goal is to improve the tournaments. I really am grateful for your running of these tournaments, and I hope they go from strength to strength. Best wishes Tom. On Wed, 2008-05-07 at 11:13 +0100, Nick Wedd wrote: > the winners of last Sunday's KGS bot tournament. > > My report is at http://www.weddslist.com/kgs/past/38/index.html > > Nick ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] Congratulations to LeelaBot2 and to CzechBot
You have a typo "deasd" should be "dead". My personal preferences would be to see the final section change its title from "Probation" to "Losses in Cleanup" or some other title addressing undesirable issue uncovered in this past tournament. I'd also appreciate it if the old versions of the report can be removed completely. The last thing I want is for a potential employer to google me and find one of those pages. They're very easy to misinterpret, especially given that my uniform-playout MC bot's performance was on par for 19x19 with relatively short time limits. Once upon a time, people thought MC bots would not scale to 19x19. Local biases in playouts, move ordering, and possibly progressive widening have changed all that. I hope to do that one day, but I don't yet. PS: You'll be happy to hear that I've been discussing alternate resignation strategies with people. The lack of resignation in this past tournament occurred because time it was a large board with short time limits (and my bot is slow). Everyone I've talked to seems to think the threshold I have for resigning are reasonable. What I have right now considers a single move in isolation, but it should be possible to consider a sequence of moves together. Essentially, it'll increase the available data to use in the resignation decision and avoid incorrect resignations at idiotic points. On Thu, May 8, 2008 at 4:21 AM, Nick Wedd <[EMAIL PROTECTED]> wrote: > In message <[EMAIL PROTECTED]>, Michael Williams < > [EMAIL PROTECTED]> writes > > > I agree with Evan 100%. > > > > I probably would have gone berserk if I were Jason. Instead he handled > > it with relative grace, considering what was in that report. > > > > Ok. I am persuaded that I have acted wrongly here. I withdraw the > probation on Jason's bot, and offer my apologies to him. I shall rewrite my > report (and archive the old one, for the historic record). > > Nick > > > Evan Daniel wrote: > > > > > On Wed, May 7, 2008 at 11:45 AM, Nick Wedd <[EMAIL PROTECTED]> wrote: > > > > > > > In message < > > > > [EMAIL PROTECTED]>, > > > > Jason House <[EMAIL PROTECTED]> writes > > > > > > > > > > > > Correction: HBotSVN was not reconfigured for speed in round 3. It > > > > > was > > > > > set to use two search threads in round 4, and was compiled in > > > > > debug > > > > > mode for the whole tournament. I apologize for the confusing PM's > > > > > during the tournament about this. > > > > > > > > > > Thank you for explaining this, I have changed the report > > > > accordingly. > > > > > > > > > > > > > > > > What is "HBotSVN's technique"? > > > > > > > > > > Its technique is to refuse to admit that its dead groups are > > > > dead, and then > > > > to waste time in the resolution phase playing meaningless stones. > > > > This > > > > sometimes gives it a win on time, and is the only way that it wins > > > > games. > > > > This is annoying for the other competitors. I know it is not your > > > > intention > > > > that it behaves like this, but it is in your power to prevent it. > > > > It is not > > > > in my power to do anything about it, except by reassigning the > > > > results of > > > > games which it wins like this: this is the purpose of the > > > > probation. It is > > > > in Bill Shubert's power to change the way the server works so that > > > > if only > > > > one player sends a final_status_list, it will accept what that > > > > player says. > > > > I shall suggest it to him. > > > > > > > > > > > > > > > > The game end protocol says "To play > > > > > in a tournament, programs must either implement both > > > > > "kgs-genmove_cleanup" and "final_status_list dead", or they must > > > > > play > > > > > until all of their opponent's dead stones are removed from the > > > > > board. > > > > > It's OK if "play until dead stones removed" is an option, but they > > > > > have > > > > > to make sure that this option is turned on whenever they are going > > > > > to > > > > > be in a tournament, or they will do poorly in the tournament!". > > > > > HouseBot (HBotSVN) handles this by playing until all of its > > > > > opponent's > > > > > dead stones are removed. > > > > > "final_status_list dead" is not supported. It's kgsGtp (not > > > > > HouseBot!) that insisists that all stones are alive. It annoys me > > > > > every time I see the description that it's the bot that's behaving > > > > > badly when it's really a problem with how the combination of > > > > > kgsGtp and > > > > > the KGS server represent this stuff. > > > > > > > > > > I have changed the wording of my report from "claimed they were > > > > alive" to > > > > "failed to admit that they were dead". I have done so because you > > > > have > > > > persuaded me that it is correct and what I said before was wrong. I > > > > do not > > > > expect you to be appeased by this. > > > > > > > > > > > > > > > > I consider it a bug in kgs that > > > > > this perpetually gets misinterpreted by spectator
Re: [computer-go] Congratulations to LeelaBot2 and to CzechBot
In message <[EMAIL PROTECTED]>, Jason House <[EMAIL PROTECTED]> writes You have a typo "deasd" should be "dead". Thank you, fixed. My personal preferences would be to see the final section change its title from "Probation" to "Losses in Cleanup" or some other title addressing undesirable issue uncovered in this past tournament. I have renamed this section to "Losses after Game Stop". Is this reasonable? I'd also appreciate it if the old versions of the report can be removed completely. The last thing I want is for a potential employer to google me and find one of those pages. Ok, understood, I have removed them. I too am glad to see them removed, as they reflect badly on me. Nick They're very easy to misinterpret, especially given that my uniform-playout MC bot's performance was on par for 19x19 with relatively short time limits. Once upon a time, people thought MC bots would not scale to 19x19. Local biases in playouts, move ordering, and possibly progressive widening have changed all that. I hope to do that one day, but I don't yet. PS: You'll be happy to hear that I've been discussing alternate resignation strategies with people. The lack of resignation in this past tournament occurred because time it was a large board with short time limits (and my bot is slow). Everyone I've talked to seems to think the threshold I have for resigning are reasonable. What I have right now considers a single move in isolation, but it should be possible to consider a sequence of moves together. Essentially, it'll increase the available data to use in the resignation decision and avoid incorrect resignations at idiotic points. On Thu, May 8, 2008 at 4:21 AM, Nick Wedd <[EMAIL PROTECTED]> wrote: In message <[EMAIL PROTECTED]>, Michael Williams < [EMAIL PROTECTED]> writes I agree with Evan 100%. I probably would have gone berserk if I were Jason. Instead he handled it with relative grace, considering what was in that report. Ok. I am persuaded that I have acted wrongly here. I withdraw the probation on Jason's bot, and offer my apologies to him. I shall rewrite my report (and archive the old one, for the historic record). Nick Evan Daniel wrote: On Wed, May 7, 2008 at 11:45 AM, Nick Wedd <[EMAIL PROTECTED] > wrote: In message < [EMAIL PROTECTED] >, Jason House <[EMAIL PROTECTED]> writes Correction: HBotSVN was not reconfigured for speed in round 3. It was set to use two search threads in round 4, and was compiled in debug mode for the whole tournament. I apologize for the confusing PM's during the tournament about this. Thank you for explaining this, I have changed the report accordingly. What is "HBotSVN's technique"? Its technique is to refuse to admit that its dead groups are dead, and then to waste time in the resolution phase playing meaningless stones. This sometimes gives it a win on time, and is the only way that it wins games. This is annoying for the other competitors. I know it is not your intention that it behaves like this, but it is in your power to prevent it. It is not in my power to do anything about it, except by reassigning the results of games which it wins like this: this is the purpose of the probation. It is in Bill Shubert's power to change the way the server works so that if only one player sends a final_status_list, it will accept what that player says. I shall suggest it to him. The game end protocol says "To play in a tournament, programs must either implement both "kgs-genmove_cleanup" and "final_status_list dead", or they must play until all of their opponent's dead stones are removed from the board. It's OK if "play until dead stones removed" is an option, but they have to make sure that this option is turned on whenever they are going to be in a tournament, or they will do poorly in the tournament!". HouseBot (HBotSVN) handles this by playing until all of its opponent's dead stones are removed. "final_status_list dead" is not supported. It's kgsGtp (not HouseBot!) that insisists that all stones are alive. It annoys me every time I see the description that it's the bot that's behaving badly when it's really a problem with how the combination of kgsGtp and the KGS server represent this stuff. I have changed the wording of my report from "claimed they were alive" to "failed to admit that they were dead". I have done so because you have persuaded me that it is correct and what
