Rémi Coulom wrote: > Don Dailey wrote: >> Hi Rémi, >> >> For a while I have considered overhauling the rating system for >> CGOS. My system is ad-hoc and based on gradually increasing K >> factor based on >> your opponents K in the standard ELO formula. >> I don't know if your idea here is feasible for a computer server, >> because presumably the players are fixed in strength, but in practice I >> think some bots change. Anyway, I'm no expert on this but want to >> find something better than what I'm doing and I have considered using >> some kind of whole history approach (such as running bayeselo after >> every round on every game, which of course is not very scalable :-) >> >> - Don >> >> > > Hi Don, > > Maybe you could consider implementing Glicko. Glicko is described there: > http://math.bu.edu/people/mg/glicko/glicko.doc/glicko.html > It should be better than any intuitive hand-made formula you could > come up with. > > Bayeselo would probably produce better ratings than Glicko. Running > Bayeselo from scratch after every round may be too costly. But it is > possible to make very efficient incremental updates: adding a few > games, and running a couple of iterations of MM should be extremely > fast. This would require keeping bayeselo in memory all the time, with > current game results. Since it cannot be done with the current program > you'd have to use my C++ code and somehow incorporate it into the > server software. This would be complicated, and may use a significant > amount of memory on the server. But computation time would be very > short (less than 0.001 second). > > The algorithm I describe in my paper may be overkill for rating > programs. If you look at table 1, you'll see that even when rating > humans, Bayeselo outperforms Glicko. Since most programs on CGOS are > constant, I believe that Bayeselo would be very difficult to beat. I much prefer bayeselo to Glicko. But it is a simple choice. From my reading of the reference you gave, it doesn't seem like the most appropriate choice for CGOS, it addresses issues that are probably of minor consequence on CGOS.
Actually, it's pretty feasible to run bayeselo between rounds at the moment - but it may not remain so forever unless I make some changes like you describe. After all the games are played, there is a significant wait before the next round is scheduled, and I use that time for bookkeeping chores anyway. I thought of using only the last year of data too, but I want this to be "all time", not "all year." Using only the last year has other problems too, a program might have a zillion games today but next month drop to nothing - in other words it would not appear very stable. There is another option too. I can run the rating calculations off-line (or as a separate process.) I hesitate to do this because we are accustomed to seeing a game played, then an updated rating. So if I did this, or if I did updates less frequently such as once per 12 hours or so, I would want to have an indicator of how many unrated games are in the queue for each program. But for a server, I don't really think this is really a very good way to do it. Of course I use bayeselo once a month to build the All Time ratings list. I've been wanting to get into the source code because bayeselo is interactive, and because of this not so easy to use in an automated process. Because of that, I had to wrap it up inside a simple to use script that takes command line arguments including sgf files. Internally, the script converts the sgf to PGN files because that's the easiest way to use it when the sgf files are available. I've been procrastinating dealing with the code directly because I dread wading in to figure it out! I don't currently have any overwhelming need. But I would love to see it accept sgf files and command line arguments and not act like an interactive shell. I think I could probably convert it to this usage as an option (still maintaining current functionality but have a switch to over-ride this. ) The server is written in tcl and it's not difficult incorporating C code, most high level languages have an interface for doing that. But I have been debating a chance to lua for quite some time because it's so efficient. In either case, it's not difficult to incorporate C code, but I'm not as sure about C++. You can also use SWIG which is very good at wrapping up functionality to create packages for high level languages. It's not that hard in either case to build a package (written in C) that is usable by the server. - Don > > Rémi > _______________________________________________ > computer-go mailing list > computer-go@computer-go.org > http://www.computer-go.org/mailman/listinfo/computer-go/ > _______________________________________________ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
