Re: [agi] Pick the Bit and Competitive Computing Platform - Towards a New Benchmark for AGI System Performance

2024-12-09 Thread Matt Mahoney
This is a multi player variant of the matching pennies game. The optimal strategy is to pick randomly. https://en.m.wikipedia.org/wiki/Matching_pennies It is also a proof of Wolpert's theorem, that two computers cannot mutually predict the other's actions. Imagine a variation of the game where eac

Re: [agi] Pick the Bit and Competitive Computing Platform - Towards a New Benchmark for AGI System Performance

2024-12-09 Thread swkane
I think static dataset benchmarks have their place, but they don't test everything. And maybe a meta-benchmark could be created that includes both static dataset benchmarks as well as things like competitions like PtB in sandboxed environments. I think any scientifically relevant benchmark that is

Re: [agi] Pick the Bit and Competitive Computing Platform - Towards a New Benchmark for AGI System Performance

2024-12-09 Thread James Bowery
I like it, even though it is inferior to lossless compression of Wikipedia as a standard benchmark. At least it conveys the central idea of Solomonoff Induction: Converging on the algorithm generating one's observations. In particular, I like the multi-agent "theory of mind" angle it takes which

Re: [agi] Pick the Bit and Competitive Computing Platform - Towards a New Benchmark for AGI System Performance

2024-12-09 Thread James Bowery
Is this inadequate to prevent the random agent strategy? > > Game Complexity: > >- PtB rewards non-random play by favoring agents that detect and >exploit patterns in opponents' choices. Random strategies are penalized >over time due to predictable health loss. > > >- Under the tok

Re: [agi] Pick the Bit and Competitive Computing Platform - Towards a New Benchmark for AGI System Performance

2024-12-09 Thread dissipate
On Monday, December 09, 2024, at 1:17 PM, James Bowery wrote: > Is this inadequate to prevent the random agent strategy? In addition to what you quoted, another point that I forgot to add is that in Matching Pennies you can play randomly strategically to observe your opponent without taking more

Re: [agi] Pick the Bit and Competitive Computing Platform - Towards a New Benchmark for AGI System Performance

2024-12-09 Thread dissipate
On Monday, December 09, 2024, at 4:21 PM, Bill Hibbard wrote: > FYI I've had some papers relevant to this. At AGI-08: https://www.ssec.wisc.edu/~billh/g/hibbard_agi.pdf At AGI-11: https://www.ssec.wisc.edu/~billh/g/hibbard_agi11a.pdf In JAGI as a co-author (Sam really wrote it, I just made a few

Re: [agi] Pick the Bit and Competitive Computing Platform - Towards a New Benchmark for AGI System Performance

2024-12-09 Thread Bill Hibbard via AGI
FYI I've had some papers relevant to this. At AGI-08: https://www.ssec.wisc.edu/~billh/g/hibbard_agi.pdf At AGI-11: https://www.ssec.wisc.edu/~billh/g/hibbard_agi11a.pdf In JAGI as a co-author (Sam really wrote it, I just made a few comments): https://sciendo.com/article/10.2478/jagi-2021-0001