See page 238 "Host phenotypes of parasite genes" in "The Extended Phenotype" for a naturalistic existence proof. <https://archive.org/details/RichardDawkinsTheExtendedPhenotypeTheLongReachOfTheGene1999OxfordUniversityPress/page/n237/mode/2up> This gets into the limits of self-modelling (Wolpert's theorem that is NOT the NFL theorem. Does it have a distinguishing name so people can refer to it by name and not invoke "NFL"?) and how it might relate to the equivalent of "reward hacking" that the ASI itself would be incapable of overcoming even if "detected" as it would not be "understood".
On Fri, May 2, 2025 at 3:37 PM Matt Mahoney <mattmahone...@gmail.com> wrote: > FLI (Max Tegmark et al) proposes controlling a superior intelligence > through a chain of progressively more intelligent agents. The paper claims > success rates up to 52% under some conditions in a series of game > simulations where there is a 400 point ELO difference (90% chance of the > better player winning), and lower as the advantage increases. > > https://arxiv.org/abs/2504.18530 > > It's not the first time I have seen the idea proposed, although it is the > first attempt I'm aware of at actually testing it. But I'm still dubious. > It's like if insects can't control humans, then maybe they could control > dogs and then the dogs would control humans. > > Or maybe it isn't a problem. I mean, our phones are already a billion > times smarter than us. So what if they control us as long as it feels like > we control them? > > -- Matt Mahoney, mattmahone...@gmail.com > *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + > delivery options <https://agi.topicbox.com/groups/agi/subscription> > Permalink > <https://agi.topicbox.com/groups/agi/Tab8f757c89546375-M3a7cac9d823d6867f7a604c7> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/Tab8f757c89546375-M6f339360310ffac7e4c576ed Delivery options: https://agi.topicbox.com/groups/agi/subscription
