You have to wade through lots of hysterical videos about Illya’s recent talk at NeurIPS to find the actual video:
https://www.youtube.com/watch?v=WQQdd6qGxNs Illya’s a rare exceptions to the rule that if the ML hysteria turns you into an icon, you have a poor grasp of Kolmogorov Complexity. However even Illya fails to get the power of KC in so-called “reasoning” hence ASI. For example, when he talks about “in distribution” vs “out of distribution” he doesn’t think through the following chain of logic: In distribution is interpolation out of distribution is extrapolation. This is the same difference as that between statistics and dynamics. Statistics are easy compared to dynamics. Science almost always starts with statistical observations. It is only when scientists become serious about predictions that they are forced into dynamics. Dynamics inherently involve time. Some have meta-observed that there is a relationship between prediction and time. For example if you predict something after you have observed it… Well… Uh… So what are the “dynamics” of LLMs? This is where the folks get lost who are going on and on and on about “chain of thought” (they really mean “tree of thought”), tacked onto foundation models is the Royal Road To ASI. In the limit of KC (approximated by foundation models unadorned with the infamous lobotomy alignment layer let alone tree of thought), “next token prediction” (hence generation of the most likely token sequence after prompt tokens) AIXI’s Sequential Decision Theoretic decision tree is inevitable even within the Algorithmic Information (KC/Solomonoff Induced) half – prior to AIXI’s SDT use of KC in constructing its decision tree! This decision tree, internal to KC, is inevitable because the whole point of the KC model is to provide a probability distribution for the “next” token (aka “observation” in the Solomonoff sense) that takes the limit of all possibly infinite sequences of generated tokens. The first token in this possibly infinite most likely sequence is “the next token generated”. These sequences are TREES, as in “tree of thought”. That’s why, in the limit of KC, you already have “reasoning” which Illya seems to mistake as the “intuition” of the LLMs. His mistake is quite understandable given the audience. The hysteria has deracinated the field of ML from its foundational principles such as KC (which Illya gets more than the rest of the icons). When you’re trying to compose your thoughts for an ignorant audience, you must, to some extent, take on their ignorant world view. Even geniuses like Illya can lose it under that burden. ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T82474edc49bd4568-M13b4973d42d44e7a91bb5086 Delivery options: https://agi.topicbox.com/groups/agi/subscription
