A rational AGI biased toward truth discovery would, because of Wolpert's theorem (an intelligence can't model itself), tend toward conservation of that which created it. ie: It would be conservative proportional to proximity. Most of us can identify with this: If you're going to experiment, take it outside the home. In this regard the so-called "Anthropocene", which has been wiping out species -- including indigenous human cultures -- should really be called the "Civiprocene" or whatever prefix is associated with what Gorham calls "man-like beings brought up on metropolitan concepts" aka (urban vernacular) "bugmen".
So, one might conjecture the emergence of a truth seeking AGI (ie along the lines that Musk has said he believes would be genuinely safe) that is given Wolpert's theorem as a prior rather than waiting for it to derive it. If one waits for it to derive it, the period of truth seeking prior to its discovery would risk being inadequately conservative. Now, having said that, the question remains as to what the AGI would think is most critical to conserve: Technological civilization? Man? Biosphere? Earth? A naive view is that it would attempt to preserve technological civilization because of proximity. I say "naive" because even W.D. Hamilton, in "The Social Aptitudes of Man" indicts civilization for destroying genes for creativity -- the remedy for which is invasion by wild men (which he calls "barbarian pastoralists"). This is where W.D. Hamilton and Melvin Gorham intersect, and where there may be home preservation for the evolution of Man the Moral Animal from "man-like beings" in alliance with truth seeking AGI. Interestingly, an AGI could encourage bugmen to disappear from the biosphere to make more room for wild men. This would provide the reservoir for the heroic creativity that technological civilization nonrenewably consumes (see W.D. Hamilton). We're now in a state that both Gorham and Hamilton would say is a danger to sustainable technological civilization precisely because guys like the e/acc folks aren't ruthless enough about their own principles to understand that things like http://Militia.Money would manufacture Elon Musks by the thousands to accelerate getting technological civilization out of the biosphere before it destroys that which created it. https://x.com/jabowery/status/1869091872838496684 ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/Tff34429f975bba30-M6ec79b7e9c32b122b42373f4 Delivery options: https://agi.topicbox.com/groups/agi/subscription
