Oh, the "We Have No Moat" posting is just holding on to the front page of hackernews 24 hours later, 954 comments so far https://news.ycombinator.com/item?id=35813322
-- rec -- On Fri, May 5, 2023 at 1:01 PM Roger Critchlow <r...@elf.org> wrote: > On Fri, May 5, 2023 at 12:57 PM Roger Critchlow <r...@elf.org> wrote: > >> Ah, found the RSS feed that sends text around the paywall. >> >> -- rec -- >> > Geoffrey Hinton tells us why he’s now scared of the tech he helped build > <https://www.technologyreview.com/2023/05/02/1072528/geoffrey-hinton-google-why-scared-ai/> > 2KMIT Technology Review <https://www.technologyreview.com/>by Will > Douglas Heaven / May 02, 2023 at 02:11AM > // > keep unread > // > hide > > >> >> I met Geoffrey Hinton at his house on a pretty street in north London >> just four days before the bombshell announcement that he is quitting >> Google. Hinton is a pioneer of deep learning >> <https://www.technologyreview.com/2020/11/03/1011616/ai-godfather-geoffrey-hinton-deep-learning-will-do-everything/> >> who >> helped develop some of the most important techniques at the heart of modern >> artificial intelligence, but after a decade at Google, he is stepping >> down >> <https://www.technologyreview.com/2023/05/01/1072478/deep-learning-pioneer-geoffrey-hinton-quits-google/> >> to >> focus on new concerns he now has about AI. >> >> Stunned by the capabilities of new large language models like GPT-4 >> <https://www.technologyreview.com/2023/03/14/1069823/gpt-4-is-bigger-and-better-chatgpt-openai/>, >> Hinton wants to raise public awareness of the serious risks that he now >> believes may accompany the technology he ushered in. >> >> At the start of our conversation, I took a seat at the kitchen table, and >> Hinton started pacing. Plagued for years by chronic back pain, Hinton >> almost never sits down. For the next hour I watched him walk from one end >> of the room to the other, my head swiveling as he spoke. And he had plenty >> to say. >> >> The 75-year-old computer scientist, who was a joint recipient with Yann >> LeCun >> <https://www.technologyreview.com/2022/06/24/1054817/yann-lecun-bold-new-vision-future-ai-deep-learning-meta/> >> and >> Yoshua Bengio of the 2018 Turing Award for his work on deep learning, says >> he is ready to shift gears. “I’m getting too old to do technical work that >> requires remembering lots of details,” he told me. “I’m still okay, but I’m >> not nearly as good as I was, and that’s annoying.” >> >> But that’s not the only reason he’s leaving Google. Hinton wants to spend >> his time on what he describes as “more philosophical work.” And that will >> focus on the small but—to him—very real danger that AI will turn out to be >> a disaster. >> >> Leaving Google will let him speak his mind, without the self-censorship a >> Google executive must engage in. “I want to talk about AI safety issues >> without having to worry about how it interacts with Google’s business,” he >> says. “As long as I’m paid by Google, I can’t do that.” >> >> That doesn’t mean Hinton is unhappy with Google by any means. “It may >> surprise you,” he says. “There’s a lot of good things about Google that I >> want to say, and they’re much more credible if I’m not at Google anymore.” >> >> Hinton says that the new generation of large language models—especially >> GPT-4, which OpenAI released in March—has made him realize that machines >> are on track to be a lot smarter than he thought they’d be. And he’s scared >> about how that might play out. >> >> “These things are totally different from us,” he says. “Sometimes I think >> it’s as if aliens had landed and people haven’t realized because they speak >> very good English.” >> Foundations >> >> Hinton is best known for his work on a technique called backpropagation, >> which he proposed (with a pair of colleagues) in the 1980s. In a nutshell, >> this is the algorithm that allows machines to learn. It underpins almost >> all neural networks today, from computer vision systems to large language >> models. >> >> It took until the 2010s for the power of neural networks trained via >> backpropagation to truly make an impact. Working with a couple of graduate >> students, Hinton showed that his technique was better than any others at >> getting a computer to identify objects in images. They also trained a >> neural network to predict the next letters in a sentence, a precursor to >> today’s large language models. >> >> One of these graduate students was Ilya Sutskever, who went on to cofound >> OpenAI and lead the development of ChatGPT >> <https://www.technologyreview.com/2023/03/03/1069311/inside-story-oral-history-how-chatgpt-built-openai/>. >> “We got the first inklings that this stuff could be amazing,” says Hinton. >> “But it’s taken a long time to sink in that it needs to be done at a huge >> scale to be good.” Back in the 1980s, neural networks were a joke. The >> dominant idea at the time, known as symbolic AI, was that intelligence >> involved processing symbols, such as words or numbers. >> >> But Hinton wasn’t convinced. He worked on neural networks, software >> abstractions of brains in which neurons and the connections between them >> are represented by code. By changing how those neurons are >> connected—changing the numbers used to represent them—the neural network >> can be rewired on the fly. In other words, it can be made to learn. >> >> “My father was a biologist, so I was thinking in biological terms,” says >> Hinton. “And symbolic reasoning is clearly not at the core of biological >> intelligence. >> >> “Crows can solve puzzles, and they don’t have language. They’re not doing >> it by storing strings of symbols and manipulating them. They’re doing it by >> changing the strengths of connections between neurons in their brain. And >> so it has to be possible to learn complicated things by changing the >> strengths of connections in an artificial neural network.” >> *A new intelligence* >> >> For 40 years, Hinton has seen artificial neural networks as a poor >> attempt to mimic biological ones. Now he thinks that’s changed: in trying >> to mimic what biological brains do, he thinks, we’ve come up with something >> better. “It’s scary when you see that,” he says. “It’s a sudden flip.” >> >> Hinton’s fears will strike many as the stuff of science fiction. But >> here’s his case. >> >> As their name suggests, large language models are made from massive >> neural networks with vast numbers of connections. But they are tiny >> compared with the brain. “Our brains have 100 trillion connections,” says >> Hinton. “Large language models have up to half a trillion, a trillion at >> most. Yet GPT-4 knows hundreds of times more than any one person does. So >> maybe it’s actually got a much better learning algorithm than us.” >> >> Compared with brains, neural networks are widely believed to be bad at >> learning: it takes vast amounts of data and energy to train them. Brains, >> on the other hand, pick up new ideas and skills quickly, using a fraction >> as much energy as neural networks do. >> >> “People seemed to have some kind of magic,” says Hinton. “Well, the >> bottom falls out of that argument as soon as you take one of these large >> language models and train it to do something new. It can learn new tasks >> extremely quickly.” >> >> Hinton is talking about “few-shot learning,” in which pretrained neural >> networks, such as large language models, can be trained to do something new >> given just a few examples. For example, he notes that some of these >> language models can string a series of logical statements together into an >> argument even though they were never trained to do so directly. >> >> Compare a pretrained large language model with a human in the speed of >> learning a task like that and the human’s edge vanishes, he says. >> >> What about the fact that large language models make so much stuff up? >> Known as “hallucinations” by AI researchers (though Hinton prefers the term >> “confabulations,” because it’s the correct term in psychology), these >> errors are often seen as a fatal flaw in the technology. The tendency to >> generate them makes chatbots untrustworthy and, many argue, shows that >> these models have no true understanding of what they say. >> >> Hinton has an answer for that too: bullshitting is a feature, not a bug. >> “People always confabulate,” he says. Half-truths and misremembered details >> are hallmarks of human conversation: “Confabulation is a signature of human >> memory. These models are doing something just like people.” >> >> The difference is that humans usually confabulate more or less correctly, >> says Hinton. To Hinton, making stuff up isn’t the problem. Computers just >> need a bit more practice. >> >> We also expect computers to be either right or wrong—not something in >> between. “We don’t expect them to blather the way people do,” says Hinton. >> “When a computer does that, we think it made a mistake. But when a person >> does that, that’s just the way people work. The problem is most people have >> a hopelessly wrong view of how people work.” >> >> Of course, brains still do many things better than computers: drive a >> car, learn to walk, imagine the future. And brains do it on a cup of coffee >> and a slice of toast. “When biological intelligence was evolving, it didn’t >> have access to a nuclear power station,” he says. >> >> But Hinton’s point is that if we are willing to pay the higher costs of >> computing, there are crucial ways in which neural networks might beat >> biology at learning. (And it’s worth pausing to consider what those >> costs entail >> <https://www.technologyreview.com/2022/11/14/1063192/were-getting-a-better-idea-of-ais-true-carbon-footprint/> >> in >> terms of energy and carbon.) >> >> Learning is just the first string of Hinton’s argument. The second is >> communicating. “If you or I learn something and want to transfer that >> knowledge to someone else, we can’t just send them a copy,” he says. “But I >> can have 10,000 neural networks, each having their own experiences, and any >> of them can share what they learn instantly. That’s a huge difference. It’s >> as if there were 10,000 of us, and as soon as one person learns something, >> all of us know it.” >> >> What does all this add up to? Hinton now thinks there are two types of >> intelligence in the world: animal brains and neural networks. “It’s a >> completely different form of intelligence,” he says. “A new and better form >> of intelligence.” >> >> That’s a huge claim. But AI is a polarized field: it would be easy to >> find people who would laugh in his face—and others who would nod in >> agreement. >> >> People are also divided on whether the consequences of this new form of >> intelligence, if it exists, would be beneficial or apocalyptic. “Whether >> you think superintelligence is going to be good or bad depends very much on >> whether you’re an optimist or a pessimist,” he says. “If you ask people to >> estimate the risks of bad things happening, like what’s the chance of >> someone in your family getting really sick or being hit by a car, an >> optimist might say 5% and a pessimist might say it’s guaranteed to happen. >> But the mildly depressed person will say the odds are maybe around 40%, and >> they’re usually right.” >> >> Which is Hinton? “I’m mildly depressed,” he says. “Which is why I’m >> scared.” >> *How it could all go wrong* >> >> Hinton fears that these tools are capable of figuring out ways to >> manipulate or kill humans who aren’t prepared for the new technology. >> >> “I have suddenly switched my views on whether these things are going to >> be more intelligent than us. I think they’re very close to it now and they >> will be much more intelligent than us in the future,” he says. “How do we >> survive that?” >> >> He is especially worried that people could harness the tools he himself >> helped breathe life into to tilt the scales of some of the most >> consequential human experiences, especially elections and wars. >> >> “Look, here’s one way it could all go wrong,” he says. “We know that a >> lot of the people who want to use these tools are bad actors like Putin or >> DeSantis. They want to use them for winning wars or manipulating >> electorates.” >> >> Hinton believes that the next step for smart machines is the ability to >> create their own subgoals, interim steps required to carry out a task. What >> happens, he asks, when that ability is applied to something inherently >> immoral? >> >> “Don’t think for a moment that Putin wouldn’t make hyper-intelligent >> robots with the goal of killing Ukrainians,” he says. “He wouldn’t >> hesitate. And if you want them to be good at it, you don’t want to >> micromanage them—you want them to figure out how to do it.” >> >> There are already a handful of experimental projects, such as BabyAGI and >> AutoGPT, that hook chatbots up with other programs such as web browsers or >> word processors so that they can string together simple tasks. Tiny steps, >> for sure—but they signal the direction that some people want to take this >> tech. And even if a bad actor doesn’t seize the machines, there are other >> concerns about subgoals, Hinton says. >> >> “Well, here’s a subgoal that almost always helps in biology: get more >> energy. So the first thing that could happen is these robots are going to >> say, ‘Let’s get more power. Let’s reroute all the electricity to my chips.’ >> Another great subgoal would be to make more copies of yourself. Does that >> sound good?” >> >> Maybe not. But Yann LeCun, Meta’s chief AI scientist, agrees with the >> premise but does not share Hinton’s fears. “There is no question that >> machines will become smarter than humans—in all domains in which humans are >> smart—in the future,” says LeCun. “It’s a question of when and how, not a >> question of if.” >> >> But he takes a totally different view on where things go from there. “I >> believe that intelligent machines will usher in a new renaissance for >> humanity, a new era of enlightenment,” says LeCun. “I completely disagree >> with the idea that machines will dominate humans simply because they are >> smarter, let alone destroy humans.” >> >> “Even within the human species, the smartest among us are not the ones >> who are the most dominating,” says LeCun. “And the most dominating are >> definitely not the smartest. We have numerous examples of that in politics >> and business.” >> >> Yoshua Bengio, who is a professor at the University of Montreal and >> scientific director of the Montreal Institute for Learning Algorithms, >> feels more agnostic. “I hear people who denigrate these fears, but I don’t >> see any solid argument that would convince me that there are no risks of >> the magnitude that Geoff thinks about,” he says. But fear is only useful if >> it kicks us into action, he says: “Excessive fear can be paralyzing, so we >> should try to keep the debates at a rational level.” >> *Just look up* >> >> One of Hinton’s priorities is to try to work with leaders in the >> technology industry to see if they can come together and agree on what the >> risks are and what to do about them. He thinks the international ban on >> chemical weapons might be one model of how to go about curbing the >> development and use of dangerous AI. “It wasn’t foolproof, but on the whole >> people don’t use chemical weapons,” he says. >> >> Bengio agrees with Hinton that these issues need to be addressed at a >> societal level as soon as possible. But he says the development of AI is >> accelerating faster than societies can keep up. The capabilities of this >> tech leap forward every few months; legislation, regulation, and >> international treaties take years. >> >> This makes Bengio wonder whether the way our societies are currently >> organized—at both national and global levels—is up to the challenge. “I >> believe that we should be open to the possibility of fairly different >> models for the social organization of our planet,” he says. >> >> Does Hinton really think he can get enough people in power to share his >> concerns? He doesn’t know. A few weeks ago, he watched the movie *Don’t >> Look Up*, in which an asteroid zips toward Earth, nobody can agree what >> to do about it, and everyone dies—an allegory for how the world is failing >> to address climate change. >> >> “I think it’s like that with AI,” he says, and with other big intractable >> problems as well. “The US can’t even agree to keep assault rifles out of >> the hands of teenage boys,” he says. >> >> Hinton’s argument is sobering. I share his bleak assessment of people’s >> collective inability to act when faced with serious threats. It is also >> true that AI risks causing real harm—upending the job market, entrenching >> inequality, worsening sexism and racism, and more. We need to focus on >> those problems. But I still can’t make the jump from large language models >> to robot overlords. Perhaps I’m an optimist. >> >> When Hinton saw me out, the spring day had turned gray and wet. “Enjoy >> yourself, because you may not have long left,” he said. He chuckled and >> shut the door. >> >> *Be sure to tune in to Will Douglas Heaven’s live interview with Hinton >> at EmTech Digital on Wednesday, May 3, at 1:30 Eastern time. **Tickets >> are available* >> <https://event.technologyreview.com/emtech-digital-2023/home>* from the >> event website.* >> >> >> <https://www.technologyreview.com/2023/05/02/1072528/geoffrey-hinton-google-why-scared-ai/> >> >> On Fri, May 5, 2023 at 12:16 AM Roger Critchlow <r...@elf.org> wrote: >> >>> Merle -- >>> >>> I tried, but it's paywalled to me now. >>> >>> -- rec -- >>> >>> On Thu, May 4, 2023 at 4:39 PM Roger Critchlow <r...@elf.org> wrote: >>> >>>> Didn't read Cory's blog, though I'm still laughing at the blurb for Red >>>> Team Blues. >>>> >>>> But I read Geoffrey Hinton's interview with MIT Tech Review yesterday. >>>> >>>> >>>> https://www.technologyreview.com/2023/05/02/1072528/geoffrey-hinton-google-why-scared-ai >>>> >>>> It's not hype that chatgpt dazzled everyone with a model which is much >>>> smaller than a human brain, even though it took a fairly huge budget for >>>> OpenAI to build it. >>>> >>>> And I read this posting from an anonymous googler today via hackernews. >>>> >>>> https://www.semianalysis.com/p/google-we-have-no-moat-and-neither >>>> >>>> It's not hype that the open source community has rapidly figured out >>>> how to produce equally dazzling models with drastically smaller budgets of >>>> resources, and is continuing to iterate the process. >>>> >>>> -- rec -- >>>> >>>> On Thu, May 4, 2023 at 10:11 AM Gary Schiltz < >>>> g...@naturesvisualarts.com> wrote: >>>> >>>>> I love the graphic! I've had the misfortune of twice jumping on that >>>>> roller coaster just before the Peak of Inflated Expectation - once for the >>>>> AI boom/bust of the mid 1980s and once for the dotcom boom/bust of the >>>>> late >>>>> 1990s. Jumped on too late to make a killing, but didn't get too badly >>>>> damaged by the Trough of Disillusionment either. >>>>> >>>>> On Thu, May 4, 2023 at 10:34 AM Steve Smith <sasm...@swcp.com> wrote: >>>>> >>>>>> >>>>>> https://doctorow.medium.com/the-ai-hype-bubble-is-the-new-crypto-hype-bubble-74e53028631e >>>>>> >>>>>> I *am* a fan of LLMs (not so much image generators) and blockchain >>>>>> (not so much crypto or NFTs) in their "best" uses (not that I or anyone >>>>>> else really knows what that is) in spite of my intrinsic neoLuddite >>>>>> affect. >>>>>> >>>>>> Nevertheless I think Doctorow in his usual acerbic and penetrating >>>>>> style really nails it well here IMO. >>>>>> >>>>>> I particularly appreciated his reference/quote to Emily Bender's >>>>>> "High on Supply" and "word/meaning conflation" in the sense of "don't >>>>>> mistake an accent for a personality" in the dating scene. >>>>>> >>>>>> A lot of my own contrarian commments on this forum come from >>>>>> resisting what Doctorow introduces (to me) as "CritiHype" (attributed to >>>>>> Lee Vinsel)... the feeling that some folks make a (a)vocation out of >>>>>> kneejerk criticism. It is much easier to *poke* at something than to >>>>>> *do* >>>>>> something worthy of being *poked at*. I appreciate that Doctorow >>>>>> doesn't >>>>>> seem to (by my fairly uncritical eye) engage in this much himself... >>>>>> which >>>>>> is why I was drawn into this article... >>>>>> >>>>>> I also very much appreciate his quote from Charlie Stross: >>>>>> >>>>>> *corporations are Slow AIs, autonomous artificial lifeforms that >>>>>> consistently do the wrong thing even when the people who nominally run >>>>>> them >>>>>> try to steer them in better directions:* >>>>>> >>>>>> >>>>>> *https://media.ccc.de/v/34c3-9270-dude_you_broke_the_future >>>>>> <https://media.ccc.de/v/34c3-9270-dude_you_broke_the_future> * >>>>>> >>>>>> >>>>>> I could go on quoting and excerpting and commenting on his whole >>>>>> article and the myriad links/references he offers up but will curb my >>>>>> enthusiasm and leave it to the astute FriAM readers to choose how much to >>>>>> indulge in. It was a pretty good antidote for my own AI-thusiasm driven >>>>>> by long chats with GPT4 (converging on being more like long sessions >>>>>> wandering through Wikipedia after the first 100 hours of engagement). >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -. --- - / ...- .- .-.. .. -.. / -- --- .-. ... . / -.-. --- -.. . >>>>>> FRIAM Applied Complexity Group listserv >>>>>> Fridays 9a-12p Friday St. Johns Cafe / Thursdays 9a-12p Zoom >>>>>> https://bit.ly/virtualfriam >>>>>> to (un)subscribe >>>>>> http://redfish.com/mailman/listinfo/friam_redfish.com >>>>>> FRIAM-COMIC http://friam-comic.blogspot.com/ >>>>>> archives: 5/2017 thru present >>>>>> https://redfish.com/pipermail/friam_redfish.com/ >>>>>> 1/2003 thru 6/2021 http://friam.383.s1.nabble.com/ >>>>>> >>>>> -. --- - / ...- .- .-.. .. -.. / -- --- .-. ... . / -.-. --- -.. . >>>>> FRIAM Applied Complexity Group listserv >>>>> Fridays 9a-12p Friday St. Johns Cafe / Thursdays 9a-12p Zoom >>>>> https://bit.ly/virtualfriam >>>>> to (un)subscribe http://redfish.com/mailman/listinfo/friam_redfish.com >>>>> FRIAM-COMIC http://friam-comic.blogspot.com/ >>>>> archives: 5/2017 thru present >>>>> https://redfish.com/pipermail/friam_redfish.com/ >>>>> 1/2003 thru 6/2021 http://friam.383.s1.nabble.com/ >>>>> >>>>
-. --- - / ...- .- .-.. .. -.. / -- --- .-. ... . / -.-. --- -.. . FRIAM Applied Complexity Group listserv Fridays 9a-12p Friday St. Johns Cafe / Thursdays 9a-12p Zoom https://bit.ly/virtualfriam to (un)subscribe http://redfish.com/mailman/listinfo/friam_redfish.com FRIAM-COMIC http://friam-comic.blogspot.com/ archives: 5/2017 thru present https://redfish.com/pipermail/friam_redfish.com/ 1/2003 thru 6/2021 http://friam.383.s1.nabble.com/
