Very nice piece by Rich Sutton! Re NanoGPT. Even though I'm retired, I wanted to learn more about how the LLMs work. So I started a project in which a number of students and i work through Karpathy's nanoGPT. We've just started, but I think it will be worth the effort. Karpathy uses non-trivial Python. Many students don't know Python at all. So learning Python along with its non-trivial features as well as becoming familiar with PyTorch, is something of a challenge for the project. Students (and I, in fact) were not prepared for the amount of scaffolding background knowledge one had to build for oneself.
The amount fo work required to build the intellectual scaffolding almost always seems to be an unanticipated problem when learning something new. In many cases the scaffolding is not just intellectual. Think of what it takes to learn to play musical instrument! -- Russ On Fri, Feb 10, 2023 at 10:09 AM Jochen Fromm <j...@cas-group.net> wrote: > Interesting article. I have found this link to "GPT in 60 Lines of NumPy" > on Hacker News > https://jaykmody.com/blog/gpt-from-scratch/ > > It is similar to nanoGPT and minGPT from Andrej Karpathy, who has joined > OpenAI again recently > https://github.com/karpathy/nanoGPT > > The secret is apparently not to use a complicated algorithm but just a > huge amount of data, as Rich Sutton writes here > http://www.incompleteideas.net/IncIdeas/BitterLesson.html > > -J. > > > -------- Original message -------- > From: Russ Abbott <russ.abb...@gmail.com> > Date: 2/10/23 5:53 PM (GMT+01:00) > To: The Friday Morning Applied Complexity Coffee Group <friam@redfish.com> > > Subject: [FRIAM] A very good piece about ChatGPT. > > It's in the New Yorker > <https://www.newyorker.com/tech/annals-of-technology/chatgpt-is-a-blurry-jpeg-of-the-web> > and intended for a non-scientific audience. > >
-. --- - / ...- .- .-.. .. -.. / -- --- .-. ... . / -.-. --- -.. . FRIAM Applied Complexity Group listserv Fridays 9a-12p Friday St. Johns Cafe / Thursdays 9a-12p Zoom https://bit.ly/virtualfriam to (un)subscribe http://redfish.com/mailman/listinfo/friam_redfish.com FRIAM-COMIC http://friam-comic.blogspot.com/ archives: 5/2017 thru present https://redfish.com/pipermail/friam_redfish.com/ 1/2003 thru 6/2021 http://friam.383.s1.nabble.com/
