Typically training includes a holdout dataset to watch how well training generalizes as it proceeds. With modern LLMs, there aren’t five points, there are trillions.
From: Friam <friam-boun...@redfish.com> on behalf of Pieter Steenekamp <piet...@randcontrols.co.za> Date: Thursday, January 30, 2025 at 7:55 PM To: The Friday Morning Applied Complexity Coffee Group <friam@redfish.com> Subject: Re: [FRIAM] ockham's razor losing its edge? In my experience, working with deep learning AI models can very easily lead to overtraining if you're low on data or computing resources. It's like trying to fit a fifth-order polynomial to five random data points - you'll get a perfect match for those points, but the model becomes useless for anything else. The thing is, with traditional methods, nobody would be foolish enough to use such a complex curve for so few points, but in deep learning, it's not always obvious when you're overtraining. The landscape has changed because we now have access to vast amounts of data and powerful computing resources. This allows us to train models with many parameters without them falling into the trap of overfitting. Essentially, the barriers of limited data and computation have been removed, enabling the creation of high-performing models even with complicated architectures. Einstein's "make it as simple as possible, but not simpler" remains relevant. Even with all the advancements, there's still a balance to strike - we need complexity to capture the nuances of real-world data, but not so much that we lose the model's ability to generalize to new situations. Note the message is 100% mine but I use AI to assist my writing. On Thu, 30 Jan 2025 at 19:24, Roger Critchlow <r...@elf.org <mailto:r...@elf.org>> wrote: This was in the Complexity Digest feed this morning, it looks like fun. https://www.pnas.org/doi/10.1073/pnas.2401230121 <https://www.pnas.org/doi/10.1073/pnas.2401230121> What makes a model good or bad or useful or risible? -- rec -- .- .-.. .-.. / ..-. --- --- - . .-. ... / .- .-. . / .-- .-. --- -. --. / ... --- -- . / .- .-. . / ..- ... . ..-. ..- .-.. FRIAM Applied Complexity Group listserv Fridays 9a-12p Friday St. Johns Cafe / Thursdays 9a-12p Zoom https://bit.ly/virtualfriam <https://bit.ly/virtualfriam> to (un)subscribe http://redfish.com/mailman/listinfo/friam_redfish.com <http://redfish.com/mailman/listinfo/friam_redfish.com> FRIAM-COMIC http://friam-comic.blogspot.com/ <http://friam-comic.blogspot.com/> archives: 5/2017 thru present https://redfish.com/pipermail/friam_redfish.com/ <https://redfish.com/pipermail/friam_redfish.com/> 1/2003 thru 6/2021 http://friam.383.s1.nabble.com/ <http://friam.383.s1.nabble.com/>
smime.p7s
Description: S/MIME cryptographic signature
.- .-.. .-.. / ..-. --- --- - . .-. ... / .- .-. . / .-- .-. --- -. --. / ... --- -- . / .- .-. . / ..- ... . ..-. ..- .-.. FRIAM Applied Complexity Group listserv Fridays 9a-12p Friday St. Johns Cafe / Thursdays 9a-12p Zoom https://bit.ly/virtualfriam to (un)subscribe http://redfish.com/mailman/listinfo/friam_redfish.com FRIAM-COMIC http://friam-comic.blogspot.com/ archives: 5/2017 thru present https://redfish.com/pipermail/friam_redfish.com/ 1/2003 thru 6/2021 http://friam.383.s1.nabble.com/
