For a gut check, I asked AI to generate a basic parser and tokenizer. It wrote it pretty much the way I would, except for a few places if I was really concerned about performance I MIGHT change. For instance, the isOperator() should have a precheck that the rune is even in the range before searching the set.
In reviewing the code, I don’t even see a single place where ‘in’ would help the readability or the performance. Clearly this has very basic functionality, and a there might be more opportunities in a production version - but I think most of the techniques used here would be performant enough - and the readability seems ok to me. The code is here https://go.dev/play/p/Zto4h7xVDE7 > On Mar 19, 2025, at 10:21 PM, Mike Schinkel <m...@newclarity.net> wrote: > >> On Mar 19, 2025, at 8:18 PM, Jason E. Aten <j.e.a...@gmail.com> wrote: >> You reach for super simple code, maybe codedgen that allows >> inlining, > > There are definitely code generators that exist for lexers and parsers — > Pigeon, goyacc, Gocc, Textmapper, etc. — and those are great projects in > their own right. > > But I am working on a fork of an existing hand-coded parser, one that was not > created with such a code generator, and that type of coding is much more > common in day-to-day work, at least for any projects I have been a part of. > > For day-to-day coding work — vs. working on a dedicated code generator > project — the idea of writing a code generator to generate a state machine > for a lexer reminds me of that old meme "The developer had a problem so they > thought 'Hey, I can use a regex for this!' Now the developer has two > problems." > >> I don't know what quirk of human nature is that causes >> so many to want to change a language to fit their whims, > > While some people view other's motivations as "whims" the people being viewed > pejoratively are often experiencing actually pain points. The pain point I > was experiencing is having to write (and later having to read) this tortured > control structure in many different places: > > switch c { > case ' ', '\t', '\n': > return Token > } > } > > Since I envisioned that Go could have a much simpler, clearer, and easier to > read alternative, it seemed to me like it would be a great fit for the Go > language especially given the more ergonomic recent improvements the Go team > made to the for loop: > > if c is ' ', '\t', '\n': > return Token > } > > This construct could have leveraged the existing multi-matching behavior of > switch-case and it felt smaller in scope but with similar utility compared to > the recent for loop enhancements. > >> when that language has as a principal virtue that it wants to be super >> readable > > Given the specific pain point that was my motivation to request a language > enhancement, I find that comment to be super ironic. > >> You can always write assembly or SIMD AVX512 code if you really need it >> to go faster. That's all supported today. > > > It sounds like you are suggesting jumping out of the frying pan — e.g. having > to write and later read five (5) lines with two (2) lines of indent vs. three > (3) lines with one (1) level of indent — and planting one's self firmly in > the fire. AFAICT, an assembly/SIMD AVX512 solution would create more pain for > this use-case, not less. > > BTW, I replied a second time to Ian after my initial email because — even > though my primary interest was in having improved readability — I discovered > that what I proposed would perform typically ~75% better than the generic > Is(), assuming the current switch case is any indication. So since it seemed > to me like a wonderful "kill two birds with one stone" kind of win — an > easier-to-write and easier-to-read construct that performs better than the > offered alternative — I made the incorrect assumption that such a feature > would be a no-brainer enhancement to Go so I asked to verify if performance > would provide any addition motivation to revive what I was told was a dead > horse. > > But given my intuition on this idea failed — and it clearly is a dead horse, > with no hope for revival on the horizon — is there any upside to continue > beating it? > > -Mike > > P.S. I was not planning to continue this thread because Ian shut down the > idea so AFAICT further discussion was a moot point. OTOH, since you took the > effort to write a detailed argument I felt I should at least give you the > courtesy of responding. > > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion visit > https://groups.google.com/d/msgid/golang-nuts/31FAFEE2-04AE-48CC-A3BB-2E5BA269D76E%40newclarity.net. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/9F303949-93F0-4645-ABCE-149C10A9BF73%40ix.netcom.com.
