I think a more important question is who wants to maintain such a codebase. If we get buy in for that then I think the actual code is much less important
On Mon, Nov 9, 2026 at 3:40 PM Andy Grove <[email protected]> wrote: > Hi Adrian, > > You guessed right. > > The high level design is genuinely mine. When I first implemented the > crate I was fresh from Java and was trying to use an object oriented > approach in Rust, which I eventually realized does not work. I wanted > one base parser then have other parsers extend it and override > specific parts of syntax while keeping the main pratt parser and > common code in the base. I couldn't achieve what I wanted and wasn't > skilled enough with generics and lifetimes. > > There is some discussion about the use of LLM in the issue as well. > This project is quite well suited to LLMs, IMO, because it really is > just refactoring existing parsing code and we have a pretty > comprehensive test suite, so as long as the new one passes all those > tests, we should have confidence that it is as correct as the current > version. > > I know there are lots of different opinions on how far to go with LLM > though, so it's a good discussion to have. > > Thanks, > > Andy. > > > On Mon, Nov 9, 2026 at 1:29 PM Adrian Garcia Badaracco > <[email protected]> wrote: > > > > I’m guessing this is mostly LLM generated code? > > > > The question I think we need to answer as a community is if we are okay > merging a large chunk of LLM generated code assuming we can put in place > the right guardrails, get it to pass all tests, all upstream tests in > datafusion, benchmarks look better and the high level design is accepted. > > > > > On Nov 9, 2026, at 5:46 AM, Andy Grove <[email protected]> wrote: > > > > > > Hi all, > > > > > > Over the weekend, I started from a clean slate and implemented a new > > > version of the sqlparser-rs crate using the design that I had > > > originally wanted but did not know how to build (I was still quite new > > > to Rust at the time). > > > > > > The new version has one trait per dialect and all dialect-specific > > > parsing lives in that trait. There is no longer one huge "core" parser > > > containing conditional logic for different dialects. Related dialects > > > can also share logic. For example, the Hive and Databricks dialects > > > both call into the Spark dialect for common pieces. > > > > > > The new version also has ~10x fewer memory allocations (no more > > > copying Strings everywhere) and is ~10x faster. > > > > > > I am nearly finished porting all of the existing dialects over. The > > > actual parsing code is taken from the current implementation, so that > > > is not re-written. I have ported over all the tests as well, and am > > > working towards 100% parity with v1, although v2 contains quite a lot > > > of correctness fixes compared to v1. > > > > > > What is the appetite in the community for me pushing this to a new 2.0 > > > branch in the Apache repo? > > > > > > Another option would be for me to file a huge issue explaining the > > > design in detail, and let the community rebuild it. > > > > > > WDYT? > > > > > > I filed an issue for this discssion as well, so that is probably the > > > best place to continue the discussion. > > > > > > https://github.com/apache/datafusion-sqlparser-rs/issues/2893 > > > > > > Thanks, > > > > > > Andy. > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
