Hi Adrian, You guessed right.
The high level design is genuinely mine. When I first implemented the crate I was fresh from Java and was trying to use an object oriented approach in Rust, which I eventually realized does not work. I wanted one base parser then have other parsers extend it and override specific parts of syntax while keeping the main pratt parser and common code in the base. I couldn't achieve what I wanted and wasn't skilled enough with generics and lifetimes. There is some discussion about the use of LLM in the issue as well. This project is quite well suited to LLMs, IMO, because it really is just refactoring existing parsing code and we have a pretty comprehensive test suite, so as long as the new one passes all those tests, we should have confidence that it is as correct as the current version. I know there are lots of different opinions on how far to go with LLM though, so it's a good discussion to have. Thanks, Andy. On Mon, Nov 9, 2026 at 1:29 PM Adrian Garcia Badaracco <[email protected]> wrote: > > I’m guessing this is mostly LLM generated code? > > The question I think we need to answer as a community is if we are okay > merging a large chunk of LLM generated code assuming we can put in place the > right guardrails, get it to pass all tests, all upstream tests in datafusion, > benchmarks look better and the high level design is accepted. > > > On Nov 9, 2026, at 5:46 AM, Andy Grove <[email protected]> wrote: > > > > Hi all, > > > > Over the weekend, I started from a clean slate and implemented a new > > version of the sqlparser-rs crate using the design that I had > > originally wanted but did not know how to build (I was still quite new > > to Rust at the time). > > > > The new version has one trait per dialect and all dialect-specific > > parsing lives in that trait. There is no longer one huge "core" parser > > containing conditional logic for different dialects. Related dialects > > can also share logic. For example, the Hive and Databricks dialects > > both call into the Spark dialect for common pieces. > > > > The new version also has ~10x fewer memory allocations (no more > > copying Strings everywhere) and is ~10x faster. > > > > I am nearly finished porting all of the existing dialects over. The > > actual parsing code is taken from the current implementation, so that > > is not re-written. I have ported over all the tests as well, and am > > working towards 100% parity with v1, although v2 contains quite a lot > > of correctness fixes compared to v1. > > > > What is the appetite in the community for me pushing this to a new 2.0 > > branch in the Apache repo? > > > > Another option would be for me to file a huge issue explaining the > > design in detail, and let the community rebuild it. > > > > WDYT? > > > > I filed an issue for this discssion as well, so that is probably the > > best place to continue the discussion. > > > > https://github.com/apache/datafusion-sqlparser-rs/issues/2893 > > > > Thanks, > > > > Andy. > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
