Luca is already aware - https://github.com/apache/datafusion-sqlparser-rs/issues/2893#issuecomment-6744577466
I could give https://github.com/andygrove/datafusion/tree/sqlparser-v2 a try soon in my current project and give feedback. We have 160 uses/imports of sqlparser. On Wed, Nov 11, 2026 at 7:59 AM Adrian Garcia Badaracco <[email protected]> wrote: > I think Andrew’s point is a good one. I am personally not prepared to take > on maintenance of sqlparser-rs. > I wonder what Luca thinks (I’m not sure if there’s any way to tag them > here). > > > On Nov 11, 2026, at 7:01 AM, Andy Grove <[email protected]> wrote: > > > > I created a draft PR against DataFusion to use the proposed v2 parser. > > All tests are now passing in DataFusion (except for a dependency check > > that cannot pass until there is a published crate for the new parser). > > > > https://github.com/apache/datafusion/pull/27304 > > > > Now that this is fully working, I would like to discuss the parser and > > dialect design in detail to get feedback on the approach. I've filed > > https://github.com/apache/datafusion-sqlparser-rs/issues/2910, > > covering: > > > > - how the parser is generic over the dialect > > - how a dialect overrides a grammar rule so the change applies > > everywhere the rule is used > > - how a dialect reuses another dialect's rules by delegating to them, > > one rule at a time > > > > Perhaps if the approach is acceptable we could start implementing in a > > series of self-contained PRs against a new empty branch in the repo so > > that we can review each piece as it lands. > > > > On Mon, Nov 9, 2026 at 1:52 PM Andrew Lamb <[email protected]> wrote: > >> > >> I think a more important question is who wants to maintain such a > codebase. > >> If we get buy in for that then I think the actual code is much less > >> important > >> > >> On Mon, Nov 9, 2026 at 3:40 PM Andy Grove <[email protected]> > wrote: > >> > >>> Hi Adrian, > >>> > >>> You guessed right. > >>> > >>> The high level design is genuinely mine. When I first implemented the > >>> crate I was fresh from Java and was trying to use an object oriented > >>> approach in Rust, which I eventually realized does not work. I wanted > >>> one base parser then have other parsers extend it and override > >>> specific parts of syntax while keeping the main pratt parser and > >>> common code in the base. I couldn't achieve what I wanted and wasn't > >>> skilled enough with generics and lifetimes. > >>> > >>> There is some discussion about the use of LLM in the issue as well. > >>> This project is quite well suited to LLMs, IMO, because it really is > >>> just refactoring existing parsing code and we have a pretty > >>> comprehensive test suite, so as long as the new one passes all those > >>> tests, we should have confidence that it is as correct as the current > >>> version. > >>> > >>> I know there are lots of different opinions on how far to go with LLM > >>> though, so it's a good discussion to have. > >>> > >>> Thanks, > >>> > >>> Andy. > >>> > >>> > >>> On Mon, Nov 9, 2026 at 1:29 PM Adrian Garcia Badaracco > >>> <[email protected]> wrote: > >>>> > >>>> I’m guessing this is mostly LLM generated code? > >>>> > >>>> The question I think we need to answer as a community is if we are > okay > >>> merging a large chunk of LLM generated code assuming we can put in > place > >>> the right guardrails, get it to pass all tests, all upstream tests in > >>> datafusion, benchmarks look better and the high level design is > accepted. > >>>> > >>>>> On Nov 9, 2026, at 5:46 AM, Andy Grove <[email protected]> > wrote: > >>>>> > >>>>> Hi all, > >>>>> > >>>>> Over the weekend, I started from a clean slate and implemented a new > >>>>> version of the sqlparser-rs crate using the design that I had > >>>>> originally wanted but did not know how to build (I was still quite > new > >>>>> to Rust at the time). > >>>>> > >>>>> The new version has one trait per dialect and all dialect-specific > >>>>> parsing lives in that trait. There is no longer one huge "core" > parser > >>>>> containing conditional logic for different dialects. Related dialects > >>>>> can also share logic. For example, the Hive and Databricks dialects > >>>>> both call into the Spark dialect for common pieces. > >>>>> > >>>>> The new version also has ~10x fewer memory allocations (no more > >>>>> copying Strings everywhere) and is ~10x faster. > >>>>> > >>>>> I am nearly finished porting all of the existing dialects over. The > >>>>> actual parsing code is taken from the current implementation, so that > >>>>> is not re-written. I have ported over all the tests as well, and am > >>>>> working towards 100% parity with v1, although v2 contains quite a lot > >>>>> of correctness fixes compared to v1. > >>>>> > >>>>> What is the appetite in the community for me pushing this to a new > 2.0 > >>>>> branch in the Apache repo? > >>>>> > >>>>> Another option would be for me to file a huge issue explaining the > >>>>> design in detail, and let the community rebuild it. > >>>>> > >>>>> WDYT? > >>>>> > >>>>> I filed an issue for this discssion as well, so that is probably the > >>>>> best place to continue the discussion. > >>>>> > >>>>> https://github.com/apache/datafusion-sqlparser-rs/issues/2893 > >>>>> > >>>>> Thanks, > >>>>> > >>>>> Andy. > >>>>> > >>>>> --------------------------------------------------------------------- > >>>>> To unsubscribe, e-mail: [email protected] > >>>>> For additional commands, e-mail: [email protected] > >>>>> > >>>> > >>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: [email protected] > >>>> For additional commands, e-mail: [email protected] > >>>> > >>> > >>> --------------------------------------------------------------------- > >>> To unsubscribe, e-mail: [email protected] > >>> For additional commands, e-mail: [email protected] > >>> > >>> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
