Re: [DISCUSS] [Spark SQL] A single-pass resolution approach for the Catalyst Analyzer

2024-08-26 Thread Wenchen Fan
+1. The analyzer rule order issue has bitten me multiple times and it's very hard to make your analyzer rule bug-free if it interacts with other rules. On Wed, Aug 21, 2024 at 2:49 AM Reynold Xin wrote: > +1 on this too > > When I implemented "group by all", I introduced at least two subtle bugs

Re: [DISCUSS] [Spark SQL] A single-pass resolution approach for the Catalyst Analyzer

2024-08-20 Thread Reynold Xin
+1 on this too When I implemented "group by all", I introduced at least two subtle bugs that many reviewers weren't able to catch and those two bugs would not have been possible to introduce if we had a single pass analyzer. Single pass can make the whole framework more robust. On Tue, Aug 2

[DISCUSS] [Spark SQL] A single-pass resolution approach for the Catalyst Analyzer

2024-08-20 Thread Xiao Li
This sounds like a good idea! The Analyzer is complex. The changes in the new Analyzer should not affect the existing one. The users could add the QO rules and rely on the existing structures and patterns of the logical plan trees generated by the current one. The new Analyzer needs to generate t