iwanttobepowerful commented on PR #50406: URL: https://github.com/apache/spark/pull/50406#issuecomment-2804214109
> Hello @iwanttobepowerful. I never worked with Calcite. > > The new Analyzer processes the logical plan in one* bottom-up traversal. For example, expression IDs get propagated bottom-up ([ExpressionIdAssigner](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/ExpressionIdAssigner.scala?rgh-link-date=2025-04-15T07%3A21%3A46Z)) or subqueries can find [outer references](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/NameScope.scala?rgh-link-date=2025-04-15T07%3A21%3A46Z#L864) from already resolved subtrees. That means that sometimes we have to build additional data-structures as we descend to leaf nodes before starting the bottom-up analysis, e.g. for [CTE resolution](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/CteScope.scala?rgh-link-date=2025-04-15T07%3A21%3A46Z) we build a stack of lookup scopes. > > * We have one pass for the main algebra (95% of the Analysis), but there is also one pre-pass to [resolve the metadata](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/MetadataResolver.scala?rgh-link-date=2025-04-15T07%3A21%3A46Z) (blocking calls), and a couple of rules running after the Analysis before the Optimizer to stay compatible with the old Analyzer in [PlanRewriter](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/PlanRewriter.scala?rgh-link-date=2025-04-15T07%3A21%3A46Z) (e.g. `CleanupAliases`). thanks!!! very nice -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
