Hello Spark developers,

Please review these two pull requests.

I examined driver overhead in large DataFrame plans with repeated branches.
Profiling showed unnecessary expression traversal and repeated
regular-expression compilation.

   1.

   *PR #59340: Prune subquery-expression traversal in DeduplicateRelations
   <https://github.com/apache/spark/pull/59340>*

   This change skips unnecessary traversal with the existing PLAN_EXPRESSION
   metadata. It keeps the existing relation renewal and attribute mapping
   unchanged.
   2.

   *PR #59341: Reuse the compiled regular expression in TreeNode.nodeName
   <https://github.com/apache/spark/pull/59341>*

   This change uses one shared, compiled Exec$ pattern. Each call uses a
   separate matcher. The change keeps the existing matching behavior.

Both PRs include regression tests and synthetic benchmarks. The synthetic
benchmarks measure isolated operations.

Please check the correctness of the changes and the test coverage.

Thank you for your time.

Hussein

Reply via email to