zhuqi-lucas opened a new pull request, #25098: URL: https://github.com/apache/datafusion/pull/25098
## Which issue does this PR close? None filed; small self-contained perf fix. Rationale below. ## Rationale for this change `get_repartition_requirement_status` creates a fresh `StatisticsContext::new()` **once per child**. `StatisticsContext::compute` recurses the child's whole subtree and carries a pointer-keyed memoization cache its own docstring describes as a *"per-call memoization cache"* meant to be reused across a walk. Allocating a new context per child discards that cache every time, so a single `ensure_distribution` pass recomputes shared subtree statistics `O(depth)` times. On a deep/wide plan this is measurable. In our deployment (`EnsureRequirements` runs several times over a ~200-node plan) sharing the cache cut physical planning by ~10% with no plan change. ## What changes are included in this PR? - Thread one `StatisticsContext` through the `ensure_distribution` `transform_up` (pass `&StatisticsContext` into `get_repartition_requirement_status`) so each subtree's statistics are computed once per pass. - `StatsCache` is keyed by raw node pointer, and `ensure_distribution` returns `Transformed::yes` unconditionally, so the cache reset is keyed on whether the node's plan pointer **actually changed** (`Arc::as_ptr` before/after). A node that changed may have freed a cached child (which would make a stale pointer key unsafe); a node that made no change cannot, so the cache safely persists across the no-op nodes that dominate a deep plan. - Adds a `repartition_through_deep_operator_stack` test that exercises the shared cache and verifies the parallelization decision is unchanged. ## Are these changes tested? Yes. `datafusion --test core_integration physical_optimizer` (569 passed) and `datafusion-physical-plan` statistics tests (96 passed) are green, plus the new test. Pure memoization: no plan output changes. ## Are there any user-facing changes? No. Internal physical-optimizer performance only; planner output is identical. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
