sunchao opened a new pull request, #6439: URL: https://github.com/apache/datafusion-comet/pull/6439
## Which issue does this PR close? Closes #6438. ## Rationale for this change Recursive aggregate conversion can leave `CometSinkPlaceHolder(CometSinkPlaceHolder(stage))` in the plan. The current top-down cleanup removes the outer wrapper but does not reapply its rule at the replacement root, so a planning-only placeholder survives. Leaf scan wrappers can similarly hide further wrappers. ## What changes are included in this PR? Remove sink placeholders from the bottom up, and explicitly recurse into the plan hidden by a leaf `CometScanWrapper`. Preserve the existing query-stage object and AQE partition specifications. Add regressions for mixed sink/scan nesting and upstream's aggregate-buffer repair path: a rejected native hash shuffle above a converted AVG partial aggregate causes repair to revisit an already wrapped input stage. ## How are these changes tested? - Built the candidate native library with `cargo build --locked`. - Spark 4.1.3 / JDK 17: all 85 tests in `CometExecRuleSuite` pass, including the two new regressions. - Before/after control: restoring only the previous cleanup makes both new tests fail; restoring the fix makes them pass in the full suite. - Maven formatting and Scala style checks pass. The aggregate regression constructs an existing query stage and exercises the actual upstream planner rule. It is a planner-level reproduction, not an end-to-end SQL-only reproduction. Broader Spark SQL and supported-version CI will be requested with `run-spark-4.1-tests` and `run-all-spark-profiles`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
