sunchao opened a new issue, #6438: URL: https://github.com/apache/datafusion-comet/issues/6438
### Describe the bug `CometExecRule` removes temporary sink and scan wrappers with a top-down `transform`. If recursive conversion creates `CometSinkPlaceHolder(CometSinkPlaceHolder(stage))`, cleanup removes only the outer wrapper: Spark does not reapply the rule at the replacement root. A `CometScanWrapper` can also hide further wrappers because it is a leaf. Upstream's `revertUnsafePartialAggregates` can revisit an already wrapped input stage when restoring a Spark aggregate, so the returned physical plan can retain a planning-only placeholder. ### Steps to reproduce A planner regression can build a grouped `AVG` plan with an existing `ShuffleQueryStageExec` below its partial aggregate. Disable native hash partitioning for the upper shuffle so the partial first converts, the final stays in Spark, and aggregate-buffer repair recursively converts the restored partial. The old cleanup leaves a sink wrapper above the input stage. The accompanying regression constructs that prior stage explicitly; this is a planner-level reproduction, not an end-to-end SQL-only reproduction. ### Expected behavior Remove all temporary wrappers and retain the original query-stage object and AQE partition specifications. ### Additional context Reproduced against main at `e6f5bf55d893daf8fb9625e302fcd832360c4397` with Spark 4.1.3. Both the aggregate-repair regression and a nested-wrapper cleanup regression fail with the old cleanup and pass with bottom-up cleanup plus explicit recursion through leaf scan wrappers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
