sunchao commented on PR #5483: URL: https://github.com/apache/datafusion-comet/pull/5483#issuecomment-5739404908
Addressed the two remaining follow-ups from [the September 9 review](https://github.com/apache/datafusion-comet/pull/5483#pullrequestreview-5154584449). The new `CometExecRuleSuite` regression executes a grouped fact/dimension join and requires both native broadcast joins and Comet DPP, as well as the expected rows. Observers around Comet's AQE preparation rules require the direct `LogicalQueryStage` broadcast-root case to occur, then check that the physical tree and standard cost are unchanged and the original temporary logical identity survives Comet's repair. A captured copy with conflicting direct and temporary links calls Spark's actual stage-link assignment helper and verifies that the temporary link wins. The observer state is scoped to the test, and Spark 3.4 skips this native AQE DPP case. The broader question of which `originalPlan` repairs remain necessary is now tracked in #6034. It covers link ownership, repeated AQE replanning, DPP broadcast roots, and the #323 clearing contract; #5482 stays focused on the reproduced stale-stage-link defect. Committed in [69b2ca86a](https://github.com/apache/datafusion-comet/commit/69b2ca86ae51601e468d23c56ce71f8df36bbafc). Local validation on Spark 4.1.3 / JDK 17 passed all 35 planner tests and three targeted execution regressions (38 total), using the native library built from the rebased source. Spotless, Scalastyle, and `git diff --check` passed. The Spark SQL and all-Spark-profile CI labels are enabled; those checks are pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
