andygrove opened a new pull request, #7338: URL: https://github.com/apache/datafusion-comet/pull/7338
## Which issue does this PR close? Backport of #6270 to `branch-1.1`. #6270 closed #6264 on `main`; the issue stays closed. `branch-1.2` already has the fix, because #6270 merged before that branch was cut, so no `branch-1.2` backport is needed (`git merge-base --is-ancestor e242944ec4 apache/branch-1.2` is true). ## Rationale for this change With AQE on, two exchanges that differ only in the dynamic partition pruning filter of a scan below them compare equal, and AQE replaces the second with a `ReusedExchange` of the first. The consumer of the second exchange then reads the other branch's rows, and the rows it should have read are silently lost. `branch-1.1` has the bug: `CometScanUtils.filterUnusedDynamicPruningExpressions` drops any DPP filter whose subquery is still the adaptive placeholder. It does not affect only an opt-in path, since it needs only AQE, DPP and a native scan. This is not a 1.1.0 regression. The bug is in every release that has the filter helper. It is a silent wrong-result bug with `priority:critical`, so it is a candidate under the backporting guide's "wrong results" rule. ## What changes are included in this PR? Cherry-pick of e242944ec4 with `-x`. The shared helper now drops only `DynamicPruningExpression(TrueLiteral)`, as `FileSourceScanExec` does, and `CometNativeScanExec.doCanonicalize` drops every DPP filter from the stale `originalPlan` copy so an unused DPP scan still matches its twin. Adaptations, all recorded in the commit message: - `contrib/delta-spark` is not on `branch-1.1`, so the changes to `CometDeltaNativeScanExec` and `CometDeltaNativeScanSuite` are dropped. - The `adding_a_new_operator.md` change is dropped. Only `docs/source/user-guide/latest/` is published from a release branch. - `CometScanUtils` keeps only the `FileSourceScanExec` import. The `FilePartition` and `NativeConfig` imports in the source commit belong to #6753, which is not on this branch. The rest of the diff (`CometNativeScanExec`, `CometTestBase`, `CometExecSuite`, `CometIcebergNativeSuite`) applied without changes. ## How are these changes tested? The source PR's tests: three `AQE DPP ... not reused across branches (#6264)` tests in `CometExecSuite` plus one in `CometIcebergNativeSuite`, and a control that scans with the same DPP filter are still reused. Run locally on this branch with Spark 3.5 and a `libcomet` built from this branch (`CometExecSuite`, `CometIcebergNativeSuite`): 263 passed, 0 failed, 10 canceled. The canceled tests are unrelated version-gated ones (variant, TIME type, V3 deletion vectors). With `spark/src/main` reverted to `branch-1.1`, the three `not reused across branches` tests fail and the control passes. So the branch has the bug and the tests catch it. `spotless:check` is clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
