ErikBPF opened a new pull request, #5828:
URL: https://github.com/apache/datafusion-comet/pull/5828
## Which issue does this PR close?
Closes #5824.
## Rationale for this change
Exchange reuse can treat plans with different semantics as equivalent. A
union of semi/anti joins can return one branch twice, while ordinary/outer
generator branches can lose the outer rows. Review also reproduced a broadcast
anti-join collision between NOT EXISTS and NOT IN when the probe contains NULL.
## What changes are included in this PR?
- Include `joinType` in equality and hashing for shuffled hash, broadcast
hash, and sort-merge joins.
- Preserve `GenerateExec.outer` in `CometExplodeExec` and include it in
equality and hashing.
- Preserve the already-serialized null-aware anti-join flag in
`CometBroadcastHashJoinExec` and include it in equality and hashing.
Native execution and protobuf schemas are unchanged.
## How are these changes tested?
Added 16 parameterized regressions in the existing join and generator
suites. They cover all three join strategies, nested/sliced array inputs,
explode/posexplode, null and empty arrays, and AQE off/on. Tests compare Spark
results and explicit expected rows, require native operators, and verify
equivalent plans still reuse post-operator exchanges.
Before the fix, six join cases, eight generator cases, and one null-aware
anti-join probe reproduced incorrect results.
Local validation:
- Final 16 regressions passed on Spark 3.4/JDK 11, 3.5/JDK 17 with Scala
2.12 and 2.13, 4.0/JDK 21, 4.1/JDK 17, and 4.2/JDK 17.
- Full Spark 4.1 join/generator suites: 98 passed. This run preceded the
final explicit Boolean-boxing adjustment for Scala 2.12; the focused runs above
validate that adjustment.
- Native build, JVM packaging, semantic and syntactic Scalafix, Spotless,
Rust formatting, Apache RAT, Prettier, workflow lint, and CI
inventory/configuration checks passed locally.
Focused command (substitute the relevant Spark/Scala profile):
```sh
./mvnw test -Pspark-4.1 -Dtest=none \
'-Dsuites=org.apache.comet.exec.CometJoinSuite
#5824,org.apache.comet.exec.CometGenerateExecSuite #5824'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
