phillipleblanc opened a new pull request, #2188: URL: https://github.com/apache/datafusion-ballista/pull/2188
# Which issue does this PR close? Closes #2187. # Rationale for this change DataFusion plans `NOT IN` predicates as null-aware `LeftAnti` hash joins. These joins cannot be swapped to `RightAnti`. They also need `PartitionMode::CollectLeft` to preserve their NULL semantics. Ballista's scheduler could violate these rules in several places. A swap fails during stage resolution with an error such as: ```text null_aware can only be true for LeftAnti joins, got RightAnti ``` The stage then cannot resolve, and the query can remain running until the client times out. A partitioned null-aware join can instead produce incorrect `NOT IN` results. This is a fresh implementation against current Apache `main`, based on the downstream report and fix in spiceai/datafusion-ballista#58. # What changes are included in this PR? - Correct an already-partitioned null-aware join to `CollectLeft` in Ballista's `JoinSelection` rule. - Match DataFusion's null-aware guard in the unbounded-input swap rule. - Check the null-aware exception before the distributed planner's generic broadcast-safety demotion. - Make adaptive query execution (AQE) force `CollectLeft` and skip size-based input swaps for null-aware joins. - Add regression tests for static join selection, unbounded inputs, broadcast demotion, and AQE. Validation: ```text cargo test -p ballista-scheduler null_aware --lib # 4 passed cargo test -p ballista-scheduler --lib # 273 passed; 2 ignored cargo clippy --all-targets --package ballista-scheduler --all-features -- -D warnings cargo fmt --all -- --check ``` # Are there any user-facing changes? `NOT IN` queries now retain valid null-aware anti-join plans across static and adaptive scheduler planning. There are no API or configuration changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
