phillipleblanc opened a new issue, #2193: URL: https://github.com/apache/datafusion-ballista/issues/2193
## Bug Ballista defaults `datafusion.optimizer.prefer_hash_join` to `false`. For a SQL `NOT IN` predicate, DataFusion's physical planner can therefore replace the logical null-aware anti join with a `SortMergeJoinExec` before Ballista's scheduler rules run. `SortMergeJoinExec` does not carry the `null_aware` flag. The scheduler cannot recover the required `NOT IN` NULL semantics afterward. ## Reproduction Use four CSV partitions for each table: - `t1(a)`: values `0..19` - `t2(b)`: values `0..4`, with one key replaced by `NULL` Run Ballista with its default join preference: ```sql select a from t1 where a not in (select b from t2) order by a; ``` Expected result: **0 rows**, because the subquery contains `NULL`. Observed result: **16 rows**. Single-process DataFusion returns the correct 0 rows. Setting `datafusion.optimizer.prefer_hash_join = true` retains a null-aware `HashJoinExec`, so the scheduler can recognize and lower it correctly. Apache #2188 addresses that hash-join and adaptive-planning path, but cannot fix a null-aware flag that was already discarded. ## Expected behavior Physical planning should retain null-aware anti-join semantics regardless of the global hash-join preference. Possible approaches include forcing a null-aware `HashJoinExec` for this logical shape or adding equivalent null-aware support to the selected physical join. Discovered while validating #2188 from the end-to-end reproduction in Andy Grove's review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
