kosiew opened a new pull request, #25994: URL: https://github.com/apache/datafusion/pull/25994
## Which issue does this PR close? Closes #25473. Closes #25474. ## Rationale for this change Issues #25473 and #25474 reported wrong results for `NOT IN (subquery)` when the subquery column is `NOT NULL` but the value being compared can evaluate to `NULL`. This includes NULL constants such as `CAST(NULL AS INT)` and nullable expressions over `NOT NULL` columns, such as a `CASE` expression without an `ELSE`. The fix itself landed in #25338, which determines null-awareness from the full `IN` operand expressions. This PR adds the missing regression coverage for both issues so this behavior does not regress. ## What changes are included in this PR? This is a test-only change. There are no production code or public API changes. It adds optimizer unit tests that verify: - A NULL constant compared against a non-nullable subquery column produces a projected `__correlated_sq_1_value` and a `null_aware` `LeftAnti` join. - A non-nullable expression such as `test.c + 1`, where `test.c` is non-nullable, continues to use a regular `LeftAnti` join without `null_aware`. It adds SQL logic tests for null-aware anti joins covering: - `CAST(NULL AS INT)`, untyped `NULL`, and `NULLIF(1, 1)`. - Nullable expressions over `NOT NULL` columns. - Nullable expressions on the subquery side. - NULL values against an empty subquery. - A non-NULL constant control. - A non-nullable `x + 1` control. - Logical and physical `EXPLAIN` plans for the NULL constant, nullable `CASE`, and non-nullable `x + 1` cases. - The NULL constant and nullable `CASE` cases with `datafusion.execution.target_partitions = 1`. It also adds SQL logic tests for the mark-join path, including the `... NOT IN (...) OR x = 99` form and a `SELECT`-list control that verifies the mark evaluates to `NULL`. ## Are these changes tested? Yes. This PR adds regression tests in: - `datafusion/optimizer/src/decorrelate_predicate_subquery.rs` - `datafusion/sqllogictest/test_files/null_aware_anti_join.slt` - `datafusion/sqllogictest/test_files/null_aware_mark_join.slt` The new tests verify both result correctness and expected logical/physical plan shapes, including that nullable operands use null-aware joins while non-nullable operands remain non-null-aware. Verification: ```bash cargo test -p datafusion-optimizer decorrelate_predicate_subquery cargo test -p datafusion-sqllogictest --test sqllogictests -- null_aware cargo test -p datafusion-sqllogictest --test sqllogictests -- subquery_projection ``` ## Are there any user-facing changes? No new user-facing behavior is introduced by this PR. The underlying correctness fix already landed in #25338. This PR only adds regression coverage for that behavior and does not change any public APIs. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
