sunchao commented on code in PR #26214:
URL: https://github.com/apache/datafusion/pull/26214#discussion_r4426151172
##########
datafusion/optimizer/src/decorrelate_predicate_subquery.rs:
##########
@@ -1037,6 +1042,38 @@ fn build_join(
}
let left = projected_left.as_ref().unwrap_or(left);
+ // The first `V` keys of a null-aware join are its `NOT IN` value keys.
+ //
+ // A tuple element that cannot be NULL in scope on either side never makes
+ // the tuple comparison UNKNOWN: compared with a definite value it is
either
+ // equal, which leaves the other elements to decide, or a definite
mismatch,
+ // which makes the comparison FALSE whatever the other elements are. It
+ // therefore only selects which subquery rows a NULL in another element is
+ // compared with, which is what a correlation scope key does. Moving such
+ // elements after the ones that can be NULL turns them into scope keys, so
+ // the hash join finds the rows a NULL-valued row is compared with through
a
+ // hash lookup instead of pairing it with every row of the other side
+ // (<https://github.com/apache/datafusion/issues/26088>).
+ //
+ // A null-aware tuple has at least one element that can be NULL, so `V` is
+ // at least one. With exactly one, the join is a scalar `NOT IN` on that
+ // element, correlated by the other elements.
+ let null_aware_value_keys = if null_aware && in_values.len() > 1 {
+ let (nullable, non_nullable): (Vec<_>, Vec<_>) = in_values
+ .into_iter()
+ .zip(elements_may_be_null)
+ .partition(|(_, may_be_null)| *may_be_null);
Review Comment:
Thanks for checking the scalar case. I agree the inferred field nullability
deserves a separate issue. I also reran the tuple query on `53cd54d96` with the
original inferred-schema `RecordBatch`. For the row with `a = 1`, the promoted
plan returns `TRUE`. The merge base and the three-valued SQL reference both
return `NULL`.
There’s a control on this same head. The only inner row is `(x, y) = (1,
2)`. Replacing `y` with `CASE WHEN x = -99 THEN NULL ELSE y END` leaves that
row as `(1, 2)`, but makes the planner keep two value keys. The tuple result
goes back to `NULL`. With plain `y` there is one value key, and a `WHERE NOT
IN` lets the outer row through.
Could we keep dictionary elements as value keys when schema nullability
alone cannot rule out a logical NULL, and capture this case in a regression
test? That would preserve the tuple query’s existing result while the broader
inference issue is investigated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
