jayzhan211 opened a new pull request, #25076:
URL: https://github.com/apache/datafusion/pull/25076
## Which issue does this PR close?
<!-- No issue filed yet; found while auditing the hash join final stage.
Happy to open one first if preferred. -->
- Closes #.
## Rationale for this change
`x NOT IN (subquery)` is planned as a null-aware `LeftAnti` hash join in
`CollectLeft` mode with several probe partitions. If any probe partition sees
a NULL key, the predicate is UNKNOWN for every build row and the join must
return nothing.
The final stage of `HashJoinStream` read the shared "probe side saw NULL"
flag *before* decrementing the probe-partition counter. That allowed this
interleaving:
1. partition B finishes its probe input and reads the flag as `false`;
2. partition A processes a batch with a NULL key, sets the flag, finishes and
decrements the counter;
3. partition B decrements the counter, becomes the last partition, and emits
the unmatched build rows.
The result then contains rows that `NOT IN` must suppress. The counter was
also decremented with `Ordering::Relaxed`, so nothing ordered the flag stores
of a finishing partition before the reads of the last one, even when the
reads happened after the decrement.
## What changes are included in this PR?
- `JoinLeftData::report_probe_completed` now decrements with `AcqRel` and
returns `Option<ProbeSideSummary>`: `Some` only for the last partition,
with
the shared flags read after the decrement. The two `AtomicBool` flags are
private; the probe phase stores through `record_probe_batch` and the early
exit reads a `probe_side_has_null_hint` that is documented as best-effort.
The final stage therefore cannot read the flags before its own decrement,
which guards the fix structurally rather than by a test that would have to
force a few-instruction race window.
- `process_unmatched_build_batch` decrements first and hands the summary to
the null-aware helpers. The NULL-probe rule of `LeftAnti` moved into
`null_aware_left_anti_final_indices`, so all its final-stage rules are in
one place.
- No change for joins that are not null-aware beyond the `AcqRel` ordering
on the counter.
## What is the testing strategy for this PR?
Two new tests in `hash_join/exec.rs`,
`test_null_aware_anti_join_probe_null_in_other_partition` and
`test_null_aware_left_mark_probe_null_in_other_partition`, run a
`CollectLeft`
join with a two-partition probe side where only partition 0 has a NULL key,
draining the partitions in both orders. They cover the cross-partition flag
propagation that had no test. The race itself cannot be reproduced
deterministically without a test hook, which is why the fix makes the wrong
call order unrepresentable instead. Existing null-aware join tests and the
join unit test suite pass unchanged.
## Are there any user-facing changes?
Wrong results from `NOT IN` under this interleaving are fixed. No API
changes;
all touched items are `pub(super)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
