kumarUjjawal opened a new pull request, #25861: URL: https://github.com/apache/datafusion/pull/25861
## Which issue does this PR close? Related to #25620. Split from #25648 following [the review request](https://github.com/apache/datafusion/pull/25648#discussion_r4122769097). This PR can be reviewed and merged independently. ## Rationale for this change Joins repeat or drop input rows, and outer joins add NULL padding. Copying each input column's null count unchanged can therefore produce inconsistent output statistics. For example, a left join can repeat a NULL in a non-key left column many times, while an unmatched left row introduces NULLs in every right column. ## What changes are included in this PR? - Estimate inner and outer join null counts from the rows carrying each side's values and the rows added as NULL padding. - Estimate unmatched input rows using the existing semi-join estimator. Inner-join row counts count matching pairs, so they cannot alone determine how many input rows are unmatched. - Account for NULL keys under ordinary and null-safe equality, while retaining unknown contributions and conservative precision. - Rescale null counts when a statistics provider replaces the row estimate. Cover projected joins and swapped inputs. The semi/anti-join estimator itself is unchanged. The rules follow join semantics and do not contain benchmark-specific cases. ## What is the testing strategy for this PR? Added unit tests cover NULL padding, repeated matches, unknown counts, null-safe keys, projection, input swapping, and provider rescaling. Existing cardinality assertions remain in place. Passed on this independent branch: - `cargo test --profile ci -p datafusion-physical-plan --lib` (2,369 tests) - `cargo clippy --profile ci -p datafusion-physical-plan --all-targets --all-features -- -D warnings` - `cargo fmt --all` and `git diff --check` Benchmarks were not rerun for the split. This extracts the join implementation already present in #25648; its earlier combined measurements are not isolated measurements of this PR. ## Are there any user-facing changes? Join row and null-count estimates can change, which can affect EXPLAIN output and plan choices. No public API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
