sunchao commented on code in PR #25693:
URL: https://github.com/apache/datafusion/pull/25693#discussion_r4147707863
##########
datafusion/physical-plan/src/joins/utils.rs:
##########
@@ -1391,12 +1391,19 @@ pub(crate) fn build_null_aware_left_mark_column(
probe_side_has_null: bool,
probe_side_non_empty: bool,
) -> ArrayRef {
+ let build_key_nulls = build_key_column.logical_nulls();
Review Comment:
[P2] Avoid rescanning the full build dictionary for every output batch
`emit_unmatched_build_rows` calls this helper once per `batch_size` chunk,
but `build_key_column` is the entire concatenated build column. In Arrow 60,
`DictionaryArray::logical_nulls()` allocates an N-bit bitmap and walks all N
keys when the dictionary values contain NULL. This makes final emission do `N *
ceil(N / batch_size)` key inspections. It also runs for correlated joins even
though `null_indices_bitmap` supplies the answer and the new buffer is unused.
I reproduced this with a null-aware `CollectLeft` `LeftMark`:
dictionary<Int32, Int32> build values `[1, NULL]`, alternating keys `[0, 1,
...]`, an empty probe, and batch size 8192. Both revisions return exactly N
non-null FALSE marks. With separate base/head target directories and the locked
Arrow 60 dependencies, median times over three unoptimized operator runs were
0.185s vs 1.165s for 524,288 rows, and 0.369s vs 4.274s for 1,048,576 rows.
Reversing run order retained the smaller-case slowdown; the equivalent plain
Int32 control stayed comparable. These are diagnostic debug-build timings, not
release query benchmarks.
Could we compute logical nulls only for the current contiguous LeftMark
chunk (adjusting the index), or retain the bitmap once for final emission, and
skip this work when the correlated bitmap is present? That preserves the fix
without repeatedly scanning the full build side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
