sunchao commented on code in PR #5194:
URL: https://github.com/apache/datafusion-comet/pull/5194#discussion_r4125102524
##########
native/spark-expr/src/array_funcs/arrays_overlap.rs:
##########
@@ -397,6 +420,30 @@ where
}
}
+/// Row overlap for nested element types using one comparator for the full
child arrays.
+fn nested_row_overlap<'a>(
+ left: &'a ArrayRef,
+ right: &'a ArrayRef,
+ comparator: &'a dyn Fn(usize, usize) -> Ordering,
+) -> impl FnMut(Range<usize>, Range<usize>) -> bool + 'a {
+ move |left_range, right_range| {
+ for li in left_range {
Review Comment:
[P2] Could we preserve shorter-side probing while hoisting the comparator?
For `arrays_overlap(a, b)` on nested integer columns where `a=[[0],...,[127]]`
and `b=[[127],...,[190]]`, both versions correctly return `true`, but the base
probes the shorter right side and finds the match after 128 comparisons. This
unconditional left-first loop performs 8,129 comparisons. Two independent
release-mode measurements reproduced approximately 50× higher kernel latency,
so this materially slows previously efficient overlapping-array workloads.
Branch once per row to preserve the previous probe order, keeping comparator
arguments in `(left_index, right_index)` order, and cover unequal lengths in
the benchmark.
Evidence: Verified the extracted base/head kernel bodies against the
requested Git commits. With locked Arrow 59.3.0, 128 rows, and every twentieth
outer row null, alternating base/head measurements produced 0.148 ms versus
7.380 ms and 0.148 ms versus 7.406 ms. Outputs matched. A separately
instrumented single-row probe counted exactly 128 versus 8,129 comparator
calls. Reproduce with `cargo run --release --offline --locked --manifest-path
/tmp/comet-5194-b641-kernel/Cargo.toml --bin comet-5194-b641-kernel` and `--bin
count`. Timing logs are `/tmp/comet-5194-final-bench1.log` and
`/tmp/comet-5194-final-bench2.log`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
