Rich-T-kid commented on code in PR #27025:
URL: https://github.com/apache/datafusion/pull/27025#discussion_r4455719783
##########
datafusion/common/src/hash_utils.rs:
##########
@@ -158,8 +181,14 @@ where
let mut buffer = cell.try_borrow_mut()
.map_err(|_| _internal_datafusion_err!("with_hashes cannot be
called reentrantly on the same thread"))?;
- // Ensure buffer has sufficient length, clearing old values
- buffer.clear();
+ // A dictionary with non-null keys and values overwrites every output
+ // slot, so retain the initialized storage instead of zeroing it first.
+ let reuse_buffer = iter
Review Comment:
Fixed in 0a7c4da0c. The fast path now debug-asserts that the chunk/tail span
covers the full output slice, which checks the overwrite invariant used by
buffer reuse.
##########
datafusion/common/src/hash_utils.rs:
##########
@@ -544,6 +573,58 @@ fn hash_generic_byte_view_array<T: ByteViewType>(
}
}
+/// Scatters non-null dictionary hashes in independent lanes to overlap
key-indexed loads.
+#[cfg(not(feature = "force_hash_collisions"))]
+#[inline(always)]
+fn scatter_non_null_dictionary<K: ArrowDictionaryKeyType, const MULTI_COL:
bool>(
+ array: &DictionaryArray<K>,
+ dict_hashes: &[u64],
+ hashes: &mut [u64],
+) {
+ const LANES: usize = 8;
+ assert_eq!(dict_hashes.len(), array.values().len());
+ assert_eq!(array.keys().null_count(), 0);
+ let keys = array.keys().values();
+ let len = keys.len().min(hashes.len());
+ let keys = &keys[..len];
+ let hashes = &mut hashes[..len];
Review Comment:
Fixed in 0a7c4da0c. These are now debug assertions, so the invariants remain
checked in debug builds without repeating the checks in the release hot path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
