zhuqi-lucas commented on code in PR #27499:
URL: https://github.com/apache/datafusion/pull/27499#discussion_r4595453902
##########
datafusion/physical-plan/src/aggregates/group_values/mod.rs:
##########
@@ -84,6 +84,14 @@ pub trait GroupValues: Send {
/// assigned.
fn intern(&mut self, cols: &[ArrayRef], groups: &mut Vec<usize>) ->
Result<()>;
+ /// Looks up the group id for each input row of `cols` without assigning
+ /// new group ids.
+ ///
+ /// When the function returns, `groups` has been cleared and contains one
+ /// entry per input row: `Some(group_id)` if the value is already stored
+ /// and `None` otherwise. Stored values and group ids are unchanged.
+ fn find(&mut self, cols: &[ArrayRef], groups: &mut Vec<Option<usize>>) ->
Result<()>;
Review Comment:
Since there is no in-tree caller yet, a default `not_impl_err!` might be
easier to land than a required method that breaks every custom `GroupValues`.
##########
datafusion/physical-expr-common/src/binary_view_map.rs:
##########
@@ -331,7 +331,7 @@ where
self.map
.find(hash, |header| {
- if header.hash != hash {
+ if header.hash != hash || header.view as u32 != len {
Review Comment:
This is a fix to the existing `intern` path: on a hash collision where a
long input shares its 4-byte prefix with an inline stored value, the inline
view was read as a buffer reference. Could we mention it in the PR description,
or split it out with its own test?
##########
datafusion/physical-plan/src/aggregates/group_values/row.rs:
##########
@@ -177,6 +177,40 @@ impl GroupValues for GroupValuesRows {
Ok(())
}
+ fn find(&mut self, cols: &[ArrayRef], groups: &mut Vec<Option<usize>>) ->
Result<()> {
+ // Same normalization as `intern`, so lookups agree with stored keys.
+ let normalized_cols: Vec<ArrayRef> = cols
+ .iter()
+ .map(|col| float_semantics::normalize(Arc::clone(col)))
+ .collect();
+ let cols = normalized_cols.as_slice();
+
+ let group_rows = &mut self.rows_buffer;
+ group_rows.clear();
+ self.row_converter.append(group_rows, cols)?;
+ let n_rows = group_rows.num_rows();
+
+ let batch_hashes = &mut self.hashes_buffer;
+ batch_hashes.clear();
+ batch_hashes.resize(n_rows, 0);
+ create_hashes(cols, &self.random_state, batch_hashes)?;
+
+ groups.clear();
+ let Some(group_values) = self.group_values.as_ref() else {
Review Comment:
nit: we could return before the row conversion and hashing when
`group_values` is `None`.
##########
datafusion/physical-plan/src/aggregates/group_values/multi_group_by/mod.rs:
##########
@@ -1213,6 +1310,8 @@ impl<const STREAMING: bool> GroupValues for
GroupValuesColumn<STREAMING> {
// a real Result rather than panicking.
let fresh = Self::build_group_columns(&self.schema)?;
let group_values = mem::replace(&mut self.group_values, fresh);
+ self.map.clear();
+ self.group_index_lists.clear();
Review Comment:
Same here, this also fixes `intern` after `emit(EmitTo::All)`: stale map
entries pointed at group ids that no longer exist. Worth calling out (or a
separate PR).
##########
datafusion/physical-expr-common/src/binary_map.rs:
##########
@@ -547,6 +547,84 @@ where
)
}
+ /// Looks up each value in `values` without inserting anything.
+ ///
+ /// `observe_payload_fn` is called once per row, in order, with
+ /// `Some(payload)` if the value is already stored and `None` otherwise. A
+ /// null row maps to the payload of the null entry if one exists, matching
+ /// [`Self::insert_if_new`].
+ pub fn find<OP>(&mut self, values: &ArrayRef, observe_payload_fn: OP)
+ where
+ OP: FnMut(Option<V>),
+ {
+ match self.output_type {
+ OutputType::Binary => {
+ assert!(matches!(
+ values.data_type(),
+ DataType::Binary | DataType::LargeBinary
+ ));
+ self.find_inner::<OP, GenericBinaryType<O>>(values,
observe_payload_fn)
+ }
+ OutputType::Utf8 => {
+ assert!(matches!(
+ values.data_type(),
+ DataType::Utf8 | DataType::LargeUtf8
+ ));
+ self.find_inner::<OP, GenericStringType<O>>(values,
observe_payload_fn)
+ }
+ _ => unreachable!("View types should use `ArrowBytesViewMap`"),
+ }
+ }
+
+ /// Generic version of [`Self::find`] that handles `ByteArrayType`
+ fn find_inner<OP, B>(&mut self, values: &ArrayRef, mut observe_payload_fn:
OP)
Review Comment:
The probe closure here (and in `binary_view_map.rs`) duplicates the one in
`insert_if_new_inner`. The length-check fix above had to be applied twice, so
maybe extract a shared helper so `intern` and `find` can't drift?
##########
datafusion/physical-plan/src/aggregates/group_values/single_group_by/primitive.rs:
##########
@@ -161,6 +161,28 @@ where
Ok(())
}
+ fn find(&mut self, cols: &[ArrayRef], groups: &mut Vec<Option<usize>>) ->
Result<()> {
+ assert_eq!(cols.len(), 1);
+ groups.clear();
+
+ for v in cols[0].as_primitive::<T>() {
+ let group_id = match v {
+ None => self.null_group,
+ Some(key) => {
+ let key = key.normalize_key();
+ let hash = key.hash(&self.random_state);
+ self.map
+ .find(hash, |&(g, h)| unsafe {
Review Comment:
nit: a `// SAFETY:` comment would help (`intern` lacks one too).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]