Rich-T-kid opened a new issue, #23342:
URL: https://github.com/apache/datafusion/issues/23342
### Is your feature request related to a problem or challenge?
In `GroupValuesColumn`, equality between rows is checked by passing a shared
boolean buffer to each column's `vectorized_equal_to` implementation. Each
trait implementation sets a bit to true if the element already exists (matches)
and leaves it false otherwise:
```
for (col_idx, group_col) in self.group_values.iter().enumerate() {
group_col.vectorized_equal_to(
&self.vectorized_operation_buffers.equal_to_group_indices,
&cols[col_idx],
&self.vectorized_operation_buffers.equal_to_row_indices,
&mut equal_to_results,
);
}
```
Since a row is only considered a match if every column reports equality,
most implementations follows a short-circuiting pattern:
```
{
if !equal_to_results.get_bit(idx) {
continue;
}
```
If an earlier column already cleared the bit for a given slot, later columns
skip the (potentially expensive) comparison for that slot.
This short-circuiting is effective, but the columns are currently compared
in input schema order, which is arbitrary with respect to comparison cost. As a
result, we may end up running expensive comparisons before cheaper ones that
could have ruled out the row earlier.
### Describe the solution you'd like
Reorder the column comparisons so that cheaper comparisons run first and
more expensive comparisons run last. This way, rows that are never going to
match are eliminated early using inexpensive checks, and costly comparisons are
only performed on rows that have survived all the cheaper filters, avoiding
unnecessary work on rows already known not to match.
### Describe alternatives you've considered
n/a
### Additional context
n/a
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
