neilconway commented on code in PR #20528:
URL: https://github.com/apache/datafusion/pull/20528#discussion_r2855227621
##########
datafusion/physical-expr/src/expressions/in_list.rs:
##########
@@ -771,32 +772,43 @@ impl PhysicalExpr for InListExpr {
}
}
None => {
- // No static filter: iterate through each expression, compare,
and OR results
+ // No static filter: iterate through each expression, compare,
and OR results.
+ // Use Arrow's vectorized eq kernel for
primitive/string/binary types,
+ // falling back to row-by-row comparator for nested types
(Struct, List, etc.)
+ // where eq semantics are ambiguous.
let value = value.into_array(num_rows)?;
+ let use_arrow_eq = !value.data_type().is_nested();
Review Comment:
Do we know for sure that Arrow's eq kernel will work for *all* non-nested
types? Perhaps that would be a bit fragile if we add new types in the future. I
wonder if we should explicitly whitelist the types we know that work?
Digging around a bit, it seems we panic if we try to pass RunEndEncoded
types to `eq`, but REE types aren't considered nested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
