adriangb commented on code in PR #14567:
URL: https://github.com/apache/datafusion/pull/14567#discussion_r1950314013
##########
datafusion/physical-optimizer/src/pruning.rs:
##########
@@ -1710,6 +1717,56 @@ fn build_like_match(
Some(combined)
}
+// For predicate `col NOT LIKE 'foo%'`, we rewrite it as `(col_min NOT LIKE
'foo%' OR col_max NOT LIKE 'foo%')`. If both col_min and col_max have the
prefix foo, we skip the entire row group (as we can be certain that all data in
this row group has the prefix foo).
+fn build_not_like_match(
+ expr_builder: &mut PruningExpressionBuilder<'_>,
+) -> Option<Arc<dyn PhysicalExpr>> {
+ // col NOT LIKE 'prefix%' -> !(col_min LIKE 'prefix%' && col_max LIKE
'prefix%') -> (col_min NOT LIKE 'prefix%' || col_max NOT LIKE 'prefix%')
+
+ let min_column_expr = expr_builder.min_column_expr().ok()?;
+ let max_column_expr = expr_builder.max_column_expr().ok()?;
+
+ let scalar_expr = expr_builder.scalar_expr();
+
+ let pattern = extract_string_literal(scalar_expr)?;
+
+ let chars: Vec<char> = pattern.chars().collect();
+ for i in 0..chars.len() - 1 {
+ // Check if current char is a wildcard and is not escaped with
backslash
+ if (chars[i] == '%' || chars[i] == '_') && (i == 0 || chars[i - 1] !=
'\\') {
+ // Example: For pattern "foo%bar", the row group might include
values like
+ // ["foobar", "food", "foodbar"], making it unsafe to prune.
Review Comment:
Makes sense thanks. @findepi do you know if Spark handles this and if so
does it do it in similar way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
