xudong963 commented on code in PR #18980:
URL: https://github.com/apache/datafusion/pull/18980#discussion_r2571476989
##########
datafusion/datasource-parquet/src/row_filter.rs:
##########
@@ -470,14 +470,33 @@ pub fn build_row_filter(
});
}
+ // To avoid double-counting metrics when multiple predicates are used:
+ // - All predicates should count rows_pruned (cumulative pruned rows)
+ // - Only the last predicate should count rows_matched (final result)
+ // This ensures: rows_matched + rows_pruned = total rows processed
+ let total_candidates = candidates.len();
+
candidates
.into_iter()
- .map(|candidate| {
+ .enumerate()
+ .map(|(idx, candidate)| {
+ let is_last = idx == total_candidates - 1;
+
+ // All predicates share the pruned counter (cumulative)
+ let predicate_rows_pruned = rows_pruned.clone();
+
+ // Only the last predicate tracks matched rows (final result)
+ let predicate_rows_matched = if is_last {
+ rows_matched.clone()
Review Comment:
yes, multiple predicates are connected with AND
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
