adriangb commented on code in PR #23696:
URL: https://github.com/apache/datafusion/pull/23696#discussion_r3757994056
##########
datafusion/core/tests/parquet/dynamic_row_group_pruning.rs:
##########
@@ -585,3 +585,76 @@ async fn
dynamic_rg_pruning_fires_for_multi_column_sort_leading_tied() {
output.description(),
);
}
+
+/// Per-RG `fully_matched` `RowFilter` skip optimization.
+///
+/// Stats prove that every row of a fully-matched row group satisfies the
+/// pushdown predicate, so the parquet decoder can skip the per-row
+/// `RowFilter` for that RG entirely. The stream rebuilds the decoder at
+/// the boundary with an empty `RowFilter` and toggles back to the real
+/// one at the next non-fully-matched RG.
+///
+/// Layout: 4 RGs of 3 values each. Predicate `v >= 3` makes RG 0 a
+/// straddler (some rows fail) but RGs 1..=3 fully matched (every value
+/// >= 3 by stats). RG 0 keeps the row filter, then the toggle flips to
+/// "no filter" when we enter the fully-matched run.
Review Comment:
Can we test `v >= 3 AND v <= 10` or make a new test for `v <= 10`? The point
is to test going from fully-matched - > non-fully-matched
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
