zhuqi-lucas opened a new pull request, #24328: URL: https://github.com/apache/datafusion/pull/24328
## Which issue does this PR close? - Part of #23696 — a benchmark that mechanistically exercises the per-RG fully-matched `RowFilter` skip. ## Rationale for this change #23696 adds a per-row-group fully-matched `RowFilter` skip, but **none of the existing benchmarks exercise it**: - `sort_tpch` / `tpch` don't enable `pushdown_filters` by default, so there is no `RowFilter` to skip; - ClickBench's `URL LIKE …` / equality predicates rarely make a row group's min/max fall entirely inside the satisfying range, so fully-matched RGs are rare. As raised in review (we should verify the optimization improves something mechanistically, otherwise add a benchmark first), this adds a suite that **necessarily** triggers the skip. ## What changes are included in this PR? A new `sql_benchmarks/parquet_row_filter_skip/` suite: - The **load SQL** enables `pushdown_filters` and `COPY`s a clustered Parquet file — a fixed-width, zero-padded, monotonically increasing string key (`skey`) so each row group holds a disjoint, sorted range — plus 14 payload columns. - The **query** applies a low-selectivity range filter (`skey >= '0000100000'`, `skey` not projected). The first row group straddles the threshold; every later RG is fully matched by statistics, so the per-row `RowFilter` is skipped on the fully-matched run (and `skey` isn't decoded there). - `bench.sh` integration: `./bench.sh run parquet_row_filter_skip`, data generated inline by the load SQL. Knobs: `PRED_ROWS` (row count), `RG_SIZE` (parquet row-group size). ## Are these changes tested? Smoke-tested locally via `cargo bench --bench sql -- --test`. Local A/B (main vs #23696, 10M rows / 10 RGs / `skey >= '0000100000'`): - **9 of 10 row groups fully matched** → `row_filter_skipped_fully_matched=9`; - **~18% faster** with the optimization (main ~0.143s → branch ~0.117s); an int64-key variant is ~12%. Once this lands, `run benchmark parquet_row_filter_skip` will compare any PR (e.g. #23696) against `main` in CI. ## Are there any user-facing changes? No — benchmark only. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
