mohitgurav20 opened a new pull request, #26897:
URL: https://github.com/apache/datafusion/pull/26897
# Which issue does this PR close?
Related to #25797 and #25811.
# Rationale for this change
To accurately evaluate performance and I/O improvements in PR #25811
(evaluating Parquet Bloom filters incrementally during row group pruning), we
need a baseline test on `main` that records the current I/O access patterns
when loading Bloom filters from storage.
Currently on `main`, candidate column Bloom filters are loaded upfront
across candidate row groups before evaluating the predicate. This PR
establishes the baseline test using a tracking reader so that PR #25811 can
directly demonstrate the skipped I/O requests in its test diff.
# What changes are included in this PR?
- Adds `RangeTracker`, `TrackingBloomAsyncReader`, and
`TrackingBloomReaderFactory` to record all byte range requests made to the
underlying `AsyncFileReader` (modeled after
`parquet/tests/arrow_reader/io/async_reader.rs` in `arrow-rs`).
- Adds `test_bloom_filter_io_pattern_baseline` in `opener/mod.rs`:
- Validates that when querying with `a = 999 AND b = 100` (where `a = 999`
is absent from column `a`'s Bloom filter), `main` fetches Bloom filters for
both column `a` and column `b` upfront (2 requests).
- Documents how incremental evaluation in #25811 will skip column `b`'s
fetch once column `a` prunes the row group.
# Are these changes tested?
Yes, via `test_bloom_filter_io_pattern_baseline` in
`datafusion-datasource-parquet`.
# Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
