mohitgurav20 commented on PR #25811:
URL: https://github.com/apache/datafusion/pull/25811#issuecomment-6486054003
@alamb Thank you for the careful review and for raising the I/O pattern
concern.
You’re right that we should make the change in request behavior explicit and
testable. Here’s the current situation and how I plan to address your points:
1. **I/O pattern change**
Yes — Bloom filters are now fetched column-by-column and evaluation
happens as soon as each filter arrives. As soon as any column can prove the row
group cannot match, we stop requesting the remaining filters for that group.
This is intentional (to avoid unnecessary object-store reads on multi-column
predicates), but I agree we need clearer evidence that the new pattern is
beneficial (or at least not harmful) under realistic latency.
2. **Code duplication**
The `in_guarantees` extraction and the direct `is_known_not_present`
checks do overlap with logic that already exists in `prune_by_bloom_filters`.
I’ll extract the shared pieces so we don’t maintain two similar paths.
3. **Better I/O visibility tests**
I’ll follow the direction you suggested and look at the arrow-rs async
reader I/O snapshot style tests. My plan is:
- Add (or extend) tests that record the sequence of Bloom-filter reads
per row group.
- Show the difference between “load everything up front” vs “incremental
+ early exit”.
- Keep the existing Criterion benchmarks for the reject-first /
reject-last / survive-all cases.
I’ll also wait for @haohuaijin’s input since he filed the original issue.
Thanks again for the guidance — I’ll push the cleanup + stronger I/O tests
in the next update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
