Gabriel39 opened a new pull request, #65487:
URL: https://github.com/apache/doris/pull/65487
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
Parquet row-group pruning cached Bloom filters only by leaf column ID. Once
a column Bloom filter was loaded for the first row group, subsequent row groups
reused that filter. An equality or IN predicate could therefore prune a later
row group based on an absence result from the first row group's Bloom filter,
silently omitting matching rows.
This change scopes both the cached Bloom filter and its loaded state by
`(row_group_idx, leaf_column_id)`, ensuring every absence decision uses the
current row group's filter.
The test fixture was generated by parquet-mr 1.17.0 with four INT32 rows
split into two row groups, real Bloom filters enabled for `id`, and
statistics/dictionary encoding disabled. The unit test verifies both row groups
expose Bloom filters and compares pruning with Bloom disabled and enabled. A
value present only in RG1 keeps RG1 selected while RG0 is pruned.
### Release note
Fix incorrect Parquet Bloom filter pruning that could silently omit rows
from later row groups.
### Check List (For Author)
- Test:
- Unit Test:
`ParquetStatisticsPruningTest.BloomFilterCacheIsScopedToRowGroupAndColumn`
- BE clang-format v16 check
- BE clang-tidy attempted; blocked by unrelated remote
toolchain/pre-existing diagnostics (`stddef.h` lookup and existing
NOLINT/readability findings)
- Behavior changed: Yes. Parquet Bloom pruning now uses each row group's own
filter.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
