Gabriel39 opened a new pull request, #65602:
URL: https://github.com/apache/doris/pull/65602
[improvement](be) Optimize filtered Parquet scans and condition cache
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Parquet v2 repeatedly advanced lazy non-predicate readers
for every fully filtered batch, and external condition cache could not safely
reuse entries when runtime filters were present. Accumulate skipped rows for
lazy readers and flush once before the next surviving batch, discard the lag at
row-group boundaries, expose Arrow skip time, and key scanner-driven condition
cache entries with the exact predicate and runtime-filter snapshot. Runtime
filters without a reliable digest and standalone callers without an explicit
split digest remain uncached.
### Release note
Improve Parquet external scan performance for highly selective predicates
and allow payload-safe condition cache reuse with ready runtime filters.
### Check List (For Author)
- Test: Unit Test
- `./run-be-ut.sh -j 64 --run
--filter='ParquetScanTest.*:NewParquetReaderTest.PageIndexFilteredGapFlushesPendingOutputSkipOnce:TableReaderTest.ConditionCache*:FileScannerV2Test.ConditionCacheDigestIncludesRuntimeFilterPayload'`
- `./run-be-ut.sh -j 64 --run --filter='TableReaderTest.ConditionCache*'`
- `build-support/check-format.sh`
- clang-tidy attempted but blocked by pre-existing remote toolchain
header and NOLINT errors
- Behavior changed: Yes. Lazy Parquet output readers defer skips across
empty selections, and condition cache supports runtime filters only with an
exact split digest.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
