suxiaogang223 opened a new pull request, #69196:
URL: https://github.com/apache/doris/pull/69196
### What problem does this PR solve?
Issue Number: N/A
Problem Summary:
Repeated scans of many small external files can outgrow the fixed
1,000-entry footer cache even after warming up. Entry count also gives the same
weight to a tiny footer and a large parsed metadata tree, so operators cannot
size the cache according to BE memory. Both scanner versions additionally use
the old entry count to bypass caching for large scans.
This change introduces `external_file_meta_cache_memory_limit`, a
startup-only budget that accepts bytes/unit suffixes (for example, `512M`) or a
percentage of the effective BE `mem_limit`. It defaults to `1%`. An empty value
restores the legacy `max_external_file_meta_cache_num` mode; `0` disables
caching. A non-positive legacy count continues to disable the cache in either
mode. Invalid budgets and budgets above the effective BE memory limit fail
initialization.
- Charge the shared cache by estimated retained bytes, including parsed
Parquet Thrift metadata, V1/V2 schema trees, nested data types, container
capacities, copied names, ORC tail strings, and cache key/handle/value-wrapper
overhead. Shared nested data types are counted once within each schema.
- Use the existing size-based LRU, cache manager and memory-pressure
capacity adjustment. An entry larger than its shard budget stays owned by the
reader, preserving successful reads without evicting reusable footers.
- Transfer Parquet memory accounting to the cache on insertion and release
it with the last handle, avoiding duplicate reader/cache charges. Pinned
entries remain valid during eviction or concurrent replacement; usage converges
to the budget when readers release them.
- Apply byte-based admission in both scanner versions while retaining the
original large-scan gate in legacy count mode. Preserve existing stale sweeps
and file-version/schema-mapping cache keys.
- Expose a separate `ExternalFileMetaCache` category through the existing
cache capacity, usage, entry-count, hit/miss and memory-tracker metrics.
The budget uses an estimate of retained allocations. String inline storage,
scalar carriers and hash-node overhead are charged conservatively; allocator
fragmentation is outside that estimate. Active readers may temporarily pin more
bytes than the configured cache capacity.
### Release note
External file metadata caching now defaults to a memory budget of 1% of the
effective BE memory limit instead of a fixed 1,000-entry limit. Configure
`external_file_meta_cache_memory_limit` in bytes or as a percentage; set it to
an empty string to retain the previous count-based configuration, or `0` to
disable caching. Configuration changes require a BE restart.
### Check List (For Author)
- Test
- [x] Unit Test: added budget parsing/validation, legacy admission and
disabling, byte eviction, oversized-entry bypass, pinned-entry release,
concurrent replacement, retained metadata/schema sizing and accounting transfer
tests. Added a real native Parquet reader test covering correct row counts with
both cache bypass and a warm footer hit.
- clang-format v16, check-format, build-hygiene and git diff checks
passed.
- clang-tidy ran against the generated UT compilation database with
unavailable PCH inputs removed for analysis; changed-line checks passed.
- Local unit test execution is blocked: `run-be-ut.sh` configured
successfully with local OpenMP discovery settings, then Ninja stopped because
the installed third-party dependencies lack `libhs.a`. `build.sh --be` also
cannot refresh dependencies because the required LLVM 20 toolchain is absent.
No unit-test runtime success or performance improvement is claimed.
- Behavior changed:
- [x] Yes. Default cache sizing and large-scan admission use a memory
budget; legacy count mode remains available. Query results and file identity
rules are preserved.
- Does this need documentation?
- [x] No. Configuration syntax, compatibility and restart requirements
are documented in the configuration declarations and release note.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
