morningman opened a new pull request, #68839:
URL: https://github.com/apache/doris/pull/68839
### What problem does this PR solve?
Issue Number: None
Related PR: #66717, #67726
Problem Summary:
**In short.** This ports the test coverage of branch-4.1 #66717 (memory
governance for external metadata caches) to master. master already has the
feature (#67726), but none of the 4.x tests came along, and several admission
rules are now unpinned. Only tests are added; no production code changes.
**Background.** #66717 bounds the estimated retained memory of external
metadata caches with three limits: the FE-wide
`external_meta_cache_max_weight`, the catalog property `meta.cache.max-weight`,
and per-entry `meta.cache.<engine>.<entry>.max-weight`. master re-implemented
it in #67726 on the connector cache framework instead of cherry-picking it, so
the 4.x tests, written against fe-core classes master does not have
(`ExternalMetaCacheBudgetManager`, `MetaCacheEntry`,
`IcebergExternalMetaCache`, ...), could not be carried over.
| | branch-4.1 #66717 | master #67726 |
|---|---|---|
| Where it lives | fe-core `datasource/metacache` | `fe-connector-cache`
plus each connector |
| Budget levels | global / catalog / entry | global / catalog / entry group
(physical caches of one logical entry share it) |
| Cached values | soft values | strong values with explicit reservations |
| Budget short on admission | synchronously evict the entry's coldest value,
then admit | reject this load, reclaim from peers in the background, admit on
the next miss |
**The problem, and what it cost.** The admission logic itself is equivalent,
but on master:
- `MetaCacheBudgetManager` has no test of its own; it is only reached
through caches built on top of it. The three-level limits, clamping a catalog
replayed from an FE with a larger global quota, the max(old, new) hand-off
between two generations of one key, force release on close, and the rules for
whom a peer reclamation may ask (only the requester's catalog when the catalog
limit is short, only the same budget group when the group limit is short,
queued requests collapsed to the largest one) could all regress without a red
test.
- At the cache level, zero and incomplete estimates, disabled entries,
oversized values, exact accounting when a key is replaced by a smaller or
larger value, reclamation across catalogs when only the global quota is short,
and failed refreshes are not pinned.
- At the connector level nothing checks which Hive and Iceberg caches a
catalog or entry limit actually reaches, that invalidating a Paimon table gives
its reservation back, or that statement copies of historical Iceberg snapshots
stay apart from the cached generation.
**How this PR fixes it.** It adds those checks in master's terms:
| Test | What it pins |
|---|---|
| `MetaCacheBudgetManagerTest` (new, 12) | global / catalog / entry limits;
200 concurrent reservations never exceed the global quota; clamping to the
smallest enclosing limit; weight parsing (binary units, heap percentages,
rejected forms); idempotent release; replacement commit and rollback; closed
budgets; close-time force release; peer reclamation coalescing and its catalog
/ group scoping; catalog and group bucket lifecycle |
| `MetaCacheWeightGovernanceTest` (+6) | zero or incomplete estimates are
returned but not cached; a disabled entry never estimates; an oversized value
does not evict cached ones; same-key replacement charges exactly the current
generation; a catalog short only on the global quota gets it back from another
catalog; a failed refresh keeps the cached value and its reservation |
| `HiveConnectorWeightGovernanceTest` (new, 2) | a catalog limit bounds the
connector's file listing and partition view caches and closing the connector
releases them; an entry limit bounds only its own entry |
| `PaimonMetaCacheCatalogTest` (+1) | invalidating a table releases its
reservation and the reload is admitted again |
| `IcebergConnectorCacheTest` (+1) | which connector caches the catalog,
manifest and partition view limits reach |
| `IcebergTableCacheTest` (+1) | statement copies of historical snapshots
are not the cached generation's objects |
Not ported, because master deliberately differs and has its own tests for
its choice: soft values, synchronous local eviction before admission, keeping
the old value when a replacement is rejected, and Iceberg generation fencing.
The `test_hive_meta_cache` message change ("is wrong" to "must be >= -1") does
not apply because master keeps the legacy message.
**Results.** All new tests pass on current master. Gaps in product behavior
found while comparing the two implementations (for example the system table
columns 4.x exposes and master does not) are left to separate PRs.
Original author: @CalvinKirs
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- `run-fe-ut.sh --run` on `MetaCacheBudgetManagerTest`,
`MetaCacheWeightGovernanceTest`, `HiveConnectorWeightGovernanceTest`,
`PaimonMetaCacheCatalogTest`, `IcebergConnectorCacheTest`,
`IcebergTableCacheTest`: 110 tests, 0 failures
- `mvn checkstyle:check` on the four touched modules: 0 violations
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
