924060929 opened a new pull request, #66260:
URL: https://github.com/apache/doris/pull/66260
### What problem does this PR solve?
Cherry-pick from master #65334.
Problem Summary:
Partition-level refresh / invalidation
(`HiveMetaStoreCache.invalidatePartitionCache`) returned early when the target
partition was absent from the partition metadata cache — evicted under
`max_hive_partition_cache_num` pressure, or simply never loaded — and left that
partition's file listing cache untouched.
The partition cache (`partitionCache`) and the file listing cache
(`fileCache`) are independent caches with independent size limits
(`max_hive_partition_cache_num` vs `max_external_file_cache_num`) and evict
separately, so the partition entry can be evicted while its file listing
survives. In that state a partition-level refresh cannot rebuild the exact
`FileCacheKey` (it needs the partition path / input format that only the cached
`HivePartition` carries), so the old code gave up and the stale file listing
remained. A later query would reload the partition metadata and re-hit the same
stale file listing — i.e. "the partition exists, but Doris keeps seeing the old
files/data".
Fix: when the exact `FileCacheKey` cannot be rebuilt, fall back to a
predicate-based invalidation on the file cache keyed by `table id + partition
values`, so the stale listing for that partition is cleared without
over-invalidating other partitions. This mirrors table-level invalidation,
which already invalidates the file cache by predicate.
The partition invalidation entry point is also decoupled from
`ExternalTable` to `NameMapping` (mirroring the existing
`invalidateTableCache(NameMapping)`) so the path can be unit tested directly.
This behavior has existed since partition-level invalidation was first
introduced in #15702 (the file cache was only invalidated inside the `if
(partition != null)` branch).
#### Backport notes
This is a manual port, not a clean cherry-pick, because branch-4.0 predates
two master changes:
- **#60937 (external meta cache framework)** — branch-4.0 still has the
legacy `HiveMetaStoreCache` rather than `HiveExternalMetaCache`, and has no
`MetaCacheEntry.invalidateIf`. The predicate invalidation therefore walks
`fileCache.asMap().keySet()`, exactly the way `invalidateTableCache` already
does on this branch.
- **#64867 (key Hive partition cache by name)** — `HivePartitionValues` here
is still keyed by local partition id (`partitionNameToIdMap` /
`partitionValuesMap`) instead of master's `nameToPartitionValues`. Replacing
that lookup with `HiveUtil.toPartitionValues(partitionName)` is
behavior-preserving on the happy path: on this branch `partitionValuesMap` is
built from `PartitionKey.createListPartitionKeyWithTypes(..., isHive=true)`,
whose `getPartitionValuesAsStringList()` returns `originHiveKeys`, i.e. exactly
the strings `HiveUtil.toPartitionValues(partitionName)` produced in the first
place.
The unit test from #65334 was ported to the branch-4.0 cache API and
verified to fail without the fix and pass with it.
### Release note
Fix a Hive external catalog consistency issue where a partition-level
refresh could leave a stale file listing cached when the partition metadata
cache had been evicted, causing queries to keep seeing outdated files for that
partition.
### Check List (For Author)
- Test
- [x] Unit Test
- [ ] Regression test
- [ ] Manual test
- [ ] No need to test
- Behavior changed:
- [ ] No.
- [x] Yes. Partition-level refresh now also invalidates the file listing
cache when the partition metadata cache entry is missing (previously it was
left stale).
- Does this need documentation?
- [x] No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
