924060929 opened a new pull request, #66259:
URL: https://github.com/apache/doris/pull/66259
### What problem does this PR solve?
Cherry-pick from master #65334.
Problem Summary:
Partition-level refresh / invalidation
(`HiveExternalMetaCache.invalidatePartitionCache`) returned early when the
target partition was absent from the partition metadata cache — evicted under
`max_hive_partition_cache_num` pressure, or simply never loaded — and left that
partition's file listing cache untouched.
The partition cache (`partition`) and the file listing cache (`file`) are
independent caches with independent size limits (`max_hive_partition_cache_num`
vs `max_external_file_cache_num`) and evict separately, so the partition entry
can be evicted while its file listing survives. In that state a partition-level
refresh cannot rebuild the exact `FileCacheKey` (it needs the partition path /
input format that only the cached `HivePartition` carries), so the old code
gave up and the stale file listing remained. A later query would reload the
partition metadata and re-hit the same stale file listing — i.e. "the partition
exists, but Doris keeps seeing the old files/data".
Fix: when the exact `FileCacheKey` cannot be rebuilt, fall back to a
predicate-based invalidation on the file cache keyed by `table id + partition
values`, so the stale listing for that partition is cleared without
over-invalidating other partitions. This mirrors table-level invalidation,
which already invalidates the file cache by predicate.
The partition invalidation coordinator is also decoupled from
`ExternalTable` to `NameMapping` (mirroring the existing
`invalidateTableCache(NameMapping)`) so the path can be unit tested directly.
This behavior has existed since partition-level invalidation was first
introduced in #15702 (the file cache was only invalidated inside the `if
(partition != null)` branch), and was later carried into the current
`HiveExternalMetaCache` by the #60937 cache-framework refactor.
#### Backport notes
Conflicts resolved by hand, because branch-4.1 does not have #64867 ("Key
Hive partition cache by name"):
- `HivePartitionValues` on this branch is still keyed by local partition id
(`partitionNameToIdMap` / `partitionValuesMap`) instead of master's
`nameToPartitionValues`, so the removed block looks different. The replacement
is unchanged: `HiveUtil.toPartitionValues(partitionName)`. This is
behavior-preserving on the happy path — on this branch `partitionValuesMap` is
built from `PartitionKey.createListPartitionKeyWithTypes(..., isHive=true)`,
whose `getPartitionValuesAsStringList()` returns `originHiveKeys`, i.e. exactly
the strings `HiveUtil.toPartitionValues(partitionName)` produced in the first
place.
- The two master-only unit tests that git pulled into the conflict hunk as
context (`testDefaultSpecsFollowConfig`,
`testHivePartitionValuesCopyKeepsIndependentNameMaps`) are not part of #65334
and were dropped; only the new test from #65334 is added.
### Release note
Fix a Hive external catalog consistency issue where a partition-level
refresh could leave a stale file listing cached when the partition metadata
cache had been evicted, causing queries to keep seeing outdated files for that
partition.
### Check List (For Author)
- Test
- [x] Unit Test
- [ ] Regression test
- [ ] Manual test
- [ ] No need to test
- Behavior changed:
- [ ] No.
- [x] Yes. Partition-level refresh now also invalidates the file listing
cache when the partition metadata cache entry is missing (previously it was
left stale).
- Does this need documentation?
- [x] No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
