This is an automated email from the ASF dual-hosted git repository.
924060929 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 90540d32bcb [fix](hive) invalidate stale file cache on partition
refresh when partition cache misses (#65334)
90540d32bcb is described below
commit 90540d32bcb7cdef54aff5502d41bd627a3583a0
Author: 924060929 <[email protected]>
AuthorDate: Mon Jul 13 09:55:41 2026 +0800
[fix](hive) invalidate stale file cache on partition refresh when partition
cache misses (#65334)
Problem Summary:
Partition-level refresh / invalidation
(`HiveExternalMetaCache.invalidatePartitionCache`) returned early when
the target partition was absent from the partition metadata cache —
evicted under `max_hive_partition_cache_num` pressure, or simply never
loaded — and left that partition's file listing cache untouched.
The partition cache (`partition`) and the file listing cache (`file`)
are independent caches with independent size limits
(`max_hive_partition_cache_num` vs `max_external_file_cache_num`) and
evict separately, so the partition entry can be evicted while its file
listing survives. In that state a partition-level refresh cannot rebuild
the exact `FileCacheKey` (it needs the partition path / input format
that only the cached `HivePartition` carries), so the old code gave up
and the stale file listing remained. A later query would reload the
partition metadata and re-hit the same stale file listing — i.e. "the
partition exists, but Doris keeps seeing the old files/data".
Fix: when the exact `FileCacheKey` cannot be rebuilt, fall back to a
predicate-based invalidation on the file cache keyed by `table id +
partition values`, so the stale listing for that partition is cleared
without over-invalidating other partitions. This mirrors table-level
invalidation, which already invalidates the file cache by predicate.
The partition invalidation coordinator is also decoupled from
`ExternalTable` to `NameMapping` (mirroring the existing
`invalidateTableCache(NameMapping)`) so the path can be unit tested
directly.
This behavior has existed since partition-level invalidation was first
introduced in #15702 (the file cache was only invalidated inside the `if
(partition != null)` branch), and was later carried into the current
`HiveExternalMetaCache` by the #60937 cache-framework refactor.
### Release note
Fix a Hive external catalog consistency issue where a partition-level
refresh could leave a stale file listing cached when the partition
metadata cache had been evicted, causing queries to keep seeing outdated
files for that partition.
---
.../datasource/hive/HiveExternalMetaCache.java | 41 +++++++-------
.../datasource/hive/HiveMetaStoreCacheTest.java | 62 ++++++++++++++++++++++
2 files changed, 85 insertions(+), 18 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveExternalMetaCache.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveExternalMetaCache.java
index abc68febcba..54a6eb75b98 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveExternalMetaCache.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveExternalMetaCache.java
@@ -582,7 +582,11 @@ public class HiveExternalMetaCache extends
AbstractExternalMetaCache {
}
public void invalidatePartitionCache(ExternalTable dorisTable, String
partitionName) {
- partitionCacheCoordinator.invalidatePartitionCache(dorisTable,
partitionName);
+ invalidatePartitionCache(dorisTable.getOrBuildNameMapping(),
partitionName);
+ }
+
+ public void invalidatePartitionCache(NameMapping nameMapping, String
partitionName) {
+ partitionCacheCoordinator.invalidatePartitionCache(nameMapping,
partitionName);
}
/**
@@ -614,33 +618,34 @@ public class HiveExternalMetaCache extends
AbstractExternalMetaCache {
}
private final class PartitionCacheCoordinator {
- private void invalidatePartitionCache(ExternalTable dorisTable, String
partitionName) {
- NameMapping nameMapping = dorisTable.getOrBuildNameMapping();
+ private void invalidatePartitionCache(NameMapping nameMapping, String
partitionName) {
long catalogId = nameMapping.getCtlId();
- MetaCacheEntry<PartitionValueCacheKey, HivePartitionValues>
partitionValuesEntry =
- partitionValuesEntryIfInitialized(catalogId);
MetaCacheEntry<PartitionCacheKey, HivePartition> partitionEntry =
partitionEntryIfInitialized(catalogId);
MetaCacheEntry<FileCacheKey, FileCacheValue> fileEntry =
fileEntryIfInitialized(catalogId);
- if (partitionValuesEntry == null || partitionEntry == null ||
fileEntry == null) {
+ if (partitionEntry == null || fileEntry == null) {
return;
}
long tableId = Util.genIdByName(nameMapping.getLocalDbName(),
nameMapping.getLocalTblName());
- PartitionValueCacheKey key = new
PartitionValueCacheKey(nameMapping, null);
- HivePartitionValues partitionValues =
partitionValuesEntry.getIfPresent(key);
- if (partitionValues == null) {
- return;
- }
-
- List<String> values =
partitionValues.nameToPartitionValues.get(partitionName);
- if (values == null) {
- return;
- }
+ // Derive the partition values directly from the partition name
(the partition-values cache is
+ // populated the same way, via HiveUtil.toPartitionValues) instead
of reading them from that
+ // cache, so file cache invalidation still runs when the table's
partition-values cache entry has
+ // been evicted while its file listings are still cached.
+ List<String> values = HiveUtil.toPartitionValues(partitionName);
PartitionCacheKey partKey = new PartitionCacheKey(nameMapping,
values);
HivePartition partition = partitionEntry.getIfPresent(partKey);
if (partition == null) {
+ // Partition metadata cache miss: the exact FileCacheKey
cannot be rebuilt here because it
+ // needs the partition path and input format carried by
HivePartition. Invalidate this
+ // table's cached file listings for the partition by (table id
+ partition values). Scoping
+ // by table id is intentional: matching partition values alone
would also drop other tables'
+ // listings that merely share the same partition value names
(e.g. dt=...) at a different
+ // location, forcing needless re-listing. The exact-key path
below (taken when the partition
+ // is cached) already clears a listing regardless of which
table id populated it.
+ fileEntry.invalidateIf(k -> k.isSameTable(tableId) &&
Objects.equals(k.getPartitionValues(), values));
+ partitionEntry.invalidateKey(partKey);
return;
}
@@ -685,7 +690,7 @@ public class HiveExternalMetaCache extends
AbstractExternalMetaCache {
List<String> modifiedPartNames,
List<String> newPartNames) {
for (String partitionName : modifiedPartNames) {
- invalidatePartitionCache(table, partitionName);
+ invalidatePartitionCache(table.getOrBuildNameMapping(),
partitionName);
}
List<String> mergedPartNames =
Lists.newArrayList(modifiedPartNames);
@@ -772,7 +777,7 @@ public class HiveExternalMetaCache extends
AbstractExternalMetaCache {
nameToPartitionValues.remove(partitionName);
if (invalidPartitionCache) {
- invalidatePartitionCache(dorisTable, partitionName);
+ invalidatePartitionCache(nameMapping, partitionName);
}
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/hive/HiveMetaStoreCacheTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/hive/HiveMetaStoreCacheTest.java
index 931c7ac7e5a..62f70b77982 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/hive/HiveMetaStoreCacheTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/hive/HiveMetaStoreCacheTest.java
@@ -86,6 +86,68 @@ public class HiveMetaStoreCacheTest {
Assertions.assertEquals(0, entrySize(partitionValuesCache));
}
+ @Test
+ public void
testInvalidatePartitionCacheClearsStaleFileCacheOnPartitionMiss() {
+ ThreadPoolExecutor executor =
ThreadPoolManager.newDaemonFixedThreadPool(
+ 1, 1, "refresh", 1, false);
+ ThreadPoolExecutor listExecutor =
ThreadPoolManager.newDaemonFixedThreadPool(
+ 1, 1, "file", 1, false);
+ try {
+ HiveExternalMetaCache cache = new HiveExternalMetaCache(executor,
listExecutor);
+ cache.initCatalog(0, new HashMap<>());
+
+ MetaCacheEntry<HiveExternalMetaCache.FileCacheKey,
HiveExternalMetaCache.FileCacheValue> fileCache =
+ cache.entry(0, HiveExternalMetaCache.ENTRY_FILE,
+ HiveExternalMetaCache.FileCacheKey.class,
+ HiveExternalMetaCache.FileCacheValue.class);
+
+ String dbName = "db";
+ String tbName = "tb";
+ NameMapping nameMapping = NameMapping.createForTest(dbName,
tbName);
+ long catalogId = nameMapping.getCtlId();
+ long tableId = Util.genIdByName(dbName, tbName);
+ long otherTableId = Util.genIdByName(dbName, "tb2");
+
+ String targetPartName = "dt=2024-01-01";
+ List<String> targetValues =
Collections.singletonList("2024-01-01");
+ String otherPartName = "dt=2024-01-02";
+ List<String> otherValues = Collections.singletonList("2024-01-02");
+
+ // Neither the `partition` cache nor the `partition_values` cache
is populated for this table,
+ // simulating entries that were evicted or never loaded.
invalidatePartitionCache must still
+ // clear the stale file listing: it derives the partition values
from the partition name and
+ // cannot rebuild the exact FileCacheKey (which needs the
partition path / input format).
+ HiveExternalMetaCache.FileCacheKey targetFileKey = new
HiveExternalMetaCache.FileCacheKey(
+ catalogId, tableId, "/wh/db/tb/" + targetPartName, "orc",
targetValues);
+ // Same table, a different partition -> must be kept.
+ HiveExternalMetaCache.FileCacheKey otherPartFileKey = new
HiveExternalMetaCache.FileCacheKey(
+ catalogId, tableId, "/wh/db/tb/" + otherPartName, "orc",
otherValues);
+ // A different table that merely shares the same partition value
names at a different location
+ // -> must be kept (the fallback is intentionally scoped by table
id, not by values alone).
+ HiveExternalMetaCache.FileCacheKey otherTableFileKey = new
HiveExternalMetaCache.FileCacheKey(
+ catalogId, otherTableId, "/wh/db/tb2/" + targetPartName,
"orc", targetValues);
+ fileCache.put(targetFileKey, new
HiveExternalMetaCache.FileCacheValue());
+ fileCache.put(otherPartFileKey, new
HiveExternalMetaCache.FileCacheValue());
+ fileCache.put(otherTableFileKey, new
HiveExternalMetaCache.FileCacheValue());
+ Assertions.assertEquals(3, entrySize(fileCache));
+
+ // Partition-level refresh for the target partition. Even though
its `partition` cache entry
+ // is missing, the stale file listing for that partition must
still be invalidated.
+ cache.invalidatePartitionCache(nameMapping, targetPartName);
+
+ Assertions.assertNull(fileCache.getIfPresent(targetFileKey),
+ "stale file cache for the refreshed partition must be
cleared even on partition cache miss");
+ Assertions.assertNotNull(fileCache.getIfPresent(otherPartFileKey),
+ "file cache for other partitions of the same table must
NOT be affected");
+ Assertions.assertNotNull(fileCache.getIfPresent(otherTableFileKey),
+ "file cache for other tables sharing the same partition
values must NOT be affected");
+ Assertions.assertEquals(2, entrySize(fileCache));
+ } finally {
+ executor.shutdownNow();
+ listExecutor.shutdownNow();
+ }
+ }
+
@Test
public void testDefaultSpecsFollowConfig() {
ThreadPoolExecutor executor =
ThreadPoolManager.newDaemonFixedThreadPool(
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
