This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new f2b8e022359 [fix](metacache) Refresh external table lookups on a cold
master (#69720)
f2b8e022359 is described below
commit f2b8e022359c06b9f095719305f16e0f72ae9582
Author: Gabriel <[email protected]>
AuthorDate: Thu Oct 29 11:55:52 2026 +0800
[fix](metacache) Refresh external table lookups on a cold master (#69720)
### What problem does this PR solve?
Related PR: #69501
`external_table_p0/hive/test_hive_external_lookup_refresh.groovy` can
fail in a multi-FE cluster depending on which FE serves the regression
connection. `SHOW TABLES` and `DESC` populate the target catalog's
lookup caches on the connected follower only. After the remote table is
dropped through a separate catalog, `REFRESH TABLE` is forwarded to the
master. If the master's target table cache is cold, the old
implementation tries to load the now-deleted table and reports that it
does not exist before invalidating caches or writing a refresh journal.
The follower therefore retains stale table membership and the suite
stops at `REFRESH TABLE`.
Use the cached local table identity when available, then share
name-based cache invalidation between command execution and journal
replay through a private helper. The command validates the
catalog/database and records the invalidation for other FEs; replay
validates the journal and dispatches rename operations separately.
Remove the unused `ignoreIfNotExists` argument to make this invalidation
contract explicit. A remote table object is no longer required just to
invalidate stale metadata. Catalog/database validation and the command's
privilege checks remain in place.
Coverage includes:
- A cold master and a warm follower, with `lower_case_table_names` set
to 0, 1, and 2; serialized refresh logs; follower table-name, object,
and ID cache eviction; and table recreation.
- Missing-database rejection without a journal, a cold table whose
remote metadata loader must not be called, repeated refresh of a
never-existing table name, and actual mixed-case master/follower
identities in modes 1/2.
- A regression suite that warms a non-master FE, issues refreshes
through both master and non-master connections, checks all live FEs,
repeats refresh after deletion, and verifies the recreated schema. The
suite also runs on a single FE.
The Iceberg nested-namespace suite also covers repeated refreshes of an
absent table name using both fully qualified and current-catalog names.
A subsequent query must still reject that table, and refresh must still
reject a missing database. This replaces the old missing-table refresh
exception assertion, which conflicts with the invalidation contract.
### Release note
Fix external table refresh failures when a dropped table is cached on a
follower but absent from the master's cache. `REFRESH TABLE` now accepts
an absent table name in an existing external database as a cache
invalidation request, so success does not depend on which FE retains an
old table object.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- Behavior changed:
- [x] Yes. An absent table name in an existing external database can be
explicitly refreshed to evict stale metadata on other FEs.
- Does this need documentation?
- [x] Yes: [English and Chinese REFRESH
documentation](https://github.com/apache/doris-website/pull/4539).
### Validation
- The new parameterized unit test fails on the original implementation
for all three case modes with the same missing-table error.
- 89 tests passed after the shared-helper refactor across
`RefreshManagerTest`, `RefreshTableTest`,
`PluginDrivenMissingTableTest`, and `ExternalDatabaseTest`.
- Repository FE Checkstyle passed after the shared-helper refactor.
- The new Groovy suite compiles.
- At commit `e77bfeab48`, on a two-FE cluster with Hive 3 Metastore,
both the original failing suite and the new multi-FE suite pass when the
regression runner connects to the non-master FE. The original suite also
passed with a master connection. The new suite issues refreshes through
both FEs and checks cache visibility on each FE.
- After the nested-namespace assertion update, `build.sh --fe -j 8`
succeeds. Using that FE build with Iceberg REST 1.10.0 and MinIO, the
original full nested-namespace suite reproduces the old exception
assertion failure, while the updated full suite passes (1 suite, 0
failures, 0 skips).
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
---
.../org/apache/doris/catalog/RefreshManager.java | 77 ++++++------
.../commands/refresh/RefreshTableCommand.java | 2 +-
.../apache/doris/catalog/RefreshManagerTest.java | 140 ++++++++++++++++++++-
.../org/apache/doris/catalog/RefreshTableTest.java | 2 +-
.../plugin/PluginDrivenMissingTableTest.java | 6 +-
...st_hive_external_lookup_refresh_multi_fe.groovy | 116 +++++++++++++++++
.../iceberg_and_internal_nested_namespace.groovy | 11 +-
7 files changed, 309 insertions(+), 45 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/RefreshManager.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/RefreshManager.java
index 9c1eaaf899d..de558d6ec3e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/RefreshManager.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/RefreshManager.java
@@ -142,8 +142,11 @@ public class RefreshManager {
LOG.info("refresh database {} in catalog {}", db.getFullName(),
db.getCatalog().getName());
}
- // Refresh table
- public void handleRefreshTable(String catalogName, String dbName, String
tableName, boolean ignoreIfNotExists)
+ /**
+ * Invalidates a table name in an existing external database without
requiring a remote table.
+ * A cold master must still invalidate stale lookups retained by other FEs.
+ */
+ public void handleRefreshTable(String catalogName, String dbName, String
tableName)
throws DdlException {
Env env = Env.getCurrentEnv();
CatalogIf catalog = catalogName != null ?
env.getCatalogMgr().getCatalog(catalogName) : env.getCurrentCatalog();
@@ -154,25 +157,19 @@ public class RefreshManager {
throw new DdlException("Only support refresh ExternalCatalog
Tables");
}
- DatabaseIf db = catalog.getDbNullable(dbName);
+ ExternalCatalog externalCatalog = (ExternalCatalog) catalog;
+ ExternalDatabase<?> db = externalCatalog.getDbNullable(dbName);
if (db == null) {
- if (!ignoreIfNotExists) {
- throw new DdlException("Database " + dbName + " does not exist
in catalog " + catalog.getName());
- }
- return;
+ throw new DdlException("Database " + dbName + " does not exist in
catalog " + catalog.getName());
}
- TableIf table = db.getTableNullable(tableName);
- if (table == null) {
- if (!ignoreIfNotExists) {
- throw new DdlException("Table " + tableName + " does not exist
in db " + dbName);
- }
- return;
- }
+ // Cached local names preserve mapped identities without depending on
remote table availability.
+ String localTableName = db.getTableForReplay(tableName)
+ .map(ExternalTable::getName).orElse(tableName);
long updateTime = System.currentTimeMillis();
- refreshTableInternal((ExternalDatabase) db, (ExternalTable) table,
updateTime, true);
ExternalObjectLog log =
ExternalObjectLog.createForExplicitRefreshTable(
- catalog.getId(), db.getFullName(), table.getName(),
updateTime);
+ catalog.getId(), db.getFullName(), localTableName, updateTime);
+ applyTableRefresh(externalCatalog, db.getFullName(), localTableName,
updateTime, true);
env.getEditLog().logRefreshExternalTable(log);
}
@@ -198,27 +195,36 @@ public class RefreshManager {
LOG.warn("refresh table replay log has no local name: {}",
log.debugForRefreshTable());
return;
}
- long dbId = Util.genIdByName(catalog.getName(), localDbName);
- long tableId = Util.genIdByName(catalog.getName(), localDbName,
localTableName);
- Optional<ExternalDatabase<? extends ExternalTable>> db =
catalog.getDbForReplay(localDbName);
if (!Strings.isNullOrEmpty(log.getNewTableName())) {
- replayRenameTable(log, catalog, db);
+ replayRenameTable(log, catalog,
catalog.getDbForReplay(localDbName));
return;
}
+ applyTableRefresh(catalog, localDbName, localTableName,
+ log.getLastUpdateTime(), log.isInvalidateTableLookup());
+ }
+
+ /** Applies command and replay cache effects without remote table loading
or journal writes. */
+ private void applyTableRefresh(ExternalCatalog catalog, String
localDbName, String localTableName,
+ long updateTime, boolean invalidateTableLookup) {
+ long catalogId = catalog.getId();
+ long dbId = Util.genIdByName(catalog.getName(), localDbName);
+ long tableId = Util.genIdByName(catalog.getName(), localDbName,
localTableName);
+ boolean invalidateDatabase = invalidateTableLookup &&
catalog.getLowerCaseTableNames() != 0;
+ ExternalMetaCacheMgr cacheMgr =
Env.getCurrentEnv().getExtMetaCacheMgr();
+ Optional<ExternalDatabase<? extends ExternalTable>> db =
catalog.getDbForReplay(localDbName);
if (!db.isPresent()) {
try {
invalidateAllConnectorCachesIfPresent(catalog);
} finally {
- if (log.isInvalidateTableLookup() &&
catalog.getLowerCaseTableNames() != 0) {
-
Env.getCurrentEnv().getExtMetaCacheMgr().invalidateDb(log.getCatalogId(), dbId,
localDbName);
+ if (invalidateDatabase) {
+ cacheMgr.invalidateDb(catalogId, dbId, localDbName);
} else {
- Env.getCurrentEnv().getExtMetaCacheMgr().invalidateTable(
- log.getCatalogId(), dbId, localDbName, tableId,
localTableName);
+ cacheMgr.invalidateTable(catalogId, dbId, localDbName,
tableId, localTableName);
}
}
- LOG.info("database object cache is cold when replaying refresh
table; "
- + "invalidated caches by local names {}.{}: {}",
- localDbName, localTableName, log.debugForRefreshTable());
+ LOG.info("database object cache is cold when refreshing table; "
+ + "invalidated caches by local names {}.{} in
catalog {}",
+ localDbName, localTableName, catalogId);
return;
}
Optional<? extends ExternalTable> table =
db.get().getTableForReplay(localTableName);
@@ -230,24 +236,23 @@ public class RefreshManager {
}
} finally {
try {
- if (log.isInvalidateTableLookup()) {
+ if (invalidateTableLookup) {
db.get().invalidateTableLookup(localTableName);
}
} finally {
- if (log.isInvalidateTableLookup() &&
catalog.getLowerCaseTableNames() != 0) {
- Env.getCurrentEnv().getExtMetaCacheMgr().invalidateDb(
- log.getCatalogId(), dbId,
db.get().getFullName());
+ if (invalidateDatabase) {
+ cacheMgr.invalidateDb(catalogId, dbId,
db.get().getFullName());
} else {
-
Env.getCurrentEnv().getExtMetaCacheMgr().invalidateTable(
- log.getCatalogId(), dbId,
db.get().getFullName(), tableId, localTableName);
+ cacheMgr.invalidateTable(catalogId, dbId,
db.get().getFullName(), tableId, localTableName);
}
}
}
- LOG.info("table object cache is cold when replaying refresh table;
invalidated caches by local name {}: {}",
- localTableName, log.debugForRefreshTable());
+ LOG.info("table object cache is cold when refreshing table; "
+ + "invalidated caches by local name {}.{} in
catalog {}",
+ db.get().getFullName(), localTableName, catalogId);
return;
}
- refreshTableInternal(db.get(), table.get(), log.getLastUpdateTime(),
log.isInvalidateTableLookup());
+ refreshTableInternal(db.get(), table.get(), updateTime,
invalidateTableLookup);
}
private void replayRenameTable(ExternalObjectLog log, ExternalCatalog
catalog,
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/refresh/RefreshTableCommand.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/refresh/RefreshTableCommand.java
index 2b88537556d..07fc1073865 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/refresh/RefreshTableCommand.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/refresh/RefreshTableCommand.java
@@ -52,7 +52,7 @@ public class RefreshTableCommand extends Command implements
ForwardWithSync {
checkRefreshTableAccess();
// refresh table execute logic
Env.getCurrentEnv().getRefreshManager()
- .handleRefreshTable(tableNameInfo.getCtl(),
tableNameInfo.getDb(), tableNameInfo.getTbl(), false);
+ .handleRefreshTable(tableNameInfo.getCtl(),
tableNameInfo.getDb(), tableNameInfo.getTbl());
}
@Override
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/catalog/RefreshManagerTest.java
b/fe/fe-core/src/test/java/org/apache/doris/catalog/RefreshManagerTest.java
index dedd1b42b82..6fc0617eeac 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/catalog/RefreshManagerTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/RefreshManagerTest.java
@@ -19,6 +19,7 @@ package org.apache.doris.catalog;
import org.apache.doris.catalog.constraint.ConstraintManager;
import org.apache.doris.catalog.info.TableNameInfo;
+import org.apache.doris.common.DdlException;
import org.apache.doris.common.jmockit.Deencapsulation;
import org.apache.doris.common.util.Util;
import org.apache.doris.connector.cache.CacheSpec;
@@ -179,7 +180,7 @@ public class RefreshManagerTest {
Assertions.assertNotNull(oldTable);
MutableCatalogProvider.putTable("new_column");
- new RefreshManager().handleRefreshTable(CATALOG_NAME, DATABASE_NAME,
TABLE_NAME, false);
+ new RefreshManager().handleRefreshTable(CATALOG_NAME, DATABASE_NAME,
TABLE_NAME);
Assertions.assertNull(database.getCachedTableNamesForTest());
Assertions.assertNull(database.getCachedTableForTest(TABLE_NAME));
@@ -190,6 +191,137 @@ public class RefreshManagerTest {
Assertions.assertTrue(log.getValue().isInvalidateTableLookup());
}
+ @ParameterizedTest
+ @ValueSource(ints = {0, 1, 2})
+ public void
testExplicitRefreshOnColdMasterInvalidatesDroppedTableOnFollower(int mode)
throws Exception {
+ useMutableCatalog(mode);
+ TestExternalCatalog followerCatalog = catalog;
+ CountingDatabase followerDatabase = database;
+ TestExternalTable staleTable =
followerDatabase.getTableNullable(TABLE_NAME);
+ Assertions.assertNotNull(staleTable);
+
Assertions.assertNotNull(followerDatabase.getCachedTableNamesForTest());
+
+ // Each FE owns its lookup cache; warming the follower does not warm
the elected master.
+ useMutableCatalog(mode);
+ Assertions.assertNull(database.getCachedTableNamesForTest());
+ Assertions.assertNull(database.getCachedTableForTest(TABLE_NAME));
+ MutableCatalogProvider.removeTable();
+
+ Assertions.assertDoesNotThrow(() ->
+ new RefreshManager().handleRefreshTable(CATALOG_NAME,
DATABASE_NAME, TABLE_NAME));
+
+ Assertions.assertEquals(0, database.buildTableCalls.get());
+ Assertions.assertNull(database.getCachedTableNamesForTest());
+ ArgumentCaptor<ExternalObjectLog> logCaptor =
ArgumentCaptor.forClass(ExternalObjectLog.class);
+ Mockito.verify(editLog).logRefreshExternalTable(logCaptor.capture());
+ ExternalObjectLog log = GsonUtils.GSON.fromJson(
+ GsonUtils.GSON.toJson(logCaptor.getValue()),
ExternalObjectLog.class);
+ Assertions.assertTrue(log.isInvalidateTableLookup());
+ Assertions.assertEquals(CATALOG_ID, log.getCatalogId());
+ Assertions.assertEquals(DATABASE_NAME, log.getDbName());
+ Assertions.assertEquals(TABLE_NAME, log.getTableName());
+
+ testingCatalogMgr.setCatalog(followerCatalog);
+ new RefreshManager().replayRefreshTable(log);
+
+ Assertions.assertNull(followerDatabase.getCachedTableNamesForTest());
+
Assertions.assertNull(followerDatabase.getCachedTableForTest(TABLE_NAME));
+
Assertions.assertNull(followerDatabase.getCachedTableNameByIdForTest(TABLE_ID));
+
Assertions.assertTrue(followerDatabase.getTableNamesWithLock().isEmpty());
+ Assertions.assertNull(followerDatabase.getTableNullable(TABLE_NAME));
+
+ MutableCatalogProvider.putTable("replacement_column");
+ followerDatabase.invalidateTableLookup(TABLE_NAME);
+ TestExternalTable replacement =
followerDatabase.getTableNullable(TABLE_NAME);
+ Assertions.assertNotNull(replacement);
+ Assertions.assertNotSame(staleTable, replacement);
+ }
+
+ @Test
+ public void testExplicitRefreshRejectsMissingDatabaseWithoutJournal() {
+ useMutableCatalog();
+
+ Assertions.assertThrows(DdlException.class, () ->
+ new RefreshManager().handleRefreshTable(CATALOG_NAME,
"missing_db", TABLE_NAME));
+
+ Mockito.verifyNoInteractions(editLog);
+ }
+
+ @Test
+ public void testExplicitRefreshWithColdTableDoesNotLoadRemoteMetadata()
throws Exception {
+ PluginCatalogFixture fixture = usePluginCatalog();
+ Mockito.when(fixture.connector.getMetadata(Mockito.any()))
+ .thenThrow(new AssertionError("Refresh must not load remote
table metadata"));
+
+ new RefreshManager().handleRefreshTable(CATALOG_NAME, DATABASE_NAME,
TABLE_NAME);
+
+ Assertions.assertEquals(0, fixture.database.buildTableCalls.get());
+ Assertions.assertNull(fixture.database.getCachedTableNamesForTest());
+ Mockito.verify(fixture.connector,
Mockito.never()).getMetadata(Mockito.any());
+ Mockito.verify(fixture.connector).invalidateDb(DATABASE_NAME);
+
Mockito.verify(editLog).logRefreshExternalTable(Mockito.any(ExternalObjectLog.class));
+ }
+
+ @Test
+ public void testExplicitRefreshAcceptsAbsentTableNameRepeatedly() throws
Exception {
+ useMutableCatalog();
+ RefreshManager manager = new RefreshManager();
+
+ manager.handleRefreshTable(CATALOG_NAME, DATABASE_NAME,
"absent_table");
+ manager.handleRefreshTable(CATALOG_NAME, DATABASE_NAME,
"absent_table");
+
+ Assertions.assertEquals(0, database.buildTableCalls.get());
+ Assertions.assertNull(database.getCachedTableNamesForTest());
+ ArgumentCaptor<ExternalObjectLog> logs =
ArgumentCaptor.forClass(ExternalObjectLog.class);
+ Mockito.verify(editLog,
Mockito.times(2)).logRefreshExternalTable(logs.capture());
+ for (ExternalObjectLog log : logs.getAllValues()) {
+ Assertions.assertEquals("absent_table", log.getTableName());
+ Assertions.assertTrue(log.isInvalidateTableLookup());
+ }
+ Assertions.assertEquals(Collections.singleton(TABLE_NAME),
database.getTableNamesWithLock());
+ }
+
+ @ParameterizedTest
+ @ValueSource(ints = {1, 2})
+ public void testColdMasterRefreshClearsMixedCaseFollowerIdentity(int mode)
throws Exception {
+ useMutableCatalog(mode);
+ MutableCatalogProvider.removeTable();
+ String remoteName = "MiXeD_Table";
+ MutableCatalogProvider.putTable(remoteName, "old_column");
+ TestExternalCatalog followerCatalog = catalog;
+ CountingDatabase followerDatabase = database;
+ TestExternalTable staleTable =
followerDatabase.getTableNullable("MIXED_TABLE");
+ Assertions.assertNotNull(staleTable);
+ String localName = staleTable.getName();
+
+ // The master receives the SQL spelling, while the follower retains
its canonical local identity.
+ useMutableCatalog(mode);
+ MutableCatalogProvider.removeTable();
+ MutableCatalogProvider.METADATA.get(DATABASE_NAME).remove(remoteName);
+ new RefreshManager().handleRefreshTable(CATALOG_NAME, DATABASE_NAME,
"MIXED_TABLE");
+ Assertions.assertEquals(0, database.buildTableCalls.get());
+ ArgumentCaptor<ExternalObjectLog> logCaptor =
ArgumentCaptor.forClass(ExternalObjectLog.class);
+ Mockito.verify(editLog).logRefreshExternalTable(logCaptor.capture());
+ ExternalObjectLog log = GsonUtils.GSON.fromJson(
+ GsonUtils.GSON.toJson(logCaptor.getValue()),
ExternalObjectLog.class);
+
+ testingCatalogMgr.setCatalog(followerCatalog);
+ new RefreshManager().replayRefreshTable(log);
+
+ Assertions.assertNull(followerDatabase.getCachedTableNamesForTest());
+
Assertions.assertNull(followerDatabase.getCachedTableForTest(localName));
+
Assertions.assertNull(followerDatabase.getCachedTableNameByIdForTest(staleTable.getId()));
+
Assertions.assertTrue(followerDatabase.getTableNamesWithLock().isEmpty());
+ Mockito.verify(metaCacheMgr,
Mockito.times(2)).invalidateDb(CATALOG_ID, DATABASE_ID, DATABASE_NAME);
+
+ MutableCatalogProvider.putTable(remoteName, "replacement_column");
+ followerDatabase.invalidateTableLookup("MIXED_TABLE");
+ TestExternalTable replacement =
followerDatabase.getTableNullable("MIXED_TABLE");
+ Assertions.assertNotNull(replacement);
+ Assertions.assertNotSame(staleTable, replacement);
+ Assertions.assertEquals(localName, replacement.getName());
+ }
+
@Test
public void
testExplicitRefreshReplayRetainsReplacementUpdateTimeBeforeSchemaLoad() {
useMutableCatalog();
@@ -576,8 +708,12 @@ public class RefreshManagerTest {
private static final Map<String, Map<String, List<Column>>> METADATA =
new HashMap<>();
static void putTable(String columnName) {
+ putTable(TABLE_NAME, columnName);
+ }
+
+ static void putTable(String tableName, String columnName) {
METADATA.computeIfAbsent(DATABASE_NAME, ignored -> new HashMap<>())
- .put(TABLE_NAME, Collections.singletonList(new
Column(columnName, PrimitiveType.INT)));
+ .put(tableName, Collections.singletonList(new
Column(columnName, PrimitiveType.INT)));
}
static void removeTable() {
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/catalog/RefreshTableTest.java
b/fe/fe-core/src/test/java/org/apache/doris/catalog/RefreshTableTest.java
index 40252053a5d..668243583bc 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/catalog/RefreshTableTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/RefreshTableTest.java
@@ -97,7 +97,7 @@ public class RefreshTableTest extends TestWithFeService {
TableNameInfo tableNameInfo = new TableNameInfo("test1", "db1",
"tbl11");
try {
Env.getCurrentEnv().getRefreshManager()
- .handleRefreshTable(tableNameInfo.getCtl(),
tableNameInfo.getDb(), tableNameInfo.getTbl(), false);
+ .handleRefreshTable(tableNameInfo.getCtl(),
tableNameInfo.getDb(), tableNameInfo.getTbl());
} catch (Exception e) {
// Do nothing
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenMissingTableTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenMissingTableTest.java
index 6062697ee67..75787df0b73 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenMissingTableTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenMissingTableTest.java
@@ -298,7 +298,7 @@ public class PluginDrivenMissingTableTest extends
TestWithFeService {
database.addTableForTest(replacement);
schemas().put(new
SchemaCacheKey(replacement.getOrBuildNameMapping()),
replacement.initSchema().get());
} else {
- new RefreshManager().handleRefreshTable(catalog.getName(), "db",
"a", false);
+ new RefreshManager().handleRefreshTable(catalog.getName(), "db",
"a");
replacement = database.getTableNullable("a");
}
List<Column> newSchema = replacement.getFullSchema();
@@ -464,7 +464,7 @@ public class PluginDrivenMissingTableTest extends
TestWithFeService {
try {
Assertions.assertTrue(captured.await(10, TimeUnit.SECONDS));
remoteTables.put("a", "new_column");
- new RefreshManager().handleRefreshTable(catalog.getName(), "db",
"a", false);
+ new RefreshManager().handleRefreshTable(catalog.getName(), "db",
"a");
Assertions.assertNull(original.getName(old.getId()));
proceed.countDown();
PluginDrivenExternalTable replacement = lookup.get(10,
TimeUnit.SECONDS);
@@ -493,7 +493,7 @@ public class PluginDrivenMissingTableTest extends
TestWithFeService {
remoteCache.put("FOO", "second");
remoteTables.remove("FOO");
remoteTables.put("Foo", "third");
- new RefreshManager().handleRefreshTable(catalog.getName(), "db",
"foo", false);
+ new RefreshManager().handleRefreshTable(catalog.getName(), "db",
"foo");
Assertions.assertTrue(remoteCache.isEmpty());
PluginDrivenExternalTable replacement =
database.getTableNullable("foo");
Assertions.assertEquals("Foo", replacement.getRemoteName());
diff --git
a/regression-test/suites/external_table_p0/hive/test_hive_external_lookup_refresh_multi_fe.groovy
b/regression-test/suites/external_table_p0/hive/test_hive_external_lookup_refresh_multi_fe.groovy
new file mode 100644
index 00000000000..4c377babbf0
--- /dev/null
+++
b/regression-test/suites/external_table_p0/hive/test_hive_external_lookup_refresh_multi_fe.groovy
@@ -0,0 +1,116 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_hive_external_lookup_refresh_multi_fe", "p0,external") {
+ String enabled = context.config.otherConfigs.get("enableHiveTest")
+ if (enabled == null || !enabled.equalsIgnoreCase("true")) {
+ logger.info("disable Hive test.")
+ return
+ }
+
+ def frontends = sql_return_maparray("SHOW FRONTENDS").findAll {
it.Alive.toBoolean() }
+ def master = frontends.find { it.IsMaster.toBoolean() }
+ assertNotNull(master)
+ def follower = frontends.find { !it.IsMaster.toBoolean() }
+ def refreshEndpoints = follower == null ? [master] : [master, follower]
+ def warmEndpoint = follower == null ? master : follower
+ def onFrontend = { frontend, action ->
+ String host = frontend.Host.contains(":") ? "[${frontend.Host}]" :
frontend.Host
+ String authority = "jdbc:mysql://${host}:${frontend.QueryPort}"
+ int queryOffset = context.config.jdbcUrl.indexOf('?')
+ String options = queryOffset < 0 ? "" :
context.config.jdbcUrl.substring(queryOffset)
+ String url = "${authority}/${options}"
+ connect(context.config.jdbcUser, context.config.jdbcPassword, url) {
+ sql "SET force_forward_all_queries = false"
+ // Wait for catalog DDL and refresh journals without warming any
external table on the master.
+ sql "SYNC"
+ action()
+ }
+ }
+
+ String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+ String hmsPort = context.config.otherConfigs.get("hive3HmsPort")
+ String hdfsPort = context.config.otherConfigs.get("hive3HdfsPort")
+ refreshEndpoints.eachWithIndex { endpoint, index ->
+ String source = "hive_refresh_multi_source_${index}"
+ String target = "hive_refresh_multi_target_${index}"
+ String database = "hive_refresh_multi_db_${index}"
+ try {
+ onFrontend(master) {
+ [source, target].each { catalog ->
+ sql "DROP CATALOG IF EXISTS ${catalog}"
+ sql """CREATE CATALOG ${catalog} PROPERTIES (
+ 'type' = 'hms',
+ 'hadoop.username' = 'hadoop',
+ 'hive.metastore.uris' =
'thrift://${externalEnvIp}:${hmsPort}',
+ 'fs.defaultFS' = 'hdfs://${externalEnvIp}:${hdfsPort}'
+ )"""
+ }
+ sql "DROP DATABASE IF EXISTS ${source}.${database} FORCE"
+ sql "CREATE DATABASE ${source}.${database}"
+ sql "CREATE TABLE ${source}.${database}.refresh_target
(before_col INT)"
+ sql "CREATE TABLE ${source}.${database}.refresh_sibling
(sibling_col INT)"
+ }
+
+ // Only the follower retains the old target object; REFRESH must
also work on a cold master.
+ onFrontend(warmEndpoint) {
+ assertEquals([["refresh_sibling"], ["refresh_target"]],
+ sql("SHOW TABLES FROM ${target}.${database}").sort {
it[0] })
+ assertEquals("before_col", sql("DESC
${target}.${database}.refresh_target")[0][0])
+ }
+ onFrontend(master) {
+ sql "DROP TABLE ${source}.${database}.refresh_target"
+ }
+ onFrontend(endpoint) {
+ sql "REFRESH TABLE ${target}.${database}.refresh_target"
+ // A second refresh must remain valid after all cached copies
of the dropped table are gone.
+ sql "REFRESH TABLE ${target}.${database}.refresh_target"
+ }
+ frontends.each { frontend ->
+ onFrontend(frontend) {
+ // Check membership before a failed lookup can repair it
as a side effect.
+ assertEquals([["refresh_sibling"]], sql("SHOW TABLES FROM
${target}.${database}"))
+ test {
+ sql "SELECT * FROM
${target}.${database}.refresh_target"
+ exception "does not exist"
+ }
+ assertEquals("sibling_col", sql("DESC
${target}.${database}.refresh_sibling")[0][0])
+ }
+ }
+
+ onFrontend(master) {
+ sql "CREATE TABLE ${source}.${database}.refresh_target
(after_col BIGINT)"
+ }
+ onFrontend(endpoint) {
+ sql "REFRESH TABLE ${target}.${database}.refresh_target"
+ }
+ frontends.each { frontend ->
+ onFrontend(frontend) {
+ assertEquals([["refresh_sibling"], ["refresh_target"]],
+ sql("SHOW TABLES FROM ${target}.${database}").sort
{ it[0] })
+ assertEquals("after_col", sql("DESC
${target}.${database}.refresh_target")[0][0])
+ }
+ }
+ } finally {
+ onFrontend(master) {
+ try_sql "DROP DATABASE IF EXISTS ${source}.${database} FORCE"
+ try_sql "DROP CATALOG IF EXISTS ${target}"
+ try_sql "DROP CATALOG IF EXISTS ${source}"
+ }
+ }
+ }
+}
diff --git
a/regression-test/suites/external_table_p0/iceberg/iceberg_and_internal_nested_namespace.groovy
b/regression-test/suites/external_table_p0/iceberg/iceberg_and_internal_nested_namespace.groovy
index abf060b85ce..f4e11f6dde1 100644
---
a/regression-test/suites/external_table_p0/iceberg/iceberg_and_internal_nested_namespace.groovy
+++
b/regression-test/suites/external_table_p0/iceberg/iceberg_and_internal_nested_namespace.groovy
@@ -204,9 +204,16 @@ suite("iceberg_and_internal_nested_namespace",
"p0,external") {
sql """refresh database `ns1.ns2`"""
sql """refresh table ${catalog_name}.`ns1.ns2`.nested_tbl2"""
sql """refresh table `ns1.ns2`.nested_tbl2"""
+ // Cache invalidation must succeed even when no table object exists on the
executing FE.
+ sql """refresh table ${catalog_name}.`ns1.ns2`.nested_tbl2xxx"""
+ sql """refresh table `ns1.ns2`.nested_tbl2xxx"""
test {
- sql """refresh table `ns1.ns2`.nested_tbl2xxx"""
- exception """Table nested_tbl2xxx does not exist in db ns1.ns2"""
+ sql """select * from `ns1.ns2`.nested_tbl2xxx"""
+ exception """Table [nested_tbl2xxx] does not exist in database
[ns1.ns2]"""
+ }
+ test {
+ sql """refresh table ${catalog_name}.`ns1.ns2.missing`.nested_tbl2"""
+ exception """Database ns1.ns2.missing does not exist in catalog
${catalog_name}"""
}
// drop ns1.ns2 first, we can still see it after refresh, because
ns1.ns2.ns3 still exists
sql """drop database `ns1.ns2` force"""
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
