This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new a5d5a52b3f2 [fix](audit) Fix the audit_log upgrade that never finishes
in cloud mode (#70393)
a5d5a52b3f2 is described below
commit a5d5a52b3f28d19dc0ffe6fa1d197048a43587e1
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Thu Nov 19 00:01:52 2026 +0800
[fix](audit) Fix the audit_log upgrade that never finishes in cloud mode
(#70393)
### Background
When a frontend becomes master, its `InternalSchemaInitializer` daemon
thread brings `__internal_schema.audit_log` to the audit schema of its
version (`InternalSchema.AUDIT_SCHEMA`), retrying every 5 seconds until
that succeeds. A schema change that only adds value columns is light:
the FE updates the metadata and nothing is rewritten. Reordering columns
turns it into a job that rewrites every tablet on a backend, and in
cloud mode that backend comes from the compute group of the current
session. Audit events are loaded by column name, so the column order
does not matter for loading.
### What problem does this PR solve?
Issue Number: None
Related PR: #51363 (introduced the add + reorder upgrade), #52432 (its
pick to branch-3.1)
PR Type: Bugfix
Problem Summary:
In cloud mode, a cluster upgraded to a release that adds audit columns
never gets them: after 4.1.3 -> 4.1.5, for example, `audit_log` has no
`queue_time_ms`, and the master FE logs `Failed to alter audit table
schema ... CONNECT_CONTEXT_NOT_SET` every 5 seconds for as long as it
runs, restarts included. The upgrade was always an add + reorder job,
which in cloud mode needs a compute group, and the initializer's thread
has no session to resolve one. This PR adds missing columns in place by
a light schema change when the table's columns are already in order, as
in every table created since 3.1, and runs the ALTER in a session of its
own, so the rewrite a 3.0 table still needs also works in cloud mode.
Most of the diff is tests.
### Details
- **Change map:** +60 -18 lines of production code, all in
`InternalSchemaInitializer`, and +371 -128 of tests (86% of the added
lines). The code has two parts: (1) `getAuditTableAlterOps` plans the
ALTER: an `ADD COLUMN ... AFTER` op per missing column when the table's
columns are in order, the existing add + reorder otherwise (about 50
lines); (2) `alterAuditSchemaIfNeeded` runs the ALTER inside
`StatisticsUtil.buildConnectContext(false)` (about 10 lines). The tests:
planner cases in `InternalSchemaInitializerTest`, both paths on an FE in
`InternalSchemaAlterTest`, and a new docker suite that restarts the FE
on an older table.
- **Frequency:** Always, in cloud mode, when the master FE finds
`audit_log` short of columns of its schema: after 3.0 -> 3.1/4.0, 4.0 or
4.1.0-4.1.3 -> 4.1.4+/4.2 (`queue_time_ms`), and 4.1/4.2 -> master
(`protocol` and two spill columns). Fresh clusters are not affected.
- **Root cause:** Since #51363, `alterAuditSchemaIfNeeded` always adds a
`ReorderColumnsOp`, so the change is a rewrite job.
`SchemaChangeHandler.createJob` asks each replica for its backend; in
cloud mode `CloudReplica.getBackendId()` ->
`CloudSystemInfoService.getCurrentClusterId()` throws
`CONNECT_CONTEXT_NOT_SET` without a `ConnectContext`, which the
initializer's thread never has. The exception is logged and the loop
retries forever.
- **Why existing tests missed it:** Unit tests:
`InternalSchemaAlterTest` creates the tables fresh, so the upgrade has
nothing to do; the two audit tests of `InternalSchemaInitializerTest`
re-implemented the old logic inside the test and never called production
code; nothing ran the initializer in cloud mode. Regression:
`audit/test_audit_log_queue_time`, `audit/test_audit_log_protocol` and
the cloud `audit/test_audit_log_hint_compute_group_docker` all start on
a fresh audit table; none restarts the FE on an older one.
- **Fix:** Missing columns follow their predecessor in the schema, in
one ALTER, when the schema's columns the table has are in order and come
first: a light change that needs no job, backend or compute group.
Anything else (a 3.0 table, an unknown column in between) keeps add +
reorder, now in a session with ADMIN and, in cloud mode, a compute
group, as `createTable()` of the class already does. The fix belongs
here: only this daemon thread created a rewrite job without a session.
- **Other paths and branches:** `modifyColumnStatsTblSchema()` in the
same thread also alters without a session, but its ops are light on
every table a cloud-capable version created, so it is unchanged;
`modifyTblReplicaCount` skips cloud mode. branch-3.1 (#52432),
branch-4.0, branch-4.1 and branch-4.2 all have the bug.
- **Behavior change:** Shared-nothing clusters, where the upgrade
worked, no longer rewrite an audit table that only lacks columns: `SHOW
ALTER TABLE COLUMN` shows a finished light change (`IndexId` =
`OriginIndexId`, `TransactionId` -1) instead of a rewrite job. In cloud
mode, a table that needs reordering is rewritten on the compute group
resolved for ADMIN, which by the code can start a suspended one.
### Testing
- FE unit tests, run locally with `run-fe-ut.sh` on this head:
`InternalSchemaInitializerTest` (8, 5 new, including a 4.1.3 table
upgraded in place with the schema's types and a 3.0 table reordered),
`InternalSchemaAlterTest` (5, 2 new, run like the daemon on a thread
without a session), `AlterJobV2QueryGlobalsTest` (1). Before the fix the
two new `InternalSchemaAlterTest` cases fail: `expected: <NORMAL> but
was: <SCHEMA_CHANGE>` and `expected: <'admin'@'%'> but was: <null>`.
- New docker suite `audit/test_audit_log_schema_upgrade_docker`: drops
three audit columns and restarts the FE, expecting them back in place
with the same tablets and filled by new audit rows; then moves `catalog`
to the end, restarts, and expects the rewrite job to finish. Run locally
before the rebase onto #63612, which only adds cache invalidation to
these alter paths, with an FE of this branch and BE/MS 4.1.3 (no master
BE image for macOS; that FE build turned two new planner features off
and set `be_exec_version = 10`, not part of this PR):
| Mode | Before the fix | This PR |
|---|---|---|
| cloud | fails: `the audit table was not upgraded`; `Failed to alter
audit table schema ... CONNECT_CONTEXT_NOT_SET` every 5 s | passes, no
such warning |
| shared-nothing | fails: the upgrade rewrote the table, all 8 tablets
replaced | passes; the in-place change also replays on restart |
- No pipeline config in this repository sets `excludeDockerTest =
false`, so whether CI runs the suite depends on TeamCity, which I could
not reach to check.
### Notes for Reviewers
- `alterAuditSchemaIfNeeded` became `@VisibleForTesting public static`,
like `createTbl` of the class, so a test can run it as the daemon does.
- The in-place ops build columns through
`ColumnDef.translateToColumnDefinition()`, as the class's stats-table
code does, because `AddColumnOp` needs a `ColumnDefinition` for its SQL
text; the tests check the resulting types against the schema.
- When picking to a release branch, `testAuditTableOf413AddedInPlace`
expects master-only columns and needs adjusting.
---
.../doris/catalog/InternalSchemaInitializer.java | 78 +++++--
.../doris/alter/InternalSchemaAlterTest.java | 126 +++++++++++
.../catalog/InternalSchemaInitializerTest.java | 234 ++++++++++-----------
.../test_audit_log_schema_upgrade_docker.groovy | 139 ++++++++++++
4 files changed, 431 insertions(+), 146 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/InternalSchemaInitializer.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/InternalSchemaInitializer.java
index ce21e38dd43..410d4deb7a0 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/catalog/InternalSchemaInitializer.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/catalog/InternalSchemaInitializer.java
@@ -20,6 +20,7 @@ package org.apache.doris.catalog;
import org.apache.doris.analysis.ColumnDef;
import org.apache.doris.analysis.ColumnNullableType;
import org.apache.doris.analysis.DbName;
+import org.apache.doris.catalog.info.ColumnPosition;
import org.apache.doris.catalog.info.TableNameInfo;
import org.apache.doris.common.Config;
import org.apache.doris.common.DdlException;
@@ -55,6 +56,7 @@ import com.google.common.collect.Sets;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
+import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
@@ -583,21 +585,70 @@ public class InternalSchemaInitializer extends Thread {
return alterAuditSchemaIfNeeded(auditTable);
}
- private boolean alterAuditSchemaIfNeeded(OlapTable auditTable) {
- List<ColumnDef> expectedSchema = InternalSchema.AUDIT_SCHEMA;
- List<String> expectedColumnNames = expectedSchema.stream()
- .map(ColumnDef::getName)
+ @VisibleForTesting
+ public static boolean alterAuditSchemaIfNeeded(OlapTable auditTable) {
+ List<String> currentColumnNames = auditTable.getBaseSchema().stream()
+ .map(Column::getName)
.map(String::toLowerCase)
.collect(Collectors.toList());
- List<Column> currentColumns = auditTable.getBaseSchema();
- List<String> currentColumnNames = currentColumns.stream()
- .map(Column::getName)
+ List<AlterTableOp> alterOps =
getAuditTableAlterOps(currentColumnNames, InternalSchema.AUDIT_SCHEMA);
+ if (alterOps.isEmpty()) {
+ return true;
+ }
+ TableNameInfo auditTableName = new
TableNameInfo(InternalCatalog.INTERNAL_CATALOG_NAME,
+ FeConstants.INTERNAL_DB_NAME, AuditLoader.AUDIT_LOG_TABLE);
+ AlterTableCommand alterTableCommand = new
AlterTableCommand(auditTableName, alterOps);
+ // A schema change that rewrites the table creates a replica on a
backend for every tablet, and in cloud
+ // mode the backend comes from the compute group of the session. This
daemon thread has no session.
+ try (AutoCloseConnectContext r =
StatisticsUtil.buildConnectContext(false)) {
+ Env.getCurrentEnv().alterTable(alterTableCommand);
+ } catch (Exception e) {
+ LOG.warn("Failed to alter audit table schema", e);
+ return false;
+ }
+ return true;
+ }
+
+ /**
+ * The ops that bring an audit table whose columns are {@code
currentColumnNames} (in lower case) to
+ * {@code expectedSchema}, keeping the columns it has beyond the schema at
the end; none when it is there.
+ *
+ * When the columns the table has of the schema are in the schema's order
and come before any other column, as
+ * in a table an earlier release created when the releases since only
added columns, each missing column is
+ * added right after the column before it in the schema: a light schema
change, which rewrites nothing.
+ * Otherwise the missing columns are appended and the table is reordered,
which rewrites it.
+ */
+ @VisibleForTesting
+ public static List<AlterTableOp> getAuditTableAlterOps(List<String>
currentColumnNames,
+ List<ColumnDef> expectedSchema) {
+ List<String> expectedColumnNames = expectedSchema.stream()
+ .map(ColumnDef::getName)
.map(String::toLowerCase)
.collect(Collectors.toList());
// check if all expected columns are exists and in the right order
if (currentColumnNames.size() >= expectedColumnNames.size()
&& expectedColumnNames.equals(currentColumnNames.subList(0,
expectedColumnNames.size()))) {
- return true;
+ return Collections.emptyList();
+ }
+
+ List<String> presentColumnNames = expectedColumnNames.stream()
+ .filter(currentColumnNames::contains)
+ .collect(Collectors.toList());
+ // With the first column of the schema there, every missing column has
a column before it to follow: one
+ // the table has, or one an earlier op of the same ALTER adds.
+ if (currentColumnNames.contains(expectedColumnNames.get(0))
+ && presentColumnNames.equals(currentColumnNames.subList(0,
presentColumnNames.size()))) {
+ List<AlterTableOp> addColumnOps = Lists.newArrayList();
+ for (int i = 1; i < expectedSchema.size(); i++) {
+ if (!presentColumnNames.contains(expectedColumnNames.get(i))) {
+ ColumnDefinition columnDefinition =
expectedSchema.get(i).translateToColumnDefinition();
+ AddColumnOp addColumnOp = new AddColumnOp(columnDefinition,
+ new ColumnPosition(expectedSchema.get(i -
1).getName()), null, null);
+
addColumnOp.setColumn(columnDefinition.translateToCatalogStyleForSchemaChange());
+ addColumnOps.add(addColumnOp);
+ }
+ }
+ return addColumnOps;
}
List<AlterTableOp> alterClauses = Lists.newArrayList();
@@ -619,16 +670,7 @@ public class InternalSchemaInitializer extends Thread {
newColumnOrders.addAll(removedColumnNames);
ReorderColumnsOp reorderColumnsOp = new
ReorderColumnsOp(newColumnOrders, null, Maps.newHashMap());
alterClauses.add(reorderColumnsOp);
- TableNameInfo auditTableName = new
TableNameInfo(InternalCatalog.INTERNAL_CATALOG_NAME,
- FeConstants.INTERNAL_DB_NAME, AuditLoader.AUDIT_LOG_TABLE);
- AlterTableCommand alterTableCommand = new
AlterTableCommand(auditTableName, alterClauses);
- try {
- Env.getCurrentEnv().alterTable(alterTableCommand);
- } catch (Exception e) {
- LOG.warn("Failed to alter audit table schema", e);
- return false;
- }
- return true;
+ return alterClauses;
}
public static boolean isStatsTableSchemaValid() {
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/alter/InternalSchemaAlterTest.java
b/fe/fe-core/src/test/java/org/apache/doris/alter/InternalSchemaAlterTest.java
index af4901864d9..c210ab37437 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/alter/InternalSchemaAlterTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/alter/InternalSchemaAlterTest.java
@@ -18,25 +18,37 @@
package org.apache.doris.alter;
import org.apache.doris.analysis.ColumnDef;
+import org.apache.doris.analysis.UserIdentity;
+import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.Database;
import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.InternalSchema;
import org.apache.doris.catalog.InternalSchemaInitializer;
import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.OlapTable.OlapTableState;
import org.apache.doris.catalog.Partition;
import org.apache.doris.catalog.PartitionInfo;
import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.catalog.Tablet;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.Config;
import org.apache.doris.common.FeConstants;
import org.apache.doris.common.util.PropertyAnalyzer;
import org.apache.doris.plugin.audit.AuditLoader;
+import org.apache.doris.qe.ConnectContext;
import org.apache.doris.statistics.StatisticConstants;
import org.apache.doris.utframe.TestWithFeService;
+import com.google.common.collect.Lists;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.FutureTask;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
public class InternalSchemaAlterTest extends TestWithFeService {
@Override
@@ -44,6 +56,12 @@ public class InternalSchemaAlterTest extends
TestWithFeService {
return 3;
}
+ @Override
+ protected void beforeCreatingConnectContext() throws Exception {
+ // the schema change handler takes this interval when the Env is
built; a job then finishes in seconds
+ FeConstants.default_scheduler_interval_millisecond = 100;
+ }
+
@Override
protected void runBeforeAll() throws Exception {
Config.allow_replica_on_same_host = true;
@@ -99,4 +117,112 @@ public class InternalSchemaAlterTest extends
TestWithFeService {
OlapTable table =
db.getOlapTableOrAnalysisException(StatisticConstants.PARTITION_STATISTIC_TBL_NAME);
Assertions.assertEquals(PrimitiveType.HLL,
table.getColumn("ndv").getType().getPrimitiveType());
}
+
+ // A table an earlier version created lacks the columns added since, and
has the others in order. They are
+ // added in place, by a light schema change: no job to wait for, and the
table keeps its index and tablets.
+ @Test
+ public void testAuditTableMissingColumnsAddedInPlace() throws Exception {
+ OlapTable table = auditTable();
+ try {
+ alterTableSync("ALTER TABLE __internal_schema.audit_log"
+ + " DROP COLUMN queue_time_ms, DROP COLUMN cpu_time_ms,
DROP COLUMN compute_group");
+ Assertions.assertNull(table.getColumn("queue_time_ms"));
+ long baseIndexId = table.getBaseIndexId();
+ Set<Long> tabletIds = baseTabletIds(table);
+
+ Assertions.assertTrue(alterAuditSchemaAsInitializer(table));
+
+ Assertions.assertEquals(OlapTableState.NORMAL, table.getState());
+ Assertions.assertEquals(auditSchemaColumnNames(),
baseColumnNames(table));
+ Assertions.assertEquals(baseIndexId, table.getBaseIndexId());
+ Assertions.assertEquals(tabletIds, baseTabletIds(table));
+ for (String columnName : Lists.newArrayList("queue_time_ms",
"cpu_time_ms", "compute_group")) {
+ ColumnDef columnDef = InternalSchema.AUDIT_SCHEMA.stream()
+ .filter(def ->
def.getName().equals(columnName)).findFirst().get();
+ Column column = table.getColumn(columnName);
+ Assertions.assertEquals(columnDef.getType(), column.getType(),
columnName);
+ Assertions.assertEquals(columnDef.isAllowNull(),
column.isAllowNull(), columnName);
+ }
+ } finally {
+ waitAlterJobsDone(table);
+ }
+ }
+
+ // A table whose columns are out of order needs the add + reorder job,
which rewrites it. The initializer
+ // creates the job in a session of its own: in cloud mode the job picks
its backends from the compute group
+ // of the session, and the initializer's daemon thread has no session.
+ @Test
+ public void testAuditTableOutOfOrderReorderedInOwnSession() throws
Exception {
+ OlapTable table = auditTable();
+ try {
+ String catalogType = table.getColumn("catalog").getType().toSql();
+ alterTableSync("ALTER TABLE __internal_schema.audit_log DROP
COLUMN catalog, DROP COLUMN queue_time_ms");
+ alterTableSync("ALTER TABLE __internal_schema.audit_log ADD COLUMN
catalog " + catalogType + " NULL");
+ List<String> columnNames = baseColumnNames(table);
+ Assertions.assertEquals("catalog",
columnNames.get(columnNames.size() - 1));
+ Set<Long> jobIds =
schemaChangeJobs(table).stream().map(AlterJobV2::getJobId)
+ .collect(Collectors.toSet());
+
+ Assertions.assertTrue(alterAuditSchemaAsInitializer(table));
+
+ List<AlterJobV2> newJobs = schemaChangeJobs(table).stream()
+ .filter(job ->
!jobIds.contains(job.getJobId())).collect(Collectors.toList());
+ Assertions.assertEquals(1, newJobs.size());
+ Assertions.assertEquals(UserIdentity.ADMIN,
newJobs.get(0).userIdentity);
+ waitAlterJobsDone(table);
+ Assertions.assertEquals(AlterJobV2.JobState.FINISHED,
newJobs.get(0).getJobState());
+ Assertions.assertEquals(auditSchemaColumnNames(),
baseColumnNames(table));
+ } finally {
+ waitAlterJobsDone(table);
+ }
+ }
+
+ private static OlapTable auditTable() throws AnalysisException {
+ return
Env.getCurrentEnv().getCatalogMgr().getInternalCatalog().getDbNullable(FeConstants.INTERNAL_DB_NAME)
+ .getOlapTableOrAnalysisException(AuditLoader.AUDIT_LOG_TABLE);
+ }
+
+ // Runs alterAuditSchemaIfNeeded the way InternalSchemaInitializer does:
on a thread of its own, which
+ // has no session.
+ private static boolean alterAuditSchemaAsInitializer(OlapTable table)
throws Exception {
+ FutureTask<Boolean> task = new FutureTask<>(() -> {
+ Assertions.assertNull(ConnectContext.get());
+ boolean altered =
InternalSchemaInitializer.alterAuditSchemaIfNeeded(table);
+ Assertions.assertNull(ConnectContext.get(), "the thread keeps the
session built for the alter");
+ return altered;
+ });
+ new Thread(task, "InternalSchemaInitializer").start();
+ return task.get(60, TimeUnit.SECONDS);
+ }
+
+ private static List<String> auditSchemaColumnNames() {
+ return
InternalSchema.AUDIT_SCHEMA.stream().map(ColumnDef::getName).collect(Collectors.toList());
+ }
+
+ private static List<String> baseColumnNames(OlapTable table) {
+ return
table.getBaseSchema().stream().map(Column::getName).collect(Collectors.toList());
+ }
+
+ private static Set<Long> baseTabletIds(OlapTable table) {
+ return table.getPartitions().stream().flatMap(partition ->
partition.getBaseIndex().getTablets().stream())
+ .map(Tablet::getId).collect(Collectors.toSet());
+ }
+
+ private static List<AlterJobV2> schemaChangeJobs(OlapTable table) {
+ return
Env.getCurrentEnv().getSchemaChangeHandler().getAlterJobsV2().values().stream()
+ .filter(job -> job.getTableId() ==
table.getId()).collect(Collectors.toList());
+ }
+
+ private static void waitAlterJobsDone(OlapTable table) throws
InterruptedException {
+ for (AlterJobV2 job : schemaChangeJobs(table)) {
+ for (int i = 0; i < 600 && !job.getJobState().isFinalState(); i++)
{
+ Thread.sleep(100);
+ }
+ Assertions.assertTrue(job.getJobState().isFinalState(),
job.toString());
+ }
+ for (int i = 0; i < 600 && table.getState() != OlapTableState.NORMAL;
i++) {
+ Thread.sleep(100);
+ }
+ Assertions.assertEquals(OlapTableState.NORMAL, table.getState());
+ }
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/catalog/InternalSchemaInitializerTest.java
b/fe/fe-core/src/test/java/org/apache/doris/catalog/InternalSchemaInitializerTest.java
index 9584aa034aa..9dc48403be6 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/catalog/InternalSchemaInitializerTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/catalog/InternalSchemaInitializerTest.java
@@ -18,21 +18,25 @@
package org.apache.doris.catalog;
import org.apache.doris.analysis.ColumnDef;
-import org.apache.doris.catalog.info.ColumnPosition;
+import org.apache.doris.analysis.ColumnNullableType;
import org.apache.doris.common.UserException;
-import org.apache.doris.nereids.trees.plans.commands.info.AlterOp;
+import org.apache.doris.nereids.trees.plans.commands.info.AddColumnOp;
+import org.apache.doris.nereids.trees.plans.commands.info.AddColumnsOp;
import org.apache.doris.nereids.trees.plans.commands.info.AlterTableOp;
import org.apache.doris.nereids.trees.plans.commands.info.ModifyColumnOp;
+import org.apache.doris.nereids.trees.plans.commands.info.ReorderColumnsOp;
import org.apache.doris.plugin.audit.AuditLoader;
import org.apache.doris.statistics.StatisticConstants;
import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.mockito.Mockito;
+import java.util.Arrays;
+import java.util.Collections;
import java.util.List;
+import java.util.stream.Collectors;
class InternalSchemaInitializerTest {
@Test
@@ -123,143 +127,117 @@ class InternalSchemaInitializerTest {
"scan_bytes_from_remote_storage field is missing from the
copied schema");
}
+ // An up-to-date table, also one with columns beyond the schema after it,
needs nothing.
@Test
- public void testStorageColumnsPositionInAuditTable() throws Exception {
- // Get storage-related column definitions directly from
InternalSchema.AUDIT_SCHEMA
- ColumnDef localStorageDef = null;
- ColumnDef remoteStorageDef = null;
-
- for (int i = 0; i < InternalSchema.AUDIT_SCHEMA.size(); i++) {
- ColumnDef def = InternalSchema.AUDIT_SCHEMA.get(i);
- if (def.getName().equals("scan_bytes_from_local_storage")) {
- localStorageDef = def;
- } else if (def.getName().equals("scan_bytes_from_remote_storage"))
{
- remoteStorageDef = def;
- }
- }
-
- Assertions.assertNotNull(localStorageDef, "The
scan_bytes_from_local_storage column should exist in AUDIT_SCHEMA");
- Assertions.assertNotNull(remoteStorageDef, "The
scan_bytes_from_remote_storage column should exist in AUDIT_SCHEMA");
-
- // Simulate column position logic in InternalSchemaInitializer
- // Note: Based on test failure, the system uses FIRST position rather
than after a specific column
- List<AlterOp> alterOps = Lists.newArrayList();
-
- // Add scan_bytes_from_local_storage column using FIRST position
- ColumnPosition localStoragePosition = ColumnPosition.FIRST;
- ModifyColumnOp modifyColumnOp = new ModifyColumnOp(
- localStorageDef.translateToColumnDefinition(),
localStoragePosition, null, Maps.newHashMap());
- modifyColumnOp.setColumn(localStorageDef.toColumn());
- alterOps.add(modifyColumnOp);
-
- // Add scan_bytes_from_remote_storage column using FIRST position
- ColumnPosition remoteStoragePosition = ColumnPosition.FIRST;
- ModifyColumnOp modifyColumnOp1 = new ModifyColumnOp(
- remoteStorageDef.translateToColumnDefinition(),
remoteStoragePosition, null, Maps.newHashMap());
- modifyColumnOp1.setColumn(remoteStorageDef.toColumn());
- alterOps.add(modifyColumnOp1);
-
- // Verify the generated AlterClauses
- Assertions.assertEquals(2, alterOps.size(), "Two AlterClauses should
be generated");
+ public void testAuditTableUpToDate() {
+ List<ColumnDef> schema = schema("a", "b", "c");
+ Assertions.assertEquals(Collections.emptyList(),
describe(alterOps(schema, "a", "b", "c")));
+ Assertions.assertEquals(Collections.emptyList(),
describe(alterOps(schema, "a", "b", "c", "x")));
+ }
- // Verify that column positions are FIRST
- Assertions.assertTrue(((ModifyColumnOp)
alterOps.get(0)).getColPos().isFirst(),
- "The position of the scan_bytes_from_local_storage column
should be FIRST");
- Assertions.assertTrue(((ModifyColumnOp)
alterOps.get(1)).getColPos().isFirst(),
- "The position of the scan_bytes_from_remote_storage column
should be FIRST");
+ // The columns the table has of the schema are in order and come first:
each missing column is added right
+ // after the column before it in the schema, by a light schema change.
+ @Test
+ public void testAuditTableMissingColumnsAddedInPlace() {
+ List<ColumnDef> schema = schema("a", "b", "c", "d", "e");
+ Assertions.assertEquals(Lists.newArrayList("ADD b AFTER a", "ADD d
AFTER c"),
+ describe(alterOps(schema, "a", "c", "e")));
+ Assertions.assertEquals(Lists.newArrayList("ADD e AFTER d"),
describe(alterOps(schema, "a", "b", "c", "d")));
+ // the second of two adjacent columns follows the first, which the
same ALTER adds
+ Assertions.assertEquals(Lists.newArrayList("ADD b AFTER a", "ADD c
AFTER b"),
+ describe(alterOps(schema, "a", "d", "e")));
+ // a column beyond the schema stays at the end
+ Assertions.assertEquals(Lists.newArrayList("ADD b AFTER a", "ADD d
AFTER c"),
+ describe(alterOps(schema, "a", "c", "e", "x")));
}
+ // Any other table gets the missing columns appended and is reordered,
which rewrites it.
@Test
- public void testDoesNotModifyExistingColumns() throws Exception {
- // Create a mock audit table with storage-related columns but with
inconsistent types (VARCHAR instead of BIGINT)
- List<Column> initialSchema = Lists.newArrayList(
- new Column("query_id", ScalarType.createVarcharType(48), true,
null, false, null, ""),
- new Column("time", ScalarType.createDatetimeV2Type(3), true,
null, false, null, ""),
- new Column("client_ip", ScalarType.createVarcharType(128),
true, null, false, null, ""),
- new Column("user", ScalarType.createVarcharType(128), true,
null, false, null, ""),
- new Column("catalog", ScalarType.createVarcharType(128), true,
null, false, null, ""),
- new Column("db", ScalarType.createVarcharType(128), true,
null, false, null, ""),
- new Column("state", ScalarType.createVarcharType(128), true,
null, false, null, ""),
- new Column("error_code", ScalarType.INT, true, null, false,
null, ""),
- new Column("error_message", ScalarType.STRING, true, null,
false, null, ""),
- new Column("query_time", ScalarType.BIGINT, true, null, false,
null, ""),
- new Column("scan_bytes", ScalarType.BIGINT, true, null, false,
null, ""),
- new Column("scan_rows", ScalarType.BIGINT, true, null, false,
null, ""),
- new Column("return_rows", ScalarType.BIGINT, true, null,
false, null, ""),
- new Column("shuffle_send_rows", ScalarType.BIGINT, true, null,
false, null, ""),
- new Column("shuffle_send_bytes", ScalarType.BIGINT, true,
null, false, null, ""),
- // Intentionally use inconsistent types (VARCHAR instead of
BIGINT)
- new Column("scan_bytes_from_local_storage",
ScalarType.createVarcharType(128), true, null, false, null, ""),
- new Column("scan_bytes_from_remote_storage",
ScalarType.createVarcharType(128), true, null, false, null, "")
- );
+ public void testAuditTableOutOfOrderReordered() {
+ List<ColumnDef> schema = schema("a", "b", "c", "d", "e");
+ // two columns of the schema swapped
+ Assertions.assertEquals(Lists.newArrayList("ADD (d)", "ORDER BY (a, b,
c, d, e)"),
+ describe(alterOps(schema, "a", "c", "b", "e")));
+ Assertions.assertEquals(Lists.newArrayList("ORDER BY (a, b, c, d, e)"),
+ describe(alterOps(schema, "b", "a", "c", "d", "e")));
+ // a column beyond the schema between two of it
+ Assertions.assertEquals(Lists.newArrayList("ADD (b)", "ORDER BY (a, b,
c, d, e, x)"),
+ describe(alterOps(schema, "a", "x", "c", "d", "e")));
+ // the first column missing: there is no column for it to follow
+ Assertions.assertEquals(Lists.newArrayList("ADD (a)", "ORDER BY (a, b,
c, d, e)"),
+ describe(alterOps(schema, "b", "c", "d", "e")));
+ }
- // Use the correct constructor to create OlapTable to ensure
nameToColumn is properly initialized
- OlapTable auditTable = new OlapTable(1000, "audit_log", initialSchema,
KeysType.AGG_KEYS,
- new SinglePartitionInfo(), new HashDistributionInfo());
+ // A table 4.1.3 created lacks the four columns added since. Each is added
in place, as the audit schema
+ // defines it.
+ @Test
+ public void testAuditTableOf413AddedInPlace() {
+ List<String> added = Lists.newArrayList("queue_time_ms",
"spill_write_bytes_to_remote_storage",
+ "spill_read_bytes_from_remote_storage", "protocol");
+ List<String> currentColumnNames = auditColumnNames();
+ currentColumnNames.removeAll(added);
+
+ List<AlterTableOp> ops =
InternalSchemaInitializer.getAuditTableAlterOps(currentColumnNames,
+ InternalSchema.AUDIT_SCHEMA);
+
+ Assertions.assertEquals(Lists.newArrayList("ADD queue_time_ms AFTER
query_time",
+ "ADD spill_write_bytes_to_remote_storage AFTER
spill_read_bytes_from_local_storage",
+ "ADD spill_read_bytes_from_remote_storage AFTER
spill_write_bytes_to_remote_storage",
+ "ADD protocol AFTER compute_group"), describe(ops));
+ for (AlterTableOp op : ops) {
+ Column column = ((AddColumnOp) op).getColumn();
+ ColumnDef columnDef = InternalSchema.AUDIT_SCHEMA.stream()
+ .filter(def ->
def.getName().equals(column.getName())).findFirst().get();
+ Assertions.assertEquals(columnDef.getType(), column.getType());
+ Assertions.assertEquals(columnDef.isAllowNull(),
column.isAllowNull());
+ Assertions.assertFalse(column.isKey());
+ }
+ }
- // Verify columns exist and have VARCHAR type
- Column localStorageCol =
auditTable.getColumn("scan_bytes_from_local_storage");
- Column remoteStorageCol =
auditTable.getColumn("scan_bytes_from_remote_storage");
+ // A table 3.0 created has its columns in the order the audit schema had
before #51363 moved them around. The
+ // missing columns are appended and the table is reordered.
+ @Test
+ public void testAuditTableOf30Reordered() {
+ List<String> currentColumnNames = Lists.newArrayList("query_id",
"time", "client_ip", "user", "catalog", "db",
+ "state", "error_code", "error_message", "query_time",
"scan_bytes", "scan_rows", "return_rows",
+ "shuffle_send_rows", "shuffle_send_bytes",
"scan_bytes_from_local_storage",
+ "scan_bytes_from_remote_storage", "stmt_id", "stmt_type",
"is_query", "is_nereids", "frontend_ip",
+ "cpu_time_ms", "sql_hash", "sql_digest", "peak_memory_bytes",
"workload_group", "compute_group",
+ "stmt");
+ List<String> missing = auditColumnNames();
+ missing.removeAll(currentColumnNames);
+
+ List<AlterTableOp> ops =
InternalSchemaInitializer.getAuditTableAlterOps(currentColumnNames,
+ InternalSchema.AUDIT_SCHEMA);
+
+ Assertions.assertEquals(Lists.newArrayList("ADD (" + String.join(", ",
missing) + ")",
+ "ORDER BY (" + String.join(", ", auditColumnNames()) + ")"),
describe(ops));
+ }
- Assertions.assertNotNull(localStorageCol, "The
scan_bytes_from_local_storage column should exist in auditTable");
- Assertions.assertNotNull(remoteStorageCol, "The
scan_bytes_from_remote_storage column should exist in auditTable");
- Assertions.assertTrue(localStorageCol.getType().isVarchar(),
- "The scan_bytes_from_local_storage column type should be
VARCHAR");
- Assertions.assertTrue(remoteStorageCol.getType().isVarchar(),
- "The scan_bytes_from_remote_storage column type should be
VARCHAR");
+ private static List<ColumnDef> schema(String... columnNames) {
+ return Arrays.stream(columnNames).map(name -> new ColumnDef(name,
ScalarType.createType(PrimitiveType.BIGINT),
+ ColumnNullableType.NULLABLE)).collect(Collectors.toList());
+ }
- // Get complete column definitions from InternalSchema.AUDIT_SCHEMA
- List<ColumnDef> expectedSchema = Lists.newArrayList();
- for (ColumnDef def : InternalSchema.AUDIT_SCHEMA) {
- expectedSchema.add(def);
- }
+ private static List<AlterTableOp> alterOps(List<ColumnDef> schema,
String... currentColumnNames) {
+ return
InternalSchemaInitializer.getAuditTableAlterOps(Lists.newArrayList(currentColumnNames),
schema);
+ }
- // Simulate column processing logic in InternalSchemaInitializer
- List<AlterOp> alterOps = Lists.newArrayList();
+ private static List<String> auditColumnNames() {
+ return
InternalSchema.AUDIT_SCHEMA.stream().map(ColumnDef::getName).collect(Collectors.toList());
+ }
- // Add columns if they don't exist
- for (int i = 0; i < expectedSchema.size(); i++) {
- ColumnDef def = expectedSchema.get(i);
- if (auditTable.getColumn(def.getName()) == null) {
- // If column doesn't exist, add it
- String afterColumn = null;
- if (i > 0) {
- for (int j = i - 1; j >= 0; j--) {
- String prevColName = expectedSchema.get(j).getName();
- if (auditTable.getColumn(prevColName) != null) {
- afterColumn = prevColName;
- break;
- }
- }
- }
- ColumnPosition position = afterColumn == null ?
ColumnPosition.FIRST :
- new ColumnPosition(afterColumn);
- ModifyColumnOp op = new
ModifyColumnOp(def.translateToColumnDefinition(), position, null,
Maps.newHashMap());
- op.setColumn(def.toColumn());
- alterOps.add(op);
+ private static List<String> describe(List<AlterTableOp> ops) {
+ return ops.stream().map(op -> {
+ if (op instanceof AddColumnOp) {
+ AddColumnOp addColumnOp = (AddColumnOp) op;
+ return "ADD " + addColumnOp.getColumn().getName() + " AFTER "
+ addColumnOp.getColPos().getLastCol();
}
- // Note: InternalSchemaInitializer.created() method does not check
if column types match
- // It only adds columns that don't exist in the table
- }
-
- // Check if AlterClauses were generated for storage-related columns
- boolean hasLocalStorageClause = false;
- boolean hasRemoteStorageClause = false;
-
- for (AlterOp op : alterOps) {
- ModifyColumnOp modifyColumnOp = (ModifyColumnOp) op;
- if
(modifyColumnOp.getColumn().getName().equals("scan_bytes_from_local_storage")) {
- hasLocalStorageClause = true;
- } else if
(modifyColumnOp.getColumn().getName().equals("scan_bytes_from_remote_storage"))
{
- hasRemoteStorageClause = true;
+ if (op instanceof AddColumnsOp) {
+ return "ADD (" + ((AddColumnsOp)
op).getColumns().stream().map(Column::getName)
+ .collect(Collectors.joining(", ")) + ")";
}
- }
-
- // Verify the system does not generate AlterClauses for columns that
already exist
- // even if their types don't match the expected types
- Assertions.assertFalse(hasLocalStorageClause,
- "The system should not generate AlterClause for the
scan_bytes_from_local_storage column that already exists");
- Assertions.assertFalse(hasRemoteStorageClause,
- "The system should not generate AlterClause for the
scan_bytes_from_remote_storage column that already exists");
+ return "ORDER BY (" + String.join(", ", ((ReorderColumnsOp)
op).getColumnsByPos()) + ")";
+ }).collect(Collectors.toList());
}
}
diff --git
a/regression-test/suites/audit/test_audit_log_schema_upgrade_docker.groovy
b/regression-test/suites/audit/test_audit_log_schema_upgrade_docker.groovy
new file mode 100644
index 00000000000..a72d5dd7df4
--- /dev/null
+++ b/regression-test/suites/audit/test_audit_log_schema_upgrade_docker.groovy
@@ -0,0 +1,139 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.apache.doris.regression.suite.ClusterOptions
+
+// When a frontend becomes master, InternalSchemaInitializer brings
__internal_schema.audit_log to the
+// audit schema of its version, so a table an earlier version created gains
the columns added since.
+// A table that only lacks columns gets them in place, by a light schema
change that rewrites nothing.
+// A table whose columns are out of order is rewritten by an add + reorder
job, which in cloud mode has
+// to run on a compute group: the initializer's daemon thread used to have
none, and the upgrade failed
+// with CONNECT_CONTEXT_NOT_SET on every retry.
+suite("test_audit_log_schema_upgrade_docker", "docker") {
+ def options = new ClusterOptions()
+ options.feNum = 1
+ options.beNum = 1
+ // Both modes: the in-place add applies to either, and the rewrite failed
in cloud mode.
+ options.cloudMode = null
+
+ docker(options) {
+ def columnNames = {
+ return (sql "desc __internal_schema.audit_log").collect { it[0] as
String }
+ }
+ // the name, type and nullability of each column, in order
+ def columns = {
+ return (sql "desc __internal_schema.audit_log").collect {
"${it[0]} ${it[1]} ${it[2]}".toString() }
+ }
+ def tabletIds = {
+ return (sql_return_maparray "show tablets from
__internal_schema.audit_log")
+ .collect { it.TabletId as String } as Set
+ }
+ def latestAlterJob = {
+ def jobs = sql_return_maparray """SHOW ALTER TABLE COLUMN FROM
__internal_schema
+ WHERE TableName = 'audit_log' ORDER BY CreateTime DESC
LIMIT 1"""
+ return jobs.isEmpty() ? null : jobs[0]
+ }
+
+ // The first master creates the table with the full schema: the shape
an upgrade must reach.
+ def expectedColumns = null
+ for (int i = 0; i < 60 && expectedColumns == null; i++) {
+ try {
+ expectedColumns = columns()
+ } catch (Exception e) {
+ logger.info("the audit table is not created yet:
${e.getMessage()}")
+ sleep(1000)
+ }
+ }
+ assertNotNull(expectedColumns)
+ assertTrue(columnNames().containsAll(["catalog", "query_time",
"queue_time_ms", "cpu_time_ms",
+ "compute_group"]), "${expectedColumns}")
+
+ // A restarted frontend becomes master again and runs
InternalSchemaInitializer against the
+ // table as it is now.
+ def restartFrontend = {
+ cluster.restartFrontends()
+ for (int i = 0; i < 60; i++) {
+ try {
+ context.reconnectFe()
+ sql "show frontends"
+ return
+ } catch (Exception e) {
+ logger.info("the restarted frontend is not ready yet:
${e.getMessage()}")
+ sleep(1000)
+ }
+ }
+ throw new IllegalStateException("cannot connect to the restarted
frontend")
+ }
+ def waitForAuditSchema = { int timeoutSeconds ->
+ def actual = columns()
+ for (int i = 0; i < timeoutSeconds && actual != expectedColumns;
i++) {
+ sleep(1000)
+ actual = columns()
+ }
+ assertEquals(expectedColumns, actual,
+ "the audit table was not upgraded; latest schema change
job: ${latestAlterJob()}")
+ }
+
+ // 1. A table of an earlier version lacks the columns added since and
has the others in order;
+ // queue_time_ms, for one, came with 4.1.4. Drop it, the column after
it, and one further on.
+ sql """ALTER TABLE __internal_schema.audit_log
+ DROP COLUMN queue_time_ms, DROP COLUMN cpu_time_ms, DROP
COLUMN compute_group"""
+ assertFalse(columnNames().contains("queue_time_ms"))
+ def tabletsBefore = tabletIds()
+
+ restartFrontend()
+ waitForAuditSchema(120)
+ // Added in place: the table keeps its tablets.
+ def tabletsAfter = tabletIds()
+ assertTrue(tabletsAfter.containsAll(tabletsBefore), "tablets before:
${tabletsBefore}, after: ${tabletsAfter}")
+
+ // New audit rows fill the columns the upgrade added.
+ sql "set global enable_audit_plugin = true"
+ def marker = "audit_schema_upgrade_marker_5E1C9A"
+ sql "select 1 as ${marker}"
+ def countMarkedRows = {
+ return (sql """select count(*) from __internal_schema.audit_log
+ where stmt like '%${marker}%' and stmt not like
'%audit_log%'
+ and queue_time_ms is not null and cpu_time_ms is not
null""")[0][0] as long
+ }
+ def found = countMarkedRows()
+ for (int i = 0; i < 60 && found == 0; i++) {
+ sleep(3000)
+ sql "call flush_audit_log()"
+ found = countMarkedRows()
+ }
+ assertTrue(found >= 1, "no audit row of the marked statement fills the
added columns")
+
+ // 2. A table created before #51363 moved the audit columns around has
them out of order. Move
+ // catalog to the end and drop queue_time_ms: the upgrade has to
rewrite the table, which in
+ // cloud mode runs on a compute group.
+ def catalogType = (sql "desc __internal_schema.audit_log").find {
it[0] == "catalog" }[1]
+ sql "ALTER TABLE __internal_schema.audit_log DROP COLUMN catalog, DROP
COLUMN queue_time_ms"
+ sql "ALTER TABLE __internal_schema.audit_log ADD COLUMN catalog
${catalogType} NULL"
+ assertEquals("catalog", columnNames().last())
+ tabletsBefore = tabletIds()
+
+ restartFrontend()
+ waitForAuditSchema(300)
+ def job = latestAlterJob()
+ assertEquals("FINISHED", job.State, "${job}")
+ // Rewritten: none of the old tablets is left.
+ tabletsAfter = tabletIds()
+ assertTrue(tabletsAfter.intersect(tabletsBefore).isEmpty(),
+ "tablets before: ${tabletsBefore}, after: ${tabletsAfter}")
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]