This is an automated email from the ASF dual-hosted git repository.
jerryshao pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new b9f4786c46 [#13346] fix(doris): Preserve table comments on Doris 2.1.0
(#13588)
b9f4786c46 is described below
commit b9f4786c4678f349d6eeba3fd18bffd8f1f48f06
Author: Qi Yu <[email protected]>
AuthorDate: Tue Sep 29 22:20:05 2026 +0800
[#13346] fix(doris): Preserve table comments on Doris 2.1.0 (#13588)
### What changes were proposed in this pull request?
After creating a Doris table, compare the stored comment with the
submitted comment and restore it with `ALTER TABLE ... MODIFY COMMENT`
when they differ. Preserve the Gravitino ID suffix and SQL escaping,
reuse the metadata lookup helper, and add unit tests, Doris 2.1.0
integration coverage, and documentation.
### Why are the changes needed?
Doris 2.1.0 with Nereids enabled can replace a supplied CREATE TABLE
comment with `OLAP`. The existing read-time fallback cannot recover a
comment that was lost during creation.
Related to #13346; follow-up to #13347.
### Does this PR introduce _any_ user-facing change?
Table comments now survive creation on affected Doris servers without
changing planner settings or adding catalog properties. Repair requires
ALTER permission. If metadata lookup or repair fails after CREATE
succeeds, the error explicitly reports that the table remains, may lack
its Gravitino identifier, and must be dropped in Doris before retrying.
Missing metadata rows raise `NoSuchTableException`.
### How was this patch tested?
- Review regression coverage verifies actionable lookup/repair errors,
missing-table handling, and existing empty comments. The repair escaping
test fails when `escapeSqlLiteral` is removed.
- The unpatched Doris 2.1.0 regression failed both comment cases with
`OLAP`; the patched run passed both `CatalogDoris2xIT` cases with
Nereids enabled, including special characters and Gravitino ID
preservation.
- All 101 Doris module unit tests passed, covering conditional repair,
escaping, failed CREATE, and lookup/repair errors.
- Comment round-trip integration tests passed on Doris 1.2.7 and 4.0.6
(5 integration tests passed in total).
- `./gradlew spotlessApply` and `git diff --check` passed.
Run the Doris module unit suite with `./gradlew
:catalogs:catalog-jdbc-doris:test -PskipITs -PskipDockerTests=true`. Run
the new integration coverage with `./gradlew
:catalogs:catalog-jdbc-doris:test --tests '*CatalogDoris2xIT'
-PskipDockerTests=false -PdorisMultiVersionTest`.
---
.../doris/operation/DorisTableOperations.java | 93 ++++++--
.../doris/integration/test/CatalogDoris2xIT.java | 171 +++++++++++++++
.../operation/TestDorisTableCommentOperations.java | 72 +++++++
.../doris/operation/TestDorisTableCreation.java | 234 +++++++++++++++++++++
docs/jdbc-doris-catalog.md | 17 +-
.../integration/test/container/DorisImageName.java | 2 +
6 files changed, 575 insertions(+), 14 deletions(-)
diff --git
a/catalogs/catalog-jdbc-doris/src/main/java/org/apache/gravitino/catalog/doris/operation/DorisTableOperations.java
b/catalogs/catalog-jdbc-doris/src/main/java/org/apache/gravitino/catalog/doris/operation/DorisTableOperations.java
index f544dfad0d..fb2cb54211 100644
---
a/catalogs/catalog-jdbc-doris/src/main/java/org/apache/gravitino/catalog/doris/operation/DorisTableOperations.java
+++
b/catalogs/catalog-jdbc-doris/src/main/java/org/apache/gravitino/catalog/doris/operation/DorisTableOperations.java
@@ -61,6 +61,8 @@ import org.apache.gravitino.catalog.jdbc.JdbcColumn;
import org.apache.gravitino.catalog.jdbc.JdbcTable;
import org.apache.gravitino.catalog.jdbc.operation.JdbcTableOperations;
import
org.apache.gravitino.catalog.jdbc.operation.JdbcTablePartitionOperations;
+import org.apache.gravitino.catalog.jdbc.utils.JdbcConnectorUtils;
+import org.apache.gravitino.exceptions.GravitinoRuntimeException;
import org.apache.gravitino.exceptions.NoSuchColumnException;
import org.apache.gravitino.exceptions.NoSuchTableException;
import org.apache.gravitino.rel.Column;
@@ -69,6 +71,7 @@ import org.apache.gravitino.rel.expressions.Expression;
import org.apache.gravitino.rel.expressions.distributions.Distribution;
import org.apache.gravitino.rel.expressions.distributions.Strategy;
import org.apache.gravitino.rel.expressions.literals.Literal;
+import org.apache.gravitino.rel.expressions.sorts.SortOrder;
import org.apache.gravitino.rel.expressions.transforms.Transform;
import org.apache.gravitino.rel.expressions.transforms.Transforms;
import org.apache.gravitino.rel.indexes.Index;
@@ -93,6 +96,62 @@ public class DorisTableOperations extends
JdbcTableOperations {
dataSource, loadedTable, exceptionMapper, typeConverter);
}
+ /** {@inheritDoc} */
+ @Override
+ public void create(
+ String databaseName,
+ String tableName,
+ JdbcColumn[] columns,
+ @Nullable String comment,
+ Map<String, String> properties,
+ Transform[] partitioning,
+ Distribution distribution,
+ Index[] indexes,
+ @Nullable SortOrder[] sortOrders) {
+ super.create(
+ databaseName,
+ tableName,
+ columns,
+ comment,
+ properties,
+ partitioning,
+ distribution,
+ indexes,
+ sortOrders);
+ if (StringUtils.isEmpty(comment)) {
+ return;
+ }
+
+ // Doris 2.1.0's Nereids CREATE TABLE path can discard the table comment.
Repair it only
+ // when necessary, without changing the planner on the pooled connection.
Keep the full
+ // comment, including the Gravitino identifier, so subsequent loads retain
table identity.
+ //
+ // The check runs on every Doris version on purpose: JdbcCatalogOperations
always appends the
+ // Gravitino identifier, so each CREATE pays one information_schema
lookup. Gating on the
+ // server version would cost a comparable extra query per CREATE, and
comparing the stored
+ // comment also covers other versions or planner settings that drop it.
ALTER privilege is
+ // needed only when the stored comment actually differs.
+ try (Connection connection = getConnection(databaseName)) {
+ if (!comment.equals(loadTableComment(connection, databaseName,
tableName))) {
+ JdbcConnectorUtils.executeUpdate(
+ connection,
+ "ALTER TABLE `"
+ + tableName
+ + "` MODIFY COMMENT \""
+ + escapeSqlLiteral(comment, '"')
+ + "\"");
+ }
+ } catch (SQLException | NoSuchTableException e) {
+ throw new GravitinoRuntimeException(
+ e,
+ "Table %s.%s was created in Doris, but its comment could not be
verified or restored. "
+ + "The table may be missing its Gravitino identifier. "
+ + "Drop the created table in Doris before retrying creation.",
+ databaseName,
+ tableName);
+ }
+ }
+
@Override
protected String generateCreateTableSql(
String tableName,
@@ -707,19 +766,8 @@ public class DorisTableOperations extends
JdbcTableOperations {
// Doris JDBC metadata can report the OLAP engine as REMARKS. Query the
actual table comment
// from information_schema when REMARKS is empty or contains that engine
name. Preserve the
// Gravitino ID suffix so JdbcCatalogOperations can extract it when
loading the table.
- StringBuilder comment = new StringBuilder();
- String sql =
- "SELECT TABLE_COMMENT FROM information_schema.TABLES WHERE
TABLE_SCHEMA = ? AND TABLE_NAME = ?";
- try (PreparedStatement preparedStatement =
connection.prepareStatement(sql)) {
- preparedStatement.setString(1, databaseName);
- preparedStatement.setString(2, tableName);
-
- try (ResultSet resultSet = preparedStatement.executeQuery()) {
- while (resultSet.next()) {
- comment.append(resultSet.getString("TABLE_COMMENT"));
- }
- }
- tableBuilder.withComment(comment.toString());
+ try {
+ tableBuilder.withComment(loadTableComment(connection, databaseName,
tableName));
} catch (SQLException e) {
throw exceptionMapper.toGravitinoException(e);
}
@@ -1265,6 +1313,25 @@ public class DorisTableOperations extends
JdbcTableOperations {
return Collections.unmodifiableMap(properties);
}
+ @Nullable
+ private String loadTableComment(Connection connection, String databaseName,
String tableName)
+ throws SQLException {
+ String sql =
+ "SELECT TABLE_COMMENT FROM information_schema.TABLES WHERE
TABLE_SCHEMA = ? AND TABLE_NAME = ?";
+ try (PreparedStatement statement = connection.prepareStatement(sql)) {
+ statement.setString(1, databaseName);
+ statement.setString(2, tableName);
+ try (ResultSet result = statement.executeQuery()) {
+ if (!result.next()) {
+ throw new NoSuchTableException(
+ "Table %s.%s does not exist in Doris when loading its comment",
+ databaseName, tableName);
+ }
+ return result.getString("TABLE_COMMENT");
+ }
+ }
+ }
+
private static String generateIndexPropertiesSql(
Index.IndexType indexType, @Nullable Map<String, String> properties) {
if (indexType != Index.IndexType.INVERTED || properties == null ||
properties.isEmpty()) {
diff --git
a/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/integration/test/CatalogDoris2xIT.java
b/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/integration/test/CatalogDoris2xIT.java
new file mode 100644
index 0000000000..c810bc6fab
--- /dev/null
+++
b/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/integration/test/CatalogDoris2xIT.java
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.catalog.doris.integration.test;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import com.google.common.collect.Maps;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.Collections;
+import java.util.Map;
+import org.apache.gravitino.Catalog;
+import org.apache.gravitino.NameIdentifier;
+import org.apache.gravitino.StringIdentifier;
+import org.apache.gravitino.catalog.jdbc.config.JdbcConfig;
+import org.apache.gravitino.client.GravitinoMetalake;
+import org.apache.gravitino.integration.test.container.ContainerSuite;
+import org.apache.gravitino.integration.test.container.DorisContainer;
+import org.apache.gravitino.integration.test.container.DorisImageName;
+import org.apache.gravitino.integration.test.util.BaseIT;
+import org.apache.gravitino.integration.test.util.GravitinoITUtils;
+import org.apache.gravitino.rel.Column;
+import org.apache.gravitino.rel.Table;
+import org.apache.gravitino.rel.TableCatalog;
+import org.apache.gravitino.rel.expressions.NamedReference;
+import org.apache.gravitino.rel.expressions.distributions.Distributions;
+import org.apache.gravitino.rel.expressions.transforms.Transforms;
+import org.apache.gravitino.rel.types.Types;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+/** Integration tests for Doris 2.1.0 with the Nereids planner enabled. */
+@Tag("gravitino-docker-test")
+@Tag("doris-multi-version")
+public class CatalogDoris2xIT extends BaseIT {
+
+ private static final String PROVIDER = "jdbc-doris";
+ private static final String DRIVER_CLASS_NAME = "com.mysql.cj.jdbc.Driver";
+ private static final ContainerSuite containerSuite =
ContainerSuite.getInstance();
+
+ private final String metalakeName =
GravitinoITUtils.genRandomName("doris2x_metalake");
+ private final String catalogName =
GravitinoITUtils.genRandomName("doris2x_catalog");
+ private final String schemaName =
GravitinoITUtils.genRandomName("doris2x_schema");
+
+ private GravitinoMetalake metalake;
+ private Catalog catalog;
+ private String jdbcUrl;
+
+ @BeforeAll
+ public void startup() {
+ containerSuite.startDorisContainer(DorisImageName.VERSION_2_1);
+ createMetalake();
+ createCatalog();
+ createSchema();
+ }
+
+ @AfterAll
+ public void stop() {
+ catalog.asSchemas().dropSchema(schemaName, true);
+ metalake.dropCatalog(catalogName, true);
+ client.dropMetalake(metalakeName, true);
+ }
+
+ @AfterEach
+ public void resetSchema() {
+ catalog.asSchemas().dropSchema(schemaName, true);
+ createSchema();
+ }
+
+ @ParameterizedTest
+ @ValueSource(
+ strings = {"a real comment", "quote \" and apostrophe ' and backslash \\
and newline\nend"})
+ void testTableCommentWithNereids(String comment) throws Exception {
+ try (Connection connection =
+ DriverManager.getConnection(
+ jdbcUrl + schemaName, DorisContainer.USER_NAME,
DorisContainer.PASSWORD);
+ Statement statement = connection.createStatement()) {
+ try (ResultSet result = statement.executeQuery("SELECT
@@enable_nereids_planner")) {
+ assertTrue(result.next());
+ assertTrue(result.getBoolean(1));
+ }
+
+ TableCatalog tables = catalog.asTableCatalog();
+ NameIdentifier identifier = NameIdentifier.of(schemaName,
"comment_with_nereids");
+ Table created =
+ tables.createTable(
+ identifier,
+ new Column[] {Column.of("id", Types.IntegerType.get(), null,
false, false, null)},
+ comment,
+ Collections.emptyMap(),
+ Transforms.EMPTY_TRANSFORM,
+ Distributions.hash(1, NamedReference.field("id")),
+ null);
+ assertEquals(comment, created.comment());
+ assertEquals(comment, tables.loadTable(identifier).comment());
+
+ try (PreparedStatement query =
+ connection.prepareStatement(
+ "SELECT TABLE_COMMENT FROM information_schema.TABLES WHERE
TABLE_SCHEMA = ? AND TABLE_NAME = ?")) {
+ query.setString(1, schemaName);
+ query.setString(2, identifier.name());
+ try (ResultSet result = query.executeQuery()) {
+ assertTrue(result.next());
+ String storedComment = result.getString(1);
+ assertNotNull(StringIdentifier.fromComment(storedComment));
+ assertEquals(comment,
StringIdentifier.removeIdFromComment(storedComment));
+ }
+ }
+
+ try (ResultSet result = statement.executeQuery("SHOW CREATE TABLE
`comment_with_nereids`")) {
+ assertTrue(result.next());
+ assertTrue(result.getString(2).contains("gravitino.v1.uid"));
+ }
+ }
+ }
+
+ private void createMetalake() {
+ client.createMetalake(metalakeName, "comment", Collections.emptyMap());
+ metalake = client.loadMetalake(metalakeName);
+ assertEquals(metalakeName, metalake.name());
+ }
+
+ private void createCatalog() {
+ DorisContainer dorisContainer =
containerSuite.getDorisContainer(DorisImageName.VERSION_2_1);
+ jdbcUrl =
+ String.format(
+ "jdbc:mysql://%s:%d/",
+ dorisContainer.getContainerIpAddress(),
dorisContainer.getFeMysqlPort());
+
+ Map<String, String> props = Maps.newHashMap();
+ props.put(JdbcConfig.JDBC_URL.getKey(), jdbcUrl);
+ props.put(JdbcConfig.JDBC_DRIVER.getKey(), DRIVER_CLASS_NAME);
+ props.put(JdbcConfig.USERNAME.getKey(), DorisContainer.USER_NAME);
+ props.put(JdbcConfig.PASSWORD.getKey(), DorisContainer.PASSWORD);
+
+ catalog =
+ metalake.createCatalog(
+ catalogName, Catalog.Type.RELATIONAL, PROVIDER, "doris 2.x
catalog", props);
+ assertEquals(catalogName, metalake.loadCatalog(catalogName).name());
+ }
+
+ private void createSchema() {
+ catalog.asSchemas().createSchema(schemaName, null, Collections.emptyMap());
+ assertEquals(schemaName,
catalog.asSchemas().loadSchema(schemaName).name());
+ }
+}
diff --git
a/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/operation/TestDorisTableCommentOperations.java
b/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/operation/TestDorisTableCommentOperations.java
index 5d16853a2d..14c1ca4588 100644
---
a/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/operation/TestDorisTableCommentOperations.java
+++
b/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/operation/TestDorisTableCommentOperations.java
@@ -19,6 +19,9 @@
package org.apache.gravitino.catalog.doris.operation;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.startsWith;
import static org.mockito.Mockito.mock;
@@ -30,6 +33,7 @@ import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import org.apache.gravitino.catalog.jdbc.JdbcTable;
+import org.apache.gravitino.exceptions.NoSuchTableException;
import org.junit.jupiter.api.Test;
class TestDorisTableCommentOperations {
@@ -79,6 +83,74 @@ class TestDorisTableCommentOperations {
assertEquals("crud probe", tableBuilder.comment());
}
+ @Test
+ void testMissingTableThrowsNoSuchTableException() throws Exception {
+ Connection connection = mock(Connection.class);
+ PreparedStatement statement = mock(PreparedStatement.class);
+ ResultSet result = mock(ResultSet.class);
+ when(connection.prepareStatement(startsWith("SELECT
TABLE_COMMENT"))).thenReturn(statement);
+ when(statement.executeQuery()).thenReturn(result);
+ when(result.next()).thenReturn(false);
+
+ NoSuchTableException error =
+ assertThrows(
+ NoSuchTableException.class,
+ () ->
+ new DorisTableOperations()
+ .correctJdbcTableFields(connection, "db", "t",
JdbcTable.builder()));
+
+ assertTrue(error.getMessage().contains("Table db.t does not exist in
Doris"));
+ verify(result).close();
+ verify(statement).close();
+ verify(connection, never()).prepareStatement(startsWith("SHOW ALTER TABLE
COLUMN"));
+ }
+
+ @Test
+ void testExistingTableWithEmptyCommentIsLoaded() throws Exception {
+ Connection connection = mock(Connection.class);
+ PreparedStatement commentStatement = mock(PreparedStatement.class);
+ ResultSet commentResult = mock(ResultSet.class);
+ PreparedStatement statusStatement = mock(PreparedStatement.class);
+ ResultSet statusResult = mock(ResultSet.class);
+ when(connection.prepareStatement(startsWith("SELECT TABLE_COMMENT")))
+ .thenReturn(commentStatement);
+ when(commentStatement.executeQuery()).thenReturn(commentResult);
+ when(commentResult.next()).thenReturn(true, false);
+ when(commentResult.getString("TABLE_COMMENT")).thenReturn("");
+ when(connection.prepareStatement(startsWith("SHOW ALTER TABLE COLUMN")))
+ .thenReturn(statusStatement);
+ when(statusStatement.executeQuery()).thenReturn(statusResult);
+
+ JdbcTable.Builder tableBuilder = JdbcTable.builder().withComment("OLAP");
+ new DorisTableOperations().correctJdbcTableFields(connection, "db", "t",
tableBuilder);
+
+ assertEquals("", tableBuilder.comment());
+ verify(commentResult).close();
+ verify(commentStatement).close();
+ }
+
+ @Test
+ void testNullInformationSchemaCommentIsLoadedAsNull() throws Exception {
+ Connection connection = mock(Connection.class);
+ PreparedStatement commentStatement = mock(PreparedStatement.class);
+ ResultSet commentResult = mock(ResultSet.class);
+ PreparedStatement statusStatement = mock(PreparedStatement.class);
+ ResultSet statusResult = mock(ResultSet.class);
+ when(connection.prepareStatement(startsWith("SELECT TABLE_COMMENT")))
+ .thenReturn(commentStatement);
+ when(commentStatement.executeQuery()).thenReturn(commentResult);
+ when(commentResult.next()).thenReturn(true, false);
+ when(commentResult.getString("TABLE_COMMENT")).thenReturn(null);
+ when(connection.prepareStatement(startsWith("SHOW ALTER TABLE COLUMN")))
+ .thenReturn(statusStatement);
+ when(statusStatement.executeQuery()).thenReturn(statusResult);
+
+ JdbcTable.Builder tableBuilder = JdbcTable.builder().withComment("OLAP");
+ new DorisTableOperations().correctJdbcTableFields(connection, "db", "t",
tableBuilder);
+
+ assertNull(tableBuilder.comment());
+ }
+
@Test
void testValidJdbcCommentNeedsNoFallback() throws Exception {
Connection connection = mock(Connection.class);
diff --git
a/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/operation/TestDorisTableCreation.java
b/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/operation/TestDorisTableCreation.java
new file mode 100644
index 0000000000..74d8d60eb2
--- /dev/null
+++
b/catalogs/catalog-jdbc-doris/src/test/java/org/apache/gravitino/catalog/doris/operation/TestDorisTableCreation.java
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.catalog.doris.operation;
+
+import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.ArgumentMatchers.startsWith;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.Collections;
+import javax.annotation.Nullable;
+import javax.sql.DataSource;
+import
org.apache.gravitino.catalog.doris.converter.DorisColumnDefaultValueConverter;
+import org.apache.gravitino.catalog.doris.converter.DorisExceptionConverter;
+import org.apache.gravitino.catalog.doris.converter.DorisTypeConverter;
+import org.apache.gravitino.catalog.jdbc.JdbcColumn;
+import org.apache.gravitino.exceptions.GravitinoRuntimeException;
+import org.apache.gravitino.exceptions.NoSuchTableException;
+import org.apache.gravitino.exceptions.TableAlreadyExistsException;
+import org.apache.gravitino.rel.expressions.distributions.Distributions;
+import org.apache.gravitino.rel.expressions.transforms.Transforms;
+import org.apache.gravitino.rel.indexes.Index;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.NullAndEmptySource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+class TestDorisTableCreation {
+ private static final String CREATE_SQL = "CREATE TABLE `t` (id INT)";
+ private static final String COMMENT =
+ "a real comment (From Gravitino, DO NOT EDIT: gravitino.v1.uid123)";
+
+ @ParameterizedTest
+ @NullAndEmptySource
+ @ValueSource(strings = {"OLAP", "different comment"})
+ void testRepairDiscardedComment(String storedComment) throws Exception {
+ CreateFixture fixture = new CreateFixture(COMMENT, storedComment);
+
+ fixture.create();
+
+ verify(fixture.createStatement).executeUpdate(CREATE_SQL);
+ verify(fixture.alterStatement)
+ .executeUpdate("ALTER TABLE `t` MODIFY COMMENT \"" + COMMENT + "\"");
+ verify(fixture.commentStatement).setString(1, "db");
+ verify(fixture.commentStatement).setString(2, "t");
+ verify(fixture.commentResult).close();
+ verify(fixture.commentStatement).close();
+ verify(fixture.connection, times(2)).close();
+ }
+
+ @Test
+ void testRepairEscapesQuotesAndBackslashes() throws Exception {
+ String comment =
+ "owner's \"comment\" C:\\tmp (From Gravitino, DO NOT EDIT:
gravitino.v1.uid123)";
+ CreateFixture fixture = new CreateFixture(comment, "OLAP");
+
+ fixture.create();
+
+ verify(fixture.alterStatement)
+ .executeUpdate(
+ "ALTER TABLE `t` MODIFY COMMENT \"owner's \"\"comment\"\"
C:\\\\tmp "
+ + "(From Gravitino, DO NOT EDIT: gravitino.v1.uid123)\"");
+ }
+
+ @Test
+ void testMissingTableIsNotTreatedAsEmptyComment() throws Exception {
+ CreateFixture fixture = new CreateFixture(COMMENT, "");
+ when(fixture.commentResult.next()).thenReturn(false);
+
+ GravitinoRuntimeException error =
+ assertThrows(GravitinoRuntimeException.class, fixture::create);
+
+ assertInstanceOf(NoSuchTableException.class, error.getCause());
+ assertTrue(error.getCause().getMessage().contains("Table db.t does not
exist in Doris"));
+ assertTrue(error.getMessage().contains("Table db.t was created in Doris"));
+ assertTrue(error.getMessage().contains("Drop the created table in Doris
before retrying"));
+ verify(fixture.alterStatement, never()).executeUpdate(anyString());
+ verify(fixture.commentResult).close();
+ verify(fixture.commentStatement).close();
+ verify(fixture.connection, times(2)).close();
+ }
+
+ @Test
+ void testPreservedCommentNeedsNoAlter() throws Exception {
+ CreateFixture fixture = new CreateFixture(COMMENT, COMMENT);
+
+ fixture.create();
+
+ verify(fixture.alterStatement, never()).executeUpdate(anyString());
+ }
+
+ @ParameterizedTest
+ @NullAndEmptySource
+ void testAbsentCommentNeedsNoRepair(String comment) throws Exception {
+ CreateFixture fixture = new CreateFixture(comment, "OLAP");
+
+ fixture.create();
+
+ verify(fixture.connection, never()).prepareStatement(anyString());
+ verify(fixture.alterStatement, never()).executeUpdate(anyString());
+ verify(fixture.connection).close();
+ }
+
+ @Test
+ void testFailedCreateDoesNotModifyExistingTable() throws Exception {
+ CreateFixture fixture = new CreateFixture(COMMENT, "existing comment");
+ when(fixture.createStatement.executeUpdate(CREATE_SQL))
+ .thenThrow(new SQLException("Table already exists", "42S01", 1050));
+
+ assertThrows(TableAlreadyExistsException.class, fixture::create);
+
+ verify(fixture.connection, never()).prepareStatement(anyString());
+ verify(fixture.alterStatement, never()).executeUpdate(anyString());
+ verify(fixture.connection).close();
+ }
+
+ @Test
+ void testCommentLookupFailureIsReported() throws Exception {
+ CreateFixture fixture = new CreateFixture(COMMENT, "OLAP");
+ SQLException failure = new SQLException("comment lookup failed");
+ when(fixture.commentStatement.executeQuery()).thenThrow(failure);
+
+ GravitinoRuntimeException error =
+ assertThrows(GravitinoRuntimeException.class, fixture::create);
+
+ assertSame(failure, error.getCause());
+ assertTrue(error.getMessage().contains("Table db.t was created in Doris"));
+ assertTrue(error.getMessage().contains("may be missing its Gravitino
identifier"));
+ assertTrue(error.getMessage().contains("Drop the created table in Doris
before retrying"));
+ verify(fixture.alterStatement, never()).executeUpdate(anyString());
+ verify(fixture.commentStatement).close();
+ verify(fixture.connection, times(2)).close();
+ }
+
+ @Test
+ void testCommentRepairFailureIsReported() throws Exception {
+ CreateFixture fixture = new CreateFixture(COMMENT, "OLAP");
+ SQLException failure = new SQLException("comment repair failed");
+ when(fixture.alterStatement.executeUpdate(anyString())).thenThrow(failure);
+
+ GravitinoRuntimeException error =
+ assertThrows(GravitinoRuntimeException.class, fixture::create);
+
+ assertSame(failure, error.getCause());
+ assertTrue(error.getMessage().contains("Table db.t was created in Doris"));
+ assertTrue(error.getMessage().contains("may be missing its Gravitino
identifier"));
+ assertTrue(error.getMessage().contains("Drop the created table in Doris
before retrying"));
+ verify(fixture.alterStatement).close();
+ verify(fixture.connection, times(2)).close();
+ }
+
+ private static class CreateFixture {
+ private final Connection connection = mock(Connection.class);
+ private final Statement createStatement = mock(Statement.class);
+ private final Statement alterStatement = mock(Statement.class);
+ private final PreparedStatement commentStatement =
mock(PreparedStatement.class);
+ private final ResultSet commentResult = mock(ResultSet.class);
+ private final DorisTableOperations operations = spy(new
DorisTableOperations());
+ private final JdbcColumn[] columns = new JdbcColumn[0];
+ private final Index[] indexes = new Index[0];
+ @Nullable private final String comment;
+
+ private CreateFixture(@Nullable String comment, @Nullable String
storedComment)
+ throws Exception {
+ this.comment = comment;
+ DataSource dataSource = mock(DataSource.class);
+ when(dataSource.getConnection()).thenReturn(connection);
+ when(connection.createStatement()).thenReturn(createStatement,
alterStatement);
+ when(connection.prepareStatement(startsWith("SELECT TABLE_COMMENT")))
+ .thenReturn(commentStatement);
+ when(commentStatement.executeQuery()).thenReturn(commentResult);
+ when(commentResult.next()).thenReturn(true, false);
+ when(commentResult.getString("TABLE_COMMENT")).thenReturn(storedComment);
+ operations.initialize(
+ dataSource,
+ new DorisExceptionConverter(),
+ new DorisTypeConverter(),
+ new DorisColumnDefaultValueConverter(),
+ Collections.emptyMap());
+ doReturn(CREATE_SQL)
+ .when(operations)
+ .generateCreateTableSql(
+ "t",
+ columns,
+ comment,
+ Collections.emptyMap(),
+ Transforms.EMPTY_TRANSFORM,
+ Distributions.NONE,
+ indexes);
+ }
+
+ private void create() {
+ operations.create(
+ "db",
+ "t",
+ columns,
+ comment,
+ Collections.emptyMap(),
+ Transforms.EMPTY_TRANSFORM,
+ Distributions.NONE,
+ indexes);
+ }
+ }
+}
diff --git a/docs/jdbc-doris-catalog.md b/docs/jdbc-doris-catalog.md
index 3a315ee69c..e7403807f8 100644
--- a/docs/jdbc-doris-catalog.md
+++ b/docs/jdbc-doris-catalog.md
@@ -25,7 +25,7 @@ Gravitino saves some system information in schema and table
comments, like
### Catalog Capabilities
- Gravitino catalog corresponds to the Doris instance.
-- Supports metadata management of Doris (1.2.x, 3.0.x, 4.0.x).
+- Supports metadata management of Doris (1.2.x, 2.1.x, 3.0.x, 4.0.x).
- Supports table index (PRIMARY_KEY, UNIQUE_KEY, INVERTED, BITMAP (legacy),
ANN/VECTOR).
- Supports [column default
value](./tables-and-views.md#table-column-default-value).
@@ -54,6 +54,21 @@ Besides the [common catalog
properties](./gravitino-server-config.md#catalog-pro
Before using the Doris Catalog, you must download the corresponding JDBC
driver to the `catalogs/jdbc-doris/libs` directory.
Gravitino doesn't package the JDBC driver for Doris due to licensing issues.
+### Doris 2.1.0 Table Comments
+
+Doris 2.1.0 can discard table comments when its Nereids planner handles
`CREATE TABLE`.
+After every table creation, on all Doris versions, Gravitino reads the stored
comment from
+`information_schema.TABLES` and, if it differs, restores it with
+`ALTER TABLE ... MODIFY COMMENT`, including Gravitino's table identifier. On
affected servers,
+the JDBC user must have permission to alter the created table. The connector
leaves the
+planner settings unchanged.
+
+If the comment lookup or restoration fails after `CREATE TABLE` succeeds,
Gravitino reports
+that the table was created but its comment could not be verified or restored.
Doris DDL is
+not rolled back, so the table remains and may be missing its Gravitino
identifier. Drop the
+created table in Doris before retrying creation; otherwise, the retry fails
because the table
+already exists.
+
### Driver Version Compatibility
The Doris catalog includes driver version compatibility checks for datetime
precision calculation:
diff --git
a/integration-test-common/src/test/java/org/apache/gravitino/integration/test/container/DorisImageName.java
b/integration-test-common/src/test/java/org/apache/gravitino/integration/test/container/DorisImageName.java
index d00f2bfb52..0135345af4 100644
---
a/integration-test-common/src/test/java/org/apache/gravitino/integration/test/container/DorisImageName.java
+++
b/integration-test-common/src/test/java/org/apache/gravitino/integration/test/container/DorisImageName.java
@@ -33,6 +33,8 @@ package org.apache.gravitino.integration.test.container;
*/
public enum DorisImageName {
VERSION_1_2("apache/gravitino-ci:doris-0.1.5"),
+ /** The official all-in-one Doris 2.1.0 image. */
+ VERSION_2_1("apache/doris:doris-all-in-one-2.1.0"),
VERSION_3_0("apache/doris:fe-3.0.6.2"),
VERSION_4_0("apache/doris:fe-4.0.6");
