hubgeter commented on code in PR #69633:
URL: https://github.com/apache/doris/pull/69633#discussion_r4390735385
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergSchemaUtils.java:
##########
@@ -102,12 +104,59 @@ public final class IcebergSchemaUtils {
static final String ICEBERG_LAST_UPDATED_SEQUENCE_NUMBER_COL =
"_last_updated_sequence_number";
static final int ICEBERG_ROW_ID_FIELD_ID = 2147483540;
static final int ICEBERG_LAST_UPDATED_SEQUENCE_NUMBER_FIELD_ID =
2147483539;
+ static final int ICEBERG_ROW_LINEAGE_MIN_VERSION = 3;
private static final Base64.Encoder BASE64_ENCODER = Base64.getEncoder();
private IcebergSchemaUtils() {
}
+ /**
+ * Validates the effective row-lineage schema before a table property
update can make it observable.
+ *
+ * <p>Iceberg keeps row-lineage columns outside the physical table schema,
so an {@code UpdateProperties}
+ * format-version upgrade does not itself validate their names against
user columns. Doris exposes both sets
+ * through one connector schema, which must remain a valid Iceberg {@link
Schema}. Delegate that contract to
+ * Iceberg instead of maintaining a second reserved-name rule here.</p>
+ */
+ static void validateRowLineageFormatVersionUpgrade(Table table,
Map<String, String> properties) {
+ int currentFormatVersion = IcebergWriterHelper.getFormatVersion(table);
+ if (currentFormatVersion >= ICEBERG_ROW_LINEAGE_MIN_VERSION) {
+ return;
+ }
+
+ String requestedFormatVersion =
properties.get(TableProperties.FORMAT_VERSION);
+ if (requestedFormatVersion == null) {
+ return;
+ }
+
+ int targetFormatVersion;
+ try {
+ targetFormatVersion = Integer.parseInt(requestedFormatVersion);
+ } catch (NumberFormatException ignored) {
+ // Leave invalid values to Iceberg's UpdateProperties validation.
+ return;
+ }
+
+ if (targetFormatVersion >= ICEBERG_ROW_LINEAGE_MIN_VERSION) {
+ validateRowLineageSchema(table.schema());
+ }
+ }
+
+ /**
+ * Verifies that the physical schema can be exposed together with
Iceberg's row-lineage metadata columns.
+ * {@link MetadataColumns#schemaWithRowLineage(Schema)} is the authority
for the combined schema, including
+ * its requirement that field names are unique and field ids match.
+ */
+ static void validateRowLineageSchema(Schema schema) {
+ try {
+ MetadataColumns.schemaWithRowLineage(schema);
Review Comment:
Fixed in 017c10d69fc. `validateRowLineageSchema` now applies the
case-insensitive Doris identifier contract after the Iceberg structural check,
at both the property-commit and effective-schema boundaries. Tests cover
mixed-case `_ROW_ID` and `_LAST_UPDATED_SEQUENCE_NUMBER` during upgrade and on
existing V3 tables.
##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java:
##########
@@ -494,6 +494,7 @@ public void renameTable(String dbName, String oldName,
String newName) {
@Override
public void setTableProperties(String dbName, String tableName,
Map<String, String> properties) {
withTable(dbName, tableName, table -> {
+
IcebergSchemaUtils.validateRowLineageFormatVersionUpgrade(table, properties);
Review Comment:
Fixed in 017c10d69fc. `setTableProperties` now uses an Iceberg-compatible
optimistic commit loop: it refreshes the metadata base on every attempt, merges
and validates the properties, reruns row-lineage validation, and only then
commits. The new tests deterministically inject `_ROW_ID` before the first
commit and before an automatic retry after `CommitFailedException`; both leave
the table at V2 and commit none of the property map.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
