airborne12 commented on code in PR #64522:
URL: https://github.com/apache/doris/pull/64522#discussion_r3465113366
##########
fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java:
##########
@@ -3274,6 +3274,16 @@ private boolean processAddIndex(CreateIndexOp
createIndexOp, OlapTable olapTable
}
AnnIndexPropertiesChecker.checkProperties(indexDef.getProperties());
}
+ //If the table format is V1, the added index is still created in V1
format.
+ // It should be necessary to upgrade the V2 table format and then add
the V2 format index.
+ if (indexDef.getIndexType() == IndexType.INVERTED
+ && olapTable.getInvertedIndexFileStorageFormat() ==
TInvertedIndexFileStorageFormat.V1
Review Comment:
Blocking: this checks only the stored enum value `V1`, but it misses the
effective V1 path through `DEFAULT`. `TableProperty` defaults
`inverted_index_storage_format` to `DEFAULT`, and cloud tablet creation later
maps `DEFAULT + Config.inverted_index_storage_format=V1` to PB `V1`. For an
existing/default table in that environment, `ALTER TABLE ADD INDEX` can still
pass this guard and create a new V1 inverted index.
Please gate on the effective inverted-index storage format, not only `==
V1`, or normalize `DEFAULT` for new index/tablet creation so cloud and
non-cloud paths cannot produce V1 when creation is disabled.
##########
regression-test/suites/fault_injection_p0/test_index_io_context.groovy:
##########
@@ -69,82 +69,82 @@ suite("test_index_io_context", "nonConcurrent") {
}
try {
- create_table(tableName1, "V1");
+ // create_table(tableName1, "V1");
Review Comment:
Blocking: commenting out the V1 side removes compatibility coverage instead
of proving that existing V1 tables still work. This PR's contract is `block new
V1 creation, keep existing V1 compatible`; changing broad suites to V2-only
weakens the second half. The same pattern appears in other load/variant/index
suites.
Please keep active V1 read/load/compaction/http-action coverage through an
isolated legacy fixture or temporary allow switch, and keep negative tests
active for new V1 creation. Dead commented blocks should be removed rather than
left as disabled coverage.
##########
regression-test/suites/inverted_index_p0/storage_format/test_inverted_index_v1_deprecated.groovy:
##########
@@ -0,0 +1,130 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_inverted_index_v1_deprecated", "p0") {
+ // AC1: explicit v1 (lowercase) must be rejected
+ test {
+ sql """
+ CREATE TABLE test_v1_rejected (
+ k INT,
+ v STRING,
+ INDEX idx_v (v) USING INVERTED
+ ) ENGINE=OLAP DUPLICATE KEY(k)
+ DISTRIBUTED BY HASH(k) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "inverted_index_storage_format" = "v1"
+ )
+ """
+ exception "Inverted index V1 is deprecated and no longer allowed for
new index creation."
+ }
+
+ // AC1 (case-insensitive): uppercase V1 must also be rejected
+ test {
+ sql """
+ CREATE TABLE test_v1_rejected_upper (
+ k INT,
+ v STRING,
+ INDEX idx_v (v) USING INVERTED
+ ) ENGINE=OLAP DUPLICATE KEY(k)
+ DISTRIBUTED BY HASH(k) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "inverted_index_storage_format" = "V1"
+ )
+ """
+ exception "Inverted index V1 is deprecated and no longer allowed for
new index creation."
+ }
+
+ // AC2: no format specified -> default succeeds and format is not V1
+ // AC3: ALTER TABLE ADD INDEX on default table succeeds
+ try {
+ sql "DROP TABLE IF EXISTS test_v1_deprecated_default"
+ sql """
+ CREATE TABLE test_v1_deprecated_default (
+ k INT,
+ v STRING
+ ) ENGINE=OLAP DUPLICATE KEY(k)
+ DISTRIBUTED BY HASH(k) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ )
+ """
+ def showCreate = sql "SHOW CREATE TABLE test_v1_deprecated_default"
+ assertTrue(showCreate.size() > 0)
+
assertFalse(showCreate[0][1].contains("\"inverted_index_storage_format\" =
\"V1\""))
+
+ sql "ALTER TABLE test_v1_deprecated_default ADD INDEX idx_v (v) USING
INVERTED"
+ def showCreateAfter = sql "SHOW CREATE TABLE
test_v1_deprecated_default"
+ assertTrue(showCreateAfter[0][1].contains("idx_v"))
+
assertFalse(showCreateAfter[0][1].contains("\"inverted_index_storage_format\" =
\"V1\""))
+ } finally {
+ sql "DROP TABLE IF EXISTS test_v1_deprecated_default"
+ }
+
+ // AC4: existing V1 table metadata load and read/write must not be
affected.
+ // Uses a deterministic fixture: temporarily unlock V1 creation via admin
config,
+ // seed the legacy table, verify reads/writes/deletes, then clean up.
+ try {
+ sql "DROP TABLE IF EXISTS test_v1_legacy_compat"
+ sql "ADMIN SET FRONTEND CONFIG ('allow_inverted_index_v1_creation' =
'true')"
+ sql """
+ CREATE TABLE test_v1_legacy_compat (
+ k INT,
+ v STRING,
+ w STRING,
+ INDEX idx_v (v) USING INVERTED
+ ) ENGINE=OLAP DUPLICATE KEY(k)
+ DISTRIBUTED BY HASH(k) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "inverted_index_storage_format" = "v1"
+ )
+ """
+ sql "ADMIN SET FRONTEND CONFIG ('allow_inverted_index_v1_creation' =
'false')"
+
+ def showCreate = sql "SHOW CREATE TABLE test_v1_legacy_compat"
+
assertTrue(showCreate[0][1].contains("\"inverted_index_storage_format\" =
\"V1\""))
+
+ // Verify basic DML works on a V1 table (tests metadata compat, not BE
index reads)
Review Comment:
Blocking: this does not prove the promised existing-V1 compatibility. The
test creates a fresh V1 table by temporarily enabling the escape hatch, then
disables the flag and only runs basic DML on `k`; it explicitly does not
exercise BE inverted-index reads on `v`. It also mutates global FE config while
the suite is only tagged `p0`, so concurrent suites can observe the temporary
permissive state.
Please make the suite isolated (`nonConcurrent` or an equivalent config
helper) and, after disabling V1 creation, run indexed-column
`MATCH`/`MATCH_PHRASE` queries with result assertions and preferably
profile/pushdown evidence. A real legacy fixture or reload/replay path would be
stronger for metadata compatibility.
##########
fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java:
##########
@@ -3274,6 +3274,16 @@ private boolean processAddIndex(CreateIndexOp
createIndexOp, OlapTable olapTable
}
AnnIndexPropertiesChecker.checkProperties(indexDef.getProperties());
}
+ //If the table format is V1, the added index is still created in V1
format.
+ // It should be necessary to upgrade the V2 table format and then add
the V2 format index.
+ if (indexDef.getIndexType() == IndexType.INVERTED
+ && olapTable.getInvertedIndexFileStorageFormat() ==
TInvertedIndexFileStorageFormat.V1
+ && !Config.allow_inverted_index_v1_creation) {
+ throw new DdlException("Inverted index V1 is deprecated and no
longer allowed for new index creation."
+ + " Please use inverted index V2."
+ + " You can upgrade the table format with: ALTER TABLE " +
olapTable.getName()
Review Comment:
Blocking: this migration hint is not executable today.
`ModifyTablePropertiesOp` rejects `inverted_index_storage_format` changes with
`Property inverted_index_storage_format is not allowed to change`, so users who
hit this error cannot actually run the suggested `ALTER TABLE ... SET` command.
Please either implement a restricted `V1/DEFAULT -> V2/V3` upgrade path, or
remove this hint and state clearly that direct table-property upgrade is not
currently supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
