u70b3 opened a new pull request, #67201: URL: https://github.com/apache/doris/pull/67201
### What problem does this PR solve? Issue Number: #66497 Related PR: #66637 (merged), #66671 (open — independent; this PR shares no files with it) Problem Summary: This is the first sub-PR (PR3A) of delivery slice 3 of the v5.1 design ([final 4.2 contract](https://github.com/apache/doris/issues/66497#issuecomment-5301314544), scope confirmed in [this review](https://github.com/apache/doris/issues/66497#issuecomment-5301637401)): the Lance index DDL surface with static validation and target-aware routing, in reject-all mode. Slice 3's remaining pieces — durable jobs, same-name fences, unresolved quotas, replay, job SQL, and dispatch with fake-worker fault tests — land as follow-up PRs; this PR contains no job admission, no index-metadata reads, and no enablement gate. What this PR adds: - Top-level `CREATE [OR REPLACE] INDEX ... USING ANN/BTREE/BITMAP` and `DROP INDEX [IF EXISTS]` parsing for Lance Directory catalog tables. `OR REPLACE` is mutually exclusive with `IF NOT EXISTS`; `BTREE` is a new non-reserved keyword. The `indexDef` rule used by `CREATE TABLE` and `ALTER TABLE ... ADD INDEX` is unchanged, and `ALTER TABLE ... ADD/DROP INDEX` remains unsupported per Section 2.1, so previously parseable internal SQL behaves byte-identically. - `IndexDefinition` carries `orReplace` and the Lance-only type name without extending the persisted internal `IndexDef.IndexType` enum (Section 4.4). New `validate()` guards reject Lance-only syntax on internal tables and fire only for SQL that was a syntax error before this PR. - `LanceIndexMutationValidator` applies the FE static bounds of Section 2.4: the ANN/`IVF_PQ` property matrix (required `index_type=IVF_PQ`, `metric` in `l2/cosine/dot`, positive required `num_partitions`/`num_sub_vectors`, fixed `num_bits=8`, unknown and case-variant duplicate properties rejected), the BTREE/BITMAP column-type sets (uint64/`LARGEINT` included as integral for both), exactly one non-null column, and a bounded index name. Arrow-level revalidation (fixed-size-list, dimension, float16-vs-float32, subvector divisibility) is deferred to the isolated worker per Sections 2.4/4.2 because `LanceTypeConverter` erases those facts. - Target-aware routing in `AlterTableCommand.validate` (after the existing table `ALTER` privilege check and catalog resolution, before any op validation): top-level CREATE/DROP INDEX on a Lance table is statically validated and then rejected with a typed per-op message before any `Env.getNextId()` allocation. Lance REST catalogs receive a fixed unsupported error mirroring PR1's `SHOW INDEX` stance. Internal tables never enter this branch. Explicitly not in this PR: durable job records, fences, quotas, and replay (PR3B); authoritative `IF` semantics, admission, and job SQL (Section 2.2, PR3C); dispatch and the isolated worker (PR3D/slice 4); `FORCE_RELEASE` (PR3E); the mutation enablement configuration (Section 9.7, arrives with admission). Property-value normalization is validation-local; persisting normalized values belongs to admission. `SHOW INDEX` behavior from PR1 and the PR2 inspection surface are untouched. Known accepted behavior change: the new `BTREE` keyword token makes stored-procedure bodies that use `btree` as a bare identifier (e.g. `CLOSE btree`) fail PL parsing, the same hazard class as the pre-existing `ANN` token; all other identifier positions are covered by `nonReserved`. ### Release note Add the `CREATE [OR REPLACE] INDEX ... USING ANN/BTREE/BITMAP` and `DROP INDEX` SQL surface for Lance Directory catalog tables with static validation; statements are currently rejected with a typed not-supported error while the mutation lifecycle is staged. ### Check List (For Author) - Test - [ ] Regression test - `regression-test/suites/external_table_p0/lance/test_lance_index_ddl.groovy` added (mirrors `test_lance_show_index`): typed rejections and static-matrix errors on the filesystem catalog, REST-specific messages, uniform `DROP INDEX IF EXISTS` rejection, the generic `ALTER TABLE ADD INDEX` rejection, and `ALTER` privilege denial preceding the typed rejection. Not executed locally — it requires the shared external docker env (MinIO + preinstalled Lance datasets + Lance REST server); pipeline/CI run requested. - [x] Unit Test - Focused run passed 187/187 with checkstyle enabled (`cd fe && mvn test -pl fe-common,fe-core -am -DfailIfNoTests=false -Dtest='...'`, no `-Dcheckstyle.skip`): new `CreateIndexParserTest` (10), `LanceIndexMutationValidatorTest` (24), `AlterTableCommandLanceIndexTest` (9), plus pre-existing `NereidsParserTest` (79), `AlterTableCommandTest` (17), `IndexDefinitionTest` (5), `ShowIndexCommandTest` (8), `LanceIndexMetadataLoaderTest` (26), `LanceFilesystemCatalogTest` (9) — zero regressions. - [x] Manual test - Parser regenerated from a clean `target/generated-sources`; `fe-common,fe-core -am` compile green with and without checkstyle; `mvn validate -pl fe-core` reports 0 Checkstyle violations (test sources included). - Internal-parity reviewed: the new guards are unreachable for previously parseable SQL (all `IndexDefinition` string-ctor call sites verified); `MTMV`/OlapTable/Iceberg/JDBC paths never enter the Lance branch; rejected Lance statements allocate no global id. - Behavior changed: - [ ] No. - [x] Yes. Top-level `CREATE`/`DROP INDEX` on Lance tables now fails with typed per-op messages after static validation (previously a generic external-table rejection after op validation); `CREATE INDEX ... USING BTREE/BITMAP` and `CREATE OR REPLACE INDEX` now parse (previously syntax errors). Internal-table index DDL is unchanged. - Does this need documentation? - [x] No. The SQL surface rejects all mutation statements; user documentation tracks the broader lifecycle work in #66497 once mutation is enabled. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
