dxyou0830 opened a new issue, #70583: URL: https://github.com/apache/doris/issues/70583
### Search before asking - [x] I searched existing issues and pull requests for this error and the index-remapping functions. The identical error message appears in #44609 (fixed by #44610), but that issue concerns duplicate column grouping with cluster keys. This report concerns index ID remapping during snapshot conversion. #59117 and its branch-4.0 backport #60415 introduced support for multiple inverted indexes on one column. ### Version Observed on **Doris 4.1.4-rc04**, commit [`ad35a140c7fd0b842f18c23300bac581f7d04326`](https://github.com/apache/doris/commit/ad35a140c7fd0b842f18c23300bac581f7d04326), using inverted index storage format **V3**. The same faulty lookup/remapping remains in both the local and cloud implementations at the following revisions, checked on November 14, 2026: | Ref | Commit | Verification | |---|---|---| | master | `e82d2650e38e57886679faecf0ad5569169c54aa` | Source inspection | | branch-4.0 | `8a9961723ea4be00cdf923c60759607202c7e2e7` | Source inspection | | branch-4.1 | `af23c8da954119f3ec6a9c0f8d18327286e940e7` | Source inspection | | 4.1.5-release | `917b20be83878e98f94e3ebf7efccc80346e3e50` | Source inspection | These additional revisions were not tested in running clusters. ### What happened A string column has two inverted indexes: an exact index and a lowercase-normalizer index. Their names, properties, and index IDs are distinct. After cloning missing versions into an **existing** tablet, the copied rowset schemas contain both indexes with the **same index ID**, while their distinct names/properties remain intact. Subsequent compaction repeatedly fails with: ```text [E-6010] IndexFileWriter::open attempted to insert a duplicate dir ``` Uncompacted versions accumulate and writes can eventually fail with E-235 / excessive tablet version count. In one observed sequence, a full clone to a new tablet succeeded. Two subsequent incremental clones copied 12 versions and then 1 version. Exactly those 13 rowset schemas contained collapsed index IDs. Compaction of the preceding version range succeeded, while compaction involving the subsequent range failed. A comparison replica's sampled rowsets retained distinct IDs. Both samples used V3 and the same schema version. ### Root cause In the affected commit: - [`SnapshotManager::_rename_index_ids`](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/storage/snapshot/snapshot_manager.cpp#L377-L397) - [`CloudSnapshotMgr::_rename_index_ids`](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/cloud/cloud_snapshot_mgr.cpp#L295-L315) Both iterate over the incoming indexes and look up the target index using only column unique ID, index type, and suffix: ```cpp for (int i = 0; i < schema_pb.index_size(); ++i) { TabletIndexPB* index_pb = schema_pb.mutable_index(i); for (int32_t col_unique_id : index_pb->col_unique_id()) { auto local_index = tablet_schema->get_index(col_unique_id, index_pb->index_type(), index_pb->index_suffix_name()); if (local_index) { if (index_pb->index_id() != local_index->index_id()) { index_pb->set_index_id(local_index->index_id()); } break; } } } ``` However, [`TabletSchema::get_index`](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/storage/tablet/tablet_schema.cpp#L1604-L1615) returns only the **first** entry in the vector associated with that tuple: ```cpp IndexKey index_key(index_type, col_unique_id, suffix_path); auto it = _col_id_suffix_to_index.find(index_key); if (it != _col_id_suffix_to_index.end()) { if (!it->second.empty() && it->second[0] < _indexes.size()) { return _indexes[it->second[0]].get(); } } ``` For example, an exact index `(ID=100, column=10, type=INVERTED, suffix="")` and a normalizer index `(ID=200, same column/type/suffix)` both resolve to target ID 100. Neither index name nor analyzer/normalizer properties participate in this lookup. This is inconsistent with the multiple-index representation: [`append_index`](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/storage/tablet/tablet_schema.cpp#L891-L904) appends all matching indexes to a vector, and `inverted_indexs()` returns the full vector. The downstream failure follows from [`IndexFileWriter::_insert_directory_into_map`](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/storage/index/index_file_writer.cpp#L69-L93), whose key is `(index_id, index_suffix)`. Opening both remapped indexes inserts the same key twice. The duplicate is an in-memory directory-map key, not evidence of a stale filesystem directory. [Index compaction enumerates the indexes and opens each destination directory](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/storage/compaction/compaction.cpp#L935-L983). ### Trigger boundary The decisive condition is whether a target tablet schema exists when index IDs are converted, rather than whether a clone is called “full” or “incremental”: - **Incremental clone into an existing tablet:** affected. - **Full clone into an existing tablet**, including fallback when the source cannot supply the requested missing versions: affected by the same conversion. - **Restore into an existing target tablet:** affected through [`SnapshotLoader`](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/runtime/snapshot_loader.cpp#L1190). Cloud restore has its own equivalent faulty remapping function. - **Local storage migration:** also reaches [`convert_rowset_ids`](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/storage/task/engine_storage_migration_task.cpp#L165). - **Full clone to a genuinely new/nonexistent target tablet:** skips this remapping because `target_tablet_schema == nullptr`; a healthy source therefore avoids this particular corruption during that initial copy. Later catch-up clones can still trigger it. Copying an already-corrupted source is not made safe by this condition. See [target schema selection](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/storage/snapshot/snapshot_manager.cpp#L186-L191) and the [clone conversion call](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/storage/task/engine_clone_task.cpp#L467-L471). Concurrent writes increase the likelihood of catch-up after an initial full clone, but are **not required** for the incorrect mapping itself. Replica repair or balancing can expose the affected paths. ### Minimal reproduce #### Synthetic table and existing-target clone scenario Use an isolated test cluster with at least three BEs. The following names and values are synthetic: ```sql CREATE DATABASE IF NOT EXISTS multi_index_repro; CREATE INVERTED INDEX NORMALIZER repro_lowercase PROPERTIES ("token_filter" = "lowercase"); CREATE TABLE multi_index_repro.items ( id BIGINT NOT NULL, label STRING NULL, INDEX idx_label_exact (label) USING INVERTED, INDEX idx_label_ci (label) USING INVERTED PROPERTIES ("normalizer" = "repro_lowercase") ) UNIQUE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ( "replication_allocation" = "tag.location.default: 3", "enable_unique_key_merge_on_write" = "true", "inverted_index_storage_format" = "V3" ); INSERT INTO multi_index_repro.items VALUES (1, 'Alpha'), (2, 'alpha'), (3, 'BETA'); ``` 1. Record the tablet and each replica's schema. Confirm that the two indexes have distinct IDs. 2. In the isolated cluster, stop one BE while retaining its tablet/data directory. Continue committing small batches of synthetic rows through the other replicas so that the stopped replica falls behind. 3. Restart that BE and request repair if necessary: ```sql ADMIN REPAIR TABLE multi_index_repro.items; ``` Verify in the logs that a clone actually occurs **into the existing tablet**. Restart/repair scheduling alone does not guarantee this: if publishing catches up without a clone, the affected path has not been exercised. 4. After the clone, inspect the copied rowset schemas. The two different index definitions can now share the first target index's ID. 5. Let compaction run, or invoke the test cluster's compaction mechanism. Check for the duplicate-directory error and validate both exact and case-insensitive query results against a scan without index filtering. An alternative integration scenario is to replace a replica while continuing writes: first copy to a nonexistent target, then verify an existing-target incremental catch-up. This is the full-copy-then-catch-up sequence observed in the incident. The SQL above is a synthetic reproduction setup; we have not run this exact isolated-cluster script. #### Deterministic extracted-function reproduction (executed) We compiled the original bodies of `TabletSchema::get_index`, `SnapshotManager::_rename_index_ids`, and `CloudSnapshotMgr::_rename_index_ids` from the fixed commit in a small C++17 harness, using minimal protobuf/container stand-ins. Both definitions use column UID 10, type INVERTED, and an empty suffix; their names/properties differ. After remapping, the harness also inserts each resulting `(index_id, suffix)` into a map, matching the writer's uniqueness rule: | Target schema | Local result IDs | Cloud result IDs | Duplicate writer key | |---|---|---|---| | No target schema | 100, 200 | 100, 200 | No | | Target contains exact=100, ci=200 | 100, 100 | 100, 100 | Yes | | Restore target contains exact=1000, ci=2000 | 1000, 1000 | 1000, 1000 | Yes | | Target order reversed: ci=200, exact=100 | 200, 200 | 200, 200 | Yes | This validates the ID collapse and duplicate key deterministically. It is not a full BE/CLucene integration test. Reordering the indexes merely changes which ID survives. The existing local snapshot conversion unit test uses a single BITMAP index. The cloud test uses two indexes on a column but with different types (BITMAP and INVERTED), so neither detects this same-column, same-type collision. ### Expected behavior - Distinct inverted index definitions on one column remain distinct after clone, restore, and storage migration. - A same-tablet clone preserves the correct existing index IDs. - Restore maps source indexes to their corresponding target definitions one-to-one when IDs change. - Both query semantics and compaction remain correct after the conversion. ### Suggested fix Please consider fixing both the local and cloud remapping implementations: - For same-tablet cloning, preserve an existing ID when it corresponds to the same index definition. - Where remapping is required, match by **index name, type, column identity, suffix, and properties**, rather than taking the first index for a column/type/suffix tuple. `TabletIndex::is_same_except_id` already compares these definition fields and may be useful. - If the required mapping is missing or ambiguous, return an explicit error instead of silently selecting the first candidate. - Do not use the BE analyzer query key alone as index identity: normalizer-only indexes and variant suffixes also need to be distinguished. - Cover visible/stale rowset schemas, the tablet schema, and the corresponding physical index IDs in the snapshot/restore tests. Suggested regression cases: same-ID clone, restore with changed IDs, reversed target index order, a missing target schema, and variant subpaths where an index ID may legitimately be shared across **different suffixes**. Also verify clone followed by compaction and indexed-versus-scan query results. Fixing the mapping prevents new corruption; it does not automatically repair rowset schemas already rewritten with duplicate IDs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
