dxyou0830 opened a new issue, #70583:
URL: https://github.com/apache/doris/issues/70583

   
   ### Search before asking
   
   - [x] I searched existing issues and pull requests for this error and the 
index-remapping functions.
   
   The identical error message appears in #44609 (fixed by #44610), but that 
issue concerns duplicate column grouping with cluster keys. This report 
concerns index ID remapping during snapshot conversion. #59117 and its 
branch-4.0 backport #60415 introduced support for multiple inverted indexes on 
one column.
   
   ### Version
   
   Observed on **Doris 4.1.4-rc04**, commit 
[`ad35a140c7fd0b842f18c23300bac581f7d04326`](https://github.com/apache/doris/commit/ad35a140c7fd0b842f18c23300bac581f7d04326),
 using inverted index storage format **V3**.
   
   The same faulty lookup/remapping remains in both the local and cloud 
implementations at the following revisions, checked on November 14, 2026:
   
   | Ref | Commit | Verification |
   |---|---|---|
   | master | `e82d2650e38e57886679faecf0ad5569169c54aa` | Source inspection |
   | branch-4.0 | `8a9961723ea4be00cdf923c60759607202c7e2e7` | Source 
inspection |
   | branch-4.1 | `af23c8da954119f3ec6a9c0f8d18327286e940e7` | Source 
inspection |
   | 4.1.5-release | `917b20be83878e98f94e3ebf7efccc80346e3e50` | Source 
inspection |
   
   These additional revisions were not tested in running clusters.
   
   ### What happened
   
   A string column has two inverted indexes: an exact index and a 
lowercase-normalizer index. Their names, properties, and index IDs are 
distinct. After cloning missing versions into an **existing** tablet, the 
copied rowset schemas contain both indexes with the **same index ID**, while 
their distinct names/properties remain intact.
   
   Subsequent compaction repeatedly fails with:
   
   ```text
   [E-6010] IndexFileWriter::open attempted to insert a duplicate dir
   ```
   
   Uncompacted versions accumulate and writes can eventually fail with E-235 / 
excessive tablet version count.
   
   In one observed sequence, a full clone to a new tablet succeeded. Two 
subsequent incremental clones copied 12 versions and then 1 version. Exactly 
those 13 rowset schemas contained collapsed index IDs. Compaction of the 
preceding version range succeeded, while compaction involving the subsequent 
range failed. A comparison replica's sampled rowsets retained distinct IDs. 
Both samples used V3 and the same schema version.
   
   ### Root cause
   
   In the affected commit:
   
   - 
[`SnapshotManager::_rename_index_ids`](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/storage/snapshot/snapshot_manager.cpp#L377-L397)
   - 
[`CloudSnapshotMgr::_rename_index_ids`](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/cloud/cloud_snapshot_mgr.cpp#L295-L315)
   
   Both iterate over the incoming indexes and look up the target index using 
only column unique ID, index type, and suffix:
   
   ```cpp
   for (int i = 0; i < schema_pb.index_size(); ++i) {
       TabletIndexPB* index_pb = schema_pb.mutable_index(i);
       for (int32_t col_unique_id : index_pb->col_unique_id()) {
           auto local_index = tablet_schema->get_index(col_unique_id, 
index_pb->index_type(),
                                                       
index_pb->index_suffix_name());
           if (local_index) {
               if (index_pb->index_id() != local_index->index_id()) {
                   index_pb->set_index_id(local_index->index_id());
               }
               break;
           }
       }
   }
   ```
   
   However, 
[`TabletSchema::get_index`](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/storage/tablet/tablet_schema.cpp#L1604-L1615)
 returns only the **first** entry in the vector associated with that tuple:
   
   ```cpp
   IndexKey index_key(index_type, col_unique_id, suffix_path);
   auto it = _col_id_suffix_to_index.find(index_key);
   if (it != _col_id_suffix_to_index.end()) {
       if (!it->second.empty() && it->second[0] < _indexes.size()) {
           return _indexes[it->second[0]].get();
       }
   }
   ```
   
   For example, an exact index `(ID=100, column=10, type=INVERTED, suffix="")` 
and a normalizer index `(ID=200, same column/type/suffix)` both resolve to 
target ID 100. Neither index name nor analyzer/normalizer properties 
participate in this lookup.
   
   This is inconsistent with the multiple-index representation: 
[`append_index`](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/storage/tablet/tablet_schema.cpp#L891-L904)
 appends all matching indexes to a vector, and `inverted_indexs()` returns the 
full vector.
   
   The downstream failure follows from 
[`IndexFileWriter::_insert_directory_into_map`](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/storage/index/index_file_writer.cpp#L69-L93),
 whose key is `(index_id, index_suffix)`. Opening both remapped indexes inserts 
the same key twice. The duplicate is an in-memory directory-map key, not 
evidence of a stale filesystem directory. [Index compaction enumerates the 
indexes and opens each destination 
directory](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/storage/compaction/compaction.cpp#L935-L983).
   
   ### Trigger boundary
   
   The decisive condition is whether a target tablet schema exists when index 
IDs are converted, rather than whether a clone is called “full” or 
“incremental”:
   
   - **Incremental clone into an existing tablet:** affected.
   - **Full clone into an existing tablet**, including fallback when the source 
cannot supply the requested missing versions: affected by the same conversion.
   - **Restore into an existing target tablet:** affected through 
[`SnapshotLoader`](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/runtime/snapshot_loader.cpp#L1190).
 Cloud restore has its own equivalent faulty remapping function.
   - **Local storage migration:** also reaches 
[`convert_rowset_ids`](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/storage/task/engine_storage_migration_task.cpp#L165).
   - **Full clone to a genuinely new/nonexistent target tablet:** skips this 
remapping because `target_tablet_schema == nullptr`; a healthy source therefore 
avoids this particular corruption during that initial copy. Later catch-up 
clones can still trigger it. Copying an already-corrupted source is not made 
safe by this condition.
   
   See [target schema 
selection](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/storage/snapshot/snapshot_manager.cpp#L186-L191)
 and the [clone conversion 
call](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/storage/task/engine_clone_task.cpp#L467-L471).
   
   Concurrent writes increase the likelihood of catch-up after an initial full 
clone, but are **not required** for the incorrect mapping itself. Replica 
repair or balancing can expose the affected paths.
   
   ### Minimal reproduce
   
   #### Synthetic table and existing-target clone scenario
   
   Use an isolated test cluster with at least three BEs. The following names 
and values are synthetic:
   
   ```sql
   CREATE DATABASE IF NOT EXISTS multi_index_repro;
   
   CREATE INVERTED INDEX NORMALIZER repro_lowercase
   PROPERTIES ("token_filter" = "lowercase");
   
   CREATE TABLE multi_index_repro.items (
       id BIGINT NOT NULL,
       label STRING NULL,
       INDEX idx_label_exact (label) USING INVERTED,
       INDEX idx_label_ci (label) USING INVERTED
           PROPERTIES ("normalizer" = "repro_lowercase")
   )
   UNIQUE KEY(id)
   DISTRIBUTED BY HASH(id) BUCKETS 1
   PROPERTIES (
       "replication_allocation" = "tag.location.default: 3",
       "enable_unique_key_merge_on_write" = "true",
       "inverted_index_storage_format" = "V3"
   );
   
   INSERT INTO multi_index_repro.items VALUES
       (1, 'Alpha'), (2, 'alpha'), (3, 'BETA');
   ```
   
   1. Record the tablet and each replica's schema. Confirm that the two indexes 
have distinct IDs.
   2. In the isolated cluster, stop one BE while retaining its tablet/data 
directory. Continue committing small batches of synthetic rows through the 
other replicas so that the stopped replica falls behind.
   3. Restart that BE and request repair if necessary:
   
      ```sql
      ADMIN REPAIR TABLE multi_index_repro.items;
      ```
   
      Verify in the logs that a clone actually occurs **into the existing 
tablet**. Restart/repair scheduling alone does not guarantee this: if 
publishing catches up without a clone, the affected path has not been exercised.
   4. After the clone, inspect the copied rowset schemas. The two different 
index definitions can now share the first target index's ID.
   5. Let compaction run, or invoke the test cluster's compaction mechanism. 
Check for the duplicate-directory error and validate both exact and 
case-insensitive query results against a scan without index filtering.
   
   An alternative integration scenario is to replace a replica while continuing 
writes: first copy to a nonexistent target, then verify an existing-target 
incremental catch-up. This is the full-copy-then-catch-up sequence observed in 
the incident. The SQL above is a synthetic reproduction setup; we have not run 
this exact isolated-cluster script.
   
   #### Deterministic extracted-function reproduction (executed)
   
   We compiled the original bodies of `TabletSchema::get_index`, 
`SnapshotManager::_rename_index_ids`, and `CloudSnapshotMgr::_rename_index_ids` 
from the fixed commit in a small C++17 harness, using minimal 
protobuf/container stand-ins. Both definitions use column UID 10, type 
INVERTED, and an empty suffix; their names/properties differ.
   
   After remapping, the harness also inserts each resulting `(index_id, 
suffix)` into a map, matching the writer's uniqueness rule:
   
   | Target schema | Local result IDs | Cloud result IDs | Duplicate writer key 
|
   |---|---|---|---|
   | No target schema | 100, 200 | 100, 200 | No |
   | Target contains exact=100, ci=200 | 100, 100 | 100, 100 | Yes |
   | Restore target contains exact=1000, ci=2000 | 1000, 1000 | 1000, 1000 | 
Yes |
   | Target order reversed: ci=200, exact=100 | 200, 200 | 200, 200 | Yes |
   
   This validates the ID collapse and duplicate key deterministically. It is 
not a full BE/CLucene integration test. Reordering the indexes merely changes 
which ID survives.
   
   The existing local snapshot conversion unit test uses a single BITMAP index. 
The cloud test uses two indexes on a column but with different types (BITMAP 
and INVERTED), so neither detects this same-column, same-type collision.
   
   ### Expected behavior
   
   - Distinct inverted index definitions on one column remain distinct after 
clone, restore, and storage migration.
   - A same-tablet clone preserves the correct existing index IDs.
   - Restore maps source indexes to their corresponding target definitions 
one-to-one when IDs change.
   - Both query semantics and compaction remain correct after the conversion.
   
   ### Suggested fix
   
   Please consider fixing both the local and cloud remapping implementations:
   
   - For same-tablet cloning, preserve an existing ID when it corresponds to 
the same index definition.
   - Where remapping is required, match by **index name, type, column identity, 
suffix, and properties**, rather than taking the first index for a 
column/type/suffix tuple. `TabletIndex::is_same_except_id` already compares 
these definition fields and may be useful.
   - If the required mapping is missing or ambiguous, return an explicit error 
instead of silently selecting the first candidate.
   - Do not use the BE analyzer query key alone as index identity: 
normalizer-only indexes and variant suffixes also need to be distinguished.
   - Cover visible/stale rowset schemas, the tablet schema, and the 
corresponding physical index IDs in the snapshot/restore tests.
   
   Suggested regression cases: same-ID clone, restore with changed IDs, 
reversed target index order, a missing target schema, and variant subpaths 
where an index ID may legitimately be shared across **different suffixes**. 
Also verify clone followed by compaction and indexed-versus-scan query results.
   
   Fixing the mapping prevents new corruption; it does not automatically repair 
rowset schemas already rewritten with duplicate IDs.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to