dxyou0830 opened a new pull request, #70585: URL: https://github.com/apache/doris/pull/70585
### Background Snapshot conversion rewrites rowset metadata when cloning replicas or restoring a snapshot into another tablet. Multiple inverted indexes can share a column, index type and suffix while having different names and properties, such as an exact index and a lowercase-normalizer index. ### What problem does this PR solve? Issue Number: close #70583 Related PR: None PR Type: Bugfix Problem Summary: Both local and cloud snapshot conversion look up the first target index for a column/type/suffix tuple. This can rewrite two distinct inverted indexes to the same ID. An incremental clone into an existing tablet can therefore leave duplicate index IDs in copied rowset schemas, causing subsequent compaction to fail with `[E-6010] IndexFileWriter::open attempted to insert a duplicate dir`. Preserve source IDs for same-tablet replica clones, and match complete index definitions where restore requires remapping. ### Details - **Frequency:** Reproducible when a column has multiple inverted indexes with the same type and suffix and snapshot conversion has an existing target schema. Reversing target index order changes which ID survives in the old implementation. - **Root cause:** `TabletSchema::get_index` returns the first matching column/type/suffix entry; that tuple does not uniquely identify an index definition. - **Why existing tests missed it:** The local conversion test uses one bitmap index. The cloud remapping test uses indexes of different types rather than two inverted indexes on the same column. - **Fix:** Same-tablet local clones retain the source index IDs, including historical rowsets. Local restore and cloud remapping use the existing `TabletIndex::is_same_except_id` comparison, which includes name, type, column IDs, suffix and properties. Existing behavior for an absent target schema or unmatched definition is retained. - **Other paths and branches:** Both `SnapshotManager` and `CloudSnapshotMgr` are covered. The local test checks tablet, visible-rowset and stale-rowset schemas. This PR targets master; no release-branch changes are included. - **Behavior change:** Distinct index definitions remain distinct after conversion. No configuration, protocol or storage-format changes are introduced. Already-corrupted rowset metadata is not repaired by this change. ### Testing - Added `SnapshotManagerTest.ConvertDualInvertedIndexesIntoExistingTablet`: create an existing target tablet, convert missing-version snapshot metadata with exact and normalizer indexes, and check distinct IDs in tablet, visible and stale schemas for clone and restore. Target index order is reversed. - Added `RenameDualInvertedIndexesByDefinition` to both snapshot-manager suites: changed target IDs, reversed order, mismatched name/properties/suffix, and absent target schema. - Local static checks: clang-format 16, build-hygiene checks and `git diff --check`. - Separate isolated validation: synthetic writes with incremental clone into an existing target reproduced E-6010 on an unpatched 4.1.4 baseline; a 4.1.5 build with the local fix retained unique index IDs and completed 28 compactions. Exact and case-insensitive query results, Stream Load and Routine Load checks passed, including an older source replica cloning into a patched target. - The new master unit tests and Cloud path have not been executed locally. The available build cache and third-party dependencies are for 4.1.5 and cannot be reused as a master incremental build. Master compilation, unit tests and clang-tidy are pending upstream CI. ### Release note Fix snapshot index-ID remapping that could collapse multiple inverted indexes on one column and cause compaction failures after replica clone or restore. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No; this restores existing clone/restore behavior. - Pre-submission status: Partially verified: master compilation, unit tests and clang-tidy are pending CI; isolated local-path validation was performed on 4.1.5. ### AI Disclosure - Tool: OpenAI Codex (GPT-6) for code, tests and description; reviewed by Anthropic Claude. - Generated: Code, tests and this description. - Human review: Not yet performed; requesting maintainer review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
