Doris-Breakwater commented on issue #69079: URL: https://github.com/apache/doris/issues/69079#issuecomment-6172721552
Breakwater-GitHub-Analysis-Slot: slot_ea1b495ba1d0 Initial triage: **the replica-membership gap is supported by source inspection and warrants high-priority correctness investigation.** The reported silent retention loss is a plausible consequence of that gap, but has not been reproduced in this investigation. In particular, permanent loss across every replica is not established by the report. I inspected upstream `master` at `2137067cccd3d647ffa36e6363e2365856864838`. The issue is open, has no labels, and supplies no reproduction or runtime evidence. The findings below concern the shared-nothing clone path; applicability to cloud mode and released versions needs separate verification. **Verified code behavior** 1. [`SchemaChangeHandler.updateRowBinlogConfig()`](https://github.com/apache/doris/blob/2137067cccd3d647ffa36e6363e2365856864838/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java#L4053) journals a new generation before RPCs. For an extension, it initially keeps the published TTL at the old value. [`updatePartitionProperties()`](https://github.com/apache/doris/blob/2137067cccd3d647ffa36e6363e2365856864838/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java#L3096) captures backend/tablet targets under the table read lock, including ROW-binlog indexes, then releases that lock before submission and acknowledgment waiting. Final publication checks the generation and partition-ID set, without checking replica membership or a per-replica configuration acknowledgment. 2. [`TabletSchedCtx.createCloneReplicaAndTask()`](https://github.com/apache/doris/blob/2137067cccd3d647ffa36e6363e2365856864838/fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java#L1106) can add a `CLONE` replica to an existing tablet without changing its partition ID. [`CloneTask.toThrift()`](https://github.com/apache/doris/blob/2137067cccd3d647ffa36e6363e2365856864838/fe/fe-core/src/main/java/org/apache/doris/task/CloneTask.java#L94) and [`TCloneReq`](https://github.com/apache/doris/blob/2137067cccd3d647ffa36e6363e2365856864838/gensrc/thrift/AgentService.thrift#L375) carry neither binlog configuration nor its generation. 3. For a destination with no existing tablet, [`EngineCloneTask`](https://github.com/apache/doris/blob/2137067cccd3d647ffa36e6363e2365856864838/be/src/storage/task/engine_clone_task.cpp#L300) installs the downloaded snapshot using [`TabletManager.load_tablet_from_dir()`](https://github.com/apache/doris/blob/2137067cccd3d647ffa36e6363e2365856864838/be/src/storage/tablet/tablet_manager.cpp#L992). [`TabletMeta`](https://github.com/apache/doris/blob/2137067cccd3d647ffa36e6363e2365856864838/be/src/storage/tablet/tablet_meta.cpp#L928) deserializes the snapshot's binlog configuration directly. This does not invoke [`Tablet::set_binlog_config()`](https://github.com/apache/doris/blob/2137067cccd3d647ffa36e6363e2365856864838/be/src/storage/tablet/tablet.cpp#L2772), which rejects lower-generation configuration updates. A snapshot captured before its source accepts the extension can therefore install the old TTL on an omitted destination. 4. [`Compaction`](https://github.com/apache/doris/blob/2137067cccd3d647ffa36e6363e2365856864838/be/src/storage/compaction/compaction.cpp#L295) derives its cutoff from the tablet's local binlog configuration and filters expired rowsets into version-covering output. Its [commit-time TTL check](https://github.com/apache/doris/blob/2137067cccd3d647ffa36e6363e2365856864838/be/src/storage/compaction/compaction.cpp#L2077) compares against that same local policy. An unchanged local 60-second TTL passes this check even if FE has published 3600 seconds. Keeping version continuity means ordinary version-catch-up checks need not detect the missing events. 5. The read consequence has prerequisites: `CLONE` itself is not queryable. [`finishCloneTask()`](https://github.com/apache/doris/blob/2137067cccd3d647ffa36e6363e2365856864838/fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java#L1209) can promote a completed replica to `NORMAL` without a binlog-config acknowledgment check. [`getQueryableReplicas()`](https://github.com/apache/doris/blob/2137067cccd3d647ffa36e6363e2365856864838/fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java#L255) checks state/version/availability, without checking retention configuration. The reproduction must establish a usable base/ROW-binlog pair and actual routing to the affected backend. Also, repair into an existing tablet uses `_finish_incremental_clone()`/`_finish_full_clone()` to revise rowsets; this inspection does **not** establish that every repair overwrites an already newer TTL. **Missing evidence and suggested reproduction** - Please provide the exact FE/BE commit hashes, deployment mode, minimal table DDL and ALTER statement, replica layout, and whether this was observed or found by source review. - Capture the source snapshot's decoded binlog configuration, destination configuration before/after installation, and FE published configuration, including `config_version`, TTL, tablet/replica/backend IDs, and timestamps. Include the relevant FE update-task/clone-completion logs and BE snapshot-install/config-update/TTL-compaction logs, with sensitive values redacted. - A deterministic test should pause the ALTER after target capture but before configuration RPCs; schedule a missing ROW-binlog replica, capture its old-TTL snapshot, and pause installation. Then let the captured targets acknowledge the extension and FE publish it, resume installation, and run TTL compaction on the destination. Keep base/ROW locality valid. Use events present in the snapshot that become older than 60 seconds but remain younger than 3600 seconds, then compare the same scan range on healthy and affected backends. Record routing and successful-but-incomplete results. Events already expired before the extension are not evidence of this race. **Next steps for the proposed PR** - Add a regression test for replica addition/replacement within an unchanged partition during publication. The existing [`RowBinlogConfigPublicationTest`](https://github.com/apache/doris/blob/2137067cccd3d647ffa36e6363e2365856864838/fe/fe-core/src/test/java/org/apache/doris/alter/RowBinlogConfigPublicationTest.java#L238) covers partition additions, but that does not establish coverage of concurrent clone installation. Add BE coverage for delayed snapshots and lower-generation configuration installation as well. - Establish an invariant that newly installed ROW-binlog replicas accept the authoritative target policy before TTL deletion or reader eligibility, and that extension publication covers the current replica identities. Revalidation/retry must also account for in-flight clones completing after validation. Keep RPC waits outside catalog locks. - Carrying a configuration version alone is insufficient: during an extension, FE temporarily publishes the **new generation with the old TTL**. Clone admission/installation must distinguish that intermediate published policy from the pending target, and prevent delayed snapshots from regressing the effective policy. Merely reconciling after activation leaves an irreversible-deletion window. Validation performed: pinned-source inspection and review of existing test coverage. No cluster reproduction, build, test execution, or code modification was performed; actual loss and release scope remain unverified. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
