gavinchou commented on PR #69129: URL: https://github.com/apache/doris/pull/69129#issuecomment-6207229936
**[Major / P1] Compact CDC offset journals discard the schema baseline on older FE readers** At commit `1a1f93842d8`, [`persistOffsetProviderIfNeed()`](https://github.com/apache/doris/blob/1a1f93842d8aeb1b39077c719df4b854a0e09af0/fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/StreamingInsertJob.java#L1770-L1784) omits `opp.ts` on ordinary offset commits while retaining the existing journal operation. Older FE readers replace `offsetProviderPersist` wholesale in `replayOnUpdated()`, so replay advances the offset but drops its previously committed schema. If an older follower becomes master, or an older binary replays these journals after rollback, provider restoration produces a null schema. A rebuilt MySQL reader then discovers the current JDBC schema. When source DDL is ahead of the committed binlog position, that schema may differ from the schema needed to decode the pending pre-DDL records, potentially preventing CDC recovery. The existing [reader recovery contract](https://github.com/apache/doris/blob/1a1f93842d8aeb1b39077c719df4b854a0e09af0/fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/reader/AbstractCdcSourceReader.java#L229-L245) explains why the schema baseline must accompany the offset. The stronger compatibility approach is to keep writing full records until all possible FE readers support merging, or gate compact writes, and test new-writer/old-reader replay followed by recovery with source DDL ahead of the offset. Validation: a standalone Gson 2.10.1 probe confirmed schema loss under the old replacement semantics and schema preservation with the new merge method. The DDL decoding failure path was traced statically; mixed-version multi-FE failover was not exercised. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
