Doris-Breakwater commented on issue #69783: URL: https://github.com/apache/doris/issues/69783#issuecomment-6473270085
Breakwater-GitHub-Analysis-Slot: slot_a536be9bc046 The three reported gaps are supported by source inspection at both `branch-4.1@5fc51d21256` and `branch-4.2@9f63be85cea`. The relevant CREATE validation, compute-group helper, ALTER command, Kafka job, metadata module ordering, and Kafka metadata routing files are identical between these revisions. This warrants a targeted FE routine-load fix on both branches, with replay cancellation the highest priority because it turns a resource lookup failure into a terminal job state. The issue is currently open with no labels; suggested triage is FE / routine load, affecting both named branch revisions. Verified code paths: 1. **Replay cancellation:** `RoutineLoadJob.gsonPostProcess()` reparses `origStmt`, sets `isReplay`, and calls `validate()`. `CreateRoutineLoadInfo.checkJobProperties()` skips compute-group validation during replay but unconditionally calls `WorkloadGroupMgr.getWorkloadGroup()` when the original CREATE contains `workload_group`. In cloud mode that resolves the CREATE-time compute group through `ConnectContext.getComputeGroup()` and `ComputeGroupMgr.getComputeGroupByName()`. An ALTER changes the job properties without rewriting that original statement. Thus dropping the original group can break reconstruction even when the current binding is valid. The outer catch directly sets `CANCELLED`; `checkStateTransform()` rejects transitions out of that state, including RESUME. [Replay and state handling](https://github.com/apache/doris/blob/5fc51d21256d598dc89afe5732550b62baeac4a9/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java), [CREATE validation](https:// github.com/apache/doris/blob/5fc51d21256d598dc89afe5732550b62baeac4a9/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateRoutineLoadInfo.java). 2. **Image-load dependency:** `Env` constructs an empty workload-group manager. `MetaWriter` writes modules in `PersistMetaModules.MODULES_IN_ORDER`, and `MetaReader` reads their recorded order. `routineLoadJobs` precedes `workloadGroups`, so job deserialization can perform this lookup before persisted workload groups have been installed. This supports the non-cloud failure mechanism independently of a missing compute group. It remains a source-supported scenario plus the reporter's unit-test evidence, not a cluster reproduction from this investigation. [Module ordering](https://github.com/apache/doris/blob/5fc51d21256d598dc89afe5732550b62baeac4a9/fe/fe-core/src/main/java/org/apache/doris/persist/meta/PersistMetaModules.java). 3. **Suspension and combined ALTER:** the branch helper checks USAGE and existence but never calls `waitForAutoStart()`, whereas `CloudEnv.changeCloudCluster()` does. Kafka metadata selection for a declared group uses `getBackendsByClusterName()` without starting it, explaining failure when suspension leaves no usable BE. Separately, `KafkaRoutineLoadJob.modifyProperties()` converts offsets before taking the write lock and installing new properties; `convertOffset()` reads the current job's group. Consequently a combined ALTER routes conversion through the old binding. [Kafka ALTER implementation](https://github.com/apache/doris/blob/5fc51d21256d598dc89afe5732550b62baeac4a9/fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaRoutineLoadJob.java). One status correction: [PR #69749](https://github.com/apache/doris/pull/69749) currently reports **open / unmerged**. I fetched and inspected its fix commit [`4ed5856f57c`](https://github.com/apache/doris/commit/4ed5856f57cd138283c1afaff3cbaf956650fc28), but it is not an ancestor of the fetched master snapshot `86d62ae00c9`. Therefore “fixed on master” should presently be read as “addressed in the master-targeted PR,” not as an available merged release fix. Recommended maintainer action is to review that fix and prepare equivalent changes for both affected branches: adopt resource names without resolving them during reconstruction; retain validation for user CREATE/ALTER; activate the explicitly declared group on the statement thread before job locking; and pass the ALTER's declared group into offset conversion, falling back to the current binding only when none is declared. Master places the Kafka job under `load/routineload/kafka/`, while these branches use `load/routineload/`, so account for that path difference when porting. The inspected diff does not change persisted formats or FE/BE protocols. Keep async-MV hint precedence and the separate refresh-time reads outside this repair. Validation still needed: - Run the five named `RoutineLoadComputeGroupTest` cases on each branch before and after the fix. They are present in the inspected PR commit, but I did not execute them or independently verify the reported red/green results. The image case exercises property validation with an empty manager; it is not a full image-load test. - Add end-to-end image-restart and create-journal replay coverage, including CREATE in A, ALTER to B, removal of A, and preservation of state, binding, and offsets. Confirm ordinary CREATE still rejects missing resources and denied privileges, and replay performs no activation RPC. - For runtime confirmation, attach sanitized CREATE/ALTER SQL, FE build hash and deployment mode, `SHOW ROUTINE LOAD` before/after restart, and the complete FE warning/stack trace beginning `error happens when parsing create routine load stmt`. Distinguish image load from journal replay and include checkpoint/journal IDs. For suspension or scale-to-zero, include group status, registration/backend availability, auto-start configuration, and the Kafka metadata error at the same time. Scale-to-zero is not established merely by the missing-group unit test: this lookup throws when the name cannot resolve to a cluster ID. No query profile is needed for these control-plane paths. For a PAUSED job needing only the combined-ALTER workaround, change its binding to a usable group first, then change offsets in a separate ALTER. Wake a suspended target with `USE @<group>` before metadata-dependent operations. Neither workaround repairs the original-statement replay problem, and the proposed fix does not revive a job already persisted as CANCELLED; recovery requires a separate decision based on its retained Kafka progress. This was a read-only code investigation; no Doris code was changed and no cluster tests were run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
