SEZ9 opened a new issue, #13153: URL: https://github.com/apache/seatunnel/issues/13153
### Search before asking - [X] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement. ### Description A pipeline that restarts repeatedly tells you that it is restarting, but not why. `SubPlan.errorByPhysicalVertex` holds the failure of the **current** attempt only, and `initStateFuture` replaces it on every restore: ```java public synchronized PassiveCompletableFuture<PipelineExecutionState> initStateFuture() { // reset errorByPhysicalVertex when restore pipeline errorByPhysicalVertex = new AtomicReference<>(); ``` So the reason each earlier run ended is discarded as the next one starts. By the time an operator — or the CLI's `--diagnose` — looks at a pipeline that has restored seven times, all seven reasons are gone, and `errorByPhysicalVertex` holds whatever the current attempt has produced so far, which may be nothing. ### Why this matters `JobRuntimeDiagnostics` publishes per pipeline: ``` pipelineId, pipelineStatus, restoreCount, maxRestoreCount, stateTimestamps ``` That is enough to **detect** a crash loop — `restoreCount` climbing toward `maxRestoreCount` while `jobStatus` stays `RUNNING` — and the `--diagnose` command added in #12631 and extended in #12688 does exactly that. It is not enough to **act** on one. The first question after "this pipeline is crash-looping" is "what is failing", and the answer existed only in the logs of a run that has already been replaced. The information is cheap to keep. It is a short string per restore, and the number of restores is already bounded by `job.retry.times` (default 3) through `checkNeedRestore`: ```java return jobMaster.isNeedRestore() && getPipelineRestoreNum() < pipelineMaxRestoreNum; ``` ### Usage Scenario A CDC pipeline whose sink credentials expire. The job stays `RUNNING`, `restoreCount` climbs, and every diagnostic surface can say only "it is restarting". With the causes retained, the same request answers it: ``` "restoreCauses": [ "restore 6: ... ErrorCode:[JDBC-05] ... Access denied for user 'etl'@'10.0.1.7'", "restore 7: ... ErrorCode:[JDBC-05] ... Access denied for user 'etl'@'10.0.1.7'" ] ``` This is the data the LLM-based root-cause diagnosis in the AI CLI needs as well; without it the model is asked to explain a failure from a restore count alone. ### Proposal Retain the cause of each restore on `SubPlan` and publish it as an additive `restoreCauses` array on each pipeline in the diagnostics block. - Capture in `prepareRestorePipeline`, inside the existing `restoreLock`, before `reset()` and before `initStateFuture` clears the reference. - Bound it twice: the existing `job.retry.times` limit, plus a fixed cap in case that option is configured high. Truncate each entry. - List a restore with no recorded error with a placeholder rather than omitting it — "restarted for an unknown reason" is a different and more alarming state than a named failure, and dropping it makes the two look alike. - No new configuration option. The retry limit already bounds this, and a knob for a list of at most sixteen truncated strings would cost a `ServerConfigOptions` entry, a YAML parser branch and another documented option for no decision the user needs to make. Additive only: a new field on an existing object, no existing field changes name, type or meaning. ### Related - #12631, #12688 — the CLI diagnostics that can detect a crash loop but not explain it - #12789 — the bounded, configurable checkpoint monitor history, the precedent for retaining a series engine-side - #12794 — `generatedAt` on the checkpoint overview, the precedent for an additive diagnostics field ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
