SEZ9 opened a new issue, #13153:
URL: https://github.com/apache/seatunnel/issues/13153

   ### Search before asking
   
   - [X] I had searched in the 
[feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22)
 and found no similar feature requirement.
   
   ### Description
   
   A pipeline that restarts repeatedly tells you that it is restarting, but not 
why.
   
   `SubPlan.errorByPhysicalVertex` holds the failure of the **current** attempt 
only, and `initStateFuture` replaces it on every restore:
   
   ```java
   public synchronized PassiveCompletableFuture<PipelineExecutionState> 
initStateFuture() {
       // reset errorByPhysicalVertex when restore pipeline
       errorByPhysicalVertex = new AtomicReference<>();
   ```
   
   So the reason each earlier run ended is discarded as the next one starts. By 
the time an operator — or the CLI's `--diagnose` — looks at a pipeline that has 
restored seven times, all seven reasons are gone, and `errorByPhysicalVertex` 
holds whatever the current attempt has produced so far, which may be nothing.
   
   ### Why this matters
   
   `JobRuntimeDiagnostics` publishes per pipeline:
   
   ```
   pipelineId, pipelineStatus, restoreCount, maxRestoreCount, stateTimestamps
   ```
   
   That is enough to **detect** a crash loop — `restoreCount` climbing toward 
`maxRestoreCount` while `jobStatus` stays `RUNNING` — and the `--diagnose` 
command added in #12631 and extended in #12688 does exactly that. It is not 
enough to **act** on one. The first question after "this pipeline is 
crash-looping" is "what is failing", and the answer existed only in the logs of 
a run that has already been replaced.
   
   The information is cheap to keep. It is a short string per restore, and the 
number of restores is already bounded by `job.retry.times` (default 3) through 
`checkNeedRestore`:
   
   ```java
   return jobMaster.isNeedRestore() && getPipelineRestoreNum() < 
pipelineMaxRestoreNum;
   ```
   
   ### Usage Scenario
   
   A CDC pipeline whose sink credentials expire. The job stays `RUNNING`, 
`restoreCount` climbs, and every diagnostic surface can say only "it is 
restarting". With the causes retained, the same request answers it:
   
   ```
   "restoreCauses": [
     "restore 6: ... ErrorCode:[JDBC-05] ... Access denied for user 
'etl'@'10.0.1.7'",
     "restore 7: ... ErrorCode:[JDBC-05] ... Access denied for user 
'etl'@'10.0.1.7'"
   ]
   ```
   
   This is the data the LLM-based root-cause diagnosis in the AI CLI needs as 
well; without it the model is asked to explain a failure from a restore count 
alone.
   
   ### Proposal
   
   Retain the cause of each restore on `SubPlan` and publish it as an additive 
`restoreCauses` array on each pipeline in the diagnostics block.
   
   - Capture in `prepareRestorePipeline`, inside the existing `restoreLock`, 
before `reset()` and before `initStateFuture` clears the reference.
   - Bound it twice: the existing `job.retry.times` limit, plus a fixed cap in 
case that option is configured high. Truncate each entry.
   - List a restore with no recorded error with a placeholder rather than 
omitting it — "restarted for an unknown reason" is a different and more 
alarming state than a named failure, and dropping it makes the two look alike.
   - No new configuration option. The retry limit already bounds this, and a 
knob for a list of at most sixteen truncated strings would cost a 
`ServerConfigOptions` entry, a YAML parser branch and another documented option 
for no decision the user needs to make.
   
   Additive only: a new field on an existing object, no existing field changes 
name, type or meaning.
   
   ### Related
   
   - #12631, #12688 — the CLI diagnostics that can detect a crash loop but not 
explain it
   - #12789 — the bounded, configurable checkpoint monitor history, the 
precedent for retaining a series engine-side
   - #12794 — `generatedAt` on the checkpoint overview, the precedent for an 
additive diagnostics field
   
   ### Are you willing to submit a PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to