Vivek1106-04 opened a new issue, #12442:
URL: https://github.com/apache/seatunnel/issues/12442
### What happened
`CheckpointCoordinator.tryTriggerPendingCheckpoint(CheckpointType)`
(`CheckpointCoordinator.java:771`) is both the periodic checkpoint trigger
and the thing that
re-arms the next one. It is scheduled with no surrounding try/catch, so if
it throws before
reaching a re-arm point, nothing schedules the next trigger and periodic
checkpointing stops
for that pipeline, silently.
The re-arm inside the pending-checkpoint branch is at
`CheckpointCoordinator.java:841`:
```java
if (pendingCounter.get() > 0) {
scheduleTriggerPendingCheckpoint(checkpointType, 500L);
LOG.debug("skip trigger checkpoint because there is already a pending
checkpoint.");
return;
}
```
A `RuntimeException` on any path that reaches neither this line nor the
normal trigger path
leaves the pipeline with no armed timer. The job keeps running; checkpoints
simply stop.
### Why it is silent
Before #12165, the body was scheduled directly on a
`ScheduledThreadPoolExecutor`. That wraps
the runnable in a future-backed task whose `run()` catches `Throwable` and
stores it via
`setException()`. Since the returned future is never queried, the exception
is swallowed with
no output at all.
After #12165 the body runs as a plain `Runnable` on the dispatch pool, so an
uncaught exception
reaches the worker thread's default uncaught-exception handler and prints a
stack trace to
stderr - strictly more visible than before, but still not routed through
SeaTunnel's logger and
still not surfaced as a job-level problem.
**This is pre-existing on `dev` and is not made worse by #12165.** It was
found while reviewing
that PR (thanks to @nzw921rx for asking the question that surfaced it, and
@DanielLeens for
tracing the difference between the two scheduling paths) and is filed
separately rather than
widening that diff.
### Expected behavior
Checkpoint triggering going silent is exactly the kind of failure an
operator needs to see.
Wrap the scheduled body - either at the call site or in
`PipelineCheckpointTask.run()` - so that
an unexpected throwable is logged at WARN or ERROR through SeaTunnel's own
logger, with the job
and pipeline id in the message.
Worth deciding at the same time whether the trigger should also re-arm
itself on an unexpected
failure rather than leaving the pipeline without a timer, or whether it
should fail the
coordinator outright. Logging is the minimum; recovering is the better
outcome but is a
behavioural change that deserves its own discussion.
### Context
- Found during review of #12165 (STIP-40: #12339)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
