[
https://issues.apache.org/jira/browse/FLINK-40238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Li updated FLINK-40238:
----------------------------
Description:
h2. Proposal
Add a new JobManager-level gauge alongside {{{}numberOfFailedCheckpoints{}}}:
{noformat}
numberOfConsecutiveFailedCheckpoints
{noformat}
The gauge mirrors {{{}CheckpointFailureManager{}}}'s internal counter:
increments on each checkpoint failure counted toward
{{{}tolerable-failed-checkpoints{}}}, resets on the next successful checkpoint.
h2. Problem it solves
The core problem this solves is letting users detect that a job is in a bad
state via external monitoring, by observing consecutive checkpoint failures.
This fires before the JobManager gives up, giving operators a window to
intervene. The alert predicate is textually the same as the config value.
h2. scope:
{{{}DefaultCheckpointStatsTracker{}}}: register the new gauge and maintain the
counter via the {{CheckpointStatsListener}} event path (increment on
{{onFailedCheckpoint}} when the failure reason is counted, reset on
{{{}onCompletedCheckpoint{}}}). No new config options, no public API or REST
changes.
Doc update in {{{}docs/content/docs/ops/metrics.md{}}}, next to{{{}
numberOfFailedCheckpoints{}}}.
was:
h2. Proposal
Add a new JobManager-level gauge alongside {{numberOfFailedCheckpoints}}:
{noformat}
numberOfConsecutiveFailedCheckpoints
{noformat}
The gauge mirrors {{CheckpointFailureManager}}'s internal counter:
increments on each checkpoint failure counted toward
{{tolerable-failed-checkpoints}}, resets on the next successful
checkpoint. Users see the same number the JobManager itself uses to
decide whether to fail the job, and the distance to
JobManager-initiated failover is simply
{{tolerable-failed-checkpoints - <gauge>}}.
Proposed scope:
{{DefaultCheckpointStatsTracker}}: register the new gauge and maintain
the counter via the {{CheckpointStatsListener}} event path (increment on
{{onFailedCheckpoint}} when the failure
reason is counted, reset on {{onCompletedCheckpoint}}).
No new config options, no public API or REST changes.
Doc update in {{docs/content/docs/ops/metrics.md}}, next to
{{numberOfFailedCheckpoints}}.
h2. Problem it solves
The core problem this solves is letting users detect that a job is in a bad
state
via external monitoring, by observing consecutive checkpoint failures.
This fires before the JobManager gives up, giving operators a window to
intervene. The alert predicate is textually the same as the config
value.
> Expose numberOfConsecutiveFailedCheckpoints as a JobManager-level gauge
> -----------------------------------------------------------------------
>
> Key: FLINK-40238
> URL: https://issues.apache.org/jira/browse/FLINK-40238
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing, Runtime / Metrics
> Reporter: Yang Li
> Priority: Minor
>
> h2. Proposal
> Add a new JobManager-level gauge alongside {{{}numberOfFailedCheckpoints{}}}:
> {noformat}
> numberOfConsecutiveFailedCheckpoints
> {noformat}
> The gauge mirrors {{{}CheckpointFailureManager{}}}'s internal counter:
> increments on each checkpoint failure counted toward
> {{{}tolerable-failed-checkpoints{}}}, resets on the next successful
> checkpoint.
> h2. Problem it solves
> The core problem this solves is letting users detect that a job is in a bad
> state via external monitoring, by observing consecutive checkpoint failures.
> This fires before the JobManager gives up, giving operators a window to
> intervene. The alert predicate is textually the same as the config value.
> h2. scope:
> {{{}DefaultCheckpointStatsTracker{}}}: register the new gauge and maintain
> the counter via the {{CheckpointStatsListener}} event path (increment on
> {{onFailedCheckpoint}} when the failure reason is counted, reset on
> {{{}onCompletedCheckpoint{}}}). No new config options, no public API or REST
> changes.
> Doc update in {{{}docs/content/docs/ops/metrics.md{}}}, next to{{{}
> numberOfFailedCheckpoints{}}}.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
