[
https://issues.apache.org/jira/browse/FLINK-40238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18100986#comment-18100986
]
Edson Wang commented on FLINK-40238:
------------------------------------
I'd like to take this one. Approach: expose a job-level gauge backed by the
continuous-failure counter that CheckpointFailureManager already tracks,
registered alongside numberOfFailedCheckpoints in the job-level metric group,
plus the generated metrics-docs update.
We monitor failed-checkpoint streaks operationally on a large production
deployment, so I'd also add an ITCase asserting the gauge resets after a
successful checkpoint.
Could a committer assign this to me?
> Expose numberOfConsecutiveFailedCheckpoints as a JobManager-level gauge
> -----------------------------------------------------------------------
>
> Key: FLINK-40238
> URL: https://issues.apache.org/jira/browse/FLINK-40238
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing, Runtime / Metrics
> Reporter: Yang Li
> Priority: Minor
>
> h2. Proposal
> Add a new JobManager-level gauge alongside {{{}numberOfFailedCheckpoints{}}}:
> {noformat}
> numberOfConsecutiveFailedCheckpoints
> {noformat}
> The gauge mirrors {{{}CheckpointFailureManager{}}}'s internal counter:
> increments on each checkpoint failure counted toward
> {{{}tolerable-failed-checkpoints{}}}, resets on the next successful
> checkpoint.
> h2. Problem it solves
> The core problem this solves is letting users detect that a job is in a bad
> state via external monitoring, by observing consecutive checkpoint failures.
> This fires before the JobManager gives up, giving operators a window to
> intervene. The alert predicate is textually the same as the config value.
> h2. scope:
> {{{}DefaultCheckpointStatsTracker{}}}: register the new gauge and maintain
> the counter via the {{CheckpointStatsListener}} event path (increment on
> {{onFailedCheckpoint}} when the failure reason is counted, reset on
> {{{}onCompletedCheckpoint{}}}). No new config options, no public API or REST
> changes.
> Doc update in {{{}docs/content/docs/ops/metrics.md{}}}, next to{{{{}}
> numberOfFailedCheckpoints{}}}.
> We've implemented this on our internal Flink fork and have been running it in
> production. Filing this ticket to check whether the community would like a PR
> against master.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
