[
https://issues.apache.org/jira/browse/FLINK-40238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Li updated FLINK-40238:
----------------------------
Description:
h2. Proposal
Add a new JobManager-level gauge alongside {{{}numberOfFailedCheckpoints{}}}:
{noformat}
numberOfConsecutiveFailedCheckpoints
{noformat}
The gauge mirrors {{{}CheckpointFailureManager{}}}'s internal counter:
increments on each checkpoint failure counted toward
{{{}tolerable-failed-checkpoints{}}}, resets on the next successful checkpoint.
h2. Problem it solves
The core problem this solves is letting users detect that a job is in a bad
state via external monitoring, by observing consecutive checkpoint failures.
This fires before the JobManager gives up, giving operators a window to
intervene. The alert predicate is textually the same as the config value.
h2. scope:
{{{}DefaultCheckpointStatsTracker{}}}: register the new gauge and maintain the
counter via the {{CheckpointStatsListener}} event path (increment on
{{onFailedCheckpoint}} when the failure reason is counted, reset on
{{{}onCompletedCheckpoint{}}}). No new config options, no public API or REST
changes.
Doc update in {{{}docs/content/docs/ops/metrics.md{}}}, next to{{{{}}
numberOfFailedCheckpoints{}}}.
We've implemented this on our internal Flink fork and have been running it in
production. Filing this ticket to check whether the community would like a PR
against master.
was:
h2. Proposal
Add a new JobManager-level gauge alongside {{{}numberOfFailedCheckpoints{}}}:
{noformat}
numberOfConsecutiveFailedCheckpoints
{noformat}
The gauge mirrors {{{}CheckpointFailureManager{}}}'s internal counter:
increments on each checkpoint failure counted toward
{{{}tolerable-failed-checkpoints{}}}, resets on the next successful checkpoint.
h2. Problem it solves
The core problem this solves is letting users detect that a job is in a bad
state via external monitoring, by observing consecutive checkpoint failures.
This fires before the JobManager gives up, giving operators a window to
intervene. The alert predicate is textually the same as the config value.
h2. scope:
{{{}DefaultCheckpointStatsTracker{}}}: register the new gauge and maintain the
counter via the {{CheckpointStatsListener}} event path (increment on
{{onFailedCheckpoint}} when the failure reason is counted, reset on
{{{}onCompletedCheckpoint{}}}). No new config options, no public API or REST
changes.
Doc update in {{{}docs/content/docs/ops/metrics.md{}}}, next to{{{}
numberOfFailedCheckpoints{}}}.
> Expose numberOfConsecutiveFailedCheckpoints as a JobManager-level gauge
> -----------------------------------------------------------------------
>
> Key: FLINK-40238
> URL: https://issues.apache.org/jira/browse/FLINK-40238
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing, Runtime / Metrics
> Reporter: Yang Li
> Priority: Minor
>
> h2. Proposal
> Add a new JobManager-level gauge alongside {{{}numberOfFailedCheckpoints{}}}:
> {noformat}
> numberOfConsecutiveFailedCheckpoints
> {noformat}
> The gauge mirrors {{{}CheckpointFailureManager{}}}'s internal counter:
> increments on each checkpoint failure counted toward
> {{{}tolerable-failed-checkpoints{}}}, resets on the next successful
> checkpoint.
> h2. Problem it solves
> The core problem this solves is letting users detect that a job is in a bad
> state via external monitoring, by observing consecutive checkpoint failures.
> This fires before the JobManager gives up, giving operators a window to
> intervene. The alert predicate is textually the same as the config value.
> h2. scope:
> {{{}DefaultCheckpointStatsTracker{}}}: register the new gauge and maintain
> the counter via the {{CheckpointStatsListener}} event path (increment on
> {{onFailedCheckpoint}} when the failure reason is counted, reset on
> {{{}onCompletedCheckpoint{}}}). No new config options, no public API or REST
> changes.
> Doc update in {{{}docs/content/docs/ops/metrics.md{}}}, next to{{{{}}
> numberOfFailedCheckpoints{}}}.
> We've implemented this on our internal Flink fork and have been running it in
> production. Filing this ticket to check whether the community would like a PR
> against master.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
