[
https://issues.apache.org/jira/browse/IGNITE-29055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitry Werner updated IGNITE-29055:
-----------------------------------
Description:
*Problem*
Since IGNITE-29047 every suppressed `SYSTEM_WORKER_BLOCKED` failure is counted
by the `failure.ignored.SYSTEM_WORKER_BLOCKED` metric (`Failure handling`
section in [Monitoring
Metrics](https://ignite.apache.org/docs/latest/monitoring-metrics/new-metrics)).
However, users monitoring blocked system-critical workers by grepping the log
for "Blocked system-critical thread has been detected" have no realistic way to
discover that an equivalent counter already exists: the metric name is
mentioned nowhere near the documentation of the blocked-worker detection itself
(Critical Workers Health Check in [Handling
Exceptions](https://ignite.apache.org/docs/latest/perf-and-troubleshooting/handling-exceptions)).
*Solution*
Documentation-only change: in the "Critical Workers Health Check" section of
`handling-exceptions.adoc`, right after the sentence describing the log
message, added a note that each detection is also counted by the
`failure.ignored.SYSTEM_WORKER_BLOCKED` metric, with a link to the Failure
Handling metrics section.
No code changes. The original metric-counting scope of this ticket is already
covered by IGNITE-29047 (see discussion); the ticket is narrowed to
cross-referencing the existing metric.
was:
Currently, the event "Blocked system-critical thread has been detected" is only
reported via the log by WorkersRegistry.
There is no way to track how often such events occur without parsing (grepping)
the log.
We need to expose this as a metric so the count of events can be monitored
through the metrics subsystem (e.g. JMX, system views, exporters) instead of
relying on log parsing.
> Document metric for blocked system workers
> ------------------------------------------
>
> Key: IGNITE-29055
> URL: https://issues.apache.org/jira/browse/IGNITE-29055
> Project: Ignite
> Issue Type: Task
> Reporter: Dmitry Werner
> Assignee: Dmitry Werner
> Priority: Major
> Labels: metrics-to-log
> Time Spent: 20m
> Remaining Estimate: 0h
>
> *Problem*
> Since IGNITE-29047 every suppressed `SYSTEM_WORKER_BLOCKED` failure is
> counted by the `failure.ignored.SYSTEM_WORKER_BLOCKED` metric (`Failure
> handling` section in [Monitoring
> Metrics](https://ignite.apache.org/docs/latest/monitoring-metrics/new-metrics)).
> However, users monitoring blocked system-critical workers by grepping the log
> for "Blocked system-critical thread has been detected" have no realistic way
> to discover that an equivalent counter already exists: the metric name is
> mentioned nowhere near the documentation of the blocked-worker detection
> itself (Critical Workers Health Check in [Handling
> Exceptions](https://ignite.apache.org/docs/latest/perf-and-troubleshooting/handling-exceptions)).
> *Solution*
> Documentation-only change: in the "Critical Workers Health Check" section of
> `handling-exceptions.adoc`, right after the sentence describing the log
> message, added a note that each detection is also counted by the
> `failure.ignored.SYSTEM_WORKER_BLOCKED` metric, with a link to the Failure
> Handling metrics section.
> No code changes. The original metric-counting scope of this ticket is already
> covered by IGNITE-29047 (see discussion); the ticket is narrowed to
> cross-referencing the existing metric.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
