[
https://issues.apache.org/jira/browse/IGNITE-29059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitry Werner updated IGNITE-29059:
-----------------------------------
Description:
Currently the "Possible starvation in striped pool" event is only reported via
the log by IgniteStripedExecutor, and there is no way to track how often such
events occur without grepping the log.
This adds a single StarvationEventsCount long metric to striped executors
(StripedExecutor, GridDataStreamExecutor) so the count of starvation episodes
can be monitored through the metrics subsystem (JMX, system views, exporters)
instead of log parsing.
The counter is updated by the periodic starvation check
(IGNITE_STARVATION_CHECK_INTERVAL, 30 s by default). A task occupying a stripe
longer than failureDetectionTimeout is counted at most once per episode, and
only if a check finds it still running past that timeout. Deduplication is done
on the detector side via a per-stripe countedStartTs CAS, with no writes on the
task hot path.
The data streamer striped pool is now also registered for periodic starvation
detection, so its StarvationEventsCount is updated without reading
DetectStarvation. As a side effect, a hung data-streamer stripe will
periodically print the existing "Possible starvation in striped pool" WARN with
a stack trace — no new kind of log noise (the "Blocked system-critical thread"
message is already written for the same stripe).
Also fixes a pre-existing volatile write order in Stripe.run() (lastStartedTs
before active = true) that could produce a false starvation WARN with a stale
start timestamp.
was:
Currently, the event "Possible starvation in striped pool" is only reported via
the log by IgniteStripedExecutor.
There is no way to track how often such events occur without parsing (grepping)
the log.
We need to expose this as a metric so the count of events can be monitored
through the metrics subsystem (e.g. JMX, system views, exporters) instead of
relying on log parsing.
> Add a metric for possible starvation in striped pool to IgniteStripedExecutor
> -----------------------------------------------------------------------------
>
> Key: IGNITE-29059
> URL: https://issues.apache.org/jira/browse/IGNITE-29059
> Project: Ignite
> Issue Type: Task
> Reporter: Dmitry Werner
> Assignee: Dmitry Werner
> Priority: Major
> Labels: metrics-to-log
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently the "Possible starvation in striped pool" event is only reported
> via the log by IgniteStripedExecutor, and there is no way to track how often
> such events occur without grepping the log.
> This adds a single StarvationEventsCount long metric to striped executors
> (StripedExecutor, GridDataStreamExecutor) so the count of starvation episodes
> can be monitored through the metrics subsystem (JMX, system views, exporters)
> instead of log parsing.
> The counter is updated by the periodic starvation check
> (IGNITE_STARVATION_CHECK_INTERVAL, 30 s by default). A task occupying a
> stripe longer than failureDetectionTimeout is counted at most once per
> episode, and only if a check finds it still running past that timeout.
> Deduplication is done on the detector side via a per-stripe countedStartTs
> CAS, with no writes on the task hot path.
> The data streamer striped pool is now also registered for periodic starvation
> detection, so its StarvationEventsCount is updated without reading
> DetectStarvation. As a side effect, a hung data-streamer stripe will
> periodically print the existing "Possible starvation in striped pool" WARN
> with a stack trace — no new kind of log noise (the "Blocked system-critical
> thread" message is already written for the same stripe).
> Also fixes a pre-existing volatile write order in Stripe.run() (lastStartedTs
> before active = true) that could produce a false starvation WARN with a stale
> start timestamp.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
