Siyao Meng created HDDS-17264:
---------------------------------
Summary: Add Grafana dashboard and Prometheus alert rules for
datanode volume health
Key: HDDS-17264
URL: https://issues.apache.org/jira/browse/HDDS-17264
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Dashboards
Reporter: Siyao Meng
The DataNode Overview dashboard (HDDS-15324) sums volume I/O metrics per host,
which hides a single slow or failing disk. No alert rules ship with the compose
monitoring setup.
Proposal:
* Add an "Ozone - Datanode Volume Health" dashboard with:
** failed volumes per datanode, by volume type
** disk I/O errors per volume ({{NumDataPathIOErrors}}, from HDDS-17261)
** per-volume read/write p99 and average latency
** each volume's p99 relative to the median of the volumes on the same datanode
** per-volume read/write operation rates
* Add {{compose/ozone/prometheus-rules.yml}}, loaded by the monitoring compose
setup, with alerts for:
** any failed volume
** more than 3 disk I/O errors on a volume within an hour
** a volume whose read or write p99 stays above 100 ms and 5x the datanode
median for 15 minutes
Validated with {{promtool check rules}} and {{promtool test rules}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
