Hello I am setting up solr monitoring using Prometheus, solr-exporter and Grafana. I also went ahead and installed alertmanager. Now my collection clusterstatus looks as below:
curl -u "solradmin:xxxxxxxxxxxxx" 'http://host01:8940/solr/admin/collections?action=CLUSTERSTATUS&wt=json&indent=true' { "responseHeader":{ "status":0, "QTime":1 }, "cluster":{ "collections":{ "test_coll1":{ "pullReplicas":0, "configName":"test_coll1", "replicationFactor":1, "router":{ "name":"compositeId" }, "nrtReplicas":1, "tlogReplicas":0, "shards":{ "shard1":{ "range":"80000000-7fffffff", "state":"active", "stateTimestamp":"1733764681612235673", "replicas":{ "core_node2":{ "core":"test_coll1_shard1_replica_n1", "node_name":"host01:8940_solr", "type":"NRT", "state":"active", "leader":"true", "force_set_state":"false", "base_url":http://host01:8940/solr } }, "health":"GREEN" } }, "health":"GREEN", "znodeVersion":39, "creationTimeMillis":1733764591523 }, "test_coll2":{ "pullReplicas":0, "configName":"test_coll2", "replicationFactor":1, "router":{ "name":"compositeId" }, "nrtReplicas":1, "tlogReplicas":0, "shards":{ "shard1":{ "range":"80000000-7fffffff", "state":"active", "stateTimestamp":"1733765402371289174", "replicas":{ "core_node2":{ "core":"test_coll2_shard1_replica_n1", "node_name":"host01:8940_solr", "type":"NRT", "state":"active", "leader":"true", "force_set_state":"false", "base_url":http://host01:8940/solr } }, "health":"GREEN" } }, "health":"GREEN", "znodeVersion":39, "creationTimeMillis":1733765317706 } }, "live_nodes":["host01:8940_solr"] } } I am looking to get alerted when the state of core changes. As per https://solr.apache.org/guide/solr/latest/deployment-guide/cluster-node-management.html#clusterstatus I can have 4 states i.e. red, orange, yellow and green. I am looking to setup an alert for the same either through Grafana or alertmanager to be emailed when either of these states happen. I have the below entry in rules.yml in Prometheus: # Alert for collection core status solr_collections_shard_state - alert: SolrCoreDown # Condition for alerting expr: solr_collections_shard_state < 1.00 for: 1m # Annotation - additional informational labels to store more information annotations: summary: "Solr shard {{ $labels.collection }}-{{ $labels.shard }} is down" description: "The Solr shard {{ $labels.collection }}-{{ $labels.shard }} has been in a non-active state for 1 minutes." I don't get alerted with the above rule. I assume that it would be the same expression or query that can be used in Grafana as well. I would appreciate guidance with writing the rule/query above either in alertmanager or Grafana to get the desired alerts. Thanks
