Hi Brian,
I thought this issue will not be faced again however we witnessed this
alert again on Saturday 09:12:18 am UTC so requesting your guideance.
Alert configuration is as below:
admin@orchestrator[nd2bwa6drm01v]# show running-config alert rule
PROCESS_STATE
alert rule PROCESS_STATE
expression "docker_service_up==1 or docker_service_up==3"
event-host-label container_name
message "{{ $labels.service_name }} instance {{
$labels.module_instance }} of module {{ $labels.module }} is in Aborted
state !"
snmp-facility application
snmp-severity critical
snmp-clear-message "{{ $labels.service_name }} instance {{
$labels.module_instance }} of module {{ $labels.module }} is moved from
Aborted state !"
!
admin@orchestrator[nd2bwa6drm01v]#
Recent alert details:
NAME EVENT HOST STATUS MESSAGE
CREATE TIME RESOLVE TIME UPDATE
TIME
PROCESS_STATE haproxy-common-s101 resolved haproxy-common instance 101 of
module haproxy-common is moved from Aborted state !
2020-05-30T09:12:18.643+00:00 2020-05-30T09:12:33.617+00:00
2020-05-30T09:27:38.659+00:00
PROCESS_STATE haproxy-common-s103 resolved haproxy-common instance 103 of
module haproxy-common is moved from Aborted state !
2020-05-30T09:12:18.644+00:00 2020-05-30T09:12:33.619+00:00
2020-05-30T09:27:38.66+00:00
Per your last suggestion, I have also verified below output but it does not
indicate 'docker_service_up' metric set to either 1 or 3 (for which alert
is configured)
"curl '
http://localhost:9090/api/v1/query_range?query=docker_service_up&start=2020-05-24T07:20:00.000Z&end=2020-05-24T08:10:00.000Z&step=1s'
> docker_service_up.log"
Please let me know if you have any comment/opinion.
On Thursday, May 28, 2020 at 11:22:21 PM UTC-7, Brian Candler wrote:
>
> Was this an alert generated by prometheus' alertmanager, or an alert
> generated by grafana's alerting system?
>
> You said alert was resolved "in 5 seconds" which sounds dubious. Maybe
> you have some extremely low interval configured for your alerting rules in
> prometheus?
>
> Nonetheless, the history is all in prometheus (at least for the TSDB
> retention period - default 15 days). You need to work out what expression
> generated the alert, and use PromQL to explore the data in prometheus.
>
> That's all we can say, unless you show the content of the alert itself
> *and* the rule which you believe generated the alert *and* the data which
> backs up your assertion that there was no triggering data in that period.
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/bc97a803-fb5a-437f-aea3-83c02374280b%40googlegroups.com.
