Brian
Let me answer you in bullet points:
1. I have tried the expression with both, ie. > 1 and also > 100. Both
don't fire.
2. I am getting other Alerts through Alertmanager for example, UP/down of
instance. So its not the Alertmanager.
Expression shows non-empty results in PromQL Query interface but still it
doesn't fire.
J
On Thu, Aug 22, 2024 at 10:20 AM 'Brian Candler' via Prometheus Users <
prometheus-users@googlegroups.com> wrote:
> Your test example in PromQL browser has:
> confluent_kafka_server_consumer_lag_offsets{job="confluent-cloud"} > 1
> and the values were 2 or 3; but the alerting expression has
> confluent_kafka_server_consumer_lag_offsets{job="confluent-cloud"} > 100
> So clearly it's not going to trigger under that condition, when the lags
> are less than 100.
>
> If that's not the probelm, then you need to determine: is the rule not
> firing? Or is Alertmanager not sending an alert?
>
> To do this, check in the Prometheus web interface under the Alerts tab. Is
> there a firing alert there? If yes, then you focus your investigation on
> the alertmanager side (e.g. check alertmanager logs). If no, then drill
> further into the expression, although if the same expression shows a
> non-empty result in the PromQL query interface, then it certainly should be
> able to fire an alert.
>
> On Wednesday 21 August 2024 at 21:24:58 UTC+1 Jay wrote:
>
>> Here is the text:
>>
>> groups:
>> - name: confluent-rules
>> rules:
>>
>> - alert: Dev-NotEqualtoBoolZero
>> expr:
>> confluent_kafka_server_consumer_lag_offsets{job="confluent-cloud"}
>> > 100
>> labels:
>> severity: critical
>> annotations:
>> description: "The consumer lags for Dev client`"
>> On Wed, Aug 21, 2024 at 1:51 PM Daz Wilkin <daz.w...@gmail.com> wrote:
>>
>>> Please include the rule.
>>>
>>> You've shown that the query returns results which is necessary but
>>> insufficient.
>>>
>>> On Wednesday, August 21, 2024 at 8:19:34 AM UTC-7 Jay P wrote:
>>>
>>>> I am not new to Prometheus however, I wrote the following rules which
>>>> never fires. (Alertmanger and all other settings are fine since I get
>>>> Alarms for other rules except this one)
>>>>
>>>> Attached here it he screenshot and i am copy pasting here as well.
>>>>
>>>> confluent_kafka_server_consumer_lag_offsets{job="confluent-cloud"} > 1
>>>>
>>>> Results:
>>>> confluent_kafka_server_consumer_lag_offsets{consumer_group_id="XXX",
>>>> instance="api.telemetry.confluent.cloud:443", job="confluent-cloud",
>>>> kafka_id="XXX", topic="XXX"}
>>>> 2
>>>> confluent_kafka_server_consumer_lag_offsets{consumer_group_id="XXX",
>>>> instance="api.telemetry.confluent.cloud:443", job="confluent-cloud",
>>>> kafka_id="XXX", topic="XXX"}
>>>> 3
>>>>
>>>> Any help is greatly appreciated. Thank you
>>>>
>>> --
>>>
>> You received this message because you are subscribed to a topic in the
>>> Google Groups "Prometheus Users" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/prometheus-users/pBEqCDIUFug/unsubscribe
>>> .
>>> To unsubscribe from this group and all its topics, send an email to
>>> prometheus-use...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/e8007886-5a85-4fdb-938e-36373409498cn%40googlegroups.com
>>> <https://groups.google.com/d/msgid/prometheus-users/e8007886-5a85-4fdb-938e-36373409498cn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Prometheus Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/prometheus-users/pBEqCDIUFug/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/8be525b8-e216-4cc4-8f05-c126bf42fc35n%40googlegroups.com
> <https://groups.google.com/d/msgid/prometheus-users/8be525b8-e216-4cc4-8f05-c126bf42fc35n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/CAN3oSLTuLZFHwey3EmkQOQWoKnbgKZn%3D9_cpNAj5C-yQNhWNkQ%40mail.gmail.com.
