Hi Neel :) I think the custom labels added in the alert rule is meant for referred by downstream apps like alertmanager. One way is to add this label in the scraping configuration for node_exporter itself (relabel_config:) and mention this label inside avg by (mode, instance, product)
--- Best Regards, *Charls P John* On Mon, Jun 19, 2023 at 7:47 PM Brian Candler <[email protected]> wrote: > I'm not sure, but I don't think that labels added *within* the alerting > rule are accessible to annotations within the same alerting rule. For for > "{{ $labels.product }}", you can simply replace it with the literal value, > i.e. "backend-app" > > On Monday, 19 June 2023 at 08:26:37 UTC NEEL THOMAS wrote: > >> I have a Prom CPU alert rule defined like below : >> >> - alert: HostHighCpuLoad expr: (sum by (instance) (avg by (mode, >> instance) (rate(node_cpu_seconds_total{mode!="idle"}[2m]))) > 0.8) * >> on(instance) group_left (nodename) node_uname_info{nodename=~".+"} for: 0m >> labels: severity: warning product : backend-app annotations: summary: Host >> high CPU load (instance {{ $labels.instance }}) description: "CPU load of >> {{ $labels.product }} is greater than > 80%" >> >> its triggering the alert, but the value of custom label product : >> backend-app is not interpolating in the alert description part, its coming >> like below >> >> "CPU load of is greater than > 80%" >> >> expected is >> >> "CPU load of backend-app is greater than > 80%" >> >> any idea why the custom label is not picking up, although the alert is >> getting triggered >> or how can we add custom alert labels that are accessible in annotation. >> >> thanks >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/86ca48d4-71be-46f9-9a1e-80342d94c212n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/86ca48d4-71be-46f9-9a1e-80342d94c212n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAJ7d%2BpSmuSMGrx4F93DZqjsJ_RwXDE1b2cqG4M5F5KVVX0aZAA%40mail.gmail.com.
