Hi Julius - thanks for replying! Yes, I'm using the latest 47.3.0 version of kube-prometheus-stack.
This appears to ship a set of Prometheus rules which includes KubeJobFailed: https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kubernetes-apps.yaml#L404C24-L428 The severity of the alert is set to "warning" but I want to override that severity and set it to "info" by adding a configuration in the top-level values.yaml, maybe under the alertmanagerSpec: section. However I searched everywhere and couldn't find a method of modifying just the severity level of a named alert without changing any other parameters. While the example here is KubeJobFailed this could apply to any alert we need to override. Dave On Mon, 3 Jul 2023 at 13:56, Julius Volz <[email protected]> wrote: > Hi David, > > Prometheus does not have built-in alerts, but I assume you are using some > framework around Prometheus that configures a "KubeJobFailed" alert? Is it > kube-prometheus with the kubernetes-mixin (since that includes an example > alert by that name: > https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/003ba5eadfbd69817d1215952133d3ecf99fbd92/alerts/apps_alerts.libsonnet#L268-L281)? > If you share more about your kube-prometheus or kubernetes-mixin jsonnet > pipeline, I'm sure someone could help. Although at that point it's probably > more of a jsonnet question than a Prometheus one. > > Kind regards, > Julius > > On Mon, Jul 3, 2023 at 9:11 AM David Dean <[email protected]> wrote: > >> Hi - >> >> How can I modify the severity of built in alert KubeJobFailed from >> warning to info? >> >> I don't want to change anything else about the built in alert, only the >> severity. >> >> And I only want to change the one alert, KubeJobFailed, not any other. >> >> I need to change the severity to info so PagerDuty will not generate >> alerts outside working hours. In our scenario, Kubernetes Jobs don't need >> an immediate callout. >> >> I've spent ages Googling and checking the docs but cannot find out how to >> do it anywhere! >> >> Thanks, Dave >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/2c6284f5-35a6-43d9-b892-41803d8bd4d9n%40googlegroups.com >> <https://groups.google.com/d/msgid/prometheus-users/2c6284f5-35a6-43d9-b892-41803d8bd4d9n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Julius Volz > PromLabs - promlabs.com > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAHtOzUfJbzLGQ9X8zZDzOB9KFPwOe%3D5bLCbEzYDDSYk031mBUA%40mail.gmail.com.
