I spend quite some time to understand how to modify Title/Subject line for
alerts firing from Alertmanager to PagerDuty or slack and still have luck
in understanding.
Is there some thread in github with description for it, which I have not
found so far? I’d like to put more meaningful information into
“Description” field as it seems like it is being use by PD to show in the
WEB UI as Subject of the alert.
Any help here would be highly appreciated.
-------------------------------------------------------------------------
Extra Details:
*Alertmanager: *As I can see Alertmanager fires messages through generic
API call to PD in JSON like:
{ "client": "Alertmanager", "client_url": "URL HERE
***********************************", "contexts": [], "description":
"[FIRING:1] flux-system (AutomationReconciliationIssue lsv-prod manager
http-prom flux-system 10.233.82.36:8080 flux-system/flux-system
Kustomization p0 critical devops monitoring-system
kustomize-controller-7b4bfdd8b-n7s4c p0
monitoring-system/kps-kube-prometheus-stack-prometheus 2y critical False
pagerduty-devops Ready)", "event_type": "trigger", "incident_key":
"*******************************************", "service_key":
"*****************************", "details": { "firing": "Labels: -
alertname = AutomationReconciliationIssue - cluster = lsv-prod - container
= manager - endpoint = http-prom - exported_namespace = flux-system -
instance = 10.233.82.36:8080 - job = flux-system/flux-system - kind =
Kustomization - label_default_priority = p0 - label_default_severity =
critical - label_team = devops - name = monitoring-system - namespace =
flux-system - pod = kustomize-controller-7b4bfdd8b-n7s4c - priority = p0 -
prometheus = monitoring-system/kps-kube-prometheus-stack-prometheus -
retention = 2y - severity = critical - status = False - target =
pagerduty-devops - type = ReadyAnnotations: - description = Kustomization
monitoring-system in flux-system is in False status - summary =
Kustomization monitoring-system in flux-system is in False statusSource:
<URL here ******************>", "num_firing": "1", "num_resolved":
"0", "resolved": "" }}
*Slack: *The same issue is valid for Slack alerts.
I have done configuration for alerts on receivers definition as follow:
- name: slack-internal
slackConfigs:
- apiURL:
key: url
name: slack-url-internal
channel: '#pgmt-k8s-alerts-internal'
username: lsv-prod
sendResolved: true
text: |-
{{ range .Alerts.Firing }}
*{{ .Labels.severity | toUpper }}* - `{{ .Labels.alertname
}}`: {{ .Annotations.summary }}{{ .Annotations.message }}
{{ .Annotations.description -}}
{{ if .Annotations.runbook_url }}{{ .Annotations.runbook_url
}}{{ end }}
*Additional Details:*
{{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value
}}`
{{ end }}
{{ end }}
And alerts looks like:
*[FIRING:5] app-prod (AutomationReconciliationIssue lsv-eng-prod manager
http-prom app-prod 10.233.111.140:8080 flux-system/flux-system HelmRelease
p0 critical si helm-controller-7868945f7f-5slwc p0
monitoring-system/kps-kube-prometheus-stack-prometheus 2y critical False
slack-internal Ready)
<http://URL.net/#/alerts?receiver=monitoring-system/global-alertmanager-configuration/slack-internal>*
*CRITICAL* - AutomationReconciliationIssue: HelmRelease app-ai in app-prod
is in False status HelmRelease app-ai in app-prod is in False status
*Additional Details:*
• *alertname:* AutomationReconciliationIssue
• *cluster:* lsv-prod
• *container:* manager
• *endpoint:* http-prom
• *exported_namespace:* app-prod
• *instance:* 10.233.111.140:8080
• *job:* flux-system/flux-system
• *kind:* HelmRelease
• *label_default_priority:* p0
• *label_default_severity:* critical
• *label_team:* si
• *name:* app-ai
• *namespace:* app-prod
• *pod:* helm-controller-7868945f7f-5slwc
• *priority:* p0
• *prometheus:* monitoring-system/kps-kube-prometheus-stack-prometheus
• *retention:* 2y
• *severity:* critical
• *status:* False
• *target:* slack-internal
• *type:* Ready
Where first line of the message is generated automatically and does not
have inheritance from the receiver configuration (as I can see).
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/ad5e21ba-02e9-486a-947f-8d93d73690b1n%40googlegroups.com.
