If we're talking about the same "calert <https://github.com/mr-karan/calert>" then it looks like it already supports user-supplied templating of the message content: https://github.com/mr-karan/calert/blob/main/internal/providers/google_chat/message.go#L18-L53
The sample config shows how to configure it: https://github.com/mr-karan/calert/blob/main/config.sample.toml#L13 https://github.com/mr-karan/calert/blob/main/static/message.tmpl If you want to do colours and highlighting, it would depend whether the Google Chat API accepts HTML tags, or some other way of marking up the content. You'll need to find that out. The variables available in the templates look to be very similar to those provided in Alertmanager templates: https://prometheus.io/docs/alerting/latest/notifications/ https://prometheus.io/docs/alerting/latest/notification_examples/ In fact, there's a test case which uses exactly the alertmanager code (an instance of alertmgrtmpl.Alert) to drive it: https://github.com/mr-karan/calert/blob/main/internal/providers/google_chat/google_chat_test.go#L27-L35 On Monday, 12 June 2023 at 11:33:32 UTC+1 Near N wrote: > Brian, thanks for pointing to other integrations. The reason why I was > wondering about curling the messages is that the existing solution (Calert) > provides little information about templating options. Alerts are incoming, > but with tight information. I'd love to highlight severity in different > colors, add warning icons, e.g. something similar to Zabbix or at least > that can suit basic user-friendly message. > > понедельник, 12 июня 2023 г. в 16:06:34 UTC+6, Brian Candler: > >> I can't speak for the project, but my guess would be it's out of scope. >> There are many existing integrations which use the existing webhook >> mechanism: >> >> https://prometheus.io/docs/operating/integrations/#alertmanager-webhook-receiver >> Doing it externally allows the request to be enriched with other data >> which wouldn't be available in a template, and to use a variety of outbound >> authentication mechanisms which Alertmanager itself doesn't implement, or >> even non-HTTP protocols like STOMP. >> >> In any case, text templating is a nasty and insecure way to create JSON >> or XML payloads. >> >> > it is possible to converse the receivable JSON on the hop and just send >> notifications via curl method >> >> You'd first have to *receive* the webhook payload from Alertmanager, >> which involves plugging your curl script into a webserver somehow. You >> could probably write one as a CGI script under Apache, for example. >> >> But why would you do that, when you have an existing tested and working >> solution? >> >> On Monday, 12 June 2023 at 08:13:50 UTC+1 Near N wrote: >> >>> Thanks for reply Brian. I do realize, that Google Chat API requires a >>> particular JSON format, but was assuming Alertmanager is capable of doing >>> it by using particular template, which is wrong. Now the questions is >>> whether it is worth creating a PR to create such a feature or it is out of >>> Alertmanager's standard scope? >>> >>> Another question is perhaps it is possible to converse the receivable >>> JSON on the hop and just send notifications via curl method? I mean if the >>> endpoint contains the following alert " >>> [{"annotations":{"server":"main","number_of_nodes":"1"},"endsAt":"2023-06-12T07:02:00.233Z","fingerprint":"b21bf244f87cd47f","receivers":[{"name":"my_alerts"}],"startsAt":"2023-06-12T06:44:00.233Z","status":{"inhibitedBy":[],"silencedBy":[],"state":"active"},"updatedAt":"2023-06-12T06:58:00.424Z","generatorURL":" >>> http://app1:8880/vmalert/alert?group_id=71689142540802366","labels":{"alertgroup":"INFRA","alertname":"The >>> >>> number of available Cassandra nodes has >>> changed","server":"main","severity":"Critical"}}]", would it be possible to >>> retrieve this json and post it via curl? This, I assume, wouldn't solve my >>> problem, cause I believe Google API expects something peculiar in terms of >>> the message it receives. >>> >>> понедельник, 12 июня 2023 г. в 13:04:17 UTC+6, Brian Candler: >>> >>>> > Invalid JSON payload received. >>>> >>>> Alertmanager's webhook sends a JSON message with a *fixed* payload >>>> format. See >>>> https://prometheus.io/docs/alerting/latest/configuration/#webhook_config >>>> *"The Alertmanager will send HTTP POST requests in the following JSON >>>> format to the configured endpoint"* >>>> >>>> You can't override it with a template. The Alertmanager JSON format is >>>> different to what Google Chat requires, and therefore you can't send it >>>> directly. >>>> >>>> That's why you need a separate piece of software to do the conversion >>>> (Calert in your case) - it receives the Alertmanager webhook JSON payload, >>>> builds a new payload for the target, and then sends it there. >>>> >>>> On Monday, 12 June 2023 at 06:59:14 UTC+1 Near N wrote: >>>> >>>>> Good day. I was wondering if there is a way to configure alertmanager >>>>> to send notifications to google chat via a webhook. >>>>> >>>>> The current setup that I use consists of vmalert (loaded with set of >>>>> promQL compatible rules) >> Alertmanager (to group them up) and Calert >>>>> (to >>>>> send them to google chat). Despite the flowchart is working fine, I found >>>>> it difficult to amend proposed message template, so decided to simplify >>>>> it >>>>> by configuring alertmanager to send the alerts. It dawns on me that it >>>>> can >>>>> operate, but I seem to have failed to fine-tune it. Any help would be >>>>> appreciated. The alertmanager.yml is below: >>>>> >>>>> global: >>>>> http_config: >>>>> proxy_url: '{{ alertmanager_proxy_url }}' >>>>> >>>>> route: >>>>> receiver: 'my_alerts' >>>>> group_wait: 30s >>>>> group_interval: 60s >>>>> repeat_interval: 15m >>>>> group_by: ['room', 'alertName'] >>>>> >>>>> receivers: >>>>> - name: 'my_alerts' >>>>> webhook_configs: >>>>> - url: '{{ alertmanager_chat_webhook }}' >>>>> send_resolved: true >>>>> >>>>> templates: >>>>> - '{{ alertmanager_working_directory }}/google-chat.tmpl' >>>>> >>>>> >>>>> And here is the error of alertmanager trying to send an alert: >>>>> >>>>> Jun 12 05:26:00 app1-cp alertmanager[140337]: >>>>> ts=2023-06-12T05:26:00.599Z caller=dispatch.go:352 level=error >>>>> component=dispatcher msg="Notify for alerts failed" num_alerts=3 >>>>> err="my_alerts/webhook[0]: notify retry canceled due to unrecoverable >>>>> error >>>>> after 1 attempts: unexpected status code 400: >>>>> https://chat.googleapis.com/v1/spaces/webook_token: {\n \"error\": >>>>> {\n \"code\": 400,\n \"message\": \"Invalid JSON payload received. >>>>> Unknown name \\\"receiver\\\" at 'message': Cannot find field.\\nInvalid >>>>> JSON payload received. Unknown name \\\"status\\\" at 'message': Cannot >>>>> find field.\\nInvalid JSON payload received. Unknown name \\\"alerts\\\" >>>>> at >>>>> 'message': Cannot find field.\\nInvalid JSON payload received. Unknown >>>>> name >>>>> \\\"groupLabels\\\" at 'message': Cannot find field.\\nInvalid JSON >>>>> payload >>>>> received. Unknown name \\\"commonLabels\\\" at 'message': Cannot find >>>>> field.\\nInvalid JSON payload received. Unknown name >>>>> \\\"commonAnnotations\\\" at 'message': Cannot find field.\\nInvalid JSON >>>>> payload received. Unknown name \\\"externalURL\\\" at 'message': Cannot >>>>> find field.\\nInvalid JSON payload received. Unknown name \\\"version\\\" >>>>> at 'message': Cannot find field.\\nInvalid JSON payload received. Unknown >>>>> name \\\"groupKey\\\" at 'message': Cannot find field.\\nInvalid JSON >>>>> payload received. Unknown name \\\"truncatedAlerts\\\" at 'message': >>>>> Cannot >>>>> find field.\",\n \"status\": \"INVALID_ARGUMENT\",\n \"details\": >>>>> [\n >>>>> {\n \"@type\": \" >>>>> type.googleapis.com/google.rpc.BadRequest\ >>>>> <http://type.googleapis.com/google.rpc.BadRequest%5C>",\n >>>>> \"fieldViolations\": [\n {\n \"field\": >>>>> \"message\",\n >>>>> \"description\": \"Invalid JSON payload received. Unknown name >>>>> \\\"receiver\\\" at 'message': Cannot find field.\"\n },\n >>>>> >>>>> {\n \"field\": \"message\",\n \"description\": >>>>> \"Invalid JSON payload received. Unknown name \\\"status\\\" at >>>>> 'message': >>>>> Cannot find field.\"\n },\n {\n \"field\": >>>>> \"message\",\n \"description\": \"Invalid JSON payload >>>>> received. >>>>> Unknown name \\\"alerts\\\" at 'message': Cannot find field.\"\n >>>>> },\n {\n \"field\": \"message\",\n >>>>> \"description\": \"Invalid JSON payload received. Unknown name >>>>> \\\"groupLabels\\\" at 'message': Cannot find field.\"\n },\n >>>>> >>>>> {\n \"field\": \"message\",\n \"description\": >>>>> \"Invalid JSON payload received. Unknown name \\\"commonLabels\\\" at >>>>> 'message': Cannot find field.\"\n },\n {\n >>>>> \"field\": \"message\",\n \"description\": \"Invalid JSON >>>>> payload received. Unknown name \\\"commonAnnotations\\\" at 'message': >>>>> Cannot find field.\"\n },\n {\n \"field\": >>>>> \"message\",\n \"description\": \"Invalid JSON payload >>>>> received. >>>>> Unknown name \\\"externalURL\\\" at 'message': Cannot find field.\"\n >>>>> >>>>> },\n {\n \"field\": \"message\",\n >>>>> \"description\": \"Invalid JSON payload received. Unknown name >>>>> \\\"version\\\" at 'message': Cannot find field.\"\n },\n >>>>> >>>>> {\n \"field\": \"message\",\n \"description\": >>>>> \"Invalid JSON payload received. Unknown name \\\"groupKey\\\" at >>>>> 'message': Cannot find field.\"\n },\n {\n >>>>> \"field\": \"message\",\n \"description\": \"Invalid JSON >>>>> payload received. Unknown name \\\"truncatedAlerts\\\" at 'message': >>>>> Cannot >>>>> find field.\"\n }\n ]\n }\n ]\n }\n}\n" >>>>> >>>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9aedd105-5284-4790-882f-11f60199aa92n%40googlegroups.com.
