> It is basically possible by converting the message in a wildcard structure acceptable by Google Chat API
I don't know what you mean by a "wildcard structure". Can you give an example, and/or point to documentation? Is it using tags within the message field, or does it require a different JSON structure entirely? On Wednesday, 14 June 2023 at 05:12:53 UTC+1 Near N wrote: > Correct, that's same calert I was mentioning previously. I understand the > use case of default template, but thought it is possible to do highlighting > within the template. It is basically possible by converting the message in > a wildcard structure acceptable by Google Chat API, but I couldn't find any > options to do so with calert template, but thanks for suggestions > > понедельник, 12 июня 2023 г. в 19:43:10 UTC+6, Brian Candler: > >> If we're talking about the same "calert >> <https://github.com/mr-karan/calert>" then it looks like it already >> supports user-supplied templating of the message content: >> >> https://github.com/mr-karan/calert/blob/main/internal/providers/google_chat/message.go#L18-L53 >> >> The sample config shows how to configure it: >> https://github.com/mr-karan/calert/blob/main/config.sample.toml#L13 >> https://github.com/mr-karan/calert/blob/main/static/message.tmpl >> >> If you want to do colours and highlighting, it would depend whether the >> Google Chat API accepts HTML tags, or some other way of marking up the >> content. You'll need to find that out. >> >> The variables available in the templates look to be very similar to those >> provided in Alertmanager templates: >> https://prometheus.io/docs/alerting/latest/notifications/ >> https://prometheus.io/docs/alerting/latest/notification_examples/ >> >> In fact, there's a test case which uses exactly the alertmanager code (an >> instance of alertmgrtmpl.Alert) to drive it: >> >> https://github.com/mr-karan/calert/blob/main/internal/providers/google_chat/google_chat_test.go#L27-L35 >> >> On Monday, 12 June 2023 at 11:33:32 UTC+1 Near N wrote: >> >>> Brian, thanks for pointing to other integrations. The reason why I was >>> wondering about curling the messages is that the existing solution (Calert) >>> provides little information about templating options. Alerts are incoming, >>> but with tight information. I'd love to highlight severity in different >>> colors, add warning icons, e.g. something similar to Zabbix or at least >>> that can suit basic user-friendly message. >>> >>> понедельник, 12 июня 2023 г. в 16:06:34 UTC+6, Brian Candler: >>> >>>> I can't speak for the project, but my guess would be it's out of scope. >>>> There are many existing integrations which use the existing webhook >>>> mechanism: >>>> >>>> https://prometheus.io/docs/operating/integrations/#alertmanager-webhook-receiver >>>> Doing it externally allows the request to be enriched with other data >>>> which wouldn't be available in a template, and to use a variety of >>>> outbound >>>> authentication mechanisms which Alertmanager itself doesn't implement, or >>>> even non-HTTP protocols like STOMP. >>>> >>>> In any case, text templating is a nasty and insecure way to create JSON >>>> or XML payloads. >>>> >>>> > it is possible to converse the receivable JSON on the hop and just >>>> send notifications via curl method >>>> >>>> You'd first have to *receive* the webhook payload from Alertmanager, >>>> which involves plugging your curl script into a webserver somehow. You >>>> could probably write one as a CGI script under Apache, for example. >>>> >>>> But why would you do that, when you have an existing tested and working >>>> solution? >>>> >>>> On Monday, 12 June 2023 at 08:13:50 UTC+1 Near N wrote: >>>> >>>>> Thanks for reply Brian. I do realize, that Google Chat API requires a >>>>> particular JSON format, but was assuming Alertmanager is capable of doing >>>>> it by using particular template, which is wrong. Now the questions is >>>>> whether it is worth creating a PR to create such a feature or it is out >>>>> of >>>>> Alertmanager's standard scope? >>>>> >>>>> Another question is perhaps it is possible to converse the receivable >>>>> JSON on the hop and just send notifications via curl method? I mean if >>>>> the >>>>> endpoint contains the following alert " >>>>> [{"annotations":{"server":"main","number_of_nodes":"1"},"endsAt":"2023-06-12T07:02:00.233Z","fingerprint":"b21bf244f87cd47f","receivers":[{"name":"my_alerts"}],"startsAt":"2023-06-12T06:44:00.233Z","status":{"inhibitedBy":[],"silencedBy":[],"state":"active"},"updatedAt":"2023-06-12T06:58:00.424Z","generatorURL":" >>>>> http://app1:8880/vmalert/alert?group_id=71689142540802366","labels":{"alertgroup":"INFRA","alertname":"The >>>>> >>>>> number of available Cassandra nodes has >>>>> changed","server":"main","severity":"Critical"}}]", would it be possible >>>>> to >>>>> retrieve this json and post it via curl? This, I assume, wouldn't solve >>>>> my >>>>> problem, cause I believe Google API expects something peculiar in terms >>>>> of >>>>> the message it receives. >>>>> >>>>> понедельник, 12 июня 2023 г. в 13:04:17 UTC+6, Brian Candler: >>>>> >>>>>> > Invalid JSON payload received. >>>>>> >>>>>> Alertmanager's webhook sends a JSON message with a *fixed* payload >>>>>> format. See >>>>>> >>>>>> https://prometheus.io/docs/alerting/latest/configuration/#webhook_config >>>>>> *"The Alertmanager will send HTTP POST requests in the following JSON >>>>>> format to the configured endpoint"* >>>>>> >>>>>> You can't override it with a template. The Alertmanager JSON format >>>>>> is different to what Google Chat requires, and therefore you can't send >>>>>> it >>>>>> directly. >>>>>> >>>>>> That's why you need a separate piece of software to do the conversion >>>>>> (Calert in your case) - it receives the Alertmanager webhook JSON >>>>>> payload, >>>>>> builds a new payload for the target, and then sends it there. >>>>>> >>>>>> On Monday, 12 June 2023 at 06:59:14 UTC+1 Near N wrote: >>>>>> >>>>>>> Good day. I was wondering if there is a way to configure >>>>>>> alertmanager to send notifications to google chat via a webhook. >>>>>>> >>>>>>> The current setup that I use consists of vmalert (loaded with set of >>>>>>> promQL compatible rules) >> Alertmanager (to group them up) and Calert >>>>>>> (to >>>>>>> send them to google chat). Despite the flowchart is working fine, I >>>>>>> found >>>>>>> it difficult to amend proposed message template, so decided to simplify >>>>>>> it >>>>>>> by configuring alertmanager to send the alerts. It dawns on me that it >>>>>>> can >>>>>>> operate, but I seem to have failed to fine-tune it. Any help would be >>>>>>> appreciated. The alertmanager.yml is below: >>>>>>> >>>>>>> global: >>>>>>> http_config: >>>>>>> proxy_url: '{{ alertmanager_proxy_url }}' >>>>>>> >>>>>>> route: >>>>>>> receiver: 'my_alerts' >>>>>>> group_wait: 30s >>>>>>> group_interval: 60s >>>>>>> repeat_interval: 15m >>>>>>> group_by: ['room', 'alertName'] >>>>>>> >>>>>>> receivers: >>>>>>> - name: 'my_alerts' >>>>>>> webhook_configs: >>>>>>> - url: '{{ alertmanager_chat_webhook }}' >>>>>>> send_resolved: true >>>>>>> >>>>>>> templates: >>>>>>> - '{{ alertmanager_working_directory }}/google-chat.tmpl' >>>>>>> >>>>>>> >>>>>>> And here is the error of alertmanager trying to send an alert: >>>>>>> >>>>>>> Jun 12 05:26:00 app1-cp alertmanager[140337]: >>>>>>> ts=2023-06-12T05:26:00.599Z caller=dispatch.go:352 level=error >>>>>>> component=dispatcher msg="Notify for alerts failed" num_alerts=3 >>>>>>> err="my_alerts/webhook[0]: notify retry canceled due to unrecoverable >>>>>>> error >>>>>>> after 1 attempts: unexpected status code 400: >>>>>>> https://chat.googleapis.com/v1/spaces/webook_token: {\n \"error\": >>>>>>> {\n \"code\": 400,\n \"message\": \"Invalid JSON payload >>>>>>> received. >>>>>>> Unknown name \\\"receiver\\\" at 'message': Cannot find >>>>>>> field.\\nInvalid >>>>>>> JSON payload received. Unknown name \\\"status\\\" at 'message': Cannot >>>>>>> find field.\\nInvalid JSON payload received. Unknown name >>>>>>> \\\"alerts\\\" at >>>>>>> 'message': Cannot find field.\\nInvalid JSON payload received. Unknown >>>>>>> name >>>>>>> \\\"groupLabels\\\" at 'message': Cannot find field.\\nInvalid JSON >>>>>>> payload >>>>>>> received. Unknown name \\\"commonLabels\\\" at 'message': Cannot find >>>>>>> field.\\nInvalid JSON payload received. Unknown name >>>>>>> \\\"commonAnnotations\\\" at 'message': Cannot find field.\\nInvalid >>>>>>> JSON >>>>>>> payload received. Unknown name \\\"externalURL\\\" at 'message': Cannot >>>>>>> find field.\\nInvalid JSON payload received. Unknown name >>>>>>> \\\"version\\\" >>>>>>> at 'message': Cannot find field.\\nInvalid JSON payload received. >>>>>>> Unknown >>>>>>> name \\\"groupKey\\\" at 'message': Cannot find field.\\nInvalid JSON >>>>>>> payload received. Unknown name \\\"truncatedAlerts\\\" at 'message': >>>>>>> Cannot >>>>>>> find field.\",\n \"status\": \"INVALID_ARGUMENT\",\n \"details\": >>>>>>> [\n >>>>>>> {\n \"@type\": \" >>>>>>> type.googleapis.com/google.rpc.BadRequest\ >>>>>>> <http://type.googleapis.com/google.rpc.BadRequest%5C>",\n >>>>>>> \"fieldViolations\": [\n {\n \"field\": >>>>>>> \"message\",\n >>>>>>> \"description\": \"Invalid JSON payload received. Unknown >>>>>>> name >>>>>>> \\\"receiver\\\" at 'message': Cannot find field.\"\n },\n >>>>>>> >>>>>>> {\n \"field\": \"message\",\n \"description\": >>>>>>> \"Invalid JSON payload received. Unknown name \\\"status\\\" at >>>>>>> 'message': >>>>>>> Cannot find field.\"\n },\n {\n \"field\": >>>>>>> \"message\",\n \"description\": \"Invalid JSON payload >>>>>>> received. >>>>>>> Unknown name \\\"alerts\\\" at 'message': Cannot find field.\"\n >>>>>>> >>>>>>> },\n {\n \"field\": \"message\",\n >>>>>>> \"description\": \"Invalid JSON payload received. Unknown name >>>>>>> \\\"groupLabels\\\" at 'message': Cannot find field.\"\n },\n >>>>>>> >>>>>>> {\n \"field\": \"message\",\n >>>>>>> \"description\": >>>>>>> \"Invalid JSON payload received. Unknown name \\\"commonLabels\\\" at >>>>>>> 'message': Cannot find field.\"\n },\n {\n >>>>>>> \"field\": \"message\",\n \"description\": \"Invalid JSON >>>>>>> payload received. Unknown name \\\"commonAnnotations\\\" at 'message': >>>>>>> Cannot find field.\"\n },\n {\n \"field\": >>>>>>> \"message\",\n \"description\": \"Invalid JSON payload >>>>>>> received. >>>>>>> Unknown name \\\"externalURL\\\" at 'message': Cannot find field.\"\n >>>>>>> >>>>>>> },\n {\n \"field\": \"message\",\n >>>>>>> \"description\": \"Invalid JSON payload received. Unknown name >>>>>>> \\\"version\\\" at 'message': Cannot find field.\"\n },\n >>>>>>> >>>>>>> {\n \"field\": \"message\",\n \"description\": >>>>>>> \"Invalid JSON payload received. Unknown name \\\"groupKey\\\" at >>>>>>> 'message': Cannot find field.\"\n },\n {\n >>>>>>> \"field\": \"message\",\n \"description\": \"Invalid JSON >>>>>>> payload received. Unknown name \\\"truncatedAlerts\\\" at 'message': >>>>>>> Cannot >>>>>>> find field.\"\n }\n ]\n }\n ]\n }\n}\n" >>>>>>> >>>>>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/698b1600-e78d-4df6-9884-f6a932270738n%40googlegroups.com.
