That's clearer, so there are two API formats: a "text <https://developers.google.com/chat/api/guides/message-formats/text>" message and a "card <https://developers.google.com/chat/api/guides/message-formats/cards>" message.
It looks like Calert is hard-coded to using the "text" format: https://github.com/mr-karan/calert/blob/main/internal/providers/google_chat/alerts.go#L29-L34 https://github.com/mr-karan/calert/blob/main/internal/providers/google_chat/message.go#L18-L53 Hence you could either extend Calert to work with cards too - or raise it as a feature request; or you could implement your own alertmanager webhook to Google Chat gateway. On Wednesday, 14 June 2023 at 08:14:03 UTC+1 Near N wrote: > Brian, a wildcard is just a gibberish of standard card messages proposed > by Google. I believe you're already familiar with > https://developers.google.com/chat/api/guides/message-formats/cards#apps-script > > Based on Mr-Karan's example ( > https://github.com/mr-karan/calert/blob/main/docs/examples/send_alert.sh) > my assumption is as follows: > 1) Calert retrieves an unstructured json alert, which alertmanager puts to > host:6000 > 2) After that calert converts such a raw json to the one suiting it's > template > 3) It is then posting it to a google chat via curl > > Hence, calert is capable of processing the message under the hood, > converting it to Google API-friendly json format and this is I guess I was > looking for, but lack general understanding of whether it is possible to > amend > среда, 14 июня 2023 г. в 12:21:49 UTC+6, Brian Candler: > >> > It is basically possible by converting the message in a wildcard >> structure acceptable by Google Chat API >> >> I don't know what you mean by a "wildcard structure". Can you give an >> example, and/or point to documentation? Is it using tags within the >> message field, or does it require a different JSON structure entirely? >> >> On Wednesday, 14 June 2023 at 05:12:53 UTC+1 Near N wrote: >> >>> Correct, that's same calert I was mentioning previously. I understand >>> the use case of default template, but thought it is possible to do >>> highlighting within the template. It is basically possible by converting >>> the message in a wildcard structure acceptable by Google Chat API, but I >>> couldn't find any options to do so with calert template, but thanks for >>> suggestions >>> >>> понедельник, 12 июня 2023 г. в 19:43:10 UTC+6, Brian Candler: >>> >>>> If we're talking about the same "calert >>>> <https://github.com/mr-karan/calert>" then it looks like it already >>>> supports user-supplied templating of the message content: >>>> >>>> https://github.com/mr-karan/calert/blob/main/internal/providers/google_chat/message.go#L18-L53 >>>> >>>> The sample config shows how to configure it: >>>> https://github.com/mr-karan/calert/blob/main/config.sample.toml#L13 >>>> https://github.com/mr-karan/calert/blob/main/static/message.tmpl >>>> >>>> If you want to do colours and highlighting, it would depend whether the >>>> Google Chat API accepts HTML tags, or some other way of marking up the >>>> content. You'll need to find that out. >>>> >>>> The variables available in the templates look to be very similar to >>>> those provided in Alertmanager templates: >>>> https://prometheus.io/docs/alerting/latest/notifications/ >>>> https://prometheus.io/docs/alerting/latest/notification_examples/ >>>> >>>> In fact, there's a test case which uses exactly the alertmanager code >>>> (an instance of alertmgrtmpl.Alert) to drive it: >>>> >>>> https://github.com/mr-karan/calert/blob/main/internal/providers/google_chat/google_chat_test.go#L27-L35 >>>> >>>> On Monday, 12 June 2023 at 11:33:32 UTC+1 Near N wrote: >>>> >>>>> Brian, thanks for pointing to other integrations. The reason why I was >>>>> wondering about curling the messages is that the existing solution >>>>> (Calert) >>>>> provides little information about templating options. Alerts are >>>>> incoming, >>>>> but with tight information. I'd love to highlight severity in different >>>>> colors, add warning icons, e.g. something similar to Zabbix or at least >>>>> that can suit basic user-friendly message. >>>>> >>>>> понедельник, 12 июня 2023 г. в 16:06:34 UTC+6, Brian Candler: >>>>> >>>>>> I can't speak for the project, but my guess would be it's out of >>>>>> scope. There are many existing integrations which use the existing >>>>>> webhook >>>>>> mechanism: >>>>>> >>>>>> https://prometheus.io/docs/operating/integrations/#alertmanager-webhook-receiver >>>>>> Doing it externally allows the request to be enriched with other data >>>>>> which wouldn't be available in a template, and to use a variety of >>>>>> outbound >>>>>> authentication mechanisms which Alertmanager itself doesn't implement, >>>>>> or >>>>>> even non-HTTP protocols like STOMP. >>>>>> >>>>>> In any case, text templating is a nasty and insecure way to create >>>>>> JSON or XML payloads. >>>>>> >>>>>> > it is possible to converse the receivable JSON on the hop and just >>>>>> send notifications via curl method >>>>>> >>>>>> You'd first have to *receive* the webhook payload from Alertmanager, >>>>>> which involves plugging your curl script into a webserver somehow. You >>>>>> could probably write one as a CGI script under Apache, for example. >>>>>> >>>>>> But why would you do that, when you have an existing tested and >>>>>> working solution? >>>>>> >>>>>> On Monday, 12 June 2023 at 08:13:50 UTC+1 Near N wrote: >>>>>> >>>>>>> Thanks for reply Brian. I do realize, that Google Chat API requires >>>>>>> a particular JSON format, but was assuming Alertmanager is capable of >>>>>>> doing >>>>>>> it by using particular template, which is wrong. Now the questions is >>>>>>> whether it is worth creating a PR to create such a feature or it is out >>>>>>> of >>>>>>> Alertmanager's standard scope? >>>>>>> >>>>>>> Another question is perhaps it is possible to converse the >>>>>>> receivable JSON on the hop and just send notifications via curl method? >>>>>>> I >>>>>>> mean if the endpoint contains the following alert " >>>>>>> [{"annotations":{"server":"main","number_of_nodes":"1"},"endsAt":"2023-06-12T07:02:00.233Z","fingerprint":"b21bf244f87cd47f","receivers":[{"name":"my_alerts"}],"startsAt":"2023-06-12T06:44:00.233Z","status":{"inhibitedBy":[],"silencedBy":[],"state":"active"},"updatedAt":"2023-06-12T06:58:00.424Z","generatorURL":" >>>>>>> http://app1:8880/vmalert/alert?group_id=71689142540802366","labels":{"alertgroup":"INFRA","alertname":"The >>>>>>> >>>>>>> number of available Cassandra nodes has >>>>>>> changed","server":"main","severity":"Critical"}}]", would it be >>>>>>> possible to >>>>>>> retrieve this json and post it via curl? This, I assume, wouldn't solve >>>>>>> my >>>>>>> problem, cause I believe Google API expects something peculiar in terms >>>>>>> of >>>>>>> the message it receives. >>>>>>> >>>>>>> понедельник, 12 июня 2023 г. в 13:04:17 UTC+6, Brian Candler: >>>>>>> >>>>>>>> > Invalid JSON payload received. >>>>>>>> >>>>>>>> Alertmanager's webhook sends a JSON message with a *fixed* payload >>>>>>>> format. See >>>>>>>> >>>>>>>> https://prometheus.io/docs/alerting/latest/configuration/#webhook_config >>>>>>>> *"The Alertmanager will send HTTP POST requests in the following >>>>>>>> JSON format to the configured endpoint"* >>>>>>>> >>>>>>>> You can't override it with a template. The Alertmanager JSON format >>>>>>>> is different to what Google Chat requires, and therefore you can't >>>>>>>> send it >>>>>>>> directly. >>>>>>>> >>>>>>>> That's why you need a separate piece of software to do the >>>>>>>> conversion (Calert in your case) - it receives the Alertmanager >>>>>>>> webhook >>>>>>>> JSON payload, builds a new payload for the target, and then sends it >>>>>>>> there. >>>>>>>> >>>>>>>> On Monday, 12 June 2023 at 06:59:14 UTC+1 Near N wrote: >>>>>>>> >>>>>>>>> Good day. I was wondering if there is a way to configure >>>>>>>>> alertmanager to send notifications to google chat via a webhook. >>>>>>>>> >>>>>>>>> The current setup that I use consists of vmalert (loaded with set >>>>>>>>> of promQL compatible rules) >> Alertmanager (to group them up) and >>>>>>>>> Calert >>>>>>>>> (to send them to google chat). Despite the flowchart is working fine, >>>>>>>>> I >>>>>>>>> found it difficult to amend proposed message template, so decided to >>>>>>>>> simplify it by configuring alertmanager to send the alerts. It dawns >>>>>>>>> on me >>>>>>>>> that it can operate, but I seem to have failed to fine-tune it. Any >>>>>>>>> help >>>>>>>>> would be appreciated. The alertmanager.yml is below: >>>>>>>>> >>>>>>>>> global: >>>>>>>>> http_config: >>>>>>>>> proxy_url: '{{ alertmanager_proxy_url }}' >>>>>>>>> >>>>>>>>> route: >>>>>>>>> receiver: 'my_alerts' >>>>>>>>> group_wait: 30s >>>>>>>>> group_interval: 60s >>>>>>>>> repeat_interval: 15m >>>>>>>>> group_by: ['room', 'alertName'] >>>>>>>>> >>>>>>>>> receivers: >>>>>>>>> - name: 'my_alerts' >>>>>>>>> webhook_configs: >>>>>>>>> - url: '{{ alertmanager_chat_webhook }}' >>>>>>>>> send_resolved: true >>>>>>>>> >>>>>>>>> templates: >>>>>>>>> - '{{ alertmanager_working_directory }}/google-chat.tmpl' >>>>>>>>> >>>>>>>>> >>>>>>>>> And here is the error of alertmanager trying to send an alert: >>>>>>>>> >>>>>>>>> Jun 12 05:26:00 app1-cp alertmanager[140337]: >>>>>>>>> ts=2023-06-12T05:26:00.599Z caller=dispatch.go:352 level=error >>>>>>>>> component=dispatcher msg="Notify for alerts failed" num_alerts=3 >>>>>>>>> err="my_alerts/webhook[0]: notify retry canceled due to unrecoverable >>>>>>>>> error >>>>>>>>> after 1 attempts: unexpected status code 400: >>>>>>>>> https://chat.googleapis.com/v1/spaces/webook_token: {\n >>>>>>>>> \"error\": {\n \"code\": 400,\n \"message\": \"Invalid JSON >>>>>>>>> payload >>>>>>>>> received. Unknown name \\\"receiver\\\" at 'message': Cannot find >>>>>>>>> field.\\nInvalid JSON payload received. Unknown name \\\"status\\\" >>>>>>>>> at >>>>>>>>> 'message': Cannot find field.\\nInvalid JSON payload received. >>>>>>>>> Unknown name >>>>>>>>> \\\"alerts\\\" at 'message': Cannot find field.\\nInvalid JSON >>>>>>>>> payload >>>>>>>>> received. Unknown name \\\"groupLabels\\\" at 'message': Cannot find >>>>>>>>> field.\\nInvalid JSON payload received. Unknown name >>>>>>>>> \\\"commonLabels\\\" >>>>>>>>> at 'message': Cannot find field.\\nInvalid JSON payload received. >>>>>>>>> Unknown >>>>>>>>> name \\\"commonAnnotations\\\" at 'message': Cannot find >>>>>>>>> field.\\nInvalid >>>>>>>>> JSON payload received. Unknown name \\\"externalURL\\\" at 'message': >>>>>>>>> Cannot find field.\\nInvalid JSON payload received. Unknown name >>>>>>>>> \\\"version\\\" at 'message': Cannot find field.\\nInvalid JSON >>>>>>>>> payload >>>>>>>>> received. Unknown name \\\"groupKey\\\" at 'message': Cannot find >>>>>>>>> field.\\nInvalid JSON payload received. Unknown name >>>>>>>>> \\\"truncatedAlerts\\\" at 'message': Cannot find field.\",\n >>>>>>>>> \"status\": \"INVALID_ARGUMENT\",\n \"details\": [\n {\n >>>>>>>>> >>>>>>>>> \"@type\": \"type.googleapis.com/google.rpc.BadRequest\ >>>>>>>>> <http://type.googleapis.com/google.rpc.BadRequest%5C>",\n >>>>>>>>> \"fieldViolations\": [\n {\n \"field\": >>>>>>>>> \"message\",\n >>>>>>>>> \"description\": \"Invalid JSON payload received. Unknown >>>>>>>>> name >>>>>>>>> \\\"receiver\\\" at 'message': Cannot find field.\"\n },\n >>>>>>>>> >>>>>>>>> {\n \"field\": \"message\",\n \"description\": >>>>>>>>> \"Invalid JSON payload received. Unknown name \\\"status\\\" at >>>>>>>>> 'message': >>>>>>>>> Cannot find field.\"\n },\n {\n >>>>>>>>> \"field\": >>>>>>>>> \"message\",\n \"description\": \"Invalid JSON payload >>>>>>>>> received. >>>>>>>>> Unknown name \\\"alerts\\\" at 'message': Cannot find field.\"\n >>>>>>>>> >>>>>>>>> },\n {\n \"field\": \"message\",\n >>>>>>>>> \"description\": \"Invalid JSON payload received. Unknown name >>>>>>>>> \\\"groupLabels\\\" at 'message': Cannot find field.\"\n >>>>>>>>> },\n >>>>>>>>> {\n \"field\": \"message\",\n >>>>>>>>> \"description\": >>>>>>>>> \"Invalid JSON payload received. Unknown name \\\"commonLabels\\\" at >>>>>>>>> 'message': Cannot find field.\"\n },\n {\n >>>>>>>>> >>>>>>>>> \"field\": \"message\",\n \"description\": \"Invalid JSON >>>>>>>>> payload received. Unknown name \\\"commonAnnotations\\\" at >>>>>>>>> 'message': >>>>>>>>> Cannot find field.\"\n },\n {\n >>>>>>>>> \"field\": >>>>>>>>> \"message\",\n \"description\": \"Invalid JSON payload >>>>>>>>> received. >>>>>>>>> Unknown name \\\"externalURL\\\" at 'message': Cannot find field.\"\n >>>>>>>>> >>>>>>>>> },\n {\n \"field\": \"message\",\n >>>>>>>>> \"description\": \"Invalid JSON payload received. Unknown name >>>>>>>>> \\\"version\\\" at 'message': Cannot find field.\"\n },\n >>>>>>>>> >>>>>>>>> {\n \"field\": \"message\",\n \"description\": >>>>>>>>> \"Invalid JSON payload received. Unknown name \\\"groupKey\\\" at >>>>>>>>> 'message': Cannot find field.\"\n },\n {\n >>>>>>>>> >>>>>>>>> \"field\": \"message\",\n \"description\": \"Invalid JSON >>>>>>>>> payload received. Unknown name \\\"truncatedAlerts\\\" at 'message': >>>>>>>>> Cannot >>>>>>>>> find field.\"\n }\n ]\n }\n ]\n }\n}\n" >>>>>>>>> >>>>>>>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ff1782ec-85c7-4a34-9c94-fe30e02f2e80n%40googlegroups.com.
