Brian, a wildcard is just a gibberish of standard card messages proposed by Google. I believe you're already familiar with https://developers.google.com/chat/api/guides/message-formats/cards#apps-script
Based on Mr-Karan's example (https://github.com/mr-karan/calert/blob/main/docs/examples/send_alert.sh) my assumption is as follows: 1) Calert retrieves an unstructured json alert, which alertmanager puts to host:6000 2) After that calert converts such a raw json to the one suiting it's template 3) It is then posting it to a google chat via curl Hence, calert is capable of processing the message under the hood, converting it to Google API-friendly json format and this is I guess I was looking for, but lack general understanding of whether it is possible to amend среда, 14 июня 2023 г. в 12:21:49 UTC+6, Brian Candler: > > It is basically possible by converting the message in a wildcard > structure acceptable by Google Chat API > > I don't know what you mean by a "wildcard structure". Can you give an > example, and/or point to documentation? Is it using tags within the > message field, or does it require a different JSON structure entirely? > > On Wednesday, 14 June 2023 at 05:12:53 UTC+1 Near N wrote: > >> Correct, that's same calert I was mentioning previously. I understand the >> use case of default template, but thought it is possible to do highlighting >> within the template. It is basically possible by converting the message in >> a wildcard structure acceptable by Google Chat API, but I couldn't find any >> options to do so with calert template, but thanks for suggestions >> >> понедельник, 12 июня 2023 г. в 19:43:10 UTC+6, Brian Candler: >> >>> If we're talking about the same "calert >>> <https://github.com/mr-karan/calert>" then it looks like it already >>> supports user-supplied templating of the message content: >>> >>> https://github.com/mr-karan/calert/blob/main/internal/providers/google_chat/message.go#L18-L53 >>> >>> The sample config shows how to configure it: >>> https://github.com/mr-karan/calert/blob/main/config.sample.toml#L13 >>> https://github.com/mr-karan/calert/blob/main/static/message.tmpl >>> >>> If you want to do colours and highlighting, it would depend whether the >>> Google Chat API accepts HTML tags, or some other way of marking up the >>> content. You'll need to find that out. >>> >>> The variables available in the templates look to be very similar to >>> those provided in Alertmanager templates: >>> https://prometheus.io/docs/alerting/latest/notifications/ >>> https://prometheus.io/docs/alerting/latest/notification_examples/ >>> >>> In fact, there's a test case which uses exactly the alertmanager code >>> (an instance of alertmgrtmpl.Alert) to drive it: >>> >>> https://github.com/mr-karan/calert/blob/main/internal/providers/google_chat/google_chat_test.go#L27-L35 >>> >>> On Monday, 12 June 2023 at 11:33:32 UTC+1 Near N wrote: >>> >>>> Brian, thanks for pointing to other integrations. The reason why I was >>>> wondering about curling the messages is that the existing solution >>>> (Calert) >>>> provides little information about templating options. Alerts are incoming, >>>> but with tight information. I'd love to highlight severity in different >>>> colors, add warning icons, e.g. something similar to Zabbix or at least >>>> that can suit basic user-friendly message. >>>> >>>> понедельник, 12 июня 2023 г. в 16:06:34 UTC+6, Brian Candler: >>>> >>>>> I can't speak for the project, but my guess would be it's out of >>>>> scope. There are many existing integrations which use the existing >>>>> webhook >>>>> mechanism: >>>>> >>>>> https://prometheus.io/docs/operating/integrations/#alertmanager-webhook-receiver >>>>> Doing it externally allows the request to be enriched with other data >>>>> which wouldn't be available in a template, and to use a variety of >>>>> outbound >>>>> authentication mechanisms which Alertmanager itself doesn't implement, or >>>>> even non-HTTP protocols like STOMP. >>>>> >>>>> In any case, text templating is a nasty and insecure way to create >>>>> JSON or XML payloads. >>>>> >>>>> > it is possible to converse the receivable JSON on the hop and just >>>>> send notifications via curl method >>>>> >>>>> You'd first have to *receive* the webhook payload from Alertmanager, >>>>> which involves plugging your curl script into a webserver somehow. You >>>>> could probably write one as a CGI script under Apache, for example. >>>>> >>>>> But why would you do that, when you have an existing tested and >>>>> working solution? >>>>> >>>>> On Monday, 12 June 2023 at 08:13:50 UTC+1 Near N wrote: >>>>> >>>>>> Thanks for reply Brian. I do realize, that Google Chat API requires >>>>>> a particular JSON format, but was assuming Alertmanager is capable of >>>>>> doing >>>>>> it by using particular template, which is wrong. Now the questions is >>>>>> whether it is worth creating a PR to create such a feature or it is out >>>>>> of >>>>>> Alertmanager's standard scope? >>>>>> >>>>>> Another question is perhaps it is possible to converse the receivable >>>>>> JSON on the hop and just send notifications via curl method? I mean if >>>>>> the >>>>>> endpoint contains the following alert " >>>>>> [{"annotations":{"server":"main","number_of_nodes":"1"},"endsAt":"2023-06-12T07:02:00.233Z","fingerprint":"b21bf244f87cd47f","receivers":[{"name":"my_alerts"}],"startsAt":"2023-06-12T06:44:00.233Z","status":{"inhibitedBy":[],"silencedBy":[],"state":"active"},"updatedAt":"2023-06-12T06:58:00.424Z","generatorURL":" >>>>>> http://app1:8880/vmalert/alert?group_id=71689142540802366","labels":{"alertgroup":"INFRA","alertname":"The >>>>>> >>>>>> number of available Cassandra nodes has >>>>>> changed","server":"main","severity":"Critical"}}]", would it be possible >>>>>> to >>>>>> retrieve this json and post it via curl? This, I assume, wouldn't solve >>>>>> my >>>>>> problem, cause I believe Google API expects something peculiar in terms >>>>>> of >>>>>> the message it receives. >>>>>> >>>>>> понедельник, 12 июня 2023 г. в 13:04:17 UTC+6, Brian Candler: >>>>>> >>>>>>> > Invalid JSON payload received. >>>>>>> >>>>>>> Alertmanager's webhook sends a JSON message with a *fixed* payload >>>>>>> format. See >>>>>>> >>>>>>> https://prometheus.io/docs/alerting/latest/configuration/#webhook_config >>>>>>> *"The Alertmanager will send HTTP POST requests in the following >>>>>>> JSON format to the configured endpoint"* >>>>>>> >>>>>>> You can't override it with a template. The Alertmanager JSON format >>>>>>> is different to what Google Chat requires, and therefore you can't send >>>>>>> it >>>>>>> directly. >>>>>>> >>>>>>> That's why you need a separate piece of software to do the >>>>>>> conversion (Calert in your case) - it receives the Alertmanager webhook >>>>>>> JSON payload, builds a new payload for the target, and then sends it >>>>>>> there. >>>>>>> >>>>>>> On Monday, 12 June 2023 at 06:59:14 UTC+1 Near N wrote: >>>>>>> >>>>>>>> Good day. I was wondering if there is a way to configure >>>>>>>> alertmanager to send notifications to google chat via a webhook. >>>>>>>> >>>>>>>> The current setup that I use consists of vmalert (loaded with set >>>>>>>> of promQL compatible rules) >> Alertmanager (to group them up) and >>>>>>>> Calert >>>>>>>> (to send them to google chat). Despite the flowchart is working fine, >>>>>>>> I >>>>>>>> found it difficult to amend proposed message template, so decided to >>>>>>>> simplify it by configuring alertmanager to send the alerts. It dawns >>>>>>>> on me >>>>>>>> that it can operate, but I seem to have failed to fine-tune it. Any >>>>>>>> help >>>>>>>> would be appreciated. The alertmanager.yml is below: >>>>>>>> >>>>>>>> global: >>>>>>>> http_config: >>>>>>>> proxy_url: '{{ alertmanager_proxy_url }}' >>>>>>>> >>>>>>>> route: >>>>>>>> receiver: 'my_alerts' >>>>>>>> group_wait: 30s >>>>>>>> group_interval: 60s >>>>>>>> repeat_interval: 15m >>>>>>>> group_by: ['room', 'alertName'] >>>>>>>> >>>>>>>> receivers: >>>>>>>> - name: 'my_alerts' >>>>>>>> webhook_configs: >>>>>>>> - url: '{{ alertmanager_chat_webhook }}' >>>>>>>> send_resolved: true >>>>>>>> >>>>>>>> templates: >>>>>>>> - '{{ alertmanager_working_directory }}/google-chat.tmpl' >>>>>>>> >>>>>>>> >>>>>>>> And here is the error of alertmanager trying to send an alert: >>>>>>>> >>>>>>>> Jun 12 05:26:00 app1-cp alertmanager[140337]: >>>>>>>> ts=2023-06-12T05:26:00.599Z caller=dispatch.go:352 level=error >>>>>>>> component=dispatcher msg="Notify for alerts failed" num_alerts=3 >>>>>>>> err="my_alerts/webhook[0]: notify retry canceled due to unrecoverable >>>>>>>> error >>>>>>>> after 1 attempts: unexpected status code 400: >>>>>>>> https://chat.googleapis.com/v1/spaces/webook_token: {\n >>>>>>>> \"error\": {\n \"code\": 400,\n \"message\": \"Invalid JSON >>>>>>>> payload >>>>>>>> received. Unknown name \\\"receiver\\\" at 'message': Cannot find >>>>>>>> field.\\nInvalid JSON payload received. Unknown name \\\"status\\\" at >>>>>>>> 'message': Cannot find field.\\nInvalid JSON payload received. Unknown >>>>>>>> name >>>>>>>> \\\"alerts\\\" at 'message': Cannot find field.\\nInvalid JSON payload >>>>>>>> received. Unknown name \\\"groupLabels\\\" at 'message': Cannot find >>>>>>>> field.\\nInvalid JSON payload received. Unknown name >>>>>>>> \\\"commonLabels\\\" >>>>>>>> at 'message': Cannot find field.\\nInvalid JSON payload received. >>>>>>>> Unknown >>>>>>>> name \\\"commonAnnotations\\\" at 'message': Cannot find >>>>>>>> field.\\nInvalid >>>>>>>> JSON payload received. Unknown name \\\"externalURL\\\" at 'message': >>>>>>>> Cannot find field.\\nInvalid JSON payload received. Unknown name >>>>>>>> \\\"version\\\" at 'message': Cannot find field.\\nInvalid JSON >>>>>>>> payload >>>>>>>> received. Unknown name \\\"groupKey\\\" at 'message': Cannot find >>>>>>>> field.\\nInvalid JSON payload received. Unknown name >>>>>>>> \\\"truncatedAlerts\\\" at 'message': Cannot find field.\",\n >>>>>>>> \"status\": \"INVALID_ARGUMENT\",\n \"details\": [\n {\n >>>>>>>> >>>>>>>> \"@type\": \"type.googleapis.com/google.rpc.BadRequest\ >>>>>>>> <http://type.googleapis.com/google.rpc.BadRequest%5C>",\n >>>>>>>> \"fieldViolations\": [\n {\n \"field\": >>>>>>>> \"message\",\n >>>>>>>> \"description\": \"Invalid JSON payload received. Unknown >>>>>>>> name >>>>>>>> \\\"receiver\\\" at 'message': Cannot find field.\"\n },\n >>>>>>>> >>>>>>>> {\n \"field\": \"message\",\n \"description\": >>>>>>>> \"Invalid JSON payload received. Unknown name \\\"status\\\" at >>>>>>>> 'message': >>>>>>>> Cannot find field.\"\n },\n {\n >>>>>>>> \"field\": >>>>>>>> \"message\",\n \"description\": \"Invalid JSON payload >>>>>>>> received. >>>>>>>> Unknown name \\\"alerts\\\" at 'message': Cannot find field.\"\n >>>>>>>> >>>>>>>> },\n {\n \"field\": \"message\",\n >>>>>>>> \"description\": \"Invalid JSON payload received. Unknown name >>>>>>>> \\\"groupLabels\\\" at 'message': Cannot find field.\"\n },\n >>>>>>>> >>>>>>>> {\n \"field\": \"message\",\n >>>>>>>> \"description\": >>>>>>>> \"Invalid JSON payload received. Unknown name \\\"commonLabels\\\" at >>>>>>>> 'message': Cannot find field.\"\n },\n {\n >>>>>>>> \"field\": \"message\",\n \"description\": \"Invalid JSON >>>>>>>> payload received. Unknown name \\\"commonAnnotations\\\" at 'message': >>>>>>>> Cannot find field.\"\n },\n {\n >>>>>>>> \"field\": >>>>>>>> \"message\",\n \"description\": \"Invalid JSON payload >>>>>>>> received. >>>>>>>> Unknown name \\\"externalURL\\\" at 'message': Cannot find field.\"\n >>>>>>>> >>>>>>>> },\n {\n \"field\": \"message\",\n >>>>>>>> \"description\": \"Invalid JSON payload received. Unknown name >>>>>>>> \\\"version\\\" at 'message': Cannot find field.\"\n },\n >>>>>>>> >>>>>>>> {\n \"field\": \"message\",\n \"description\": >>>>>>>> \"Invalid JSON payload received. Unknown name \\\"groupKey\\\" at >>>>>>>> 'message': Cannot find field.\"\n },\n {\n >>>>>>>> \"field\": \"message\",\n \"description\": \"Invalid JSON >>>>>>>> payload received. Unknown name \\\"truncatedAlerts\\\" at 'message': >>>>>>>> Cannot >>>>>>>> find field.\"\n }\n ]\n }\n ]\n }\n}\n" >>>>>>>> >>>>>>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c4d7d5a4-25e9-4c7f-9b23-4c568163732bn%40googlegroups.com.
