Correct, that's same calert I was mentioning previously. I understand the use case of default template, but thought it is possible to do highlighting within the template. It is basically possible by converting the message in a wildcard structure acceptable by Google Chat API, but I couldn't find any options to do so with calert template, but thanks for suggestions
понедельник, 12 июня 2023 г. в 19:43:10 UTC+6, Brian Candler: > If we're talking about the same "calert > <https://github.com/mr-karan/calert>" then it looks like it already > supports user-supplied templating of the message content: > > https://github.com/mr-karan/calert/blob/main/internal/providers/google_chat/message.go#L18-L53 > > The sample config shows how to configure it: > https://github.com/mr-karan/calert/blob/main/config.sample.toml#L13 > https://github.com/mr-karan/calert/blob/main/static/message.tmpl > > If you want to do colours and highlighting, it would depend whether the > Google Chat API accepts HTML tags, or some other way of marking up the > content. You'll need to find that out. > > The variables available in the templates look to be very similar to those > provided in Alertmanager templates: > https://prometheus.io/docs/alerting/latest/notifications/ > https://prometheus.io/docs/alerting/latest/notification_examples/ > > In fact, there's a test case which uses exactly the alertmanager code (an > instance of alertmgrtmpl.Alert) to drive it: > > https://github.com/mr-karan/calert/blob/main/internal/providers/google_chat/google_chat_test.go#L27-L35 > > On Monday, 12 June 2023 at 11:33:32 UTC+1 Near N wrote: > >> Brian, thanks for pointing to other integrations. The reason why I was >> wondering about curling the messages is that the existing solution (Calert) >> provides little information about templating options. Alerts are incoming, >> but with tight information. I'd love to highlight severity in different >> colors, add warning icons, e.g. something similar to Zabbix or at least >> that can suit basic user-friendly message. >> >> понедельник, 12 июня 2023 г. в 16:06:34 UTC+6, Brian Candler: >> >>> I can't speak for the project, but my guess would be it's out of scope. >>> There are many existing integrations which use the existing webhook >>> mechanism: >>> >>> https://prometheus.io/docs/operating/integrations/#alertmanager-webhook-receiver >>> Doing it externally allows the request to be enriched with other data >>> which wouldn't be available in a template, and to use a variety of outbound >>> authentication mechanisms which Alertmanager itself doesn't implement, or >>> even non-HTTP protocols like STOMP. >>> >>> In any case, text templating is a nasty and insecure way to create JSON >>> or XML payloads. >>> >>> > it is possible to converse the receivable JSON on the hop and just >>> send notifications via curl method >>> >>> You'd first have to *receive* the webhook payload from Alertmanager, >>> which involves plugging your curl script into a webserver somehow. You >>> could probably write one as a CGI script under Apache, for example. >>> >>> But why would you do that, when you have an existing tested and working >>> solution? >>> >>> On Monday, 12 June 2023 at 08:13:50 UTC+1 Near N wrote: >>> >>>> Thanks for reply Brian. I do realize, that Google Chat API requires a >>>> particular JSON format, but was assuming Alertmanager is capable of doing >>>> it by using particular template, which is wrong. Now the questions is >>>> whether it is worth creating a PR to create such a feature or it is out of >>>> Alertmanager's standard scope? >>>> >>>> Another question is perhaps it is possible to converse the receivable >>>> JSON on the hop and just send notifications via curl method? I mean if the >>>> endpoint contains the following alert " >>>> [{"annotations":{"server":"main","number_of_nodes":"1"},"endsAt":"2023-06-12T07:02:00.233Z","fingerprint":"b21bf244f87cd47f","receivers":[{"name":"my_alerts"}],"startsAt":"2023-06-12T06:44:00.233Z","status":{"inhibitedBy":[],"silencedBy":[],"state":"active"},"updatedAt":"2023-06-12T06:58:00.424Z","generatorURL":" >>>> http://app1:8880/vmalert/alert?group_id=71689142540802366","labels":{"alertgroup":"INFRA","alertname":"The >>>> >>>> number of available Cassandra nodes has >>>> changed","server":"main","severity":"Critical"}}]", would it be possible >>>> to >>>> retrieve this json and post it via curl? This, I assume, wouldn't solve my >>>> problem, cause I believe Google API expects something peculiar in terms of >>>> the message it receives. >>>> >>>> понедельник, 12 июня 2023 г. в 13:04:17 UTC+6, Brian Candler: >>>> >>>>> > Invalid JSON payload received. >>>>> >>>>> Alertmanager's webhook sends a JSON message with a *fixed* payload >>>>> format. See >>>>> >>>>> https://prometheus.io/docs/alerting/latest/configuration/#webhook_config >>>>> *"The Alertmanager will send HTTP POST requests in the following JSON >>>>> format to the configured endpoint"* >>>>> >>>>> You can't override it with a template. The Alertmanager JSON format is >>>>> different to what Google Chat requires, and therefore you can't send it >>>>> directly. >>>>> >>>>> That's why you need a separate piece of software to do the conversion >>>>> (Calert in your case) - it receives the Alertmanager webhook JSON >>>>> payload, >>>>> builds a new payload for the target, and then sends it there. >>>>> >>>>> On Monday, 12 June 2023 at 06:59:14 UTC+1 Near N wrote: >>>>> >>>>>> Good day. I was wondering if there is a way to configure alertmanager >>>>>> to send notifications to google chat via a webhook. >>>>>> >>>>>> The current setup that I use consists of vmalert (loaded with set of >>>>>> promQL compatible rules) >> Alertmanager (to group them up) and Calert >>>>>> (to >>>>>> send them to google chat). Despite the flowchart is working fine, I >>>>>> found >>>>>> it difficult to amend proposed message template, so decided to simplify >>>>>> it >>>>>> by configuring alertmanager to send the alerts. It dawns on me that it >>>>>> can >>>>>> operate, but I seem to have failed to fine-tune it. Any help would be >>>>>> appreciated. The alertmanager.yml is below: >>>>>> >>>>>> global: >>>>>> http_config: >>>>>> proxy_url: '{{ alertmanager_proxy_url }}' >>>>>> >>>>>> route: >>>>>> receiver: 'my_alerts' >>>>>> group_wait: 30s >>>>>> group_interval: 60s >>>>>> repeat_interval: 15m >>>>>> group_by: ['room', 'alertName'] >>>>>> >>>>>> receivers: >>>>>> - name: 'my_alerts' >>>>>> webhook_configs: >>>>>> - url: '{{ alertmanager_chat_webhook }}' >>>>>> send_resolved: true >>>>>> >>>>>> templates: >>>>>> - '{{ alertmanager_working_directory }}/google-chat.tmpl' >>>>>> >>>>>> >>>>>> And here is the error of alertmanager trying to send an alert: >>>>>> >>>>>> Jun 12 05:26:00 app1-cp alertmanager[140337]: >>>>>> ts=2023-06-12T05:26:00.599Z caller=dispatch.go:352 level=error >>>>>> component=dispatcher msg="Notify for alerts failed" num_alerts=3 >>>>>> err="my_alerts/webhook[0]: notify retry canceled due to unrecoverable >>>>>> error >>>>>> after 1 attempts: unexpected status code 400: >>>>>> https://chat.googleapis.com/v1/spaces/webook_token: {\n \"error\": >>>>>> {\n \"code\": 400,\n \"message\": \"Invalid JSON payload received. >>>>>> Unknown name \\\"receiver\\\" at 'message': Cannot find field.\\nInvalid >>>>>> JSON payload received. Unknown name \\\"status\\\" at 'message': Cannot >>>>>> find field.\\nInvalid JSON payload received. Unknown name \\\"alerts\\\" >>>>>> at >>>>>> 'message': Cannot find field.\\nInvalid JSON payload received. Unknown >>>>>> name >>>>>> \\\"groupLabels\\\" at 'message': Cannot find field.\\nInvalid JSON >>>>>> payload >>>>>> received. Unknown name \\\"commonLabels\\\" at 'message': Cannot find >>>>>> field.\\nInvalid JSON payload received. Unknown name >>>>>> \\\"commonAnnotations\\\" at 'message': Cannot find field.\\nInvalid >>>>>> JSON >>>>>> payload received. Unknown name \\\"externalURL\\\" at 'message': Cannot >>>>>> find field.\\nInvalid JSON payload received. Unknown name >>>>>> \\\"version\\\" >>>>>> at 'message': Cannot find field.\\nInvalid JSON payload received. >>>>>> Unknown >>>>>> name \\\"groupKey\\\" at 'message': Cannot find field.\\nInvalid JSON >>>>>> payload received. Unknown name \\\"truncatedAlerts\\\" at 'message': >>>>>> Cannot >>>>>> find field.\",\n \"status\": \"INVALID_ARGUMENT\",\n \"details\": >>>>>> [\n >>>>>> {\n \"@type\": \" >>>>>> type.googleapis.com/google.rpc.BadRequest\ >>>>>> <http://type.googleapis.com/google.rpc.BadRequest%5C>",\n >>>>>> \"fieldViolations\": [\n {\n \"field\": >>>>>> \"message\",\n >>>>>> \"description\": \"Invalid JSON payload received. Unknown >>>>>> name >>>>>> \\\"receiver\\\" at 'message': Cannot find field.\"\n },\n >>>>>> >>>>>> {\n \"field\": \"message\",\n \"description\": >>>>>> \"Invalid JSON payload received. Unknown name \\\"status\\\" at >>>>>> 'message': >>>>>> Cannot find field.\"\n },\n {\n \"field\": >>>>>> \"message\",\n \"description\": \"Invalid JSON payload >>>>>> received. >>>>>> Unknown name \\\"alerts\\\" at 'message': Cannot find field.\"\n >>>>>> },\n {\n \"field\": \"message\",\n >>>>>> \"description\": \"Invalid JSON payload received. Unknown name >>>>>> \\\"groupLabels\\\" at 'message': Cannot find field.\"\n },\n >>>>>> >>>>>> {\n \"field\": \"message\",\n >>>>>> \"description\": >>>>>> \"Invalid JSON payload received. Unknown name \\\"commonLabels\\\" at >>>>>> 'message': Cannot find field.\"\n },\n {\n >>>>>> \"field\": \"message\",\n \"description\": \"Invalid JSON >>>>>> payload received. Unknown name \\\"commonAnnotations\\\" at 'message': >>>>>> Cannot find field.\"\n },\n {\n \"field\": >>>>>> \"message\",\n \"description\": \"Invalid JSON payload >>>>>> received. >>>>>> Unknown name \\\"externalURL\\\" at 'message': Cannot find field.\"\n >>>>>> >>>>>> },\n {\n \"field\": \"message\",\n >>>>>> \"description\": \"Invalid JSON payload received. Unknown name >>>>>> \\\"version\\\" at 'message': Cannot find field.\"\n },\n >>>>>> >>>>>> {\n \"field\": \"message\",\n \"description\": >>>>>> \"Invalid JSON payload received. Unknown name \\\"groupKey\\\" at >>>>>> 'message': Cannot find field.\"\n },\n {\n >>>>>> \"field\": \"message\",\n \"description\": \"Invalid JSON >>>>>> payload received. Unknown name \\\"truncatedAlerts\\\" at 'message': >>>>>> Cannot >>>>>> find field.\"\n }\n ]\n }\n ]\n }\n}\n" >>>>>> >>>>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0eed443b-cf2d-46b4-8a17-c4654cfc3054n%40googlegroups.com.
