> Invalid JSON payload received.
Alertmanager's webhook sends a JSON message with a *fixed* payload format.
See
https://prometheus.io/docs/alerting/latest/configuration/#webhook_config
*"The Alertmanager will send HTTP POST requests in the following JSON
format to the configured endpoint"*
You can't override it with a template. The Alertmanager JSON format is
different to what Google Chat requires, and therefore you can't send it
directly.
That's why you need a separate piece of software to do the conversion
(Calert in your case) - it receives the Alertmanager webhook JSON payload,
builds a new payload for the target, and then sends it there.
On Monday, 12 June 2023 at 06:59:14 UTC+1 Near N wrote:
> Good day. I was wondering if there is a way to configure alertmanager to
> send notifications to google chat via a webhook.
>
> The current setup that I use consists of vmalert (loaded with set of
> promQL compatible rules) >> Alertmanager (to group them up) and Calert (to
> send them to google chat). Despite the flowchart is working fine, I found
> it difficult to amend proposed message template, so decided to simplify it
> by configuring alertmanager to send the alerts. It dawns on me that it can
> operate, but I seem to have failed to fine-tune it. Any help would be
> appreciated. The alertmanager.yml is below:
>
> global:
> http_config:
> proxy_url: '{{ alertmanager_proxy_url }}'
>
> route:
> receiver: 'my_alerts'
> group_wait: 30s
> group_interval: 60s
> repeat_interval: 15m
> group_by: ['room', 'alertName']
>
> receivers:
> - name: 'my_alerts'
> webhook_configs:
> - url: '{{ alertmanager_chat_webhook }}'
> send_resolved: true
>
> templates:
> - '{{ alertmanager_working_directory }}/google-chat.tmpl'
>
>
> And here is the error of alertmanager trying to send an alert:
>
> Jun 12 05:26:00 app1-cp alertmanager[140337]: ts=2023-06-12T05:26:00.599Z
> caller=dispatch.go:352 level=error component=dispatcher msg="Notify for
> alerts failed" num_alerts=3 err="my_alerts/webhook[0]: notify retry
> canceled due to unrecoverable error after 1 attempts: unexpected status
> code 400: https://chat.googleapis.com/v1/spaces/webook_token: {\n
> \"error\": {\n \"code\": 400,\n \"message\": \"Invalid JSON payload
> received. Unknown name \\\"receiver\\\" at 'message': Cannot find
> field.\\nInvalid JSON payload received. Unknown name \\\"status\\\" at
> 'message': Cannot find field.\\nInvalid JSON payload received. Unknown name
> \\\"alerts\\\" at 'message': Cannot find field.\\nInvalid JSON payload
> received. Unknown name \\\"groupLabels\\\" at 'message': Cannot find
> field.\\nInvalid JSON payload received. Unknown name \\\"commonLabels\\\"
> at 'message': Cannot find field.\\nInvalid JSON payload received. Unknown
> name \\\"commonAnnotations\\\" at 'message': Cannot find field.\\nInvalid
> JSON payload received. Unknown name \\\"externalURL\\\" at 'message':
> Cannot find field.\\nInvalid JSON payload received. Unknown name
> \\\"version\\\" at 'message': Cannot find field.\\nInvalid JSON payload
> received. Unknown name \\\"groupKey\\\" at 'message': Cannot find
> field.\\nInvalid JSON payload received. Unknown name
> \\\"truncatedAlerts\\\" at 'message': Cannot find field.\",\n
> \"status\": \"INVALID_ARGUMENT\",\n \"details\": [\n {\n
> \"@type\": \"type.googleapis.com/google.rpc.BadRequest\
> <http://type.googleapis.com/google.rpc.BadRequest%5C>",\n
> \"fieldViolations\": [\n {\n \"field\": \"message\",\n
> \"description\": \"Invalid JSON payload received. Unknown name
> \\\"receiver\\\" at 'message': Cannot find field.\"\n },\n
> {\n \"field\": \"message\",\n \"description\":
> \"Invalid JSON payload received. Unknown name \\\"status\\\" at 'message':
> Cannot find field.\"\n },\n {\n \"field\":
> \"message\",\n \"description\": \"Invalid JSON payload received.
> Unknown name \\\"alerts\\\" at 'message': Cannot find field.\"\n
> },\n {\n \"field\": \"message\",\n
> \"description\": \"Invalid JSON payload received. Unknown name
> \\\"groupLabels\\\" at 'message': Cannot find field.\"\n },\n
> {\n \"field\": \"message\",\n \"description\":
> \"Invalid JSON payload received. Unknown name \\\"commonLabels\\\" at
> 'message': Cannot find field.\"\n },\n {\n
> \"field\": \"message\",\n \"description\": \"Invalid JSON
> payload received. Unknown name \\\"commonAnnotations\\\" at 'message':
> Cannot find field.\"\n },\n {\n \"field\":
> \"message\",\n \"description\": \"Invalid JSON payload received.
> Unknown name \\\"externalURL\\\" at 'message': Cannot find field.\"\n
> },\n {\n \"field\": \"message\",\n
> \"description\": \"Invalid JSON payload received. Unknown name
> \\\"version\\\" at 'message': Cannot find field.\"\n },\n
> {\n \"field\": \"message\",\n \"description\":
> \"Invalid JSON payload received. Unknown name \\\"groupKey\\\" at
> 'message': Cannot find field.\"\n },\n {\n
> \"field\": \"message\",\n \"description\": \"Invalid JSON
> payload received. Unknown name \\\"truncatedAlerts\\\" at 'message': Cannot
> find field.\"\n }\n ]\n }\n ]\n }\n}\n"
>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/bc0c1939-fe6d-42ff-88bf-928b45f3bd8dn%40googlegroups.com.
