Then something strange is going on. Are you 100% sure that you pasted
*exactly* the configuration file you're using, with *exactly* the correct
contents and indentation?
You can add labels in your service discovery, e.g. if you are using
file_sd_configs then inside the targets file you can put
- labels:
owner: teamA
targets:
- foo
- bar
- baz
- labels:
owner: teamB
targets:
- qux
In this case, all timeseries which are scraped from the hosts will have
those labels added as attributes of every timeseries.
Or you can add labels in your alerting rules:
groups:
- name: UpDown
rules:
- alert: UpDownTeamA
expr: up{instance=~"foo|bar|baz"} == 0
for: 3m
keep_firing_for: 3m
labels:
owner: teamA
- alert: UpDownTeamB
expr: up{instance=~"qux"} == 0
for: 3m
keep_firing_for: 3m
labels:
owner: teamB
It's up to you on whether you want to label the timeseries themselves
(which is simpler but less flexible), or do the labelling in the alerting
rules (which means you may need to look at other labels to decide how to
route things, but allows you to change your alert routing without changing
the underlying timeseries)
In alertmanager you'd then match labels to decide where to deliver.
route:
# Default receiver (for all alerts that don't match any specific route)
receiver: "everyone"
routes:
- matchers:
- owner=teamA
receiver: "gmail"
- matchers:
- owner=teamB
receiver: "dev-receiver"
This simple example says alerts with owner=teamA go to "gmail", alerts with
owner=teamB go to "dev-receiver", and everything else goes to "everyone".
(The first match stops any subsequent processing, unless that rule has
"continue: true")
There is a tool you can test your alertmanager config with here:
https://prometheus.io/webtools/alerting/routing-tree-editor/
Try pasting in the above block, then click "Draw routing tree". Then next
to Match Label Set, enter
{instance="foo",owner="teamA"}
and click Match Label Set.
HTH.
On Monday, 6 January 2025 at 16:39:49 UTC Chrysale Tchako wrote:
>
> I believe this is the correct config file because when I mute an email
> from one of the receivers that email is no longer included in any
> notification that gets sent out. As far as the labels you mentioned
> earlier, could you share an example of how to implement or format those in
> my alertmanager.yml and alert.rules.yml files.
> On Friday, January 3, 2025 at 11:15:20 AM UTC-5 Brian Candler wrote:
>
>> I have retested this with alertmanager 0.25.0, and it still rejects the
>> config file you supplied as invalid, refusing to start. Therefore, I think
>> you're *not* running with the configuration file you think you are. This is
>> the first issue to resolve.
>>
>> You can prove this to yourself by running a separate instance of
>> alertmanager on the command line, pointing it explicitly at the config file
>> in question:
>>
>> # /opt/alertmanager-0.25.0.linux-amd64/alertmanager
>> --config.file=amtest.conf.orig --web.listen-address=:9193
>> --cluster.listen-address=0.0.0.0:9194
>> ts=2025-01-03T16:10:12.215Z caller=main.go:240 level=info msg="Starting
>> Alertmanager" version="(version=0.25.0, branch=HEAD,
>> revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)"
>> ts=2025-01-03T16:10:12.215Z caller=main.go:241 level=info
>> build_context="(go=go1.19.4, user=root@abe866dd5717,
>> date=20221222-14:51:36)"
>> ts=2025-01-03T16:10:12.217Z caller=cluster.go:185 level=info
>> component=cluster msg="setting advertise address explicitly"
>> addr=10.12.255.33 port=9194
>> ts=2025-01-03T16:10:12.219Z caller=cluster.go:681 level=info
>> component=cluster msg="Waiting for gossip to settle..." interval=2s
>> ts=2025-01-03T16:10:12.249Z caller=coordinator.go:113 level=info
>> component=configuration msg="Loading configuration file"
>> file=amtest.conf.orig
>> ts=2025-01-03T16:10:12.249Z caller=coordinator.go:118 level=error
>> component=configuration *msg="Loading configuration file failed"
>> file=amtest.conf.orig err="yaml: unmarshal errors:\n line 12: field routes
>> not found in type config.plain\n line 23: field routes not found in type
>> config.plain"*
>> ts=2025-01-03T16:10:12.249Z caller=cluster.go:690 level=info
>> component=cluster msg="gossip not settled but continuing anyway" polls=0
>> elapsed=30.684346ms
>>
>> (The default config file is "alertmanager.yml", but as there is no
>> absolute path, it will be in whatever the current working directory is when
>> alertmanager is started. This may not be where you think it is, and in
>> particular, is not necessarily the same directory as alertmanager itself)
>>
>> > Let's say I have Team A receiver for Team A resources and Team B
>> receiver for Team B resources, I want to isolate the notifications for each
>> team. Neither team should be aware of or get notified about the other teams
>> resources. How can I accomplish that?
>>
>> You set labels on the Team A / Team B alerts, and use those labels in
>> routing rules in alertmanager to route them appropriately. But there's no
>> point trying to do that until you've worked out why the system isn't using
>> the configuration file you think it is.
>>
>> On Friday, 3 January 2025 at 15:43:30 UTC Chrysale Tchako wrote:
>>
>>> Currently I am running alertmanager version 0.25.0, and I am making
>>> changes/modifications directly in the alertmanager.yml file, this is the
>>> file containing the information I shared in my initial inquiry. Or are
>>> referring to another file? I updated my routes config to match what you
>>> recommended but, email notifications are still getting sent to all emails
>>> in the alertmanager.yml file regardless of the receiver they are associated
>>> with. What I am trying to accomplish is a custom alert/notification for
>>> each receiver.
>>>
>>> Let's say I have Team A receiver for Team A resources and Team B
>>> receiver for Team B resources, I want to isolate the notifications for each
>>> team. Neither team should be aware of or get notified about the other teams
>>> resources. How can I accomplish that? What modifications should I make to
>>> my alertmanager.yml file? Also, do I need to make any modifications to
>>> my alert.rules.yml file or any other file that is?
>>>
>>> Please let me know if you need any additional information.
>>>
>>> On Thursday, January 2, 2025 at 11:56:08 AM UTC-5 Brian Candler wrote:
>>>
>>>> What version of alertmanager are you using? Your configuration is
>>>> invalid, so I don't know how you're even getting alertmanager to start.
>>>>
>>>> You have a top-level "routes:" block (before "inhibit_rules"), and you
>>>> have another "routes:" block nested under inhibit_rules, but neither of
>>>> these are allowed by Alertmanager. If I try to run your config with
>>>> alertmanager-0.27.0 I get an explicit error:
>>>>
>>>> ts=2025-01-02T16:41:05.905Z caller=coordinator.go:118 level=error
>>>> component=configuration msg="Loading configuration file failed"
>>>> file=amtest.conf err="yaml: unmarshal errors:\n* line 12: field
>>>> routes not found in type config.plain\n line 23: field routes not found
>>>> in
>>>> type config.plain*"
>>>>
>>>> It would be valid to nest "routes" under "route", like this:
>>>>
>>>> route:
>>>> group_wait: 1m
>>>> group_interval: 5m
>>>> repeat_interval: 12h
>>>> group_by: ["alertname", "severity"]
>>>> # Default receiver (for all alerts that don't match any specific
>>>> route)
>>>> receiver: "gmail"
>>>>
>>>> routes:
>>>> - receiver: "gmail"
>>>> continue: true
>>>> - receiver: "dev-receiver"
>>>>
>>>> Note the spacing: "routes:" is indented to line up with other
>>>> attributes like "receiver", not at the top level. Furthermore, if you did,
>>>> it could explain the behaviour you're seeing, since this configuration
>>>> requests all messages to be sent to both receivers. But since you say
>>>> that's *not* your actual configuration, I think we have to start somewhere
>>>> else.
>>>>
>>>> How are you getting your config into alertmanager? Are you editing the
>>>> alertmanager configuration directly, or is it coming via some frontend
>>>> like
>>>> a helm chart which spits out the config? If so, can you show the generated
>>>> config, which is the actual file that alertmanager consumes?
>>>>
>>>> > What changes can I make so that each receiver gets it's own
>>>> individual email notification instead of grouping them together?
>>>>
>>>> What do you mean by "individual email notification instead of grouping
>>>> them together"? Do you want some alerts to go to one receiver, and some
>>>> alerts to go to a different receiver? If so, how do you want to choose
>>>> which alerts go where?
>>>>
>>>> On Thursday, 2 January 2025 at 15:16:26 UTC Chrysale Tchako wrote:
>>>>
>>>>> This is my current alertmanager.yml config, but all notifications sent
>>>>> out get sent to all emails added in the "to" section of the email
>>>>> configs.
>>>>> What changes can I make so that each receiver gets it's own individual
>>>>> email notification instead of grouping them together? Note the "from"
>>>>> email
>>>>> is the same for both receivers.
>>>>>
>>>>> global:
>>>>> smtp_hello: '*******.net'
>>>>>
>>>>> route:
>>>>> group_wait: 1m
>>>>> group_interval: 5m
>>>>> repeat_interval: 12h
>>>>> group_by: ["alertname", "severity"]
>>>>> # Default receiver (for all alerts that don't match any specific
>>>>> route)
>>>>> receiver: "gmail"
>>>>>
>>>>> routes:
>>>>> - receiver: "gmail"
>>>>> continue: true
>>>>> - receiver: "dev-receiver"
>>>>>
>>>>> inhibit_rules:
>>>>> - source_matchers:
>>>>> - severity="critical"
>>>>> target_matchers:
>>>>> - severity="warning"
>>>>> equal: ['alertname', 'instance']
>>>>> routes:
>>>>> - receiver: "gmail"
>>>>> continue: true
>>>>> - receiver: "dev-receiver"
>>>>>
>>>>> receivers:
>>>>> # Default receiver for other alerts
>>>>> - name: "gmail"
>>>>> email_configs:
>>>>> - to: "email1.net, email2.net"
>>>>> from: '*******.net'
>>>>> smarthost: 'smtp-relay.gmail.com:587'
>>>>> send_resolved: true
>>>>>
>>>>> - name: "dev-receiver"
>>>>> email_configs:
>>>>> - to: 'email3.net'
>>>>> from: '*******.net'
>>>>> smarthost: 'smtp-relay.gmail.com:587'
>>>>> send_resolved: true
>>>>>
>>>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion visit
https://groups.google.com/d/msgid/prometheus-users/b18a66f5-61de-4a94-b3df-270c595614a8n%40googlegroups.com.
