I have retested this with alertmanager 0.25.0, and it still rejects the config file you supplied as invalid, refusing to start. Therefore, I think you're *not* running with the configuration file you think you are. This is the first issue to resolve.
You can prove this to yourself by running a separate instance of alertmanager on the command line, pointing it explicitly at the config file in question: # /opt/alertmanager-0.25.0.linux-amd64/alertmanager --config.file=amtest.conf.orig --web.listen-address=:9193 --cluster.listen-address=0.0.0.0:9194 ts=2025-01-03T16:10:12.215Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" ts=2025-01-03T16:10:12.215Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" ts=2025-01-03T16:10:12.217Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=10.12.255.33 port=9194 ts=2025-01-03T16:10:12.219Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s ts=2025-01-03T16:10:12.249Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=amtest.conf.orig ts=2025-01-03T16:10:12.249Z caller=coordinator.go:118 level=error component=configuration *msg="Loading configuration file failed" file=amtest.conf.orig err="yaml: unmarshal errors:\n line 12: field routes not found in type config.plain\n line 23: field routes not found in type config.plain"* ts=2025-01-03T16:10:12.249Z caller=cluster.go:690 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=30.684346ms (The default config file is "alertmanager.yml", but as there is no absolute path, it will be in whatever the current working directory is when alertmanager is started. This may not be where you think it is, and in particular, is not necessarily the same directory as alertmanager itself) > Let's say I have Team A receiver for Team A resources and Team B receiver for Team B resources, I want to isolate the notifications for each team. Neither team should be aware of or get notified about the other teams resources. How can I accomplish that? You set labels on the Team A / Team B alerts, and use those labels in routing rules in alertmanager to route them appropriately. But there's no point trying to do that until you've worked out why the system isn't using the configuration file you think it is. On Friday, 3 January 2025 at 15:43:30 UTC Chrysale Tchako wrote: > Currently I am running alertmanager version 0.25.0, and I am making > changes/modifications directly in the alertmanager.yml file, this is the > file containing the information I shared in my initial inquiry. Or are > referring to another file? I updated my routes config to match what you > recommended but, email notifications are still getting sent to all emails > in the alertmanager.yml file regardless of the receiver they are associated > with. What I am trying to accomplish is a custom alert/notification for > each receiver. > > Let's say I have Team A receiver for Team A resources and Team B receiver > for Team B resources, I want to isolate the notifications for each team. > Neither team should be aware of or get notified about the other teams > resources. How can I accomplish that? What modifications should I make to > my alertmanager.yml file? Also, do I need to make any modifications to > my alert.rules.yml file or any other file that is? > > Please let me know if you need any additional information. > > On Thursday, January 2, 2025 at 11:56:08 AM UTC-5 Brian Candler wrote: > >> What version of alertmanager are you using? Your configuration is >> invalid, so I don't know how you're even getting alertmanager to start. >> >> You have a top-level "routes:" block (before "inhibit_rules"), and you >> have another "routes:" block nested under inhibit_rules, but neither of >> these are allowed by Alertmanager. If I try to run your config with >> alertmanager-0.27.0 I get an explicit error: >> >> ts=2025-01-02T16:41:05.905Z caller=coordinator.go:118 level=error >> component=configuration msg="Loading configuration file failed" >> file=amtest.conf err="yaml: unmarshal errors:\n* line 12: field routes >> not found in type config.plain\n line 23: field routes not found in type >> config.plain*" >> >> It would be valid to nest "routes" under "route", like this: >> >> route: >> group_wait: 1m >> group_interval: 5m >> repeat_interval: 12h >> group_by: ["alertname", "severity"] >> # Default receiver (for all alerts that don't match any specific route) >> receiver: "gmail" >> >> routes: >> - receiver: "gmail" >> continue: true >> - receiver: "dev-receiver" >> >> Note the spacing: "routes:" is indented to line up with other attributes >> like "receiver", not at the top level. Furthermore, if you did, it could >> explain the behaviour you're seeing, since this configuration requests all >> messages to be sent to both receivers. But since you say that's *not* your >> actual configuration, I think we have to start somewhere else. >> >> How are you getting your config into alertmanager? Are you editing the >> alertmanager configuration directly, or is it coming via some frontend like >> a helm chart which spits out the config? If so, can you show the generated >> config, which is the actual file that alertmanager consumes? >> >> > What changes can I make so that each receiver gets it's own individual >> email notification instead of grouping them together? >> >> What do you mean by "individual email notification instead of grouping >> them together"? Do you want some alerts to go to one receiver, and some >> alerts to go to a different receiver? If so, how do you want to choose >> which alerts go where? >> >> On Thursday, 2 January 2025 at 15:16:26 UTC Chrysale Tchako wrote: >> >>> This is my current alertmanager.yml config, but all notifications sent >>> out get sent to all emails added in the "to" section of the email configs. >>> What changes can I make so that each receiver gets it's own individual >>> email notification instead of grouping them together? Note the "from" email >>> is the same for both receivers. >>> >>> global: >>> smtp_hello: '*******.net' >>> >>> route: >>> group_wait: 1m >>> group_interval: 5m >>> repeat_interval: 12h >>> group_by: ["alertname", "severity"] >>> # Default receiver (for all alerts that don't match any specific route) >>> receiver: "gmail" >>> >>> routes: >>> - receiver: "gmail" >>> continue: true >>> - receiver: "dev-receiver" >>> >>> inhibit_rules: >>> - source_matchers: >>> - severity="critical" >>> target_matchers: >>> - severity="warning" >>> equal: ['alertname', 'instance'] >>> routes: >>> - receiver: "gmail" >>> continue: true >>> - receiver: "dev-receiver" >>> >>> receivers: >>> # Default receiver for other alerts >>> - name: "gmail" >>> email_configs: >>> - to: "email1.net, email2.net" >>> from: '*******.net' >>> smarthost: 'smtp-relay.gmail.com:587' >>> send_resolved: true >>> >>> - name: "dev-receiver" >>> email_configs: >>> - to: 'email3.net' >>> from: '*******.net' >>> smarthost: 'smtp-relay.gmail.com:587' >>> send_resolved: true >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/prometheus-users/36ef364b-9d41-411f-b005-e5122393730an%40googlegroups.com.
