Thanks Brian, this is helpful. It seems like the "active" flag would be a nice way to take devices out of monitoring for long-term outages.
I'm curious what naming convention you use for devices in Netbox. Since device names are required to be unique within a site, I'm thinking about some short "computer-friendly" convention and then put a "human-readable" functional name in the "description" field. This functional name will be used to label Grafana panels, so it only needs to be unique within its local context (room, rack, workflow, etc., depending on the situation). Site-wide uniqueness would be excessively long and hard to read here. My problem is, the "description" field in Netbox is not exposed by the Prometheus SD plugin. So I'm wondering how other people approach this? On Thu, Dec 14, 2023 at 1:02 AM 'Brian Candler' via Prometheus Users < [email protected]> wrote: > Right now I use Netbox to control monitoring with node_exporter and > snmp_exporter. > > I have three custom fields on Device and VirtualMachine, all of which are > optional: > > * "monitoring" is a selection field with values "node", "snmp" and "icmp" > * "snmp_auth" is a selection field with the names of the configured auths > * "snmp_module" is a multi-select field with the names of the snmp modules > I use. (snmp_exporter now supports selection of multiple modules in a > single scrape, but sadly combined with how netbox-plugin-prometheus-sd > exposes lists, that makes the rewriting config messy) > > Prometheus configuration: > > - job_name: exporter > scrape_interval: 1m > metrics_path: /metrics > static_configs: > - targets: > - localhost:9115 # blackbox_exporter > - localhost:9116 # snmp_exporter > > - job_name: snmp > scrape_interval: 15s > http_sd_configs: > # > https://github.com/netbox-community/netbox/issues/11538#issuecomment-1635839720 > - url: > https://netbox.example.net/api/plugins/prometheus-sd/devices/?cf_monitoring=snmp&status=active > refresh_interval: 10m > authorization: > type: Token > credentials_file: /etc/prometheus/netbox.token > - url: > https://netbox.example.net/api/plugins/prometheus-sd/virtual-machines/?cf_monitoring=snmp&status=active > refresh_interval: 10m > authorization: > type: Token > credentials_file: /etc/prometheus/netbox.token > metrics_path: /snmp > relabel_configs: > # Labels which control scraping > - source_labels: [__address__] > target_label: instance > - source_labels: [__address__] > target_label: __param_target > - source_labels: [__meta_netbox_primary_ip] > regex: '(.+)' > target_label: __param_target > - source_labels: [__meta_netbox_custom_field_snmp_module] > target_label: __param_module > # Ugh: multiselect is of form ['foo', 'bar'] and we need foo,bar. > There is no gsub. > - source_labels: [__param_module] > regex: "\\['(.*)'\\]" > target_label: __param_module > - source_labels: [__param_module] > regex: "(.*)', *'(.*)" > replacement: "$1,$2" > target_label: __param_module > - source_labels: [__param_module] > regex: "(.*)', *'(.*)" > replacement: "$1,$2" > target_label: __param_module > - source_labels: [__param_module] > regex: "(.*)', *'(.*)" > replacement: "$1,$2" > target_label: __param_module > - source_labels: [__meta_netbox_custom_field_snmp_auth] > target_label: __param_auth > - target_label: __address__ > replacement: 127.0.0.1:9116 # SNMP exporter > # Optional extra metadata labels > - source_labels: [__param_module] > target_label: module > - source_labels: [__meta_netbox_cluster_slug] > target_label: cluster > - source_labels: [__meta_netbox_device_type_slug] > target_label: device_type > - source_labels: [__meta_netbox_model] > target_label: netbox_model > - source_labels: [__meta_netbox_platform_slug] > target_label: platform > - source_labels: [__meta_netbox_role_slug] > target_label: role > - source_labels: [__meta_netbox_site_slug] > target_label: site > - source_labels: [__meta_netbox_tag_slugs] > target_label: tags > - source_labels: [__meta_netbox_tenant_slug] > target_label: tenant > > - job_name: icmp > scrape_interval: 1m > http_sd_configs: > - url: > https://netbox.example.net/api/plugins/prometheus-sd/devices/?cf_monitoring=icmp&status=active > refresh_interval: 10m > authorization: > type: Token > credentials_file: /etc/prometheus/netbox.token > - url: > https://netbox.example.net/api/plugins/prometheus-sd/virtual-machines/?cf_monitoring=icmp&status=active > refresh_interval: 10m > authorization: > type: Token > credentials_file: /etc/prometheus/netbox.token > metrics_path: /probe > params: > module: [icmp] > relabel_configs: > # Labels which control scraping > - source_labels: [__address__] > target_label: instance > - source_labels: [__address__] > target_label: __param_target > - source_labels: [__meta_netbox_primary_ip] > regex: '(.+)' > target_label: __param_target > - target_label: __address__ > replacement: 127.0.0.1:9115 # Blackbox exporter > # Optional extra metadata labels > - source_labels: [__meta_netbox_cluster_slug] > target_label: cluster > - source_labels: [__meta_netbox_device_type_slug] > target_label: device_type > - source_labels: [__meta_netbox_model] > target_label: netbox_model > - source_labels: [__meta_netbox_platform_slug] > target_label: platform > - source_labels: [__meta_netbox_role_slug] > target_label: role > - source_labels: [__meta_netbox_site_slug] > target_label: site > - source_labels: [__meta_netbox_tag_slugs] > target_label: tags > - source_labels: [__meta_netbox_tenant_slug] > target_label: tenant > > - job_name: node > scrape_interval: 1m > scrape_timeout: 50s > http_sd_configs: > - url: > https://netbox.example.net/api/plugins/prometheus-sd/devices/?cf_monitoring=node&status=active > refresh_interval: 10m > authorization: > type: Token > credentials_file: /etc/prometheus/netbox.token > - url: > https://netbox.example.net/api/plugins/prometheus-sd/virtual-machines/?cf_monitoring=node&status=active > refresh_interval: 10m > authorization: > type: Token > credentials_file: /etc/prometheus/netbox.token > metrics_path: /metrics > relabel_configs: > # Labels which control scraping > - source_labels: [__address__] > target_label: instance > - source_labels: [__meta_netbox_primary_ip4] > regex: '(.+)' > target_label: __address__ > - source_labels: [__meta_netbox_primary_ip6] > regex: '(.+)' > target_label: __address__ > replacement: '[${1}]' > - source_labels: [__address__] > target_label: __address__ > replacement: '${1}:9100' > # Optional extra metadata labels > - source_labels: [__meta_netbox_cluster_slug] > target_label: cluster > - source_labels: [__meta_netbox_device_type_slug] > target_label: device_type > - source_labels: [__meta_netbox_model] > target_label: netbox_model > - source_labels: [__meta_netbox_platform_slug] > target_label: platform > - source_labels: [__meta_netbox_role_slug] > target_label: role > - source_labels: [__meta_netbox_site_slug] > target_label: site > - source_labels: [__meta_netbox_tag_slugs] > target_label: tags > - source_labels: [__meta_netbox_tenant_slug] > target_label: tenant > > # Monitoring of netbox itself > - job_name: netbox > scrape_interval: 2m > scheme: https > authorization: > type: Token > credentials_file: /etc/prometheus/netbox.token > static_configs: > - targets: ['netbox.example.net:443'] > > I added the "icmp" monitoring option only for completeness and haven't > really tested it yet. > > I'm still on static files for other blackbox_exporter tests though. > Although Netbox does have a "service" model, there are things I want to > monitor which are not attached to devices or VMs. I'm currently thinking > about using config contexts and/or attaching blackbox monitoring directly > to an IP address object rather than a device or VM. Discussion: > https://github.com/netbox-community/netbox/discussions/14261 > > (That shows a slightly older Prometheus config where I was using tags to > enable the various monitoring types; a custom field is actually simpler) > > On Thursday 14 December 2023 at 00:27:08 UTC Elliott Balsley wrote: > >> I've just started looking at Netbox as a way to store "inventory" for >> Prometheus using this plugin: >> https://github.com/FlxPeters/netbox-plugin-prometheus-sd >> >> The documentation is a bit light and I'm wondering if anyone can share a >> good example of how to use the multi-target exporter pattern. My initial >> thought is to add custom fields in netbox like this: >> *prom_job*: (multi-selection with choices like blackbox, snmp, json) >> *prom_blackbox_module*: (selection with choices like ping, http, >> tcp3389, etc.) >> *prom_snmp_module*: (selection with choices for all my snmp modules) >> *prom_snmp_auth*: (selection with choices like public_v2, public_v3, >> etc. to be used by snmp). >> >> Then in Prometheus, I would have just one blackbox job which chooses the >> right module based on that field, and its target URL includes a Netbox >> filter for prom_job to only return targets which have that module selected. >> >> Similarly, I would have one snmp job, etc. Does this sound like the best >> approach? >> > -- > You received this message because you are subscribed to a topic in the > Google Groups "Prometheus Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/prometheus-users/q99ZvG_ke9w/unsubscribe > . > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/b95127d2-c1f2-47d1-b6d2-1f1b8e2bc949n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/b95127d2-c1f2-47d1-b6d2-1f1b8e2bc949n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CALajkdgt383E46zF6F1rVOr%3DLjChEuOvsR1VFOtoHTJcAadzvA%40mail.gmail.com.
