Right now I use Netbox to control monitoring with node_exporter and
snmp_exporter.
I have three custom fields on Device and VirtualMachine, all of which are
optional:
* "monitoring" is a selection field with values "node", "snmp" and "icmp"
* "snmp_auth" is a selection field with the names of the configured auths
* "snmp_module" is a multi-select field with the names of the snmp modules
I use. (snmp_exporter now supports selection of multiple modules in a
single scrape, but sadly combined with how netbox-plugin-prometheus-sd
exposes lists, that makes the rewriting config messy)
Prometheus configuration:
- job_name: exporter
scrape_interval: 1m
metrics_path: /metrics
static_configs:
- targets:
- localhost:9115 # blackbox_exporter
- localhost:9116 # snmp_exporter
- job_name: snmp
scrape_interval: 15s
http_sd_configs:
#
https://github.com/netbox-community/netbox/issues/11538#issuecomment-1635839720
- url:
https://netbox.example.net/api/plugins/prometheus-sd/devices/?cf_monitoring=snmp&status=active
refresh_interval: 10m
authorization:
type: Token
credentials_file: /etc/prometheus/netbox.token
- url:
https://netbox.example.net/api/plugins/prometheus-sd/virtual-machines/?cf_monitoring=snmp&status=active
refresh_interval: 10m
authorization:
type: Token
credentials_file: /etc/prometheus/netbox.token
metrics_path: /snmp
relabel_configs:
# Labels which control scraping
- source_labels: [__address__]
target_label: instance
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__meta_netbox_primary_ip]
regex: '(.+)'
target_label: __param_target
- source_labels: [__meta_netbox_custom_field_snmp_module]
target_label: __param_module
# Ugh: multiselect is of form ['foo', 'bar'] and we need foo,bar.
There is no gsub.
- source_labels: [__param_module]
regex: "\\['(.*)'\\]"
target_label: __param_module
- source_labels: [__param_module]
regex: "(.*)', *'(.*)"
replacement: "$1,$2"
target_label: __param_module
- source_labels: [__param_module]
regex: "(.*)', *'(.*)"
replacement: "$1,$2"
target_label: __param_module
- source_labels: [__param_module]
regex: "(.*)', *'(.*)"
replacement: "$1,$2"
target_label: __param_module
- source_labels: [__meta_netbox_custom_field_snmp_auth]
target_label: __param_auth
- target_label: __address__
replacement: 127.0.0.1:9116 # SNMP exporter
# Optional extra metadata labels
- source_labels: [__param_module]
target_label: module
- source_labels: [__meta_netbox_cluster_slug]
target_label: cluster
- source_labels: [__meta_netbox_device_type_slug]
target_label: device_type
- source_labels: [__meta_netbox_model]
target_label: netbox_model
- source_labels: [__meta_netbox_platform_slug]
target_label: platform
- source_labels: [__meta_netbox_role_slug]
target_label: role
- source_labels: [__meta_netbox_site_slug]
target_label: site
- source_labels: [__meta_netbox_tag_slugs]
target_label: tags
- source_labels: [__meta_netbox_tenant_slug]
target_label: tenant
- job_name: icmp
scrape_interval: 1m
http_sd_configs:
- url:
https://netbox.example.net/api/plugins/prometheus-sd/devices/?cf_monitoring=icmp&status=active
refresh_interval: 10m
authorization:
type: Token
credentials_file: /etc/prometheus/netbox.token
- url:
https://netbox.example.net/api/plugins/prometheus-sd/virtual-machines/?cf_monitoring=icmp&status=active
refresh_interval: 10m
authorization:
type: Token
credentials_file: /etc/prometheus/netbox.token
metrics_path: /probe
params:
module: [icmp]
relabel_configs:
# Labels which control scraping
- source_labels: [__address__]
target_label: instance
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__meta_netbox_primary_ip]
regex: '(.+)'
target_label: __param_target
- target_label: __address__
replacement: 127.0.0.1:9115 # Blackbox exporter
# Optional extra metadata labels
- source_labels: [__meta_netbox_cluster_slug]
target_label: cluster
- source_labels: [__meta_netbox_device_type_slug]
target_label: device_type
- source_labels: [__meta_netbox_model]
target_label: netbox_model
- source_labels: [__meta_netbox_platform_slug]
target_label: platform
- source_labels: [__meta_netbox_role_slug]
target_label: role
- source_labels: [__meta_netbox_site_slug]
target_label: site
- source_labels: [__meta_netbox_tag_slugs]
target_label: tags
- source_labels: [__meta_netbox_tenant_slug]
target_label: tenant
- job_name: node
scrape_interval: 1m
scrape_timeout: 50s
http_sd_configs:
- url:
https://netbox.example.net/api/plugins/prometheus-sd/devices/?cf_monitoring=node&status=active
refresh_interval: 10m
authorization:
type: Token
credentials_file: /etc/prometheus/netbox.token
- url:
https://netbox.example.net/api/plugins/prometheus-sd/virtual-machines/?cf_monitoring=node&status=active
refresh_interval: 10m
authorization:
type: Token
credentials_file: /etc/prometheus/netbox.token
metrics_path: /metrics
relabel_configs:
# Labels which control scraping
- source_labels: [__address__]
target_label: instance
- source_labels: [__meta_netbox_primary_ip4]
regex: '(.+)'
target_label: __address__
- source_labels: [__meta_netbox_primary_ip6]
regex: '(.+)'
target_label: __address__
replacement: '[${1}]'
- source_labels: [__address__]
target_label: __address__
replacement: '${1}:9100'
# Optional extra metadata labels
- source_labels: [__meta_netbox_cluster_slug]
target_label: cluster
- source_labels: [__meta_netbox_device_type_slug]
target_label: device_type
- source_labels: [__meta_netbox_model]
target_label: netbox_model
- source_labels: [__meta_netbox_platform_slug]
target_label: platform
- source_labels: [__meta_netbox_role_slug]
target_label: role
- source_labels: [__meta_netbox_site_slug]
target_label: site
- source_labels: [__meta_netbox_tag_slugs]
target_label: tags
- source_labels: [__meta_netbox_tenant_slug]
target_label: tenant
# Monitoring of netbox itself
- job_name: netbox
scrape_interval: 2m
scheme: https
authorization:
type: Token
credentials_file: /etc/prometheus/netbox.token
static_configs:
- targets: ['netbox.example.net:443']
I added the "icmp" monitoring option only for completeness and haven't
really tested it yet.
I'm still on static files for other blackbox_exporter tests though.
Although Netbox does have a "service" model, there are things I want to
monitor which are not attached to devices or VMs. I'm currently thinking
about using config contexts and/or attaching blackbox monitoring directly
to an IP address object rather than a device or VM. Discussion:
https://github.com/netbox-community/netbox/discussions/14261
(That shows a slightly older Prometheus config where I was using tags to
enable the various monitoring types; a custom field is actually simpler)
On Thursday 14 December 2023 at 00:27:08 UTC Elliott Balsley wrote:
> I've just started looking at Netbox as a way to store "inventory" for
> Prometheus using this plugin:
> https://github.com/FlxPeters/netbox-plugin-prometheus-sd
>
> The documentation is a bit light and I'm wondering if anyone can share a
> good example of how to use the multi-target exporter pattern. My initial
> thought is to add custom fields in netbox like this:
> *prom_job*: (multi-selection with choices like blackbox, snmp, json)
> *prom_blackbox_module*: (selection with choices like ping, http, tcp3389,
> etc.)
> *prom_snmp_module*: (selection with choices for all my snmp modules)
> *prom_snmp_auth*: (selection with choices like public_v2, public_v3, etc.
> to be used by snmp).
>
> Then in Prometheus, I would have just one blackbox job which chooses the
> right module based on that field, and its target URL includes a Netbox
> filter for prom_job to only return targets which have that module selected.
>
> Similarly, I would have one snmp job, etc. Does this sound like the best
> approach?
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/b95127d2-c1f2-47d1-b6d2-1f1b8e2bc949n%40googlegroups.com.
