There are a variety of tools for this: - tools which capture packets on the interface and classify them into their own format: e.g. packetbeat <https://www.elastic.co/beats/packetbeat>, SiLK <https://tools.netsa.cert.org/silk/> + flowviewer <http://flowviewer.net/>, pmacct <http://www.pmacct.net/>, NTOPng <https://www.ntop.org/products/traffic-analysis/ntop/> [some features of the latter are commercial only]
- tools which capture packets on the interface and turn them into standard Netflow/IPFIX records: e.g. softflowd <https://github.com/irino/softflowd>, goflow <https://github.com/cloudflare/goflow> => you then also need something which stores and queries these flow records, e.g. elastiflow <https://github.com/robcowart/elastiflow>, nfdump <https://github.com/phaag/nfdump>/nfsen <http://nfsen.sourceforge.net/> - IDS tools which perform deeper packet analysis, e.g. zeek (formerly bro <https://bricata.com/blog/bro-ids-renames-zeek-ids/>) - or you can just capture raw packets on the wire using tcpdump/wireshark and analyse them later, e.g. with packetq <https://www.dns-oarc.net/tools/packetq> Most of these don't work directly with prometheus, because of the high cardinality issue that Julius highlighted. You will need a proper log storage backend like elasticsearch or loki. However, I have managed to get one of them to work in a prometheus-exporter type of way, and that's pmacct. This could be useful if you can pre-define the ranges of addresses that you want to expose, so that you have limited cardinality. To make this work I used exporter_exporter <https://github.com/QubitProducts/exporter_exporter> to run a small bit of python code (included below) to query pmacct's own in-RAM counter store. pmacct is configured to aggregate non-local source and destination addresses. With this configuration, the results you get look like this: # TYPE flow_bytes untyped flow_bytes{net_src="::",ip_dst="2001:db8:1234:8fa::2",aggregate="inbound"} 7695 flow_bytes{net_src="0.0.0.0",ip_dst="10.12.255.33",aggregate="inbound"} 566 flow_bytes{net_src="0.0.0.0",ip_dst="10.12.255.1",aggregate="inbound"} 581964 flow_bytes{net_src="::",ip_dst="2001:db8:1234:800::1",aggregate="inbound"} 10883 flow_bytes{net_src="::",ip_dst="2001:db8:1234:8ff::32",aggregate="inbound"} 1300 flow_bytes{net_src="::",ip_dst="2001:db8:1234:8f9::4",aggregate="inbound"} 2138 flow_bytes{net_src="0.0.0.0",ip_dst="192.0.2.234",aggregate="inbound"} 530 flow_bytes{net_src="::",ip_dst="2001:db8:1234:8ff::58",aggregate="inbound"} 912 flow_bytes{net_src="0.0.0.0",ip_dst="10.12.255.47",aggregate="inbound"} 84 flow_bytes{net_src="0.0.0.0",ip_dst="192.0.2.233",aggregate="inbound"} 12084 flow_bytes{net_src="::",ip_dst="2001:db8:1234:8ff::33",aggregate="inbound"} 107517 flow_bytes{net_src="::",ip_dst="2001:db8:1234:800:e809:5c7e:b33c:4f48",aggregate="inbound"} 1796 flow_bytes{ip_src="10.12.255.31",net_dst="0.0.0.0",aggregate="outbound"} 1981 flow_bytes{ip_src="2001:db8:1234:8ff::58",net_dst="::",aggregate="outbound"} 904 flow_bytes{ip_src="10.12.255.56",net_dst="0.0.0.0",aggregate="outbound"} 52 flow_bytes{ip_src="2001:db8:1234:8fa::2",net_dst="::",aggregate="outbound"} 43842 flow_bytes{ip_src="2001:db8:1234:800::1",net_dst="::",aggregate="outbound"} 63675 flow_bytes{ip_src="10.12.255.254",net_dst="0.0.0.0",aggregate="outbound"} 209 flow_bytes{ip_src="2001:db8:1234:8ff::33",net_dst="::",aggregate="outbound"} 18578 flow_bytes{ip_src="2001:db8:1234:800:e809:5c7e:b33c:4f48",net_dst="::",aggregate="outbound"} 1709 flow_bytes{ip_src="10.12.0.100",net_dst="0.0.0.0",aggregate="outbound"} 4250 flow_bytes{ip_src="10.12.255.33",net_dst="0.0.0.0",aggregate="outbound"} 2887 flow_bytes{ip_src="192.0.2.234",net_dst="0.0.0.0",aggregate="outbound"} 566 flow_bytes{ip_src="10.12.255.1",net_dst="0.0.0.0",aggregate="outbound"} 84 flow_bytes{ip_src="2001:db8:1234:8f9::4",net_dst="::",aggregate="outbound"} 2140 flow_bytes{ip_src="2001:db8:1234:8ff::32",net_dst="::",aggregate="outbound"} 1298 flow_bytes{ip_src="10.12.255.47",net_dst="0.0.0.0",aggregate="outbound"} 581964 That is: for "inbound" traffic, the individual destination addresses are shown, and the source is always 0.0.0.0 or :: (i.e. "the Intenet"), aggregated together. For "outbound" traffic, the destination is always 0.0.0.0 or ::, and individual source IP addresses are shown. It lets you answer questions like "which devices on my local network are sending (or receiving) the most traffic?", and the cardinality is limited by the number of active local IPs you have. But it won't answer questions like "where on the Internet is my outbound traffic going?"; that's too high cardinality for Prometheus. The configs to achieve this are below. Note: I was using nfacctd to receive Netflow records generated by my router. If you want to sniff packets going through a Linux server interface, then you'll need to run pmacctd instead of nfacctd. More info here <https://github.com/pmacct/pmacct/blob/master/QUICKSTART>. *# On pmacct server## /etc/pmacct/nfacctd.conf* nfacctd_port: 2055 plugins: memory[inbound], memory[outbound] imt_path[inbound]: /tmp/inbound.pipe aggregate_filter[inbound]: dst net 10.0.0.0/8 or dst net 192.0.2.232/30 or dst net 2001:db8:1234::/56 aggregate[inbound]: dst_host imt_mem_pools_number[inbound]: 64 imt_mem_pools_size[inbound]: 65536 imt_path[outbound]: /tmp/outbound.pipe aggregate_filter[outbound]: src net 10.0.0.0/8 or src net 192.0.2.232/30 or src net 2001:db8:1234::/56 aggregate[outbound]: src_host imt_mem_pools_number[outbound]: 64 imt_mem_pools_size[outbound]: 65536 *## /etc/systemd/system/nfacct.service* [Unit] Description=nfacctd Documentation=https://github.com/pmacct/pmacct/wiki After=network-online.target [Service] User=nobody Group=nogroup ExecStart=/usr/sbin/nfacctd -f /etc/pmacct/nfacctd.conf Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target *## /etc/prometheus/expexp.yaml* modules: pmacct: method: exec timeout: 5s exec: command: /usr/local/bin/pmacct.py *## /etc/systemd/system/exporter_exporter.service* [Unit] Description=Prometheus exporter proxy Documentation=https://github.com/QubitProducts/exporter_exporter After=network-online.target [Service] User=nobody Group=nogroup ExecStart=/usr/local/bin/exporter_exporter -config.file /etc/prometheus/expexp.yaml Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target *## /usr/local/bin/pmacct.py* #!/usr/bin/python3 import json import subprocess LABELS={} # add any static labels here, eg hostname def export(metric, labels, value): lstr = ",".join(("%s=\"%s\"" % (k,v) for k,v in labels.items())) print("%s{%s} %d" % (metric, lstr, value)) for aggregate in ["inbound", "outbound"]: res = subprocess.run(["pmacct", "-s", "-p", "/tmp/%s.pipe" % aggregate, "-O", "json"], stdout=subprocess.PIPE) if res.returncode: print(res.stdout) res.check_returncode() for line in res.stdout.splitlines(): data = json.loads(line) b = data.pop("bytes") p = data.pop("packets") data.update(LABELS) data["aggregate"] = aggregate export("flow_bytes", data, b) export("flow_packets", data, p) *## Test* curl localhost:9999/proxy?module=pmacct *# On prometheus server* - job_name: pmacct scrape_interval: 1m metrics_path: /proxy params: module: [pmacct] static_configs: - targets: - pmacct.example.net:9999 > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f14d0014-49f0-48c6-bd49-14a4a79877a6n%40googlegroups.com.
