Thanks a lot juliu...@ @ Brian Candler for valuable answer and time. Let me go through
Thanks once again. Thanks DV On Saturday, June 4, 2022 at 5:32:33 PM UTC+5:30 Brian Candler wrote: > There are a variety of tools for this: > > - tools which capture packets on the interface and classify them into > their own format: e.g. packetbeat > <https://www.elastic.co/beats/packetbeat>, SiLK > <https://tools.netsa.cert.org/silk/> + flowviewer <http://flowviewer.net/>, > pmacct <http://www.pmacct.net/>, NTOPng > <https://www.ntop.org/products/traffic-analysis/ntop/> [some features of > the latter are commercial only] > > - tools which capture packets on the interface and turn them into standard > Netflow/IPFIX records: e.g. softflowd <https://github.com/irino/softflowd>, > goflow <https://github.com/cloudflare/goflow> > => you then also need something which stores and queries these flow > records, e.g. elastiflow <https://github.com/robcowart/elastiflow>, nfdump > <https://github.com/phaag/nfdump>/nfsen <http://nfsen.sourceforge.net/> > > - IDS tools which perform deeper packet analysis, e.g. zeek (formerly bro > <https://bricata.com/blog/bro-ids-renames-zeek-ids/>) > > - or you can just capture raw packets on the wire using tcpdump/wireshark > and analyse them later, e.g. with packetq > <https://www.dns-oarc.net/tools/packetq> > > Most of these don't work directly with prometheus, because of the high > cardinality issue that Julius highlighted. You will need a proper log > storage backend like elasticsearch or loki. > > However, I have managed to get one of them to work in a > prometheus-exporter type of way, and that's pmacct. This could be useful > if you can pre-define the ranges of addresses that you want to expose, so > that you have limited cardinality. > > To make this work I used exporter_exporter > <https://github.com/QubitProducts/exporter_exporter> to run a small bit > of python code (included below) to query pmacct's own in-RAM counter > store. pmacct is configured to aggregate non-local source and destination > addresses. With this configuration, the results you get look like this: > > # TYPE flow_bytes untyped > flow_bytes{net_src="::",ip_dst="2001:db8:1234:8fa::2",aggregate="inbound"} > 7695 > flow_bytes{net_src="0.0.0.0",ip_dst="10.12.255.33",aggregate="inbound"} 566 > flow_bytes{net_src="0.0.0.0",ip_dst="10.12.255.1",aggregate="inbound"} > 581964 > flow_bytes{net_src="::",ip_dst="2001:db8:1234:800::1",aggregate="inbound"} > 10883 > flow_bytes{net_src="::",ip_dst="2001:db8:1234:8ff::32",aggregate="inbound"} > 1300 > flow_bytes{net_src="::",ip_dst="2001:db8:1234:8f9::4",aggregate="inbound"} > 2138 > flow_bytes{net_src="0.0.0.0",ip_dst="192.0.2.234",aggregate="inbound"} 530 > flow_bytes{net_src="::",ip_dst="2001:db8:1234:8ff::58",aggregate="inbound"} > 912 > flow_bytes{net_src="0.0.0.0",ip_dst="10.12.255.47",aggregate="inbound"} 84 > flow_bytes{net_src="0.0.0.0",ip_dst="192.0.2.233",aggregate="inbound"} > 12084 > flow_bytes{net_src="::",ip_dst="2001:db8:1234:8ff::33",aggregate="inbound"} > 107517 > flow_bytes{net_src="::",ip_dst="2001:db8:1234:800:e809:5c7e:b33c:4f48",aggregate="inbound"} > > 1796 > flow_bytes{ip_src="10.12.255.31",net_dst="0.0.0.0",aggregate="outbound"} > 1981 > flow_bytes{ip_src="2001:db8:1234:8ff::58",net_dst="::",aggregate="outbound"} > 904 > flow_bytes{ip_src="10.12.255.56",net_dst="0.0.0.0",aggregate="outbound"} 52 > flow_bytes{ip_src="2001:db8:1234:8fa::2",net_dst="::",aggregate="outbound"} > 43842 > flow_bytes{ip_src="2001:db8:1234:800::1",net_dst="::",aggregate="outbound"} > 63675 > flow_bytes{ip_src="10.12.255.254",net_dst="0.0.0.0",aggregate="outbound"} > 209 > flow_bytes{ip_src="2001:db8:1234:8ff::33",net_dst="::",aggregate="outbound"} > 18578 > flow_bytes{ip_src="2001:db8:1234:800:e809:5c7e:b33c:4f48",net_dst="::",aggregate="outbound"} > > 1709 > flow_bytes{ip_src="10.12.0.100",net_dst="0.0.0.0",aggregate="outbound"} > 4250 > flow_bytes{ip_src="10.12.255.33",net_dst="0.0.0.0",aggregate="outbound"} > 2887 > flow_bytes{ip_src="192.0.2.234",net_dst="0.0.0.0",aggregate="outbound"} 566 > flow_bytes{ip_src="10.12.255.1",net_dst="0.0.0.0",aggregate="outbound"} 84 > flow_bytes{ip_src="2001:db8:1234:8f9::4",net_dst="::",aggregate="outbound"} > 2140 > flow_bytes{ip_src="2001:db8:1234:8ff::32",net_dst="::",aggregate="outbound"} > 1298 > flow_bytes{ip_src="10.12.255.47",net_dst="0.0.0.0",aggregate="outbound"} > 581964 > > That is: for "inbound" traffic, the individual destination addresses are > shown, and the source is always 0.0.0.0 or :: (i.e. "the Intenet"), > aggregated together. For "outbound" traffic, the destination is always > 0.0.0.0 or ::, and individual source IP addresses are shown. It lets you > answer questions like "which devices on my local network are sending (or > receiving) the most traffic?", and the cardinality is limited by the number > of active local IPs you have. But it won't answer questions like "where on > the Internet is my outbound traffic going?"; that's too high cardinality > for Prometheus. > > The configs to achieve this are below. Note: I was using nfacctd to > receive Netflow records generated by my router. If you want to sniff > packets going through a Linux server interface, then you'll need to run > pmacctd instead of nfacctd. More info here > <https://github.com/pmacct/pmacct/blob/master/QUICKSTART>. > > > > *# On pmacct server## /etc/pmacct/nfacctd.conf* > > nfacctd_port: 2055 > plugins: memory[inbound], memory[outbound] > > imt_path[inbound]: /tmp/inbound.pipe > aggregate_filter[inbound]: dst net 10.0.0.0/8 or dst net 192.0.2.232/30 > or dst net 2001:db8:1234::/56 > aggregate[inbound]: dst_host > imt_mem_pools_number[inbound]: 64 > imt_mem_pools_size[inbound]: 65536 > > imt_path[outbound]: /tmp/outbound.pipe > aggregate_filter[outbound]: src net 10.0.0.0/8 or src net 192.0.2.232/30 > or src net 2001:db8:1234::/56 > aggregate[outbound]: src_host > imt_mem_pools_number[outbound]: 64 > imt_mem_pools_size[outbound]: 65536 > > *## /etc/systemd/system/nfacct.service* > > [Unit] > Description=nfacctd > Documentation=https://github.com/pmacct/pmacct/wiki > After=network-online.target > > [Service] > User=nobody > Group=nogroup > ExecStart=/usr/sbin/nfacctd -f /etc/pmacct/nfacctd.conf > Restart=on-failure > RestartSec=5 > > [Install] > WantedBy=multi-user.target > > *## /etc/prometheus/expexp.yaml* > > modules: > pmacct: > method: exec > timeout: 5s > exec: > command: /usr/local/bin/pmacct.py > > *## /etc/systemd/system/exporter_exporter.service* > > [Unit] > Description=Prometheus exporter proxy > Documentation=https://github.com/QubitProducts/exporter_exporter > After=network-online.target > > [Service] > User=nobody > Group=nogroup > ExecStart=/usr/local/bin/exporter_exporter -config.file > /etc/prometheus/expexp.yaml > Restart=on-failure > RestartSec=5 > > [Install] > WantedBy=multi-user.target > > *## /usr/local/bin/pmacct.py* > > #!/usr/bin/python3 > > import json > import subprocess > > LABELS={} # add any static labels here, eg hostname > > def export(metric, labels, value): > lstr = ",".join(("%s=\"%s\"" % (k,v) for k,v in labels.items())) > print("%s{%s} %d" % (metric, lstr, value)) > > for aggregate in ["inbound", "outbound"]: > res = subprocess.run(["pmacct", "-s", "-p", "/tmp/%s.pipe" % > aggregate, "-O", "json"], > stdout=subprocess.PIPE) > if res.returncode: > print(res.stdout) > res.check_returncode() > for line in res.stdout.splitlines(): > data = json.loads(line) > b = data.pop("bytes") > p = data.pop("packets") > data.update(LABELS) > data["aggregate"] = aggregate > export("flow_bytes", data, b) > export("flow_packets", data, p) > > *## Test* > > curl localhost:9999/proxy?module=pmacct > > *# On prometheus server* > > - job_name: pmacct > scrape_interval: 1m > metrics_path: /proxy > params: > module: [pmacct] > static_configs: > - targets: > - pmacct.example.net:9999 > >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0644a9d6-db46-489d-a06c-e70250121d6cn%40googlegroups.com.
