Hi Brian,
Thank you for your excellent step of diagnostic steps you have provided. I
think I'm close to resolving this now.
ps auxwww | grep exporter
prometh+ 75709 0.0 0.2 1457064 16460 ? Ssl 09:59 0:00
/usr/bin/prometheus-node-exporter
Telnet fails to connect from a remote host.
However, the tcpdump indicates some sort of firewalling. Do you think I
need to add something else into iptables to make this work?
$ sudo tcpdump -i ens160 -nn tcp port 9100 or icmp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), snapshot length 262144
bytes
12:25:25.232349 IP 10.215.85.120.49725 > 10.4.121.23.9100: Flags [S], seq
239820249, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK],
length 0
12:25:25.232440 IP 10.4.121.23 > 10.215.85.120: ICMP host 10.4.121.23
unreachable - admin prohibited filter, length 60
12:25:26.245900 IP 10.215.85.120.49725 > 10.4.121.23.9100: Flags [S], seq
239820249, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK],
length 0
12:25:26.245983 IP 10.4.121.23 > 10.215.85.120: ICMP host 10.4.121.23
unreachable - admin prohibited filter, length 60
12:25:28.259816 IP 10.215.85.120.49725 > 10.4.121.23.9100: Flags [S], seq
239820249, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK],
length 0
12:25:28.259921 IP 10.4.121.23 > 10.215.85.120: ICMP host 10.4.121.23
unreachable - admin prohibited filter, length 60
12:25:32.273658 IP 10.215.85.120.49725 > 10.4.121.23.9100: Flags [S], seq
239820249, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK],
length 0
12:25:32.273727 IP 10.4.121.23 > 10.215.85.120: ICMP host 10.4.121.23
unreachable - admin prohibited filter, length 60
12:25:40.279317 IP 10.215.85.120.49725 > 10.4.121.23.9100: Flags [S], seq
239820249, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK],
length 0
12:25:40.279392 IP 10.4.121.23 > 10.215.85.120: ICMP host 10.4.121.23
unreachable - admin prohibited filter, length 60
12:25:50.768290 IP 10.4.10.217 > 10.4.121.23: ICMP echo request, id 11871,
seq 0, length 76
12:25:50.768409 IP 10.4.121.23 > 10.4.10.217: ICMP echo reply, id 11871,
seq 0, length 76
iptables -L -n -v shows the following . I have added a couple of rules to
make sure that port 9100 wasn't blocked.
The rest of the rules just seem related to docker.
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination
32 1664 tcp -- ens160 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:9100
91 4837 ACCEPT tcp -- * * 0.0.0.0/0
0.0.0.0/0 tcp dpt:9100
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination
28315 9833K DOCKER-USER all -- * * 0.0.0.0/0
0.0.0.0/0
28315 9833K DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0
0.0.0.0/0
0 0 ACCEPT all -- * docker0 0.0.0.0/0
0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- * docker0 0.0.0.0/0
0.0.0.0/0
0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0
0.0.0.0/0
0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0
0.0.0.0/0
199 91210 ACCEPT all -- * br-6f8875048c08 0.0.0.0/0
0.0.0.0/0 ctstate RELATED,ESTABLISHED
21 1092 DOCKER all -- * br-6f8875048c08 0.0.0.0/0
0.0.0.0/0
200 99188 ACCEPT all -- br-6f8875048c08 !br-6f8875048c08 0.0.0.0/0
0.0.0.0/0
0 0 ACCEPT all -- br-6f8875048c08 br-6f8875048c08 0.0.0.0/0
0.0.0.0/0
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination
Chain DOCKER (2 references)
pkts bytes target prot opt in out source
destination
15 780 ACCEPT tcp -- !br-6f8875048c08 br-6f8875048c08 0.0.0.0/0
172.19.0.2 tcp dpt:443
6 312 ACCEPT tcp -- !br-6f8875048c08 br-6f8875048c08 0.0.0.0/0
172.19.0.2 tcp dpt:80
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source
destination
0 0 DOCKER-ISOLATION-STAGE-2 all -- docker0 !docker0 0.0.0.0/0
0.0.0.0/0
200 99188 DOCKER-ISOLATION-STAGE-2 all -- br-6f8875048c08
!br-6f8875048c08 0.0.0.0/0 0.0.0.0/0
28315 9833K RETURN all -- * * 0.0.0.0/0
0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (2 references)
pkts bytes target prot opt in out source
destination
0 0 DROP all -- * docker0 0.0.0.0/0
0.0.0.0/0
0 0 DROP all -- * br-6f8875048c08 0.0.0.0/0
0.0.0.0/0
286 103K RETURN all -- * * 0.0.0.0/0
0.0.0.0/0
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source
destination
28315 9833K RETURN all -- * * 0.0.0.0/0
0.0.0.0/0
On Wednesday, 1 March 2023 at 11:35:07 UTC Brian Candler wrote:
> On Wednesday, 1 March 2023 at 11:11:19 UTC Nicholas Reid wrote:
>
> Has anyone else had trouble getting the node exporter working on Ubuntu
> 22.04.
>
>
> Nope, it works perfectly for me using the release binaries from github.
>
> There's another guide you can look at here, and see if you did anything
> differently:
> https://nsrc.org/workshops/2022/rwnog/nmm/netmgmt/en/prometheus/ex-node-exporter.html
>
>
> The service starts up OK and I can curl it locally
> (localhost:9100/metrics). However, when I try to access it remotely I'm not
> able to connect on 9100, or any port I set it to listen on .
>
>
> What does "ps auxwww | grep node_exporter" show on the host where
> node_exporter is running?
>
> What does "iptables -L -n -v" on the host where node_exporter is running
> show?
>
> What does "telnet x.x.x.x 9100" from a remote host show? (where x.x.x.x is
> the host where node_exporter is running)
>
> Also: try running "tcpdump -i eth0 -nn tcp port 9100 or icmp" on the
> node_exporter host (replace 'eth0' with the actual network interface),
> *then* run "telnet x.x.x.x 9100" from a remote host. What *exactly* does
> tcpdump show? The details of this will help determine the problem.
>
> (e.g. do you just see SYN packets arriving? Do you see SYNs followed by
> ICMP messages, if so what exactly is the ICMP type/code? Do you see SYNs
> followed by SYN ACK responses, but repeated SYNs?)
>
>
> I've confirmed that ufw is inactive, and the service has bound to port
> 9100 with netstat .
>
>
> It will need to be bound to :::9100 or 0.0.0.0:9100 to accept connections
> from outside on all interfaces. If it's bound to a single interface like
> 127.0.0.1:9100 then it will only accept connections on that address.
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/bfc2049c-d4fb-4f64-9943-9ecdfacd042an%40googlegroups.com.
