*> The main issue is that I am not able to see the metrics in Prometheus UI. I am unable to understand the root cause.*
Almost certainly because they are not being collected. So you need to break this down further to find out why. (1) Show the output of this command: *ls -l /var/lib/node_exporter* * If the filenames don't end with ".prom" then that is your root cause. node_exporter only picks up files whose names match the pattern *.prom * If the files aren't readable by the user that node_exporter is running as, then that's the root cause. * Are you running under RedHat/CentOS with SELinux? If so, try putting it into permissive mode. Maybe there's some SELinux policy which prevents access to files in /var/lib/node_exporter. (2) Then show the contents of one of those files: *head /var/lib/node_exporter/foobar.prom* If the files are not in valid openmetrics format, then they won't be picked up either. You can test them with "promtool check metrics": /path/to/promtool check metrics </var/lib/node_exporter/foobar.prom If that shows an error, then modify your script to generate files in the correct format. There are examples you can copy from here: https://github.com/prometheus-community/node-exporter-textfile-collector-scripts (3) Show the command line that you're running node_exporter with: *ps auxwww | grep node_exporter* If the flag "--collector.textfile.directory=/var/lib/node_exporter" is not present, then that's your root cause. (4) Show the output of this command: *curl localhost:9100/metrics | grep foobar* where "foobar" is one of the custom metrics you are generating. Does it appear here? If not, then you can rule out any issues with prometheus itself, it's definitely node_exporter not picking up your custom metrics. If it does, then you'll need to dig further onto prometheus side (e.g. your scrape job config) *> In /var/lib/node_exporter --> In this folder, the scripts(custom metrics script) will be available.* You don't want the scripts in that directory - only the *output* from those scripts. However, as long as the scripts themselves don't end with ".prom" then they will be ignored, so that won't be a problem. On Wednesday, 1 March 2023 at 05:24:40 UTC BHARATH KUMAR wrote: > The main issue is that I am not able to see the metrics in Prometheus UI. > I am unable to understand the root cause. > > Am I missing anything? > > On Tuesday, 28 February 2023 at 23:44:59 UTC+5:30 Brian Candler wrote: > >> Aside: you might also want to look at kube-state-metrics >> <https://github.com/kubernetes/kube-state-metrics>, which can give you a >> range of pod metrics >> <https://github.com/kubernetes/kube-state-metrics/blob/main/docs/pod-metrics.md> >> >> including the pod start time. Then an expression like >> time()-kube_pod_start_time should give you the pod age. >> >> On Tuesday, 28 February 2023 at 17:55:20 UTC Brian Candler wrote: >> >>> The problem is that the filename must end with ".prom", i.e. you need to >>> create "apt1.prom" not "apt1-prom" >>> >>> Also, you need to enable the functionality with a flag to node_exporter: >>> >>> node_exporter --collector.textfile.directory=/var/lib/node_exporter >>> >>> See the documentation here: >>> https://github.com/prometheus/node_exporter#textfile-collector >>> >>> On Tuesday, 28 February 2023 at 17:16:06 UTC Stuart Clark wrote: >>> >>>> On 28/02/2023 17:12, BHARATH KUMAR wrote: >>>> > Hello All, >>>> > >>>> > I want to calculate the pod age running on every server. I wrote a >>>> > shell script. I added this script under the folder >>>> /var/lib/node_exporter. >>>> > >>>> > I created a cronjob for this script to run every minute >>>> > * * * * * root bash /var/lib/node_exporter/custom_metrics.sh > >>>> > /var/lib/node_exporter/apt1-prom >>>> > >>>> > I store the above cronjob in the /etc/cron.d folder with filename: >>>> > prom-apt1. >>>> > >>>> > But I am not able to see the metrics I created in Prometheus UI. >>>> > >>>> > >>>> > >>>> > But similarly, I created another shell script file to fetch some >>>> > metrics. I followed the same procedure as above. >>>> > >>>> > * * * * * root bash /var/lib/node_exporter/custom1.sh > >>>> > /var/lib/node_exporter/apt-prom >>>> > >>>> > I store the above cronjob in the /etc/cron.d folder with filename: >>>> > prom-apt. >>>> > >>>> > The metrics which I mentioned in custom1.sh, I am able to see those >>>> > metrics in Prometheus UI. >>>> > >>>> > >>>> > Could anyone help me? >>>> > >>>> What is the contents of those files in /var/lib/node_exporter? >>>> >>>> -- >>>> Stuart Clark >>>> >>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2c04d777-3ced-41c9-bbda-f4e10ba4fd3bn%40googlegroups.com.
