Also: you should clarify where exactly you're running these scripts, i.e.
which host has the /var/lib/node_exporter/*.prom files
* If it's on the prometheus server itself, then you'll need to run
node_exporter on the prometheus server, *and* you'll need to get prometheus
to scrape itself - i.e. a scrape job that collects from
localhost:9100/metrics
- job_name: 'node'
static_configs:
- targets: ['localhost:9100']
* If it's on some remote host, then you'll need to get prometheus to scrape
*that* host on x.x.x.x:9100/metrics
If you're already scraping node_exporter on all your hosts - for example,
you can see metrics like "node_filesystem_avail_bytes", on the instance
where your custom metrics are being created - then you should be fine.
On Wednesday, 1 March 2023 at 08:58:25 UTC Brian Candler wrote:
> *> The main issue is that I am not able to see the metrics in Prometheus
> UI. I am unable to understand the root cause.*
>
> Almost certainly because they are not being collected. So you need to
> break this down further to find out why.
>
> (1) Show the output of this command:
>
> *ls -l /var/lib/node_exporter*
>
> * If the filenames don't end with ".prom" then that is your root cause.
> node_exporter only picks up files whose names match the pattern *.prom
>
> * If the files aren't readable by the user that node_exporter is running
> as, then that's the root cause.
>
> * Are you running under RedHat/CentOS with SELinux? If so, try putting it
> into permissive mode. Maybe there's some SELinux policy which prevents
> access to files in /var/lib/node_exporter.
>
> (2) Then show the contents of one of those files:
>
> *head /var/lib/node_exporter/foobar.prom*
>
> If the files are not in valid openmetrics format, then they won't be
> picked up either. You can test them with "promtool check metrics":
>
> /path/to/promtool check metrics </var/lib/node_exporter/foobar.prom
>
> If that shows an error, then modify your script to generate files in the
> correct format. There are examples you can copy from here:
> https://github.com/prometheus-community/node-exporter-textfile-collector-scripts
>
> (3) Show the command line that you're running node_exporter with:
>
> *ps auxwww | grep node_exporter*
>
> If the flag "--collector.textfile.directory=/var/lib/node_exporter" is not
> present, then that's your root cause.
>
> (4) Show the output of this command:
>
> *curl localhost:9100/metrics | grep foobar*
>
> where "foobar" is one of the custom metrics you are generating. Does it
> appear here? If not, then you can rule out any issues with prometheus
> itself, it's definitely node_exporter not picking up your custom metrics.
> If it does, then you'll need to dig further onto prometheus side (e.g. your
> scrape job config)
>
>
> *> In /var/lib/node_exporter --> In this folder, the scripts(custom
> metrics script) will be available.*
>
> You don't want the scripts in that directory - only the *output* from
> those scripts. However, as long as the scripts themselves don't end with
> ".prom" then they will be ignored, so that won't be a problem.
>
> On Wednesday, 1 March 2023 at 05:24:40 UTC BHARATH KUMAR wrote:
>
>> The main issue is that I am not able to see the metrics in Prometheus UI.
>> I am unable to understand the root cause.
>>
>> Am I missing anything?
>>
>> On Tuesday, 28 February 2023 at 23:44:59 UTC+5:30 Brian Candler wrote:
>>
>>> Aside: you might also want to look at kube-state-metrics
>>> <https://github.com/kubernetes/kube-state-metrics>, which can give you
>>> a range of pod metrics
>>> <https://github.com/kubernetes/kube-state-metrics/blob/main/docs/pod-metrics.md>
>>>
>>> including the pod start time. Then an expression like
>>> time()-kube_pod_start_time should give you the pod age.
>>>
>>> On Tuesday, 28 February 2023 at 17:55:20 UTC Brian Candler wrote:
>>>
>>>> The problem is that the filename must end with ".prom", i.e. you need
>>>> to create "apt1.prom" not "apt1-prom"
>>>>
>>>> Also, you need to enable the functionality with a flag to node_exporter:
>>>>
>>>> node_exporter --collector.textfile.directory=/var/lib/node_exporter
>>>>
>>>> See the documentation here:
>>>> https://github.com/prometheus/node_exporter#textfile-collector
>>>>
>>>> On Tuesday, 28 February 2023 at 17:16:06 UTC Stuart Clark wrote:
>>>>
>>>>> On 28/02/2023 17:12, BHARATH KUMAR wrote:
>>>>> > Hello All,
>>>>> >
>>>>> > I want to calculate the pod age running on every server. I wrote a
>>>>> > shell script. I added this script under the folder
>>>>> /var/lib/node_exporter.
>>>>> >
>>>>> > I created a cronjob for this script to run every minute
>>>>> > * * * * * root bash /var/lib/node_exporter/custom_metrics.sh >
>>>>> > /var/lib/node_exporter/apt1-prom
>>>>> >
>>>>> > I store the above cronjob in the /etc/cron.d folder with filename:
>>>>> > prom-apt1.
>>>>> >
>>>>> > But I am not able to see the metrics I created in Prometheus UI.
>>>>> >
>>>>> >
>>>>> >
>>>>> > But similarly, I created another shell script file to fetch some
>>>>> > metrics. I followed the same procedure as above.
>>>>> >
>>>>> > * * * * * root bash /var/lib/node_exporter/custom1.sh >
>>>>> > /var/lib/node_exporter/apt-prom
>>>>> >
>>>>> > I store the above cronjob in the /etc/cron.d folder with filename:
>>>>> > prom-apt.
>>>>> >
>>>>> > The metrics which I mentioned in custom1.sh, I am able to see those
>>>>> > metrics in Prometheus UI.
>>>>> >
>>>>> >
>>>>> > Could anyone help me?
>>>>> >
>>>>> What is the contents of those files in /var/lib/node_exporter?
>>>>>
>>>>> --
>>>>> Stuart Clark
>>>>>
>>>>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/280247db-682f-494c-a5e2-b8a5179babe1n%40googlegroups.com.
