I have following docker-compose file for deploying my monitoring stack:
version: '3'
services:
prometheus:
image: prom/prometheus:latest
volumes:
- ./prometheus/prometheus.yaml:/etc/prometheus/prometheus.yaml
# - prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yaml'
# - '--storage.tsdb.path=/prometheus'
- '--storage.tsdb.retention.time=24w'
ports:
- 9090:9090
node-exporter:
image: prom/node-exporter
ports:
- 9100:9100
cadvisor:
image: google/cadvisor:latest
ports:
- 8081:8080
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
grafana:
image: grafana/grafana
user: "1000"
volumes:
-
./grafana/provisioning/datasources:/etc/grafana/provisioning/datasources
-
./grafana/provisioning/dashboards:/etc/grafana/provisioning/dashboards
- ./grafana/grafana.ini:/etc/grafana/grafana.ini
depends_on:
- prometheus
ports:
- 3032:3000
# volumes:
# prometheus_data:
It works perfectly fine:
[image: enter image description here]
I also have some grafana panels embedded in my web app, which also get
rendered.
I wanted to persist the monitoring data. So I need to add named docker
volume. To this, I un-commented the commented lines in above docker compose
file. This made cadvisor unavailable to prometheus:
[image: enter image description here]
The embedded panels contained information from both node-exporter (CPU, RAM
etc information for host machine) and cadvisor (CPU, RAM etc information
for each docker container). All these panels started showing no data after
un-commenting those lines! I felt only docker container related metrics
should become unavailable given that only cadvisor is unavailable. But I
was wrong, even host machine related panels too were down! However, I am
still able to access all metrics end points from browser and they were
returning huge list of metrics:
1. Prometheus endpoint: http://192.168.0.104:9090/metrics
2. cAdvisor endpoint: http://192.168.0.104:8081/metrics
3. node-exporter endpoint: http://192.168.0.104:9100/metrics
*What mistake I am making here?*
I also tried to check permissions of the volume directory in my laptop. It
is created in my laptop at /var/lib/docker/volumes:
/var/lib/docker/volumes# ls -l
total 904
brw------- 1 root root 259, 4 May 28 17:59 backingFsBlockDev
drwx-----x 3 root root 4096 Jan 14 2022
myapp_demo_myapp_nginx_volume
drwx-----x 3 root root 4096 Feb 5 2022
myapp_demo_myapp_pgadmin_volume
drwx-----x 3 root root 4096 Jan 14 2022
myapp_demo_myapp_postgres_volume
drwx-----x 3 root root 4096 Jan 14 2022
myapp_demo_myapp_redis_volume
drwx-----x 3 root root 4096 Feb 14 2022
myapp_demo_fm_v2_myapp_nginx_volume
drwx-----x 3 root root 4096 Feb 14 2022
myapp_demo_fm_v2_myapp_pgadmin_volume
drwx-----x 3 root root 4096 Feb 14 2022
myapp_demo_fm_v2_myapp_postgres_volume
drwx-----x 3 root root 4096 Feb 14 2022
myapp_demo_fm_v2_myapp_redis_volume
drwx-----x 3 root root 4096 May 25 20:10
myapp_ip_logging_myapp_nginx_volume
drwx-----x 3 root root 4096 May 25 20:11
myapp_ip_logging_myapp_pgadmin_volume
drwx-----x 3 root root 4096 May 25 20:10
myapp_ip_logging_myapp_postgres_volume
drwx-----x 3 root root 4096 May 25 20:11
myapp_ip_logging_myapp_redis_volume
drwx-----x 3 root root 4096 Jun 4 2022
myapp_stack_1_myapp_nginx_volume
drwx-----x 3 root root 4096 Jun 4 2022
myapp_stack_1_myapp_pgadmin_volume
drwx-----x 3 root root 4096 Jun 4 2022
myapp_stack_1_myapp_postgres_volume
drwx-----x 3 root root 4096 Jun 4 2022
myapp_stack_1_myapp_redis_volume
drwx-----x 3 root root 4096 May 31 06:28
myapp_stack_1_monitoring_prometheus_data <---------
drwx-----x 3 root root 4096 Jul 1 2022
myapp_stack_2_myapp_nginx_volume
drwx-----x 3 root root 4096 Jul 1 2022
myapp_stack_2_myapp_pgadmin_volume
drwx-----x 3 root root 4096 Jul 1 2022
myapp_stack_2_myapp_postgres_volume
drwx-----x 3 root root 4096 Jul 1 2022
myapp_stack_2_myapp_redis_volume
drwx-----x 3 root root 4096 Jul 1 2022
myapp_stack_3_myapp_nginx_volume
drwx-----x 3 root root 4096 Jul 1 2022
myapp_stack_3_myapp_pgadmin_volume
drwx-----x 3 root root 4096 Jul 1 2022
myapp_stack_3_myapp_postgres_volume
drwx-----x 3 root root 4096 Jul 1 2022
myapp_stack_3_myapp_redis_volume
drwx-----x 3 root root 4096 Aug 18 2022
myapp_stack_py3_1_myapp_nginx_volume
drwx-----x 3 root root 4096 Aug 18 2022
myapp_stack_py3_1_myapp_pgadmin_volume
drwx-----x 3 root root 4096 Aug 18 2022
myapp_stack_py3_1_myapp_postgres_volume
drwx-----x 3 root root 4096 Aug 18 2022
myapp_stack_py3_1_myapp_redis_volume
drwx-----x 3 root root 4096 Sep 23 2022
myapp_stats_myapp_nginx_volume
drwx-----x 3 root root 4096 Sep 23 2022
myapp_stats_myapp_postgres_volume
drwx-----x 3 root root 4096 Sep 23 2022
myapp_stats_myapp_redis_volume
drwx-----x 3 root root 4096 Apr 11 2022 myapp_myapp_nginx_volume
drwx-----x 3 root root 4096 Apr 11 2022 myapp_myapp_pgadmin_volume
drwx-----x 3 root root 4096 Apr 11 2022
myapp_myapp_postgres_volume
drwx-----x 3 root root 4096 Apr 11 2022 myapp_myapp_redis_volume
drwx-----x 3 root root 4096 May 30 2022
myapp_master_myapp_nginx_volume
drwx-----x 3 root root 4096 May 30 2022
myapp_master_myapp_pgadmin_volume
drwx-----x 3 root root 4096 May 30 2022
myapp_master_myapp_postgres_volume
drwx-----x 3 root root 4096 May 30 2022
myapp_master_myapp_redis_volume
drwx-----x 3 root root 4096 Feb 24 2022 elk_certs
drwx-----x 3 root root 4096 Feb 24 2022 elk_esdata01
drwx-----x 3 root root 4096 Feb 24 2022 elk_esdata02
drwx-----x 3 root root 4096 Feb 24 2022 elk_esdata03
drwx-----x 3 root root 4096 Feb 24 2022 elk_kibanadata
drwx-----x 3 root root 4096 Mar 31 2022 jenkins_home
-rw------- 1 root root 131072 May 31 04:08 metadata.db
drwx-----x 3 root root 4096 Apr 5 2022 minikube
drwx-----x 3 root root 4096 Feb 16 2022 vscode
Notice below line:
drwx-----x 3 root root 4096 May 31 06:28
myapp_stack_1_monitoring_prometheus_data <---------
It shows prometheus_data volumne too have same permissions as others i.e.
drwx-----x. But when I check permissions on directory individually, it
gives different permissions:
# ls -l myapp_stack_1_monitoring_prometheus_data
total 4
drwxr-xr-x 5 nobody nogroup 4096 May 31 13:16 _ data
Notice nobody nogroup in above output. *Q. Could this be a problem?*
This differs from whats shown for other volumes:
# ls -l myapp_stack_1_myapp_postgres_volume
total 4
drwx------ 19 systemd-coredump systemd-coredump 4096 May 29 19:05
_data
# ls -l myapp_stack_1_myapp_nginx_volume
total 4
drwxr-xr-x 2 root root 4096 Jun 4 2022 _data
# ls -l myapp_stack_1_myapp_redis_volume
total 4
drwxr-xr-x 2 systemd-coredump systemd-coredump 4096 May 31 13:16
_data
Another difference I noticed is the suffix _data in
myapp_stack_1_monitoring_prometheus_data, while others have suffix _volume.*
Q. Could that be a problem?*
*PS:* My prometheus.yaml looks like this:
global:
scrape_interval: 5s
external_labels:
monitor: 'Monitoring'
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['prometheus:9090']
- job_name: 'node-exporter'
static_configs:
- targets: ['node-exporter:9100']
- job_name: 'cAdvisor'
static_configs:
- targets: ['cadvisor:8080']
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/320f20d8-ccaa-4e6c-acfd-85c6ac2a64b8n%40googlegroups.com.
