Hi Ben,
After chown tp 472, “systemctl daemon-reload” changes it back to 167.
I also notice that these are still from docker.io while the rest are from quay
/home/general# docker ps --no-trunc | grep docker
93b8c3aa33580fb6f4951849a6ff9c2e66270eb913b8579aca58371ef41f2d6c
docker.io/grafana/grafana:6.7.4
"/run.sh"
10 days ago Up 10 days
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201
df6b7368a54d0af7d2cdd45c0c9bad0999d58c144cb99927a3f76683652b00f2
docker.io/prom/alertmanager:v0.16.2
"/bin/alertmanager --cluster.listen-address=:9094
--web.listen-address=:9093
--cluster.peer=fl31ca104ja0201.deacluster.intel.com:9094
--config.file=/etc/alertmanager/alertmanager.yml"
10 days ago Up 10 days
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-alertmanager-fl31ca104ja0201
aa2055733fe8d426312af5572c94558e89e7cf350e7baba2c22eb6a0e20682fc
docker.io/prom/prometheus:v2.7.2
"/bin/prometheus --config.file=/etc/prometheus/prometheus.yml
--storage.tsdb.path=/prometheus --web.listen-address=:9095
--storage.tsdb.retention.time=15d --storage.tsdb.retention.size=0
--web.external-url=http://fl31ca104ja0201:9095"; 10 days ago
Up 10 days
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-prometheus-fl31ca104ja0201
a9526f50dfacad47af298c0c1b2cf6cfd74b796b6df1945325529c79658d7356
docker.io/prom/node-exporter:v0.17.0
"/bin/node_exporter --no-collector.timex --web.listen-address=:9100
--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs"
10 days ago Up 10 days
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-node-exporter-fl31ca104ja0201
440926ce479bdd114f43e3228cc8cbfe48b4e1a6c2c7fab58c4cd103bc0f3a0e
docker.io/arcts/keepalived
"./init.sh"
3 weeks ago Up 3 weeks
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-keepalived-rgw-default-default-fl31ca104ja0201-yiasjs
2813ca859a7ba0de7fcb6be74a00b9b11a23e79636c5f35fb2b6b4be31a29f89
docker.io/library/haproxy:2.3
"docker-entrypoint.sh haproxy -f /var/lib/haproxy/haproxy.cfg"
3 weeks ago Up 3 weeks
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-haproxy-rgw-default-default-fl31ca104ja0201-yvwsmz
d68e2f68c45f2ea9a10267c8d964c2aaf026b4291918f4f3fb306da20a532db9
docker.io/arcts/keepalived
"./init.sh"
3 weeks ago Up 3 weeks
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-keepalived-nfs-nfs-1-fl31ca104ja0201-dsynjg
40f3c0b7455f5540fdb4f428bef4e9032b0ff0f50d302352551abb208eff1f28
docker.io/library/haproxy:2.3
"docker-entrypoint.sh haproxy -f /var/lib/haproxy/haproxy.cfg"
3 weeks ago Up 3 weeks
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-haproxy-nfs-nfs-1-fl31ca104ja0201-zdbzvv
From: Ben <ruidong....@gmail.com>
Sent: Wednesday, May 17, 2023 6:32 PM
To: Adiga, Anantha <anantha.ad...@intel.com>
Cc: ceph-users@ceph.io
Subject: Re: [ceph-users] Grafana service fails to start due to bad directory
name after Quincy upgrade
use this to get relevant long lines in log:
journalctl -u ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201
| less -S
it is '--user 472' by content of unit.run, not the default ceph user 167.
Maybe set the directory owner to 472 could help.
Hope it helps
Ben
Adiga, Anantha <anantha.ad...@intel.com<mailto:anantha.ad...@intel.com>>
于2023年5月18日周四 01:15写道：
Ben,
Thanks for the suggestion.
Changed the user and group to 167 for all files in the data and etc folders in
the grafana service folder were not 167. Did a systemctl daemon-reload and
restarted the grafana service ,
but still seeing the same error
-- Logs begin at Mon 2023-05-15 19:39:34 UTC, end at Wed 2023-05-17 17:08:02
UTC. --
May 17 17:07:44 fl31ca104ja0201 systemd[1]:
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main
process exited, code=exite>
May 17 17:07:44 fl31ca104ja0201 bash[148899]: /bin/bash:
/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop:
No >
May 17 17:07:44 fl31ca104ja0201 systemd[1]:
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service:
Failed with result 'exit-code'.
May 17 17:07:54 fl31ca104ja0201 systemd[1]:
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service:
Scheduled restart job, restart >
May 17 17:07:54 fl31ca104ja0201 systemd[1]: Stopped Ceph
grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
May 17 17:07:54 fl31ca104ja0201 systemd[1]: Started Ceph
grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
May 17 17:07:54 fl31ca104ja0201 bash[149116]: /bin/bash:
/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run:
No such >
May 17 17:07:54 fl31ca104ja0201 systemd[1]:
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main
process exited, code=exite>
May 17 17:07:54 fl31ca104ja0201 bash[149118]: /bin/bash:
/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop:
No >
May 17 17:07:54 fl31ca104ja0201 systemd[1]:
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service:
Failed with result 'exit-code'.
ESCOC
2 UTC. --
a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>:
Main process exited, code=exited, status=127/n/a
b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such
file or directory
a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>:
Failed with result 'exit-code'.
a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>:
Scheduled restart job, restart counter is at 3.
a0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
a0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No such file
or directory
a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>:
Main process exited, code=exited, status=127/n/a
b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop: No such
file or directory
a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:a3a1d7a87e@grafana-fl31ca104ja0201.service>:
Failed with result 'exit-code'.
~
Thank you,
Anantha
From: Ben <ruidong....@gmail.com<mailto:ruidong....@gmail.com>>
Sent: Wednesday, May 17, 2023 2:29 AM
To: Adiga, Anantha <anantha.ad...@intel.com<mailto:anantha.ad...@intel.com>>
Cc: ceph-users@ceph.io<mailto:ceph-users@ceph.io>
Subject: Re: [ceph-users] Grafana service fails to start due to bad directory
name after Quincy upgrade
you could check owner of /var/lib/ceph on host with grafana container running.
If its owner is root, change to 167:167 recursively.
Then systemctl daemon-reload and restart the service. Good luck.
Ben
Adiga, Anantha <anantha.ad...@intel.com<mailto:anantha.ad...@intel.com>>
于2023年5月17日周三 03:57写道：
Hi
Upgraded from Pacific 16.2.5 to 17.2.6 on May 8th
However, Grafana fails to start due to bad folder path
:/tmp# journalctl -u
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201 -n 25
-- Logs begin at Sun 2023-05-14 20:05:52 UTC, end at Tue 2023-05-16 19:07:51
UTC. --
May 16 19:05:00 fl31ca104ja0201 systemd[1]: Stopped Ceph
grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
May 16 19:05:00 fl31ca104ja0201 systemd[1]: Started Ceph
grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
May 16 19:05:00 fl31ca104ja0201 bash[2575021]: /bin/bash:
/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u>
May 16 19:05:00 fl31ca104ja0201 systemd[1]:
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main
process ex>
May 16 19:05:00 fl31ca104ja0201 bash[2575030]: /bin/bash:
/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u>
May 16 19:05:00 fl31ca104ja0201 systemd[1]:
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service:
Failed with res>
May 16 19:05:10 fl31ca104ja0201 systemd[1]:
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service:
Scheduled resta>
May 16 19:05:10 fl31ca104ja0201 systemd[1]: Stopped Ceph
grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
May 16 19:05:10 fl31ca104ja0201 systemd[1]: Started Ceph
grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
May 16 19:05:10 fl31ca104ja0201 bash[2575273]: /bin/bash:
/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u>
May 16 19:05:10 fl31ca104ja0201 systemd[1]:
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main
process ex>
May 16 19:05:10 fl31ca104ja0201 bash[2575282]: /bin/bash:
/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u>
May 16 19:05:10 fl31ca104ja0201 systemd[1]:
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service:
Failed with res>
May 16 19:05:20 fl31ca104ja0201 systemd[1]:
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service:
Scheduled resta>
May 16 19:05:20 fl31ca104ja0201 systemd[1]: Stopped Ceph
grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
May 16 19:05:20 fl31ca104ja0201 systemd[1]: Started Ceph
grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
May 16 19:05:20 fl31ca104ja0201 bash[2575369]: /bin/bash:
/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u>
May 16 19:05:20 fl31ca104ja0201 systemd[1]:
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service: Main
process ex>
May 16 19:05:20 fl31ca104ja0201 bash[2575370]: /bin/bash:
/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/u>
May 16 19:05:20 fl31ca104ja0201 systemd[1]:
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service:
Failed with res>
May 16 19:05:30 fl31ca104ja0201 systemd[1]:
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service:
Scheduled resta>
May 16 19:05:30 fl31ca104ja0201 systemd[1]: Stopped Ceph
grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
May 16 19:05:30 fl31ca104ja0201 systemd[1]:
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service:
Start request r>
May 16 19:05:30 fl31ca104ja0201 systemd[1]:
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service:
Failed with res>
May 16 19:05:30 fl31ca104ja0201 systemd[1]: Failed to start Ceph
grafana-fl31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
ESCOC
19:07:51 UTC. --
31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No
such file or directory
-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>:
Main process exited, code=exited, status=127/n/a
ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop:
No such file or directory
-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>:
Failed with result 'exit-code'.
-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>:
Scheduled restart job, restart counter is at 3.
31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run: No
such file or directory
-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>:
Main process exited, code=exited, status=127/n/a
ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.poststop:
No such file or directory
-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>:
Failed with result 'exit-code'.
-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service<mailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service%3cmailto:-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201.service>>:
Scheduled restart job, restart counter is at 4.
31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
31ca104ja0201 for d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e.
Check if path exists:
#<mailto:root@fl31ca104ja0201<mailto:root@fl31ca104ja0201>:/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201#>
ls
/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run
ls: cannot access
'/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana-fl31ca104ja0201/unit.run':
No such file or directory
Check if grafana.fl31ca104ja0201 directory exists:
#<mailto:root@fl31ca104ja0201<mailto:root@fl31ca104ja0201>:/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201#>
ls -l
rt/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/unit.run
total 36
drwxr-xr-x 4 167 167 4096 Apr 20 08:05 data
drwxr-xr-x 3 167 167 4096 Apr 20 08:13 etc
-rw------- 1 167 167 48 Apr 20 08:13 unit.created
-rw------- 1 167 167 390 May 8 16:12 unit.stop
-rw------- 1 167 167 390 May 8 16:12 unit.poststop
-rw------- 1 167 167 365 May 8 16:12 unit.meta
-rw------- 1 167 167 32 May 8 16:12 unit.image
-rw------- 1 167 167 38 May 8 16:12 unit.configured
-rw------- 1 167 167 1063 May 8 16:12 unit.run
cat unit.run
set -e
# grafana.fl31ca104ja0201
! /usr/bin/docker rm -f
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana.fl31ca104ja0201 2> /dev/null
! /usr/bin/docker rm -f
ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 2> /dev/null
/usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --init
--name ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-grafana-fl31ca104ja0201 --user
472 -e
CONTAINER_IMAGE=docker.io/grafana/grafana:6.7.4<http://docker.io/grafana/grafana:6.7.4>
-e NODE_NAME=fl31ca104ja0201 -e CEPH_USE_RANDOM_NONCE=1 -v
/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/grafana.ini:/etc/grafana/grafana.ini:Z
-v
/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:Z
-v
/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/etc/grafana/certs:/etc/grafana/certs:Z
-v
/var/lib/ceph/d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e/grafana.fl31ca104ja0201/data/grafana.db:/var/lib/grafana/grafana.db:Z
docker.io/grafana/grafana:6.7.4<http://docker.io/grafana/grafana:6.7.4>
Thank you,
Anantha
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-le...@ceph.io<mailto:ceph-users-le...@ceph.io>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
