Hi,
I just noticed this error
"failed to probe daemon or devices"
which seems to be related to /var/lib/ceph being read only
I checked the host ( ceph-host-1) and it seems fine so it must be a docker
instance
How do I identify which one and how can I fix it please ?
Many thanks
Steven
root@ceph-host-1:~/utilities# ceph -s
cluster:
id: 0cfa836d-68b5-11f0-90bf-7cc2558e5ce8
health: HEALTH_WARN
failed to probe daemons or devices
services:
mon: 5 daemons, quorum
ceph-host-1,ceph-host-2,ceph-host-3,ceph-host-7,ceph-host-6 (age 8h)
mgr: ceph-host-1.lqlece(active, since 5w), standbys: ceph-host-2.suiuxi
mds: 19/19 daemons up, 7 standby
osd: 185 osds: 185 up (since 4w), 185 in (since 4w)
flags noautoscale
rgw: 2 daemons active (2 hosts, 1 zones)
data:
volumes: 4/4 healthy
pools: 16 pools, 7137 pgs
objects: 359.48M objects, 811 TiB
usage: 1.2 PiB used, 1.1 PiB / 2.3 PiB avail
pgs: 7034 active+clean
72 active+clean+scrubbing
31 active+clean+scrubbing+deep
io:
client: 30 MiB/s rd, 34 KiB/s wr, 11 op/s rd, 1 op/s wr
cephadm shell -- ceph-volume lvm list --format json
Inferring fsid 0cfa836d-68b5-11f0-90bf-7cc2558e5ce8
Inferring config
/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/mon.ceph-host-1/config
docker: Error response from daemon: error while creating mount source path
'/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/crash': mkdir
/var/lib/ceph: read-only file system.
root@ceph-host-1:~/utilities# ceph health detail
HEALTH_WARN failed to probe daemons or devices
[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices
host ceph-host-1 `cephadm ceph-volume` failed: cephadm exited with an
error code: 1, stderr: Inferring config
/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/mon.ceph-host-1/config
Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host
--stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint
/usr/sbin/ceph-volume --privileged --group-add=disk --init -e
CONTAINER_IMAGE=
quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6
-e NODE_NAME=ceph-host-1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e
CEPH_VOLUME_DEBUG=1 -v
/var/run/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8:/var/run/ceph:z -v
/var/log/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8:/var/log/ceph:z -v
/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/crash:/var/lib/ceph/crash:z
-v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v
/run/lock/lvm:/run/lock/lvm -v /:/rootfs:rslave -v
/tmp/ceph-tmptx7tjpjz:/etc/ceph/ceph.conf:z
quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6
inventory --format=json-pretty --filter-for-batch
/usr/bin/docker: stderr docker: Error response from daemon: error while
creating mount source path
'/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/crash': mkdir
/var/lib/ceph: read-only file system.
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File
"/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/cephadm.e2b78d38c8b7ec4ef00612ed046678feceb37baccd3990bc21df1538095d27c9/__main__.py",
line 5581, in <module>
File
"/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/cephadm.e2b78d38c8b7ec4ef00612ed046678feceb37baccd3990bc21df1538095d27c9/__main__.py",
line 5569, in main
File
"/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/cephadm.e2b78d38c8b7ec4ef00612ed046678feceb37baccd3990bc21df1538095d27c9/__main__.py",
line 409, in _infer_config
File
"/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/cephadm.e2b78d38c8b7ec4ef00612ed046678feceb37baccd3990bc21df1538095d27c9/__main__.py",
line 324, in _infer_fsid
File
"/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/cephadm.e2b78d38c8b7ec4ef00612ed046678feceb37baccd3990bc21df1538095d27c9/__main__.py",
line 437, in _infer_image
File
"/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/cephadm.e2b78d38c8b7ec4ef00612ed046678feceb37baccd3990bc21df1538095d27c9/__main__.py",
line 311, in _validate_fsid
File
"/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/cephadm.e2b78d38c8b7ec4ef00612ed046678feceb37baccd3990bc21df1538095d27c9/__main__.py",
line 3314, in command_ceph_volume
File
"/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/cephadm.e2b78d38c8b7ec4ef00612ed046678feceb37baccd3990bc21df1538095d27c9/cephadmlib/call_wrappers.py",
line 307, in call_throws
RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host
--stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint
/usr/sbin/ceph-volume --privileged --group-add=disk --init -e
CONTAINER_IMAGE=
quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6
-e NODE_NAME=ceph-host-1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e
CEPH_VOLUME_DEBUG=1 -v
/var/run/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8:/var/run/ceph:z -v
/var/log/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8:/var/log/ceph:z -v
/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/crash:/var/lib/ceph/crash:z
-v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v
/run/lock/lvm:/run/lock/lvm -v /:/rootfs:rslave -v
/tmp/ceph-tmptx7tjpjz:/etc/ceph/ceph.conf:z
quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6
inventory --format=json-pretty --filter-for-batch: docker: Error response
from daemon: error while creating mount source path
'/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/crash': mkdir
/var/lib/ceph: read-only file system.
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
