Hello,
recently I checked my server logs for error events and got some hits for
the ceph crash service, deployed with cephadm.
When restarted, crash service logs to journald:
auth: unable to find a keyring on
/etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin:
(2) No such file or directory
AuthRegistry(0x7f21a0068da0) no keyring found at
/etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,
disabling cephx
auth: unable to find a keyring on
/etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin:
(2) No such file or directory
AuthRegistry(0x7f21a861bff0) no keyring found at
/etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,
disabling cephx
monclient(hunting): handle_auth_bad_method server allowed_methods [2]
but i only support [1]
monclient(hunting): handle_auth_bad_method server allowed_methods [2]
but i only support [1]
monclient: authenticate NOTE: no keyring found; disabled cephx
authentication
[errno 13] RADOS permission denied (error connecting to the cluster)
The image is mounted with:
-v
/var/lib/ceph/7644057a-00f6-11f0-9a0c-eac00fed9338/crash.virt-master3/keyring:/etc/ceph/ceph.client.crash.virt-master3.keyring
so I assume there should be a key available, but crash daemon searches
for a admin keyring instead of "client.crash.virt-master3".
Any advice how to fix this error? I see these error logs on all machines
where crash service is deployed. I've tried a redeployment without any
effect.
# ceph -v
ceph version 19.2.1 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid
(stable)
# systemctl restart
ceph-7644057a-00f6-11f0-9a0c-eac00fed9...@crash.virt-master3.service
# journalctl -u
ceph-7644057a-00f6-11f0-9a0c-eac00fed9...@crash.virt-master3.service -ef
Output:
Apr 12 18:36:24 virt-master3
ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1402272]:
*** Interrupted with signal 15 ***
Apr 12 18:36:24 virt-master3 podman[1408122]: 2025-04-12
18:36:24.992684396 +0200 CEST m=+0.043782773 container died
6c538b6216744326c00d9b3989cfe8d498629fa2f008a62fe184f367b01cbf6b
(image=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de,
name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3,
org.label-schema.name=CentOS Stream 9 Base Image,
org.opencontainers.image.authors=Ceph Release Team
<ceph-maintain...@ceph.io>, org.label-schema.vendor=CentOS,
org.label-schema.license=GPLv2, CEPH_REF=squid,
org.label-schema.schema-version=1.0,
CEPH_GIT_REPO=https://github.com/ceph/ceph.git,
io.buildah.version=1.33.7,
org.opencontainers.image.documentation=https://docs.ceph.com/,
CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb,
GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/,
FROM_IMAGE=quay.io/centos/centos:stream9,
org.label-schema.build-date=20250124, OSD_FLAVOR=default)
Apr 12 18:36:25 virt-master3 podman[1408122]: 2025-04-12
18:36:25.016797163 +0200 CEST m=+0.067895534 container remove
6c538b6216744326c00d9b3989cfe8d498629fa2f008a62fe184f367b01cbf6b
(image=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de,
name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3,
org.opencontainers.image.documentation=https://docs.ceph.com/,
org.label-schema.build-date=20250124,
FROM_IMAGE=quay.io/centos/centos:stream9,
GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/,
io.buildah.version=1.33.7, org.label-schema.name=CentOS Stream 9 Base
Image, org.label-schema.vendor=CentOS,
org.label-schema.schema-version=1.0,
CEPH_GIT_REPO=https://github.com/ceph/ceph.git,
CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb, OSD_FLAVOR=default,
org.label-schema.license=GPLv2, CEPH_REF=squid,
org.opencontainers.image.authors=Ceph Release Team
<ceph-maintain...@ceph.io>)
Apr 12 18:36:25 virt-master3 bash[1408104]:
ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3
Apr 12 18:36:25 virt-master3 systemd[1]:
ceph-7644057a-00f6-11f0-9a0c-eac00fed9...@crash.virt-master3.service:
Deactivated successfully.
Apr 12 18:36:25 virt-master3 systemd[1]: Stopped
ceph-7644057a-00f6-11f0-9a0c-eac00fed9...@crash.virt-master3.service -
Ceph crash.virt-master3 for 7644057a-00f6-11f0-9a0c-eac00fed9338.
Apr 12 18:36:25 virt-master3 systemd[1]:
ceph-7644057a-00f6-11f0-9a0c-eac00fed9...@crash.virt-master3.service:
Consumed 1.540s CPU time.
Apr 12 18:36:25 virt-master3 systemd[1]: Starting
ceph-7644057a-00f6-11f0-9a0c-eac00fed9...@crash.virt-master3.service -
Ceph crash.virt-master3 for 7644057a-00f6-11f0-9a0c-eac00fed9338...
Apr 12 18:36:25 virt-master3 podman[1408259]:
Apr 12 18:36:25 virt-master3 podman[1408259]: 2025-04-12
18:36:25.498991532 +0200 CEST m=+0.071935136 container create
bb21751667c7c4dac8f5624fb2e84f79facd5e809ff08c5168216cd591470b63
(image=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de,
name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3,
org.label-schema.build-date=20250124, org.label-schema.name=CentOS
Stream 9 Base Image, OSD_FLAVOR=default,
org.label-schema.schema-version=1.0,
CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb,
io.buildah.version=1.33.7,
CEPH_GIT_REPO=https://github.com/ceph/ceph.git,
org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2,
FROM_IMAGE=quay.io/centos/centos:stream9,
org.opencontainers.image.documentation=https://docs.ceph.com/,
GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/,
CEPH_REF=squid, org.opencontainers.image.authors=Ceph Release Team
<ceph-maintain...@ceph.io>)
Apr 12 18:36:25 virt-master3 podman[1408259]: 2025-04-12
18:36:25.462816456 +0200 CEST m=+0.035760071 image pull
quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de
Apr 12 18:36:25 virt-master3 podman[1408259]: 2025-04-12
18:36:25.589479015 +0200 CEST m=+0.162422619 container init
bb21751667c7c4dac8f5624fb2e84f79facd5e809ff08c5168216cd591470b63
(image=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de,
name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3,
org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0,
CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb,
org.label-schema.name=CentOS Stream 9 Base Image, OSD_FLAVOR=default,
org.label-schema.build-date=20250124, org.label-schema.vendor=CentOS,
org.opencontainers.image.authors=Ceph Release Team
<ceph-maintain...@ceph.io>, FROM_IMAGE=quay.io/centos/centos:stream9,
org.opencontainers.image.documentation=https://docs.ceph.com/,
GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/,
CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_REF=squid,
io.buildah.version=1.33.7)
Apr 12 18:36:25 virt-master3 podman[1408259]: 2025-04-12
18:36:25.595018205 +0200 CEST m=+0.167961840 container start
bb21751667c7c4dac8f5624fb2e84f79facd5e809ff08c5168216cd591470b63
(image=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de,
name=ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3,
org.label-schema.license=GPLv2,
FROM_IMAGE=quay.io/centos/centos:stream9,
org.label-schema.build-date=20250124, io.buildah.version=1.33.7,
CEPH_GIT_REPO=https://github.com/ceph/ceph.git,
org.label-schema.vendor=CentOS,
CEPH_SHA1=58a7fab8be0a062d730ad7da874972fd3fba59fb, OSD_FLAVOR=default,
org.opencontainers.image.documentation=https://docs.ceph.com/,
CEPH_REF=squid, org.opencontainers.image.authors=Ceph Release Team
<ceph-maintain...@ceph.io>,
GANESHA_REPO_BASEURL=https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/,
org.label-schema.name=CentOS Stream 9 Base Image,
org.label-schema.schema-version=1.0)
Apr 12 18:36:25 virt-master3 bash[1408259]:
bb21751667c7c4dac8f5624fb2e84f79facd5e809ff08c5168216cd591470b63
Apr 12 18:36:25 virt-master3 systemd[1]: Started
ceph-7644057a-00f6-11f0-9a0c-eac00fed9...@crash.virt-master3.service -
Ceph crash.virt-master3 for 7644057a-00f6-11f0-9a0c-eac00fed9338.
Apr 12 18:36:25 virt-master3
ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]:
INFO:ceph-crash:pinging cluster to exercise our key
Apr 12 18:36:26 virt-master3
ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]:
2025-04-12T16:36:26.131+0000 7f21a861d640 -1 auth: unable to find a
keyring on
/etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin:
(2) No such file or directory
Apr 12 18:36:26 virt-master3
ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]:
2025-04-12T16:36:26.131+0000 7f21a861d640 -1
AuthRegistry(0x7f21a0068da0) no keyring found at
/etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,
disabling cephx
Apr 12 18:36:26 virt-master3
ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]:
2025-04-12T16:36:26.131+0000 7f21a861d640 -1 auth: unable to find a
keyring on
/etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin:
(2) No such file or directory
Apr 12 18:36:26 virt-master3
ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]:
2025-04-12T16:36:26.131+0000 7f21a861d640 -1
AuthRegistry(0x7f21a861bff0) no keyring found at
/etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,
disabling cephx
Apr 12 18:36:26 virt-master3
ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]:
2025-04-12T16:36:26.131+0000 7f21a5b91640 -1 monclient(hunting):
handle_auth_bad_method server allowed_methods [2] but i only support [1]
Apr 12 18:36:26 virt-master3
ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]:
2025-04-12T16:36:26.131+0000 7f21a6392640 -1 monclient(hunting):
handle_auth_bad_method server allowed_methods [2] but i only support [1]
Apr 12 18:36:26 virt-master3
ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]:
2025-04-12T16:36:26.131+0000 7f21a861d640 -1 monclient: authenticate
NOTE: no keyring found; disabled cephx authentication
Apr 12 18:36:26 virt-master3
ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]:
[errno 13] RADOS permission denied (error connecting to the cluster)
Apr 12 18:36:26 virt-master3
ceph-7644057a-00f6-11f0-9a0c-eac00fed9338-crash-virt-master3[1408301]:
INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s
--
Best regards / Mit freundlichen Grüßen
Daniel Vogelbacher
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
