Hi Robert,
were you able to resolve this issue? I haven't faced that error myself
yet, so I can't really comment. But it would be interesting to know if
and how you got out of it.
Thanks,
Eugen
Zitat von Robert Sander <r.san...@heinlein-support.de>:
Hi,
Am 6/30/25 um 16:50 schrieb Robert Sander:
With marking the MDS as repaired you mean the command "ceph mds
repaired storage_cluster:0", right?
It looks like we hit this bug: https://tracker.ceph.com/issues/65094
From the MDS log: No subtrees found for root MDS rank!
Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.0.1753398 handle_mds_map
i am now mds.0.1753398
Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.0.1753398 handle_mds_map
state change up:reconnect --> up:rejoin
Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.0.1753398 rejoin_start
Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.0.1753398 rejoin_joint_start
Jul 01 08:50:56 sn04 ceph-mds[1563881]: mds.0.1753398 rejoin_done
Jul 01 08:50:56 sn04 ceph-mds[1563881]: log_channel(cluster) log
[ERR] : No subtrees found for root MDS rank!
Jul 01 08:50:56 sn04 ceph-mds[1563881]:
mds.beacon.storage_cluster.sn04.cbvzzu set_want_state: up:rejoin ->
down:damaged
Jul 01 08:50:56 sn04 ceph-mds[1563881]: log_client log_queue is 1
last_log 1 sent 0 num 1 unsent 1 sending 1
Jul 01 08:50:56 sn04 ceph-mds[1563881]: log_client will send
2025-07-01T06:50:56.151556+0000 mds.storage_cluster.sn04.cbvzzu
(mds.0) 1 : cluster [ERR] No subtrees found for root MDS rank!
Jul 01 08:50:56 sn04 ceph-mds[1563881]: monclient: _send_mon_message
to mon.sn03 at v2:192.168.91.53:3300/0
Jul 01 08:50:56 sn04 ceph-mds[1563881]:
mds.beacon.storage_cluster.sn04.cbvzzu Sending beacon down:damaged
seq 11440
Jul 01 08:50:56 sn04 ceph-mds[1563881]: monclient: _send_mon_message
to mon.sn03 at v2:192.168.91.53:3300/0
Jul 01 08:50:56 sn04 ceph-mds[1563881]:
mds.beacon.storage_cluster.sn04.cbvzzu received beacon reply
up:rejoin seq 11439 rtt 1.011
Jul 01 08:50:56 sn04 ceph-mds[1563881]:
mds.beacon.storage_cluster.sn04.cbvzzu received beacon reply
down:damaged seq 11440 rtt 0.0880002
Jul 01 08:50:56 sn04 ceph-mds[1563881]:
mds.storage_cluster.sn04.cbvzzu respawn!
Jul 01 08:50:56 sn04
ceph-28ca2bfa-d87e-11ed-83a3-1070fddda30f-mds-storage_cluster-sn04-cbvzzu[1563770]: -9999> 2025-07-01T06:50:56.150+0000 7f5daca13640 -1 log_channel(cluster) log [ERR] : No subtrees found for root MDS
rank!
Jul 01 08:50:56 sn04 ceph-mds[1563881]:
mds.storage_cluster.sn04.cbvzzu e: '/usr/bin/ceph-mds'
Jul 01 08:50:56 sn04 ceph-mds[1563881]:
mds.storage_cluster.sn04.cbvzzu 0: '/usr/bin/ceph-mds'
Jul 01 08:50:56 sn04 ceph-mds[1563881]:
mds.storage_cluster.sn04.cbvzzu 1: '-n'
Jul 01 08:50:56 sn04 ceph-mds[1563881]:
mds.storage_cluster.sn04.cbvzzu 2: 'mds.storage_cluster.sn04.cbvzzu'
Jul 01 08:50:56 sn04 ceph-mds[1563881]:
mds.storage_cluster.sn04.cbvzzu 3: '-f'
Jul 01 08:50:56 sn04 ceph-mds[1563881]:
mds.storage_cluster.sn04.cbvzzu 4: '--setuser'
Jul 01 08:50:56 sn04 ceph-mds[1563881]:
mds.storage_cluster.sn04.cbvzzu 5: 'ceph'
Jul 01 08:50:56 sn04 ceph-mds[1563881]:
mds.storage_cluster.sn04.cbvzzu 6: '--setgroup'
Jul 01 08:50:56 sn04 ceph-mds[1563881]:
mds.storage_cluster.sn04.cbvzzu 7: 'ceph'
Jul 01 08:50:56 sn04 ceph-mds[1563881]:
mds.storage_cluster.sn04.cbvzzu 8: '--default-log-to-file=false'
Jul 01 08:50:56 sn04 ceph-mds[1563881]:
mds.storage_cluster.sn04.cbvzzu 9: '--default-log-to-journald=true'
Jul 01 08:50:56 sn04 ceph-mds[1563881]:
mds.storage_cluster.sn04.cbvzzu 10: '--default-log-to-stderr=false'
Jul 01 08:50:56 sn04 ceph-mds[1563881]:
mds.storage_cluster.sn04.cbvzzu respawning with exe /usr/bin/ceph-mds
Jul 01 08:50:56 sn04 ceph-mds[1563881]:
mds.storage_cluster.sn04.cbvzzu exe_path /proc/self/exe
Jul 01 08:50:56 sn04
ceph-28ca2bfa-d87e-11ed-83a3-1070fddda30f-mds-storage_cluster-sn04-cbvzzu[1563770]: ignoring --setuser ceph since I am not
root
Jul 01 08:50:56 sn04
ceph-28ca2bfa-d87e-11ed-83a3-1070fddda30f-mds-storage_cluster-sn04-cbvzzu[1563770]: ignoring --setgroup ceph since I am not
root
Jul 01 08:50:56 sn04 ceph-mds[1563881]: ceph version 18.2.4
(e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process
ceph-mds, pid 2
Jul 01 08:50:56 sn04 ceph-mds[1563881]: main not setting numa affinity
Jul 01 08:50:56 sn04 ceph-mds[1563881]: pidfile_write: ignore empty
--pid-file
Jul 01 08:50:56 sn04
ceph-28ca2bfa-d87e-11ed-83a3-1070fddda30f-mds-storage_cluster-sn04-cbvzzu[1563770]: starting mds.storage_cluster.sn04.cbvzzu
at
Jul 01 08:50:56 sn04 ceph-mds[1563881]:
mds.storage_cluster.sn04.cbvzzu Updating MDS map to version 1753401
from mon.2
Jul 01 08:50:57 sn04 ceph-mds[1563881]:
mds.storage_cluster.sn04.cbvzzu Updating MDS map to version 1753402
from mon.2
Jul 01 08:50:57 sn04 ceph-mds[1563881]:
mds.storage_cluster.sn04.cbvzzu Monitors have assigned me to become
a standby.
And that's it. The MDS journal integrity seems to be OK:
# /usr/bin/cephfs-journal-tool --rank=storage_cluster:all journal inspect
Overall journal integrity: OK
How do we get this filesystem online again?
Regards
--
Robert Sander
Linux Consultant
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0
Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B
Geschäftsführer: Peer Heinlein - Sitz: Berlin
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
