Those tools are specifically for a disaster recovery, meaning there's
no mds running (otherwise you wouldn't need them ;-) ).
>> What would this command do ?
>> ceph fs reset-mds filesystem --yes-i-really-mean-it
I can only quote the docs, I don't know what exactly happens in the
background, I guess the devs would need to answer that. But maybe you
don't even need it, sometimes it's sufficient to reset the journal.
A few months ago I asked to improve the disaster recovery docs since
it's not really clear what exactly the described steps do and how
dangerous which operation is. Zac created a tracker for this, but I
haven't checked in a while for any progress. But most of the times (at
least that's my perception) the recovery seems to work.
Zitat von Amudhan P <amudha...@gmail.com>:
> Also, I don't have any active mds running, not sure if I can do all the
> recovery steps.
>
> On Thu, Apr 17, 2025 at 5:42 PM Amudhan P <amudha...@gmail.com> wrote:
>
>> I didn't face any network related issues but in my email start. I have
>> mentioned about one of the mon node was out of quorum.
>> Even after restart of mon node had few issues in log like slow ops in
mon
>> node and eventually it cleared but same mon node went out of quorum and
>> comes back online. During the above said issue observed 40% cpu in
wait in
>> top command in the same node where mon had issue. But there is no
errors
>> logged in dmesg related to drive error or network. Only after this mon
>> error may be after 24 hours MDS failed. MON issue cleared now after all
>> nodes were reboot cycled.
>>
>> Yeah, disaster recovery will be performed only as last resort.
>>
>> What would this command do ?
>>
>> ceph fs reset-mds filesystem --yes-i-really-mean-it
>>
>>
>> On Thu, Apr 17, 2025 at 5:19 PM Eugen Block <ebl...@nde.ag> wrote:
>>
>>> Was there any issue in your network or anything? Something that would
>>> explain the MDS crash? I would recommend to scan syslog, dmesg etc.
>>> for anything suspicious. If you don't find anything, you might need to
>>> go through
>>> https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/ to
>>> get your CephFS back up.
>>>
>>> Don't forget to create a backup of the journals. Read the instructions
>>> carefully, you might not need to do all the mentioned steps. I would
>>> start with (for both journals where applicable):
>>>
>>> cephfs-journal-tool journal export backup.bin
>>> cephfs-journal-tool event recover_dentries summary
>>> cephfs-journal-tool [--rank=<fs_name>:{mds-rank|all}] journal reset
>>> --yes-i-really-really-mean-it
>>> cephfs-table-tool all reset session
>>>
>>> But again, read the instructions carefully. This procedure is usually
>>> the last resort, I would first try to find out what happened causing
>>> this situation.
>>>
>>> Zitat von Amudhan P <amudha...@gmail.com>:
>>>
>>> > I don't think I have a memory issue.
>>> > Sorry, for sending log as file Pastebin is not working due to size
>>> > limitation.
>>> >
>>> > Crash info
>>> >
>>> >> ceph crash info
>>> >> 2025-04-17T08:50:35.931485Z_0109f173-206c-471e-afac-c0d3e0aa2785
>>> >> {
>>> >> "backtrace": [
>>> >> "/lib64/libpthread.so.0(+0x12b20) [0x7fbd42bbdb20]",
>>> >> "/usr/lib64/ceph/libceph-common.so.2(+0x8ec7a00)
>>> [0x7fbd4c81da00]"
>>> >> ],
>>> >> "ceph_version": "16.2.6",
>>> >> "crash_id":
>>> >> "2025-04-17T08:50:35.931485Z_0109f173-206c-471e-afac-c0d3e0aa2785",
>>> >> "entity_name": "mds.mummasstrg.strg-node1.gchapr",
>>> >> "os_id": "centos",
>>> >> "os_name": "CentOS Linux",
>>> >> "os_version": "8",
>>> >> "os_version_id": "8",
>>> >> "process_name": "ceph-mds",
>>> >> "stack_sig":
>>> >> "5238fe1be0b82b479b10ee7d17b5ad3182cdc93ede581af63d627a472a4fcf9e",
>>> >> "timestamp": "2025-04-17T08:50:35.931485Z",
>>> >> "utsname_hostname": "strg-node1",
>>> >> "utsname_machine": "x86_64",
>>> >> "utsname_release": "5.10.0-8-amd64",
>>> >> "utsname_sysname": "Linux",
>>> >> "utsname_version": "#1 SMP Debian 5.10.46-5 (2021-09-23)"
>>> >> }
>>> >>
>>> >
>>> > On Thu, Apr 17, 2025 at 2:55 PM Eugen Block <ebl...@nde.ag> wrote:
>>> >
>>> >> Oh right, I just noticed you had the status in your first message.
Is
>>> >> there any chance the MDS is out of memory? Can you paste a complete
>>> >> (fresh) startup log on some pastebin or so? Also, a complete 'ceph
>>> >> crash info <crash>' could be useful.
>>> >>
>>> >> Zitat von Amudhan P <amudha...@gmail.com>:
>>> >>
>>> >> > I have run the status and stat command below is the output.
>>> >> >
>>> >> > ceph -s
>>> >> >
>>> >> > cluster:
>>> >> > id: 7b3a4952-2131-11ec-94ce-0cc47a5ec98a
>>> >> > health: HEALTH_WARN
>>> >> > 2 failed cephadm daemon(s)
>>> >> > 1 filesystem is degraded
>>> >> > insufficient standby MDS daemons available
>>> >> > 7 daemons have recently crashed
>>> >> >
>>> >> > services:
>>> >> > mon: 3 daemons, quorum strg-node1,strg-node2,strg-node3 (age
20h)
>>> >> > mgr: strg-node2.unyimy(active, since 20h), standbys:
>>> >> strg-node1.ivkfid
>>> >> > mds: 1/1 daemons up
>>> >> > osd: 32 osds: 32 up (since 20h), 32 in (since 10w)
>>> >> >
>>> >> > data:
>>> >> > volumes: 0/1 healthy, 1 recovering
>>> >> > pools: 3 pools, 321 pgs
>>> >> > objects: 15.49M objects, 54 TiB
>>> >> > usage: 109 TiB used, 66 TiB / 175 TiB avail
>>> >> > pgs: 317 active+clean
>>> >> > 4 active+clean+scrubbing+deep
>>> >> >
>>> >> >
>>> >> > ceph mds stat
>>> >> > mumstrg:1/1 {0=mumstrg.strg-node1.gchapr=up:replay(laggy or
crashed)}
>>> >> >
>>> >> > ceph osd lspools
>>> >> > 1 device_health_metrics
>>> >> > 2 cephfs.mumstrg.meta
>>> >> > 3 cephfs.mumstrg.data
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Thu, Apr 17, 2025 at 10:33 AM Eugen Block <ebl...@nde.ag>
wrote:
>>> >> >
>>> >> >> What’s your overall Ceph status? It says data pool 3 not found.
>>> >> >>
>>> >> >> Zitat von Amudhan P <amudha...@gmail.com>:
>>> >> >>
>>> >> >> > There are few more logs in MDS. I have highlighted few log
lines
>>> >> which I
>>> >> >> am
>>> >> >> > not sure what it is.
>>> >> >> >
>>> >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -79>
>>> >> >> > 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5
asok(0x560a2c44e000)
>>> >> >> > register_command dump inode hook 0x560a2c354580
>>> >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -78>
>>> >> >> > 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5
asok(0x560a2c44e000)
>>> >> >> > register_command exit hook 0x560a2c354580
>>> >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -77>
>>> >> >> > 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5
asok(0x560a2c44e000)
>>> >> >> > register_command respawn hook 0x560a2c354580
>>> >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -76>
>>> >> >> > 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5
asok(0x560a2c44e000)
>>> >> >> > register_command heap hook 0x560a2c354580
>>> >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -75>
>>> >> >> > 2025-04-16T14:43:59.170+0000 7f74b5030700 1
>>> >> >> mds.mumstrg.strg-node3.xhxbwx
>>> >> >> > Updating MDS map to version 127517 f
>>> >> >> > rom mon.2
>>> >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -74>
>>> >> >> > 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5
asok(0x560a2c44e000)
>>> >> >> > register_command cpu_profiler hook 0x560a2c35458
>>> >> >> >
>>> >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -73>
>>> >> >> > 2025-04-16T14:43:59.170+0000 7f74b302c700 5
>>> >> >> > mds.beacon.mumstrg.strg-node3.xhxbwx Sending beacon up:boot
seq 1
>>> >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -72>
>>> >> >> > 2025-04-16T14:43:59.170+0000 7f74b302c700 10 monclient:
>>> >> _send_mon_message
>>> >> >> > to mon.strg-node3 at v2:10.0.103.3:3300/
>>> >> >> >
>>> >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -71>
>>> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 1
>>> >> >> mds.mumstrg.strg-node3.xhxbwx
>>> >> >> > Updating MDS map to version 127518 f
>>> >> >> > rom mon.2
>>> >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -70>
>>> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient:
>>> _renew_subs
>>> >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -69>
>>> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient:
>>> >> _send_mon_message
>>> >> >> > to mon.strg-node3 at v2:10.0.103.3:3300/
>>> >> >> >
>>> >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -68>
>>> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.purge_queue
>>> >> >> operator():
>>> >> >> > data pool 3 not found in OSDMap
>>> >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -67>
>>> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 5
asok(0x560a2c44e000)
>>> >> >> > register_command objecter_requests hook 0x560a2c
>>> >> >> > 3544c0
>>> >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -66>
>>> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient:
>>> _renew_subs
>>> >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -65>
>>> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient:
>>> >> _send_mon_message
>>> >> >> > to mon.strg-node3 at v2:10.0.103.3:3300/
>>> >> >> >
>>> >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -64>
>>> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 10
log_channel(cluster)
>>> >> >> > update_config to_monitors: true to_syslog: false
>>> >> >> > syslog_facility: daemon prio: info to_graylog: false
>>> graylog_host:
>>> >> >> > 127.0.0.1 graylog_port: 12201)
>>> >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -63>
>>> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.purge_queue
>>> >> >> operator():
>>> >> >> > data pool 3 not found in OSDMap
>>> >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -62>
>>> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.0
>>> handle_osd_map
>>> >> epoch
>>> >> >> > 0, 0 new blocklist entries
>>> >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -61>
>>> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518
>>> >> handle_mds_map
>>> >> >> i
>>> >> >> > am now mds.0.127518
>>> >> >> >
>>> >> >> >> Apr 16 20:13:59 strg-node3 bash[59114]: debug -60>
>>> >> >> >> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518
>>> >> handle_mds_map
>>> >> >> >> state change up:boot --> up:replay
>>> >> >> >> Apr 16 20:13:59 strg-node3 bash[59114]: debug -59>
>>> >> >> >> 2025-04-16T14:43:59.254+0000 7f74b5030700 5
>>> >> >> >> mds.beacon.mummasstrg.strg-node3.xhxbwx set_want_state:
up:boot
>>> ->
>>> >> >> up:replay
>>> >> >> >> Apr 16 20:13:59 strg-node3 bash[59114]: debug -58>
>>> >> >> >> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518
>>> >> replay_start
>>> >> >> >> *Apr 16 20:13:59 strg-node3 bash[59114]: debug -57>
>>> >> >> >> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518
>>> waiting
>>> >> for
>>> >> >> >> osdmap 45749 (which blocklists prior instance)*
>>> >> >> >> Apr 16 20:13:59 strg-node3 bash[59114]: debug -56>
>>> >> >> >> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient:
>>> >> >> _send_mon_message
>>> >> >> >> to mon.strg-node3 at v2:10.0.103.3:3300/0
>>> >> >> >> *Apr 16 20:13:59 strg-node3 bash[59114]: debug -55>
>>> >> >> >> 2025-04-16T14:43:59.254+0000 7f74b5030700 4
mds.0.purge_queue
>>> >> >> operator():
>>> >> >> >> data pool 3 not found in OSDMap*
>>> >> >> >>
>>> >> >> >
>>> >> >> >
>>> >> >> > On Thu, Apr 17, 2025 at 7:06 AM Amudhan P <
amudha...@gmail.com>
>>> >> wrote:
>>> >> >> >
>>> >> >> >> Eugen,
>>> >> >> >>
>>> >> >> >> This is the output for the command
>>> >> >> >> cephfs-journal-tool --rank=mumstrg:all --journal=purge_queue
>>> journal
>>> >> >> >> inspect
>>> >> >> >> Overall journal integrity: OK
>>> >> >> >> cephfs-journal-tool --rank=mumstrg:all --journal=mdlog
journal
>>> >> inspect
>>> >> >> >> Overall journal integrity: OK
>>> >> >> >>
>>> >> >> >> On Thu, Apr 17, 2025 at 2:59 AM Eugen Block <ebl...@nde.ag>
>>> wrote:
>>> >> >> >>
>>> >> >> >>> I think either your mdlog or the purge_queue journal is
>>> corrupted:
>>> >> >> >>>
>>> >> >> >>> 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506
>>> Booting:
>>> >> 2:
>>> >> >> >>> waiting for purge queue recovered
>>> >> >> >>> Apr 16 15:29:30 strg-node4 bash[7566]: debug -1>
>>> >> >> >>> 2025-04-16T09:59:30.146+0000 7f43d9085700 10 monclient:
>>> >> >> get_auth_request
>>> >> >> >>> con 0x562856a25400 auth_method 0
>>> >> >> >>> Apr 16 15:29:30 strg-node4 bash[7566]: debug 0>
>>> >> >> >>> 2025-04-16T09:59:30.230+0000 7f43ce06f700 -1 *** Caught
signal
>>> >> >> >>> (Segmentation fault) **
>>> >> >> >>> Apr 16 15:29:30 strg-node4 bash[7566]: in thread
7f43ce06f700
>>> >> >> >>> thread_name:md_log_replay
>>> >> >> >>>
>>> >> >> >>> Can you paste the output of this command?
>>> >> >> >>>
>>> >> >> >>> cephfs-journal-tool --rank={YOUR_CEPH_FS}:all
>>> --journal=purge_queue
>>> >> >> >>> journal inspect
>>> >> >> >>> cephfs-journal-tool --rank={YOUR_CEPH_FS}:all
--journal=mdlog
>>> >> journal
>>> >> >> >>> inspect
>>> >> >> >>>
>>> >> >> >>> I expect one or more damaged entries. Check this thread for
more
>>> >> >> details:
>>> >> >> >>>
>>> >> >> >>> https://www.spinics.net/lists/ceph-users/msg80124.html
>>> >> >> >>>
>>> >> >> >>> You should try to backup the journal, but in my case that
wasn't
>>> >> >> >>> possible, so I had no other choice than resetting it.
>>> >> >> >>>
>>> >> >> >>> Zitat von Amudhan P <amudha...@gmail.com>:
>>> >> >> >>>
>>> >> >> >>> > Hi,
>>> >> >> >>> >
>>> >> >> >>> > I am having 2 problems with my Ceph version 16.2.6
>>> >> >> >>> > (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific
(stable)
>>> >> deployed
>>> >> >> >>> thru
>>> >> >> >>> > cephadm.
>>> >> >> >>> >
>>> >> >> >>> > First issue :-
>>> >> >> >>> > 1 out 3 mon service went out of quorum .
>>> >> >> >>> > When restarted service it comes normal but after a few
>>> minutes in
>>> >> >> ceph
>>> >> >> >>> > watch log it reports slow ops and mon goes out of quorum.
>>> >> >> >>> > Node where this mon service failed had one weird thing
that I
>>> >> could
>>> >> >> see
>>> >> >> >>> 40%
>>> >> >> >>> > of wait in the top command. But I don't see any error in
>>> dmesg or
>>> >> >> >>> anything
>>> >> >> >>> > related to drive IO error.
>>> >> >> >>> > Below are the logs that were printed in ceph watch
command.
>>> >> >> >>> >
>>> >> >> >>> > 2025-04-16T09:30:00.000393+0530 mon.strg-node2 [WRN] [WRN]
>>> >> MON_DOWN:
>>> >> >> 1/3
>>> >> >> >>> > mons down, quorum strg-node2,strg-node3
>>> >> >> >>> > 2025-04-16T09:30:00.000416+0530 mon.strg-node2 [WRN]
>>> >> >> mon.strg-node1
>>> >> >> >>> > (rank 0) addr [v2:10.0.103.1:3300/0,v1:10.0.103.1:6789/0]
is
>>> down
>>> >> >> (out
>>> >> >> >>> of
>>> >> >> >>> > quorum)
>>> >> >> >>> >
>>> >> >> >>> > For now this is not appearing again.
>>> >> >> >>> >
>>> >> >> >>> >
>>> >> >> >>> > Second issue Cephfs degraded :-
>>> >> >> >>> > I have 2 MDS services running in 2 different nodes. Both
are
>>> in a
>>> >> >> >>> stopped
>>> >> >> >>> > state.
>>> >> >> >>> > when running Ceph -s command
>>> >> >> >>> >
>>> >> >> >>> > cluster:
>>> >> >> >>> > id: 7b3a4952-2131-11ec-94ce-0cc47a5ec98a
>>> >> >> >>> > health: HEALTH_WARN
>>> >> >> >>> > 2 failed cephadm daemon(s)
>>> >> >> >>> > 1 filesystem is degraded
>>> >> >> >>> > insufficient standby MDS daemons available
>>> >> >> >>> >
>>> >> >> >>> > services:
>>> >> >> >>> > mon: 3 daemons, quorum
strg-node1,strg-node2,strg-node3
>>> (age
>>> >> 4h)
>>> >> >> >>> > mgr: strg-node2.unyimy(active, since 4h), standbys:
>>> >> >> >>> strg-node1.ivkfid
>>> >> >> >>> > mds: 1/1 daemons up
>>> >> >> >>> > osd: 32 osds: 32 up (since 4h), 32 in (since 10w)
>>> >> >> >>> >
>>> >> >> >>> > data:
>>> >> >> >>> > volumes: 0/1 healthy, 1 recovering
>>> >> >> >>> > pools: 3 pools, 321 pgs
>>> >> >> >>> > objects: 15.49M objects, 54 TiB
>>> >> >> >>> > usage: 109 TiB used, 66 TiB / 175 TiB avail
>>> >> >> >>> > pgs: 321 active+clean
>>> >> >> >>> >
>>> >> >> >>> > Volume shows recovering but there wasn't any progress
till now
>>> >> even
>>> >> >> >>> manual
>>> >> >> >>> > start mds service fails again. In Ceph -s command under
>>> services
>>> >> it
>>> >> >> >>> shows
>>> >> >> >>> > mds up no any mds service is running.
>>> >> >> >>> >
>>> >> >> >>> > Below is a log snip from one of the mds service.
>>> >> >> >>> >
>>> >> >> >>> >
>>> >> >> >>> > -25> 2025-04-16T09:59:29.954+0000
7f43d0874700 1
>>> >> >> >>> > mds.0.journaler.pq(ro) _finish_read_head loghead(trim
>>> 13967032320,
>>> >> >> ex>
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -24>
>>> >> >> >>> > 2025-04-16T09:59:29.954+0000 7f43d0874700 1
>>> >> mds.0.journaler.pq(ro)
>>> >> >> >>> probing
>>> >> >> >>> > for end of the log
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -23>
>>> >> >> >>> > 2025-04-16T09:59:29.954+0000 7f43d9085700 10 monclient:
>>> >> >> get_auth_request
>>> >> >> >>> > con 0x562856a17400 auth_method 0
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -22>
>>> >> >> >>> > 2025-04-16T09:59:29.954+0000 7f43d8884700 10 monclient:
>>> >> >> get_auth_request
>>> >> >> >>> > con 0x562856a17c00 auth_method 0
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -21>
>>> >> >> >>> > 2025-04-16T09:59:29.974+0000 7f43cf071700 1
>>> >> >> mds.0.journaler.mdlog(ro)
>>> >> >> >>> > recover start
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -20>
>>> >> >> >>> > 2025-04-16T09:59:29.974+0000 7f43cf071700 1
>>> >> >> mds.0.journaler.mdlog(ro)
>>> >> >> >>> > read_head
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -19>
>>> >> >> >>> > 2025-04-16T09:59:29.974+0000 7f43cf071700 4 mds.0.log
>>> Waiting for
>>> >> >> >>> journal
>>> >> >> >>> > 0x200 to recover...
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -18>
>>> >> >> >>> > 2025-04-16T09:59:29.974+0000 7f43d8083700 10 monclient:
>>> >> >> get_auth_request
>>> >> >> >>> > con 0x562856a25000 auth_method 0
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -17>
>>> >> >> >>> > 2025-04-16T09:59:29.998+0000 7f43d0874700 1
>>> >> mds.0.journaler.pq(ro)
>>> >> >> >>> > _finish_probe_end write_pos = 13968309289 (hea>
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -16>
>>> >> >> >>> > 2025-04-16T09:59:29.998+0000 7f43d0874700 4
mds.0.purge_queue
>>> >> >> >>> operator():
>>> >> >> >>> > open complete
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -15>
>>> >> >> >>> > 2025-04-16T09:59:29.998+0000 7f43d0874700 1
>>> >> mds.0.journaler.pq(ro)
>>> >> >> >>> > set_writeable
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -14>
>>> >> >> >>> > 2025-04-16T09:59:29.998+0000 7f43cf872700 1
>>> >> >> mds.0.journaler.mdlog(ro)
>>> >> >> >>> > _finish_read_head loghead(trim 189741504921>
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -13>
>>> >> >> >>> > 2025-04-16T09:59:29.998+0000 7f43cf872700 1
>>> >> >> mds.0.journaler.mdlog(ro)
>>> >> >> >>> > probing for end of the log
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -12>
>>> >> >> >>> > 2025-04-16T09:59:30.002+0000 7f43d9085700 10 monclient:
>>> >> >> get_auth_request
>>> >> >> >>> > con 0x562856a25c00 auth_method 0
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -11>
>>> >> >> >>> > 2025-04-16T09:59:30.098+0000 7f43cf872700 1
>>> >> >> mds.0.journaler.mdlog(ro)
>>> >> >> >>> > _finish_probe_end write_pos = 1897428915052>
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -10>
>>> >> >> >>> > 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log
Journal
>>> >> 0x200
>>> >> >> >>> > recovered.
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -9>
>>> >> >> >>> > 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log
>>> Recovered
>>> >> >> journal
>>> >> >> >>> > 0x200 in format 1
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -8>
>>> >> >> >>> > 2025-04-16T09:59:30.098+0000 7f43cf071700 2 mds.0.127506
>>> >> Booting: 1:
>>> >> >> >>> > loading/discovering base inodes
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -7>
>>> >> >> >>> > 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache
>>> creating
>>> >> >> system
>>> >> >> >>> > inode with ino:0x100
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -6>
>>> >> >> >>> > 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache
>>> creating
>>> >> >> system
>>> >> >> >>> > inode with ino:0x1
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -5>
>>> >> >> >>> > 2025-04-16T09:59:30.098+0000 7f43d8884700 10 monclient:
>>> >> >> get_auth_request
>>> >> >> >>> > con 0x562856a25800 auth_method 0
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -4>
>>> >> >> >>> > 2025-04-16T09:59:30.098+0000 7f43d8083700 10 monclient:
>>> >> >> get_auth_request
>>> >> >> >>> > con 0x562856a5dc00 auth_method 0
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -3>
>>> >> >> >>> > 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506
>>> >> Booting: 2:
>>> >> >> >>> > replaying mds log
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -2>
>>> >> >> >>> > 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506
>>> >> Booting: 2:
>>> >> >> >>> > waiting for purge queue recovered
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -1>
>>> >> >> >>> > 2025-04-16T09:59:30.146+0000 7f43d9085700 10 monclient:
>>> >> >> get_auth_request
>>> >> >> >>> > con 0x562856a25400 auth_method 0
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug 0>
>>> >> >> >>> > 2025-04-16T09:59:30.230+0000 7f43ce06f700 -1 *** Caught
signal
>>> >> >> >>> > (Segmentation fault) **
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: in thread
7f43ce06f700
>>> >> >> >>> > thread_name:md_log_replay
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: ceph version
16.2.6
>>> >> >> >>> > (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific
(stable)
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: 1:
>>> >> >> >>> /lib64/libpthread.so.0(+0x12b20)
>>> >> >> >>> > [0x7f43dd293b20]
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: 2:
>>> >> >> >>> > /usr/lib64/ceph/libceph-common.so.2(+0x8ec7a00)
>>> [0x7f43e6ef3a00]
>>> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: NOTE: a copy of
the
>>> >> >> executable,
>>> >> >> >>> or
>>> >> >> >>> > `objdump -rdS <executable>` is needed to interpret this.
>>> >> >> >>> >
>>> >> >> >>> >
>>> >> >> >>> > Not sure what caused the issue. I couldn't find any
resources
>>> to
>>> >> fix
>>> >> >> >>> this
>>> >> >> >>> > issue.
>>> >> >> >>> > Need help from someone to bring the ceph cluster online.
>>> >> >> >>> > _______________________________________________
>>> >> >> >>> > ceph-users mailing list -- ceph-users@ceph.io
>>> >> >> >>> > To unsubscribe send an email to ceph-users-le...@ceph.io
>>> >> >> >>>
>>> >> >> >>>
>>> >> >> >>> _______________________________________________
>>> >> >> >>> ceph-users mailing list -- ceph-users@ceph.io
>>> >> >> >>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>> >> >> >>>
>>> >> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>>
>>>
>>>
>>>
