Was there any issue in your network or anything? Something that would
explain the MDS crash? I would recommend to scan syslog, dmesg etc.
for anything suspicious. If you don't find anything, you might need to
go through
https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/ to
get your CephFS back up.
Don't forget to create a backup of the journals. Read the instructions
carefully, you might not need to do all the mentioned steps. I would
start with (for both journals where applicable):
cephfs-journal-tool journal export backup.bin
cephfs-journal-tool event recover_dentries summary
cephfs-journal-tool [--rank=<fs_name>:{mds-rank|all}] journal reset
--yes-i-really-really-mean-it
cephfs-table-tool all reset session
But again, read the instructions carefully. This procedure is usually
the last resort, I would first try to find out what happened causing
this situation.
Zitat von Amudhan P <amudha...@gmail.com>:
> I don't think I have a memory issue.
> Sorry, for sending log as file Pastebin is not working due to size
> limitation.
>
> Crash info
>
>> ceph crash info
>> 2025-04-17T08:50:35.931485Z_0109f173-206c-471e-afac-c0d3e0aa2785
>> {
>> "backtrace": [
>> "/lib64/libpthread.so.0(+0x12b20) [0x7fbd42bbdb20]",
>> "/usr/lib64/ceph/libceph-common.so.2(+0x8ec7a00)
[0x7fbd4c81da00]"
>> ],
>> "ceph_version": "16.2.6",
>> "crash_id":
>> "2025-04-17T08:50:35.931485Z_0109f173-206c-471e-afac-c0d3e0aa2785",
>> "entity_name": "mds.mummasstrg.strg-node1.gchapr",
>> "os_id": "centos",
>> "os_name": "CentOS Linux",
>> "os_version": "8",
>> "os_version_id": "8",
>> "process_name": "ceph-mds",
>> "stack_sig":
>> "5238fe1be0b82b479b10ee7d17b5ad3182cdc93ede581af63d627a472a4fcf9e",
>> "timestamp": "2025-04-17T08:50:35.931485Z",
>> "utsname_hostname": "strg-node1",
>> "utsname_machine": "x86_64",
>> "utsname_release": "5.10.0-8-amd64",
>> "utsname_sysname": "Linux",
>> "utsname_version": "#1 SMP Debian 5.10.46-5 (2021-09-23)"
>> }
>>
>
> On Thu, Apr 17, 2025 at 2:55 PM Eugen Block <ebl...@nde.ag> wrote:
>
>> Oh right, I just noticed you had the status in your first message. Is
>> there any chance the MDS is out of memory? Can you paste a complete
>> (fresh) startup log on some pastebin or so? Also, a complete 'ceph
>> crash info <crash>' could be useful.
>>
>> Zitat von Amudhan P <amudha...@gmail.com>:
>>
>> > I have run the status and stat command below is the output.
>> >
>> > ceph -s
>> >
>> > cluster:
>> > id: 7b3a4952-2131-11ec-94ce-0cc47a5ec98a
>> > health: HEALTH_WARN
>> > 2 failed cephadm daemon(s)
>> > 1 filesystem is degraded
>> > insufficient standby MDS daemons available
>> > 7 daemons have recently crashed
>> >
>> > services:
>> > mon: 3 daemons, quorum strg-node1,strg-node2,strg-node3 (age 20h)
>> > mgr: strg-node2.unyimy(active, since 20h), standbys:
>> strg-node1.ivkfid
>> > mds: 1/1 daemons up
>> > osd: 32 osds: 32 up (since 20h), 32 in (since 10w)
>> >
>> > data:
>> > volumes: 0/1 healthy, 1 recovering
>> > pools: 3 pools, 321 pgs
>> > objects: 15.49M objects, 54 TiB
>> > usage: 109 TiB used, 66 TiB / 175 TiB avail
>> > pgs: 317 active+clean
>> > 4 active+clean+scrubbing+deep
>> >
>> >
>> > ceph mds stat
>> > mumstrg:1/1 {0=mumstrg.strg-node1.gchapr=up:replay(laggy or crashed)}
>> >
>> > ceph osd lspools
>> > 1 device_health_metrics
>> > 2 cephfs.mumstrg.meta
>> > 3 cephfs.mumstrg.data
>> >
>> >
>> >
>> > On Thu, Apr 17, 2025 at 10:33 AM Eugen Block <ebl...@nde.ag> wrote:
>> >
>> >> What’s your overall Ceph status? It says data pool 3 not found.
>> >>
>> >> Zitat von Amudhan P <amudha...@gmail.com>:
>> >>
>> >> > There are few more logs in MDS. I have highlighted few log lines
>> which I
>> >> am
>> >> > not sure what it is.
>> >> >
>> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -79>
>> >> > 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000)
>> >> > register_command dump inode hook 0x560a2c354580
>> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -78>
>> >> > 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000)
>> >> > register_command exit hook 0x560a2c354580
>> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -77>
>> >> > 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000)
>> >> > register_command respawn hook 0x560a2c354580
>> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -76>
>> >> > 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000)
>> >> > register_command heap hook 0x560a2c354580
>> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -75>
>> >> > 2025-04-16T14:43:59.170+0000 7f74b5030700 1
>> >> mds.mumstrg.strg-node3.xhxbwx
>> >> > Updating MDS map to version 127517 f
>> >> > rom mon.2
>> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -74>
>> >> > 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000)
>> >> > register_command cpu_profiler hook 0x560a2c35458
>> >> >
>> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -73>
>> >> > 2025-04-16T14:43:59.170+0000 7f74b302c700 5
>> >> > mds.beacon.mumstrg.strg-node3.xhxbwx Sending beacon up:boot seq 1
>> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -72>
>> >> > 2025-04-16T14:43:59.170+0000 7f74b302c700 10 monclient:
>> _send_mon_message
>> >> > to mon.strg-node3 at v2:10.0.103.3:3300/
>> >> >
>> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -71>
>> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 1
>> >> mds.mumstrg.strg-node3.xhxbwx
>> >> > Updating MDS map to version 127518 f
>> >> > rom mon.2
>> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -70>
>> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient:
_renew_subs
>> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -69>
>> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient:
>> _send_mon_message
>> >> > to mon.strg-node3 at v2:10.0.103.3:3300/
>> >> >
>> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -68>
>> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.purge_queue
>> >> operator():
>> >> > data pool 3 not found in OSDMap
>> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -67>
>> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 5 asok(0x560a2c44e000)
>> >> > register_command objecter_requests hook 0x560a2c
>> >> > 3544c0
>> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -66>
>> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient:
_renew_subs
>> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -65>
>> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient:
>> _send_mon_message
>> >> > to mon.strg-node3 at v2:10.0.103.3:3300/
>> >> >
>> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -64>
>> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 10 log_channel(cluster)
>> >> > update_config to_monitors: true to_syslog: false
>> >> > syslog_facility: daemon prio: info to_graylog: false
graylog_host:
>> >> > 127.0.0.1 graylog_port: 12201)
>> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -63>
>> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.purge_queue
>> >> operator():
>> >> > data pool 3 not found in OSDMap
>> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -62>
>> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.0
handle_osd_map
>> epoch
>> >> > 0, 0 new blocklist entries
>> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -61>
>> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518
>> handle_mds_map
>> >> i
>> >> > am now mds.0.127518
>> >> >
>> >> >> Apr 16 20:13:59 strg-node3 bash[59114]: debug -60>
>> >> >> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518
>> handle_mds_map
>> >> >> state change up:boot --> up:replay
>> >> >> Apr 16 20:13:59 strg-node3 bash[59114]: debug -59>
>> >> >> 2025-04-16T14:43:59.254+0000 7f74b5030700 5
>> >> >> mds.beacon.mummasstrg.strg-node3.xhxbwx set_want_state: up:boot
->
>> >> up:replay
>> >> >> Apr 16 20:13:59 strg-node3 bash[59114]: debug -58>
>> >> >> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518
>> replay_start
>> >> >> *Apr 16 20:13:59 strg-node3 bash[59114]: debug -57>
>> >> >> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518
waiting
>> for
>> >> >> osdmap 45749 (which blocklists prior instance)*
>> >> >> Apr 16 20:13:59 strg-node3 bash[59114]: debug -56>
>> >> >> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient:
>> >> _send_mon_message
>> >> >> to mon.strg-node3 at v2:10.0.103.3:3300/0
>> >> >> *Apr 16 20:13:59 strg-node3 bash[59114]: debug -55>
>> >> >> 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.purge_queue
>> >> operator():
>> >> >> data pool 3 not found in OSDMap*
>> >> >>
>> >> >
>> >> >
>> >> > On Thu, Apr 17, 2025 at 7:06 AM Amudhan P <amudha...@gmail.com>
>> wrote:
>> >> >
>> >> >> Eugen,
>> >> >>
>> >> >> This is the output for the command
>> >> >> cephfs-journal-tool --rank=mumstrg:all --journal=purge_queue
journal
>> >> >> inspect
>> >> >> Overall journal integrity: OK
>> >> >> cephfs-journal-tool --rank=mumstrg:all --journal=mdlog journal
>> inspect
>> >> >> Overall journal integrity: OK
>> >> >>
>> >> >> On Thu, Apr 17, 2025 at 2:59 AM Eugen Block <ebl...@nde.ag>
wrote:
>> >> >>
>> >> >>> I think either your mdlog or the purge_queue journal is
corrupted:
>> >> >>>
>> >> >>> 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506
Booting:
>> 2:
>> >> >>> waiting for purge queue recovered
>> >> >>> Apr 16 15:29:30 strg-node4 bash[7566]: debug -1>
>> >> >>> 2025-04-16T09:59:30.146+0000 7f43d9085700 10 monclient:
>> >> get_auth_request
>> >> >>> con 0x562856a25400 auth_method 0
>> >> >>> Apr 16 15:29:30 strg-node4 bash[7566]: debug 0>
>> >> >>> 2025-04-16T09:59:30.230+0000 7f43ce06f700 -1 *** Caught signal
>> >> >>> (Segmentation fault) **
>> >> >>> Apr 16 15:29:30 strg-node4 bash[7566]: in thread 7f43ce06f700
>> >> >>> thread_name:md_log_replay
>> >> >>>
>> >> >>> Can you paste the output of this command?
>> >> >>>
>> >> >>> cephfs-journal-tool --rank={YOUR_CEPH_FS}:all
--journal=purge_queue
>> >> >>> journal inspect
>> >> >>> cephfs-journal-tool --rank={YOUR_CEPH_FS}:all --journal=mdlog
>> journal
>> >> >>> inspect
>> >> >>>
>> >> >>> I expect one or more damaged entries. Check this thread for more
>> >> details:
>> >> >>>
>> >> >>> https://www.spinics.net/lists/ceph-users/msg80124.html
>> >> >>>
>> >> >>> You should try to backup the journal, but in my case that wasn't
>> >> >>> possible, so I had no other choice than resetting it.
>> >> >>>
>> >> >>> Zitat von Amudhan P <amudha...@gmail.com>:
>> >> >>>
>> >> >>> > Hi,
>> >> >>> >
>> >> >>> > I am having 2 problems with my Ceph version 16.2.6
>> >> >>> > (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable)
>> deployed
>> >> >>> thru
>> >> >>> > cephadm.
>> >> >>> >
>> >> >>> > First issue :-
>> >> >>> > 1 out 3 mon service went out of quorum .
>> >> >>> > When restarted service it comes normal but after a few
minutes in
>> >> ceph
>> >> >>> > watch log it reports slow ops and mon goes out of quorum.
>> >> >>> > Node where this mon service failed had one weird thing that I
>> could
>> >> see
>> >> >>> 40%
>> >> >>> > of wait in the top command. But I don't see any error in
dmesg or
>> >> >>> anything
>> >> >>> > related to drive IO error.
>> >> >>> > Below are the logs that were printed in ceph watch command.
>> >> >>> >
>> >> >>> > 2025-04-16T09:30:00.000393+0530 mon.strg-node2 [WRN] [WRN]
>> MON_DOWN:
>> >> 1/3
>> >> >>> > mons down, quorum strg-node2,strg-node3
>> >> >>> > 2025-04-16T09:30:00.000416+0530 mon.strg-node2 [WRN]
>> >> mon.strg-node1
>> >> >>> > (rank 0) addr [v2:10.0.103.1:3300/0,v1:10.0.103.1:6789/0] is
down
>> >> (out
>> >> >>> of
>> >> >>> > quorum)
>> >> >>> >
>> >> >>> > For now this is not appearing again.
>> >> >>> >
>> >> >>> >
>> >> >>> > Second issue Cephfs degraded :-
>> >> >>> > I have 2 MDS services running in 2 different nodes. Both are
in a
>> >> >>> stopped
>> >> >>> > state.
>> >> >>> > when running Ceph -s command
>> >> >>> >
>> >> >>> > cluster:
>> >> >>> > id: 7b3a4952-2131-11ec-94ce-0cc47a5ec98a
>> >> >>> > health: HEALTH_WARN
>> >> >>> > 2 failed cephadm daemon(s)
>> >> >>> > 1 filesystem is degraded
>> >> >>> > insufficient standby MDS daemons available
>> >> >>> >
>> >> >>> > services:
>> >> >>> > mon: 3 daemons, quorum strg-node1,strg-node2,strg-node3
(age
>> 4h)
>> >> >>> > mgr: strg-node2.unyimy(active, since 4h), standbys:
>> >> >>> strg-node1.ivkfid
>> >> >>> > mds: 1/1 daemons up
>> >> >>> > osd: 32 osds: 32 up (since 4h), 32 in (since 10w)
>> >> >>> >
>> >> >>> > data:
>> >> >>> > volumes: 0/1 healthy, 1 recovering
>> >> >>> > pools: 3 pools, 321 pgs
>> >> >>> > objects: 15.49M objects, 54 TiB
>> >> >>> > usage: 109 TiB used, 66 TiB / 175 TiB avail
>> >> >>> > pgs: 321 active+clean
>> >> >>> >
>> >> >>> > Volume shows recovering but there wasn't any progress till now
>> even
>> >> >>> manual
>> >> >>> > start mds service fails again. In Ceph -s command under
services
>> it
>> >> >>> shows
>> >> >>> > mds up no any mds service is running.
>> >> >>> >
>> >> >>> > Below is a log snip from one of the mds service.
>> >> >>> >
>> >> >>> >
>> >> >>> > -25> 2025-04-16T09:59:29.954+0000 7f43d0874700 1
>> >> >>> > mds.0.journaler.pq(ro) _finish_read_head loghead(trim
13967032320,
>> >> ex>
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -24>
>> >> >>> > 2025-04-16T09:59:29.954+0000 7f43d0874700 1
>> mds.0.journaler.pq(ro)
>> >> >>> probing
>> >> >>> > for end of the log
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -23>
>> >> >>> > 2025-04-16T09:59:29.954+0000 7f43d9085700 10 monclient:
>> >> get_auth_request
>> >> >>> > con 0x562856a17400 auth_method 0
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -22>
>> >> >>> > 2025-04-16T09:59:29.954+0000 7f43d8884700 10 monclient:
>> >> get_auth_request
>> >> >>> > con 0x562856a17c00 auth_method 0
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -21>
>> >> >>> > 2025-04-16T09:59:29.974+0000 7f43cf071700 1
>> >> mds.0.journaler.mdlog(ro)
>> >> >>> > recover start
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -20>
>> >> >>> > 2025-04-16T09:59:29.974+0000 7f43cf071700 1
>> >> mds.0.journaler.mdlog(ro)
>> >> >>> > read_head
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -19>
>> >> >>> > 2025-04-16T09:59:29.974+0000 7f43cf071700 4 mds.0.log
Waiting for
>> >> >>> journal
>> >> >>> > 0x200 to recover...
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -18>
>> >> >>> > 2025-04-16T09:59:29.974+0000 7f43d8083700 10 monclient:
>> >> get_auth_request
>> >> >>> > con 0x562856a25000 auth_method 0
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -17>
>> >> >>> > 2025-04-16T09:59:29.998+0000 7f43d0874700 1
>> mds.0.journaler.pq(ro)
>> >> >>> > _finish_probe_end write_pos = 13968309289 (hea>
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -16>
>> >> >>> > 2025-04-16T09:59:29.998+0000 7f43d0874700 4 mds.0.purge_queue
>> >> >>> operator():
>> >> >>> > open complete
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -15>
>> >> >>> > 2025-04-16T09:59:29.998+0000 7f43d0874700 1
>> mds.0.journaler.pq(ro)
>> >> >>> > set_writeable
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -14>
>> >> >>> > 2025-04-16T09:59:29.998+0000 7f43cf872700 1
>> >> mds.0.journaler.mdlog(ro)
>> >> >>> > _finish_read_head loghead(trim 189741504921>
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -13>
>> >> >>> > 2025-04-16T09:59:29.998+0000 7f43cf872700 1
>> >> mds.0.journaler.mdlog(ro)
>> >> >>> > probing for end of the log
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -12>
>> >> >>> > 2025-04-16T09:59:30.002+0000 7f43d9085700 10 monclient:
>> >> get_auth_request
>> >> >>> > con 0x562856a25c00 auth_method 0
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -11>
>> >> >>> > 2025-04-16T09:59:30.098+0000 7f43cf872700 1
>> >> mds.0.journaler.mdlog(ro)
>> >> >>> > _finish_probe_end write_pos = 1897428915052>
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -10>
>> >> >>> > 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log Journal
>> 0x200
>> >> >>> > recovered.
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -9>
>> >> >>> > 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log
Recovered
>> >> journal
>> >> >>> > 0x200 in format 1
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -8>
>> >> >>> > 2025-04-16T09:59:30.098+0000 7f43cf071700 2 mds.0.127506
>> Booting: 1:
>> >> >>> > loading/discovering base inodes
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -7>
>> >> >>> > 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache
creating
>> >> system
>> >> >>> > inode with ino:0x100
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -6>
>> >> >>> > 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache
creating
>> >> system
>> >> >>> > inode with ino:0x1
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -5>
>> >> >>> > 2025-04-16T09:59:30.098+0000 7f43d8884700 10 monclient:
>> >> get_auth_request
>> >> >>> > con 0x562856a25800 auth_method 0
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -4>
>> >> >>> > 2025-04-16T09:59:30.098+0000 7f43d8083700 10 monclient:
>> >> get_auth_request
>> >> >>> > con 0x562856a5dc00 auth_method 0
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -3>
>> >> >>> > 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506
>> Booting: 2:
>> >> >>> > replaying mds log
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -2>
>> >> >>> > 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506
>> Booting: 2:
>> >> >>> > waiting for purge queue recovered
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -1>
>> >> >>> > 2025-04-16T09:59:30.146+0000 7f43d9085700 10 monclient:
>> >> get_auth_request
>> >> >>> > con 0x562856a25400 auth_method 0
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug 0>
>> >> >>> > 2025-04-16T09:59:30.230+0000 7f43ce06f700 -1 *** Caught signal
>> >> >>> > (Segmentation fault) **
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: in thread 7f43ce06f700
>> >> >>> > thread_name:md_log_replay
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: ceph version 16.2.6
>> >> >>> > (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable)
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: 1:
>> >> >>> /lib64/libpthread.so.0(+0x12b20)
>> >> >>> > [0x7f43dd293b20]
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: 2:
>> >> >>> > /usr/lib64/ceph/libceph-common.so.2(+0x8ec7a00)
[0x7f43e6ef3a00]
>> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: NOTE: a copy of the
>> >> executable,
>> >> >>> or
>> >> >>> > `objdump -rdS <executable>` is needed to interpret this.
>> >> >>> >
>> >> >>> >
>> >> >>> > Not sure what caused the issue. I couldn't find any resources
to
>> fix
>> >> >>> this
>> >> >>> > issue.
>> >> >>> > Need help from someone to bring the ceph cluster online.
>> >> >>> > _______________________________________________
>> >> >>> > ceph-users mailing list -- ceph-users@ceph.io
>> >> >>> > To unsubscribe send an email to ceph-users-le...@ceph.io
>> >> >>>
>> >> >>>
>> >> >>> _______________________________________________
>> >> >>> ceph-users mailing list -- ceph-users@ceph.io
>> >> >>> To unsubscribe send an email to ceph-users-le...@ceph.io
>> >> >>>
>> >> >>
>> >>
>> >>
>> >>
>> >>
>>
>>
>>
>>
