[ceph-users] Re: Mds daemon damaged - assert failed

2024-09-27 Thread Kyriazis, George
I am running 18.2.2, which apparently is the latest one available for proxmox at this time (9/2024). I’d rather not mess around with backporting and testing fixes at this point, since this is our “production” cluster.. If it was not a production one, then I could possibly play around with this

[ceph-users] Re: Mds daemon damaged - assert failed

2024-09-27 Thread Konstantin Shalygin
Hi, The [2] is the fix for [1] and should be backported? Currently fields are not filled, so no one knows that backports are needed k > On 27 Sep 2024, at 11:01, Frédéric Nass > wrote: > > Hi George, > > Looks like you hit this one [1]. Can't find the fix [2] in Reef release notes > [3].

[ceph-users] Re: Mds daemon damaged - assert failed

2024-09-27 Thread Frédéric Nass
Hi George, Looks like you hit this one [1]. Can't find the fix [2] in Reef release notes [3]. You'll have to cherry pick it and build sources or wait for it to come to next build. Regards, Frédéric. [1] https://tracker.ceph.com/issues/58878 [2] https://github.com/ceph/ceph/pull/55265 [3] https

[ceph-users] Re: Mds daemon damaged - assert failed

2024-09-26 Thread Eugen Block
It could be a bug, sure, but I haven't searched tracker too long, maybe there is an existing bug, I'd leave it to the devs to comment on that. But the assert alone isn't of much help (to me), more mds logs could help track this down. Zitat von "Kyriazis, George" : On Sep 25, 2024, at 1:05

[ceph-users] Re: Mds daemon damaged - assert failed

2024-09-25 Thread Kyriazis, George
> On Sep 25, 2024, at 1:05 AM, Eugen Block wrote: > > Great that you got your filesystem back. > >> cephfs-journal-tool journal export >> cephfs-journal-tool event recover_dentries summary >> >> Both failed > > Your export command seems to be missing the output file, or was it not the > exa

[ceph-users] Re: Mds daemon damaged - assert failed

2024-09-24 Thread Eugen Block
Great that you got your filesystem back. cephfs-journal-tool journal export cephfs-journal-tool event recover_dentries summary Both failed Your export command seems to be missing the output file, or was it not the exact command? Also, I understand that the metadata itself is sitting on th

[ceph-users] Re: Mds daemon damaged - assert failed

2024-09-24 Thread Kyriazis, George
I managed to recover my filesystem. cephfs-journal-tool journal export cephfs-journal-tool event recover_dentries summary Both failed But truncating the journal and following some of the instructions in https://people.redhat.com/bhubbard/nature/default/cephfs/disaster-recovery-experts/ helped

[ceph-users] Re: Mds daemon damaged - assert failed

2024-09-24 Thread Eugen Block
Hi, I would probably start by inspecting the journal with the cephfs-journal-tool [0]: cephfs-journal-tool [--rank=:{mds-rank|all}] journal inspect And it could be helful to have the logs prior to the assert. [0] https://docs.ceph.com/en/latest/cephfs/cephfs-journal-tool/#example-journal-