[ceph-users] Re: mds dump inode crashes file system

2023-05-26 Thread Frank Schilder
_ From: Xiubo Li Sent: Wednesday, May 17, 2023 7:43 AM To: Gregory Farnum; Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: mds dump inode crashes file system On 5/16/23 21:55, Gregory Farnum wrote: > On Fri, May 12, 2023 at 5:28 AM Frank Schilder wrote: >> De

[ceph-users] Re: mds dump inode crashes file system

2023-05-16 Thread Xiubo Li
On 5/16/23 21:55, Gregory Farnum wrote: On Fri, May 12, 2023 at 5:28 AM Frank Schilder wrote: Dear Xiubo and others. I have never heard about that option until now. How do I check that and how to I disable it if necessary? I'm in meetings pretty much all day and will try to send some more i

[ceph-users] Re: mds dump inode crashes file system

2023-05-16 Thread Xiubo Li
the mds dump inode command. Thanks a lot and best regards, = Frank Schilder AIT Risø Campus Bygning 109, rum S14 From: Frank Schilder Sent: Thursday, May 11, 2023 12:26 PM To: Xiubo Li; ceph-users@ceph.io Subject: [ceph-users] Re: mds dump inode cr

[ceph-users] Re: mds dump inode crashes file system

2023-05-16 Thread Gregory Farnum
On Fri, May 12, 2023 at 5:28 AM Frank Schilder wrote: > > Dear Xiubo and others. > > >> I have never heard about that option until now. How do I check that and > >> how to I disable it if necessary? > >> I'm in meetings pretty much all day and will try to send some more info > >> later. > > > >

[ceph-users] Re: mds dump inode crashes file system

2023-05-16 Thread Xiubo Li
__ From: Frank Schilder Sent: Monday, May 15, 2023 6:33 PM To: Xiubo Li; ceph-users@ceph.io Subject: [ceph-users] Re: mds dump inode crashes file system Dear Xiubo, I uploaded the cache dump, the MDS log and the dmesg log containing the snaptrace dump to ceph-post-file: 763955a3-7d37-408a-

[ceph-users] Re: mds dump inode crashes file system

2023-05-16 Thread Xiubo Li
Thanks Best regards, = Frank Schilder AIT Risø Campus Bygning 109, rum S14 From: Xiubo Li Sent: Friday, May 12, 2023 3:44 PM To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: mds dump inode crashes file system On 5/12

[ceph-users] Re: mds dump inode crashes file system

2023-05-16 Thread Frank Schilder
specific commands). Thanks and best regards, = Frank Schilder AIT Risø Campus Bygning 109, rum S14 From: Frank Schilder Sent: Monday, May 15, 2023 6:33 PM To: Xiubo Li; ceph-users@ceph.io Subject: [ceph-users] Re: mds dump inode cras

[ceph-users] Re: mds dump inode crashes file system

2023-05-15 Thread Frank Schilder
From: Xiubo Li Sent: Friday, May 12, 2023 3:44 PM To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: mds dump inode crashes file system On 5/12/23 20:27, Frank Schilder wrote: > Dear Xiubo and others. > >>> I have never heard about that option until now.

[ceph-users] Re: mds dump inode crashes file system

2023-05-12 Thread Xiubo Li
_________________ From: Xiubo Li Sent: Friday, May 12, 2023 8:04 AM To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: mds dump inode crashes file system On 5/11/23 20:12, Frank Schilder wrote: Dear Xiubo, please see also my previous e-mail about the async

[ceph-users] Re: mds dump inode crashes file system

2023-05-12 Thread Frank Schilder
napshot tomorrow we end up in the degraded situation again. I really need instructions for how to check what is broken without an MDS crash and then how to fix it. Thanks and best regards, = Frank Schilder AIT Risø Campus Bygning 109, rum S14 ____________

[ceph-users] Re: mds dump inode crashes file system

2023-05-11 Thread Xiubo Li
the metadata in cephfs. Thanks Thanks a lot and best regards, = Frank Schilder AIT Risø Campus Bygning 109, rum S14 From: Frank Schilder Sent: Thursday, May 11, 2023 12:26 PM To: Xiubo Li;ceph-users@ceph.io Subject: [ceph-users] Re: mds d

[ceph-users] Re: mds dump inode crashes file system

2023-05-11 Thread Xiubo Li
y, May 11, 2023 3:45 AM To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: mds dump inode crashes file system Hey Frank, On 5/10/23 21:44, Frank Schilder wrote: The kernel message that shows up on boot on the file server in text format: May 10 13:56:59 rit-pfile01 kernel: WAR

[ceph-users] Re: mds dump inode crashes file system

2023-05-11 Thread Frank Schilder
nk Schilder AIT Risø Campus Bygning 109, rum S14 From: Frank Schilder Sent: Thursday, May 11, 2023 12:26 PM To: Xiubo Li; ceph-users@ceph.io Subject: [ceph-users] Re: mds dump inode crashes file system Dear Xiubo, thanks for your reply. > BTW, did y

[ceph-users] Re: mds dump inode crashes file system

2023-05-11 Thread Frank Schilder
= Frank Schilder AIT Risø Campus Bygning 109, rum S14 From: Frank Schilder Sent: Thursday, May 11, 2023 12:34 PM To: Gregory Farnum Cc: ceph-users@ceph.io Subject: [ceph-users] Re: mds dump inode crashes file system Dear Gregory, > I would start by looki

[ceph-users] Re: mds dump inode crashes file system

2023-05-11 Thread Frank Schilder
y pointer you can provide. Thanks and best regards, = Frank Schilder AIT Risø Campus Bygning 109, rum S14 From: Frank Schilder Sent: Wednesday, May 10, 2023 6:18 PM To: Gregory Farnum Cc: ceph-users@ceph.io Subject: [ceph-users] Re: mds d

[ceph-users] Re: mds dump inode crashes file system

2023-05-11 Thread Frank Schilder
ilder AIT Risø Campus Bygning 109, rum S14 From: Xiubo Li Sent: Thursday, May 11, 2023 3:45 AM To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: mds dump inode crashes file system Hey Frank, On 5/10/23 21:44, Frank Schilder wrote: > T

[ceph-users] Re: mds dump inode crashes file system

2023-05-10 Thread Xiubo Li
Frank Schilder AIT Risø Campus Bygning 109, rum S14 From: Frank Schilder Sent: Wednesday, May 10, 2023 3:36 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: mds dump inode crashes file system For the "mds dump inode" command I c

[ceph-users] Re: mds dump inode crashes file system

2023-05-10 Thread Frank Schilder
p and best regards, = Frank Schilder AIT Risø Campus Bygning 109, rum S14 From: Gregory Farnum Sent: Wednesday, May 10, 2023 4:37 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: mds dump inode crashes file system On W

[ceph-users] Re: mds dump inode crashes file system

2023-05-10 Thread Gregory Farnum
to head home > now ... > > Thanks and best regards, > = > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > > From: Gregory Farnum > Sent: Wednesday, May 10, 2023 4:26 PM > To: Frank Schilder >

[ceph-users] Re: mds dump inode crashes file system

2023-05-10 Thread Frank Schilder
From: Gregory Farnum Sent: Wednesday, May 10, 2023 4:26 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: mds dump inode crashes file system This is a very strange assert to be hitting. From a code skim my best guess is the inode somehow has an

[ceph-users] Re: mds dump inode crashes file system

2023-05-10 Thread Gregory Farnum
This is a very strange assert to be hitting. From a code skim my best guess is the inode somehow has an xattr with no value, but that's just a guess and I've no idea how it would happen. Somebody recently pointed you at the (more complicated) way of identifying an inode path by looking at its RADOS

[ceph-users] Re: mds dump inode crashes file system

2023-05-10 Thread Frank Schilder
Campus Bygning 109, rum S14 From: Frank Schilder Sent: Wednesday, May 10, 2023 3:44 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: mds dump inode crashes file system The kernel message that shows up on boot on the file server in text format: May 1

[ceph-users] Re: mds dump inode crashes file system

2023-05-10 Thread Frank Schilder
ome help is appreciated. Best regards, = Frank Schilder AIT Risø Campus Bygning 109, rum S14 From: Frank Schilder Sent: Wednesday, May 10, 2023 3:36 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: mds dump inode crashes file system For the &quo

[ceph-users] Re: mds dump inode crashes file system

2023-05-10 Thread Frank Schilder
For the "mds dump inode" command I could find the crash in the log; see below. Most of the log contents is the past OPS dump from the 3 MDS restarts that happened. It contains the 1 last OPS before the crash and I can upload the log if someone can use it. The crash stack trace somewhat trunc