Hi, What is the actual backtrace from the crash? We occasionally had dup inode errors like this in the past but they never escalated to a crash.
You can see my old thread here: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-July/036294.html The developer at that time suggested some manipulations to remove the omap entries for the older version of the inode from the stray dirs, and also to eventually run cephfs-data-scan scan_links. Cheers, Dan On Mon, Aug 23, 2021 at 9:01 PM Pickett, Neale T <ne...@lanl.gov> wrote: > > hello, ceph-users! > > > We have an old cephfs that is ten different kinds of broken, which we are > attempting to (slowly) pull files from. The most recent issue we've hit is > that the mds will start up, log hundreds of messages like below, then crash. > This is happening in a loop; we can never actually get an mds to stay up long > enough to try and fix it. > > > The most recent thing we tried was to do a extents scan, followed by an inode > scan. No luck. > > > Any suggestions? Could I maybe just manually delete the stray inodes? > > > 2021-08-23T12:59:40.803056-0600 mds.dellingr [ERR] loaded dup inode > 0x100028e324a [2,head] v265523356 at ~mds0/stray3/100028e324a, but inode > 0x100028e324a.head v265540146 already exists at ~mds0/stray1/100028e324a > 2021-08-23T12:59:40.803082-0600 mds.dellingr [ERR] loaded dup inode > 0x100028e3249 [2,head] v265523350 at ~mds0/stray3/100028e3249, but inode > 0x100028e3249.head v265540138 already exists at ~mds0/stray1/100028e3249 > 2021-08-23T12:59:40.803109-0600 mds.dellingr [ERR] loaded dup inode > 0x100028e3247 [2,head] v265523354 at ~mds0/stray3/100028e3247, but inode > 0x100028e3247.head v265540140 already exists at ~mds0/stray1/100028e3247 > 2021-08-23T12:59:40.803136-0600 mds.dellingr [ERR] loaded dup inode > 0x100028e3246 [2,head] v265523344 at ~mds0/stray3/100028e3246, but inode > 0x100028e3246.head v265540130 already exists at ~mds0/stray1/100028e3246 > 2021-08-23T12:59:40.803162-0600 mds.dellingr [ERR] loaded dup inode > 0x100028e3245 [2,head] v265523342 at ~mds0/stray3/100028e3245, but inode > 0x100028e3245.head v265540128 already exists at ~mds0/stray1/100028e3245 > 2021-08-23T12:59:40.803188-0600 mds.dellingr [ERR] loaded dup inode > 0x100028e3244 [2,head] v265523348 at ~mds0/stray3/100028e3244, but inode > 0x100028e3244.head v265540126 already exists at ~mds0/stray1/100028e3244 > 2021-08-23T12:59:40.803214-0600 mds.dellingr [ERR] loaded dup inode > 0x100028e3243 [2,head] v265523346 at ~mds0/stray3/100028e3243, but inode > 0x100028e3243.head v265540136 already exists at ~mds0/stray1/100028e3243 > 2021-08-23T12:59:40.803241-0600 mds.dellingr [ERR] loaded dup inode > 0x100028e323f [2,head] v265523338 at ~mds0/stray3/100028e323f, but inode > 0x100028e323f.head v265540124 already exists at ~mds0/stray1/100028e323f > > > > Neale Pickett <ne...@lanl.gov> > A-4: Advanced Research in Cyber Systems > Los Alamos National Laboratory > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
