Re: [ceph-users] ceph mds crashing constantly : ceph_assert fail … prepare_new_inode

On Sat, Aug 11, 2018 at 1:21 PM Amit Handa wrote: > > Thanks for the response, gregory. > > We need to support a couple of production services we have migrated to ceph. > So we are in a bit of soup. > > cluster is as follows: > ``` > ceph osd tree > ID CLASS WEIGHT TYPE NAME STATUS REWEI

Re: [ceph-users] ceph mds crashing constantly : ceph_assert fail … prepare_new_inode

Thanks for the response, gregory. We need to support a couple of production services we have migrated to ceph. So we are in a bit of soup. cluster is as follows: ``` ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 11.06848 root default -75.45799 hos

Re: [ceph-users] ceph mds crashing constantly : ceph_assert fail … prepare_new_inode

As Paul said, the MDS is loading "duplicate inodes" and that's very bad. If you've already gone through some of the disaster recovery steps, that's likely the cause. But you'll need to provide a *lot* more information about what you've already done to the cluster for people to be sure. The backwar

Re: [ceph-users] ceph mds crashing constantly : ceph_assert fail … prepare_new_inode

Thanks for the info , paul. Our cluster is 130gb in size, at present. We are starting out in ceph adoption in our company. At present, I am looking for guidance from the community. It ll help us, as well in learning more about the product and available support. Thanks, On Fri, 10 Aug 2018 at

Re: [ceph-users] ceph mds crashing constantly : ceph_assert fail … prepare_new_inode

Sorry, a step-by-step guide through something like that is beyond the scope of what we can do on a mailing list. But what I would do here is carefully asses the situation/ the damage. My wild guess would be to reset and rebuild the inode table but that might be incorrect and unsafe without further

Re: [ceph-users] ceph mds crashing constantly : ceph_assert fail … prepare_new_inode

Thanks alot, Paul. we did (hopefully) follow through with the disaster recovery. however, please guide me in how to get the cluster back up ! Thanks, On Fri, Aug 10, 2018 at 9:32 PM Paul Emmerich wrote: > Looks like you got some duplicate inodes due to corrupted metadata, you > likely tried to

Re: [ceph-users] ceph mds crashing constantly : ceph_assert fail … prepare_new_inode

Looks like you got some duplicate inodes due to corrupted metadata, you likely tried to a disaster recovery and didn't follow through it completely or you hit some bug in Ceph. The solution here is probably to do a full recovery of the metadata/full backwards scan after resetting the inodes. I've

[ceph-users] ceph mds crashing constantly : ceph_assert fail … prepare_new_inode

We are facing constant crash from ceph mds. We have installed mimic (v13.2.1). mds: cephfs-1/1/1 up {0=node2=up:active(laggy or crashed)} *mds logs: https://pastebin.com/AWGMLRm0 * we have followed the DR steps listed at *http://docs.ceph.com/docs/mimic/cephfs/dis