Hi Luke, May be you can set these flags:
ceph osd set nodown ceph osd set noout Regards Sahana On Wed, Mar 4, 2015 at 2:32 PM, Luke Kao <luke....@mycom-osi.com> wrote: > Hello ceph community, > We need some immediate help that our cluster is in a very strange and bad > status after unexpected reboot of many OSD nodes in a very short time frame. > > We have a cluster with 195 osd configured on 9 different OSD nodes, > original version 0.80.5. > After some issue of the datacenter, at least 5 OSD nodes rebooted and > after reboot not all OSDs goes up then trigger a lot of recovery, also many > PGs goes into dead / incomplete state. > > Then we try to restart OSD, and found OSD keep crashes with error "FAILED > assert(log.head >= olog.tail && olog.head >= log.tail)", so we upgrade to > 0.80.7 which covers fix of #9482, however we still see the error with > different behavior: > 0.80.5: once OSD crashes with this error, any trial to restart the OSD, it > will crash with same error at the end > 0.80.7: OSD can be restarted, but after some time, there will be another > OSD will crash with this error > > We also tried to set nobackfill and norecover flag but doesn't help. > > > So the cluster get stuck that we cannot bring more osd back. > > Any suggestion that we may have the chance to recover the cluster? > Many thanks, > > > Luke Kao > > MYCOM-OSI > <http://www.mycom-osi.com> > > ------------------------------ > > This electronic message contains information from Mycom which may be > privileged or confidential. The information is intended to be for the use > of the individual(s) or entity named above. If you are not the intended > recipient, be aware that any disclosure, copying, distribution or any other > use of the contents of this information is prohibited. If you have received > this electronic message in error, please notify us by post or telephone (to > the numbers or correspondence address above) or by email (at the email > address above) immediately. > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
