Hi, On Apr 3, 2014 4:49 AM, Christian Balzer <ch...@gol.com> wrote: > > On Tue, 1 Apr 2014 14:18:51 +0000 Dan Van Der Ster wrote: > > [snip] > > > > > > http://www.slideshare.net/Inktank_Ceph/scaling-ceph-at-cern > > > > [snap] > > In that slide it says that replacing failed OSDs is automated via puppet. > > I'm very curious on how exactly this happens, as in: > Do you just fire up a "spare" OSD somewhere in the cluster and then > deal with the failed OSD manually? > > Or do you bring down that OSD first including umounting it (so that any > locks on the physical device are gone), hotswap disks, format the disk > (same OSD as before or a new one as far as keyrings go?) and bring the OSD > up? >
When the disk fails, the OSD process exits and after 15 minutes Ceph starts backfilling. Within an hour or two Ceph is healthy again. The cleanup steps to remove the fail OSD logically from Ceph are all manual (umount if possible, crush rm, osd rm, and auth del). Then we physically replace the failed drive. When Puppet next runs, it detects the new (unpartitioned) disk then prepares it to become a new OSD. This is all done using a slightly modified version of enovance's puppet-ceph module. The last step, starting the OSD process, is manual again (by our preference). We usually rm the old OSD and start the new one around the same time, since those both trigger backfilling. Cheers, Dan > Regards, > > Christian > -- > Christian Balzer Network/Systems Engineer > ch...@gol.com Global OnLine Japan/Fusion Communications > http://www.gol.com/
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
