> On 28 Oct 2014, at 08:25, Robert van Leeuwen > <robert.vanleeu...@spilgames.com> wrote: > >> By now we decide use a SuperMicro's SKU with 72 bays for HDD = 22 SSD + >> 50 SATA drives. >> Our racks can hold 10 this servers and 50 this racks in ceph cluster = >> 36000 OSD's, >> With 4tb SATA drives and replica = 2 and nerfull ratio = 0.8 we have 40 >> Petabyte of useful capacity. >> >> It's too big or normal use case for ceph? > > I'm a bit worried about the replica count: > The chances of 2 disks failing of 25000 at the same time becomes very > significant. (or a disk + server failure) > Without doing any math my gut feeling says that 3 replica's is still not very > comfortable. (especially if the disks come from the same batch)
It doesn’t quite work like that. You’re not going to lose data if _any_ two disks out of 25000 fail. You’ll only lose data if two disks that are coupled in a PG are lost. So, while there are 25000^2 ways to lose two disks, there are only nPGs disk pairs that matter for data loss. Said another way, suppose you have one disk failed, what is the probability of losing data? Well, the data loss scenario is going to happen if one of the ~100 disks coupled with the failed disk also fails. So you see, the chance of data loss with 2 replicas is roughly equivalent whether you have 1000 OSDs or 25000 OSDs. Cheers, Dan > > Cheers, > Robert van Leeuwen > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
