We've had a couple of puzzling experiences recently with unfound objects, and I wonder if anyone can shed some light.
This happened with Hammer 0.94.7 on a cluster with 1,309 OSDs. Our use case is exclusively RBD in this cluster, so it's naturally replicated. The rbd pool size is 3, min_size is 2. The crush map is flat, so each host is a failure domain. The OSD hosts are 4U Supermicro chassis with 32 OSDs each. Drive failures have caused the OSD count to be 1,309 instead of 1,312. Twice in the last few weeks we've experienced issues where the cluster was HEALTH_OK but was frequently getting some blocked requests. In each of the two occurrences we investigated and discovered that the blocked requests resulted from two drives in the same host that were misbehaving (different set of 2 drives in each occurrence). We decided to remove the misbehaving OSDs and let things backfill to see if that would address the issue. Removing the drives resulted in a small number of unfound objects, which was surprising. We were able to add the OSDs back with 0 weight and recover the unfound objects in both cases, but removing two OSDs from a single failure domain shouldn't have resulted in unfound objects in an otherwise healthy cluster, correct? ________________________________ [cid:image575d42.JPG@8ddd3310.40afc06a]<https://storagecraft.com> Steve Taylor | Senior Software Engineer | StorageCraft Technology Corporation<https://storagecraft.com> 380 Data Drive Suite 300 | Draper | Utah | 84020 Office: 801.871.2799 | ________________________________ If you are not the intended recipient of this message or received it erroneously, please notify the sender and delete it, together with any attachments, and be advised that any dissemination or copying of this message is prohibited. ________________________________
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
