We just had to look into a similiar problem (missing clone objects, extraneous clone objects, wrong sizes on few objects...)
You should do something like this: 1) find all OSDs hosting the PG ceph pg map 8.e82 2) find the directory with the object on the OSDs should be something like /var/lib/ceph/ceph-XX/current/8.e82_head/ 3) look in this directory for files named like what you see in logs (rb.0.bfcb12.238e1f29.0000002acd39) there are _head_ objects that contain the original data, and then objects named with the snapshot id instead (_1fc8ce82_ instead of _head_) 4) compare what files are there on the OSDs 5a) you are lucky and one of the OSDs has them - in that case you could either copy them to the others (don't forget xattrs!) or rebuild them via backfills from the good OSD 5b) you are not that lucky and the files are not there - I'm not that sure what to do then You could in theory just copy the _head_ object contents to the missing objects and then drop the image. Or you could maybe just delete the _head_ objects (since you don't need that image anymore), but I don't know whether there's some info stored (in leveldb, or somewhere else) about the rbd image or if all the info is in the objects themselves. I think others here will help you more in that case. I'm not sure if there's an option to "delete rbd image, ignore missing files, call it a day" - that one would be handy for situations like this. Jan > On 21 Oct 2015, at 09:01, Chris Taylor <ctay...@eyonic.com> wrote: > > Is there some way to manually correct this error while this bug is still > needing review? I have one PG that is stuck inconsistent with the same error. > I already created a new RBD image and migrated the data to it. The original > RBD image was "rb.0.ac3386.238e1f29". The new image is "rb.0.bfcb12.238e1f29". > > > 2015-10-20 19:18:07.686783 7f50e4c1d700 0 log_channel(cluster) log [INF] : > 8.e82 repair starts > 2015-10-20 19:18:40.300721 7f50e4c1d700 -1 log_channel(cluster) log [ERR] : > repair 8.e82 1fc8ce82/rb.0.ac3386.238e1f29.00000008776e/snapdir//8 missing > clones > 2015-10-20 19:18:40.301094 7f50e4c1d700 -1 log_channel(cluster) log [ERR] : > repair 8.e82 9cc8ce82/rb.0.bfcb12.238e1f29.0000002acd39/head//8 expected > clone 1fc8ce82/rb.0.ac3386.238e1f29.00000008776e/44//8 > 2015-10-20 19:18:40.301124 7f50e4c1d700 -1 log_channel(cluster) log [ERR] : > repair 8.e82 fb78ce82/rb.0.bfcb12.238e1f29.0000000e69a3/head//8 expected > clone 9cc8ce82/rb.0.bfcb12.238e1f29.0000002acd39/44//8 > 2015-10-20 19:18:40.301140 7f50e4c1d700 -1 log_channel(cluster) log [ERR] : > repair 8.e82 8038ce82/rb.0.bfcb12.238e1f29.0000002b7781/head//8 expected > clone fb78ce82/rb.0.bfcb12.238e1f29.0000000e69a3/44//8 > 2015-10-20 19:18:40.301155 7f50e4c1d700 -1 log_channel(cluster) log [ERR] : > repair 8.e82 c8b7ce82/rb.0.bfcb12.238e1f29.000000059252/head//8 expected > clone 8038ce82/rb.0.bfcb12.238e1f29.0000002b7781/44//8 > 2015-10-20 19:18:40.301170 7f50e4c1d700 -1 log_channel(cluster) log [ERR] : > repair 8.e82 9d26ce82/rb.0.bfcb12.238e1f29.0000000cd86d/head//8 expected > clone c8b7ce82/rb.0.bfcb12.238e1f29.000000059252/44//8 > 2015-10-20 19:18:40.301185 7f50e4c1d700 -1 log_channel(cluster) log [ERR] : > repair 8.e82 c006ce82/rb.0.bfcb12.238e1f29.0000000c53d6/head//8 expected > clone 9d26ce82/rb.0.bfcb12.238e1f29.0000000cd86d/44//8 > 2015-10-20 19:18:40.301200 7f50e4c1d700 -1 log_channel(cluster) log [ERR] : > repair 8.e82 3434ce82/rb.0.bfcb12.238e1f29.0000002cb957/head//8 expected > clone c006ce82/rb.0.bfcb12.238e1f29.0000000c53d6/44//8 > 2015-10-20 19:18:47.724047 7f50e4c1d700 -1 log_channel(cluster) log [ERR] : > 8.e82 repair 8 errors, 0 fixed > 2 > > > Thanks, > > Chris > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
