To add -- I thought I was running 0.67.4 on my test cluster (fc 19), but I
appear to be running 0.69. Not sure how that happened as my yum config is
still pointing to dumpling. :)
On Thu, Oct 24, 2013 at 10:52 AM, Matt Thompson <watering...@gmail.com>wrote:
> Hi Harry,
>
> I was able to replicate this.
>
> What does appear to work (for me) is to do an osd scrub followed by a pg
> repair. I've tried this 2x now and in each case the deleted file gets
> copied over to the OSD from where it was removed. However, I've tried a
> few pg scrub / pg repairs after manually deleting a file and have yet to
> see the file get copied back to the OSD on which it was deleted. Like you
> said, the pg repair sets the health of the PG back to active+clean, but
> then re-running the pg scrub detects the file as missing again and sets it
> back to active+clean+inconsistent.
>
> Regards,
> Matt
>
>
> On Wed, Oct 23, 2013 at 3:45 PM, Harry Harrington <git-ha...@live.co.uk>wrote:
>
>> Hi,
>>
>> I've been taking a look at the repair functionality in ceph. As I
>> understand it the osds should try to copy an object from another member of
>> the pg if it is missing. I have been attempting to test this by manually
>> removing a file from one of the osds however each time the repair
>> completes the the file has not been restored. If I run another scrub on the
>> pg it gets flagged as inconsistent. See below for the output from my
>> testing. I assume I'm missing something obvious, any insight into this
>> process would be greatly appreciated.
>>
>> Thanks,
>> Harry
>>
>> # ceph --version
>> ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)
>> # ceph status
>> cluster a4e417fe-0386-46a5-4475-ca7e10294273
>> health HEALTH_OK
>> monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum
>> 0 ceph1
>> osdmap e13: 3 osds: 3 up, 3 in
>> pgmap v232: 192 pgs: 192 active+clean; 44 bytes data, 15465 MB used,
>> 164 GB / 179 GB avail
>> mdsmap e1: 0/0/1 up
>>
>> file removed from osd.2
>>
>> # ceph pg scrub 0.b
>> instructing pg 0.b on osd.1 to scrub
>>
>> # ceph status
>> cluster a4e417fe-0386-46a5-4475-ca7e10294273
>> health HEALTH_ERR 1 pgs inconsistent; 1 scrub errors
>> monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum
>> 0 ceph1
>> osdmap e13: 3 osds: 3 up, 3 in
>> pgmap v233: 192 pgs: 191 active+clean, 1 active+clean+inconsistent;
>> 44 bytes data, 15465 MB used, 164 GB / 179 GB avail
>> mdsmap e1: 0/0/1 up
>>
>> # ceph pg repair 0.b
>> instructing pg 0.b on osd.1 to repair
>>
>> # ceph status
>> cluster a4e417fe-0386-46a5-4475-ca7e10294273
>> health HEALTH_OK
>> monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum
>> 0 ceph1
>> osdmap e13: 3 osds: 3 up, 3 in
>> pgmap v234: 192 pgs: 192 active+clean; 44 bytes data, 15465 MB used,
>> 164 GB / 179 GB avail
>> mdsmap e1: 0/0/1 up
>>
>> # ceph pg scrub 0.b
>> instructing pg 0.b on osd.1 to scrub
>>
>> # ceph status
>> cluster a4e417fe-0386-46a5-4475-ca7e10294273
>> health HEALTH_ERR 1 pgs inconsistent; 1 scrub errors
>> monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum
>> 0 ceph1
>> osdmap e13: 3 osds: 3 up, 3 in
>> pgmap v236: 192 pgs: 191 active+clean, 1 active+clean+inconsistent;
>> 44 bytes data, 15465 MB used, 164 GB / 179 GB avail
>> mdsmap e1: 0/0/1 up
>>
>>
>>
>> The logs from osd.1:
>> 2013-10-23 14:12:31.188281 7f02a5161700 0 log [ERR] : 0.b osd.2 missing
>> 3a643fcb/testfile1/head//0
>> 2013-10-23 14:12:31.188312 7f02a5161700 0 log [ERR] : 0.b scrub 1
>> missing, 0 inconsistent objects
>> 2013-10-23 14:12:31.188319 7f02a5161700 0 log [ERR] : 0.b scrub 1 errors
>> 2013-10-23 14:13:03.197802 7f02a5161700 0 log [ERR] : 0.b osd.2 missing
>> 3a643fcb/testfile1/head//0
>> 2013-10-23 14:13:03.197837 7f02a5161700 0 log [ERR] : 0.b repair 1
>> missing, 0 inconsistent objects
>> 2013-10-23 14:13:03.197850 7f02a5161700 0 log [ERR] : 0.b repair 1
>> errors, 1 fixed
>> 2013-10-23 14:14:47.232953 7f02a5161700 0 log [ERR] : 0.b osd.2 missing
>> 3a643fcb/testfile1/head//0
>> 2013-10-23 14:14:47.232985 7f02a5161700 0 log [ERR] : 0.b scrub 1
>> missing, 0 inconsistent objects
>> 2013-10-23 14:14:47.232991 7f02a5161700 0 log [ERR] : 0.b scrub 1 errors
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
