Hi,
I've been taking a look at the repair functionality in ceph. As I understand it
the osds should try to copy an object from another member of the pg if it is
missing. I have been attempting to test this by manually removing a file from
one of the osds however each time the repair completes the the file has not
been restored. If I run another scrub on the pg it gets flagged as
inconsistent. See below for the output from my testing. I assume I'm missing
something obvious, any insight into this process would be greatly appreciated.
Thanks,
Harry
# ceph --version
ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)
# ceph status
cluster a4e417fe-0386-46a5-4475-ca7e10294273
health HEALTH_OK
monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum 0 ceph1
osdmap e13: 3 osds: 3 up, 3 in
pgmap v232: 192 pgs: 192 active+clean; 44 bytes data, 15465 MB used, 164 GB
/ 179 GB avail
mdsmap e1: 0/0/1 up
file removed from osd.2
# ceph pg scrub 0.b
instructing pg 0.b on osd.1 to scrub
# ceph status
cluster a4e417fe-0386-46a5-4475-ca7e10294273
health HEALTH_ERR 1 pgs inconsistent; 1 scrub errors
monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum 0 ceph1
osdmap e13: 3 osds: 3 up, 3 in
pgmap v233: 192 pgs: 191 active+clean, 1 active+clean+inconsistent; 44
bytes data, 15465 MB used, 164 GB / 179 GB avail
mdsmap e1: 0/0/1 up
# ceph pg repair 0.b
instructing pg 0.b on osd.1 to repair
# ceph status
cluster a4e417fe-0386-46a5-4475-ca7e10294273
health HEALTH_OK
monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum 0 ceph1
osdmap e13: 3 osds: 3 up, 3 in
pgmap v234: 192 pgs: 192 active+clean; 44 bytes data, 15465 MB used, 164 GB
/ 179 GB avail
mdsmap e1: 0/0/1 up
# ceph pg scrub 0.b
instructing pg 0.b on osd.1 to scrub
# ceph status
cluster a4e417fe-0386-46a5-4475-ca7e10294273
health HEALTH_ERR 1 pgs inconsistent; 1 scrub errors
monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum 0 ceph1
osdmap e13: 3 osds: 3 up, 3 in
pgmap v236: 192 pgs: 191 active+clean, 1 active+clean+inconsistent; 44
bytes data, 15465 MB used, 164 GB / 179 GB avail
mdsmap e1: 0/0/1 up
The logs from osd.1:
2013-10-23 14:12:31.188281 7f02a5161700 0 log [ERR] : 0.b osd.2 missing
3a643fcb/testfile1/head//0
2013-10-23 14:12:31.188312 7f02a5161700 0 log [ERR] : 0.b scrub 1 missing, 0
inconsistent objects
2013-10-23 14:12:31.188319 7f02a5161700 0 log [ERR] : 0.b scrub 1 errors
2013-10-23 14:13:03.197802 7f02a5161700 0 log [ERR] : 0.b osd.2 missing
3a643fcb/testfile1/head//0
2013-10-23 14:13:03.197837 7f02a5161700 0 log [ERR] : 0.b repair 1 missing, 0
inconsistent objects
2013-10-23 14:13:03.197850 7f02a5161700 0 log [ERR] : 0.b repair 1 errors, 1
fixed
2013-10-23 14:14:47.232953 7f02a5161700 0 log [ERR] : 0.b osd.2 missing
3a643fcb/testfile1/head//0
2013-10-23 14:14:47.232985 7f02a5161700 0 log [ERR] : 0.b scrub 1 missing, 0
inconsistent objects
2013-10-23 14:14:47.232991 7f02a5161700 0 log [ERR] : 0.b scrub 1 errors
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
