i would only tar the pg you have missing objects from, trying to inject
older objects when the pg is correct can not be good.
scrub errors is kind of the issue with only 2 replicas. when you have 2
different objects. how to know witch one is correct and witch one is bad..
and as you have read on
http://ceph.com/geen-categorie/ceph-manually-repair-object/ and on
http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-pg/
you need to
- find the pg :: rados list-inconsistent-pg [pool]
- find the problem :: rados list-inconsistent-obj 0.6
--format=json-pretty ; give you the object name look for hints to what
is the bad object
- find the object :: manually check the objects, check the object
metadata, run md5sum on them all and compare. check objects on the
nonrunning osd's and compare there as well. anything to try to determine
what object is ok and what is bad.
- fix the problem :: assuming you find the bad object, stop the
affected osd with the bad object, remove the object manually, restart
osd. issue repair command.
if the rados commands does not give you the info you need to do it all
manually as on http://ceph.com/geen-categorie/ceph-manually-repair-object/
good luck
Ronny Aasen
On 20.09.2017 22:17, hjcho616 wrote:
Thanks Ronny.
I decided to try to tar everything under current directory. Is this
correct command for it? Is there any directory we do not want in the
new drive? commit_op_seq, meta, nosnap, omap?
tar --xattrs --preserve-permissions -zcvf osd.4.tar.gz .
As far as inconsistent PGs... I am running in to these errors. I
tried moving one copy of pg to other location, but it just says moved
shard is missing. Tried setting 'noout ' and turn one of them down,
seems to work on something but then back to same error. Currently
trying to move to different osd... making sure the drive is not
faulty, got few of them.. but still persisting.. I've been kicking
off ceph pg repair PG#, hoping it would fix them. =P Any other
suggestion?
2017-09-20 09:39:48.481400 7f163c5fa700 0 log_channel(cluster) log
[INF] : 0.29 repair starts
2017-09-20 09:47:37.384921 7f163c5fa700 -1 log_channel(cluster) log
[ERR] : 0.29 shard 6: soid 0:97126ead:::200014ce4c3.0000028f:head
data_digest 0x8f679a50 != data_digest 0x979f2ed4 from auth oi
0:97126ead:::200014ce4c3.0000028f:head(19366'539375
client.535319.1:2361163 dirty|data_digest|omap_digest s 4194304 uv
539375 dd 979f2ed4 od ffffffff alloc_hint [0 0])
2017-09-20 09:47:37.384931 7f163c5fa700 -1 log_channel(cluster) log
[ERR] : 0.29 shard 7: soid 0:97126ead:::200014ce4c3.0000028f:head
data_digest 0x8f679a50 != data_digest 0x979f2ed4 from auth oi
0:97126ead:::200014ce4c3.0000028f:head(19366'539375
client.535319.1:2361163 dirty|data_digest|omap_digest s 4194304 uv
539375 dd 979f2ed4 od ffffffff alloc_hint [0 0])
2017-09-20 09:47:37.384936 7f163c5fa700 -1 log_channel(cluster) log
[ERR] : 0.29 soid 0:97126ead:::200014ce4c3.0000028f:head: failed to
pick suitable auth object
2017-09-20 09:48:11.138566 7f1639df5700 -1 log_channel(cluster) log
[ERR] : 0.29 shard 6: soid 0:97d5c15a:::100000101b4.00006892:head
data_digest 0xd65b4014 != data_digest 0xf41cfab8 from auth oi
0:97d5c15a:::100000101b4.00006892:head(12962'65557 osd.4.0:42234
dirty|data_digest|omap_digest s 4194304 uv 776 dd f41cfab8 od ffffffff
alloc_hint [0 0])
2017-09-20 09:48:11.138575 7f1639df5700 -1 log_channel(cluster) log
[ERR] : 0.29 shard 7: soid 0:97d5c15a:::100000101b4.00006892:head
data_digest 0xd65b4014 != data_digest 0xf41cfab8 from auth oi
0:97d5c15a:::100000101b4.00006892:head(12962'65557 osd.4.0:42234
dirty|data_digest|omap_digest s 4194304 uv 776 dd f41cfab8 od ffffffff
alloc_hint [0 0])
2017-09-20 09:48:11.138581 7f1639df5700 -1 log_channel(cluster) log
[ERR] : 0.29 soid 0:97d5c15a:::100000101b4.00006892:head: failed to
pick suitable auth object
2017-09-20 09:48:55.584022 7f1639df5700 -1 log_channel(cluster) log
[ERR] : 0.29 repair 4 errors, 0 fixed
Latest health...
HEALTH_ERR 1 pgs are stuck inactive for more than 300 seconds; 1 pgs
down; 1 pgs incomplete; 9 pgs inconsistent; 1 pgs repair; 1 pgs stuck
inactive; 1 pgs stuck unclean; 68 scrub errors; mds rank 0 has failed;
mds cluster is degraded; no legacy OSD present but 'sortbitwise' flag
is not set
Regards,
Hong
On Wednesday, September 20, 2017 11:53 AM, Ronny Aasen
<ronny+ceph-us...@aasen.cx> wrote:
On 20.09.2017 16:49, hjcho616 wrote:
Anyone? Can this page be saved? If not what are my options?
Regards,
Hong
On Saturday, September 16, 2017 1:55 AM, hjcho616
<hjcho...@yahoo.com> <mailto:hjcho...@yahoo.com> wrote:
Looking better... working on scrubbing..
HEALTH_ERR 1 pgs are stuck inactive for more than 300 seconds; 1 pgs
incomplete; 12 pgs inconsistent; 2 pgs repair; 1 pgs stuck inactive;
1 pgs stuck unclean; 109 scrub errors; too few PGs per OSD (29 < min
30); mds rank 0 has failed; mds cluster is degraded; noout flag(s)
set; no legacy OSD present but 'sortbitwise' flag is not set
Now PG1.28.. looking at all old osds dead or alive. Only one with
DIR_* directory is in osd.4. This appears to be metadata pool! 21M
of metadata can be quite a bit of stuff.. so I would like to rescue
this! But I am not able to start this OSD. exporting through
ceph-objectstore-tool appears to crash. Even with
--skip-journal-replay and --skip-mount-omap (different failure). As
I mentioned in earlier email, that exception thrown message is bogus...
# ceph-objectstore-tool --op export --pgid 1.28 --data-path
/var/lib/ceph/osd/ceph-4 --journal-path
/var/lib/ceph/osd/ceph-4/journal --file ~/1.28.export
terminate called after throwing an instance of 'std::domain_error'
[SNIP]
What can I do to save that PG1.28? Please let me know if you need
more information. So close!... =)
Regards,
Hong
12 inconsistent and 109 scrub errors is something you should fix first
of all.
also you can consider using the paid-services of many ceph support
companies. that specialize in these kind of situations.
--
that beeing said, here are some suggestions...
when it comes to lost object recovery you have come about as far as i
have ever experienced. so everything after here is just assumptions
and wild guesswork to what you can try. I hope others shouts out if i
tell you wildly wrong things.
if you have found date pg1.28 from the broken osd and have checked all
other working and nonworking drives, for that pg. then you need to try
and extract the pg from the broken drive. As always in recovery cases,
take a dd clone of the drive and work from the cloned image. to avoid
more damage to the drive, and to allow you to try multiple times.
you should add a temporary injection drive large enough for that pg,
and set its crush weight to 0 so it always drains. make sure it is up
and registered properly in ceph.
the idea is to copy the pg manually from broken-osd to the injection
drive, since the export/import fails.. making sure you get all xattrs
included. one can either copy the whole pg, or just the "missing"
objects. if there are few objects i would go for that, if there are
many i would take the whole pg. you wont get data from leveldb. so i
am not at all sure this would work. but worth a shot.
- stop your injection osd, verify it is down and the proccess not running.
- from the mountpoint of your broken-osd go into the current
directory. and tar up the pg1.28 make sure you use -p and --xattrs
when you create the archive.
- if tar errors out on unreadable files, just rm those (since you are
working on a copy of your rescue image, you can allways try again)
- copy the tar file to the injection drive and extract while sitting
in the current directory (remember --xattrs)
- set debug options on the injection drive in ceph.conf
- start the injection drive, and follow along in the log file.
hopefully it should scan, locate the pg, and replicate the pg1.28
objects off to the current primary drive for pg1.28. and since it have
crush weight 0 it should drain out.
- if that works, verify the injection drive is drained, stop it and
remove it from ceph. zap the drive.
this is all as i said guesstimates so your mileage may vary
good luck
Ronny Aasen
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
