Hi
I have single node cephfs on top of EC pool on top of bluestore ( my dream
setup ;) ) .
I hit bug that is crushing my osd ( first only one , then second before data
backfilled ):
http://tracker.ceph.com/issues/20997
After some investigation i found that there is object
("3#22:addebba8:::10000014f8e.00000145:head#32ce3" ( please note abnormal
"#32ce3" at the end ) )
that is probably causing this.
- ceph-bluestore-tool deep fsck : ceph-bluestore-tool --path
/var/lib/ceph/osd/ceph-1/ --deep yes --command fsck
"finish with 0 errors"
- ceph-objectstore-tool fsck : ceph-objectstore-tool --data-path
/var/lib/ceph/osd/ceph-1/ --op fsck
"fsck found no errors"
- export of the affected pg (22.5s3) is failing:
ceph-objectstore-tool --force --data-path /var/lib/ceph/osd/ceph-1/ --pgid
22.5s3 --op export --file /tank1/ceph-recovery/recover.22.5s3v2 2>&1 | tee
recover.22.5s3v2.log
....
Read 3#22:addea9c4:::1000000d42b.00000073:head#
Read 3#22:addeb3f7:::10000009d3b.00000440:head#
Read 3#22:addeb487:::1000000a84f.00000e50:head#
Read 3#22:addebba8:::10000014f8e.00000145:head#32ce3
getattr failure object_info -61
export_files error -61
- tools like ceph-objectstore-tool shows warning when they for example list
objects for pg :
Error getting attr on :
22.5s3_head,3#22:addebba8:::10000014f8e.00000145:head#/32ce3, (61) No data
available
- I can clearly see directory for this strange object, when i fuse-mount
bluestore.
/mnt/target/22.5s3_head/all/3#22:addebba8:::10000014f8e.00000145:head#32ce3
/mnt/target/22.5s3_head/all/3#22:addebba8:::10000014f8e.00000145:head#32ce3/bitwise_hash
/mnt/target/22.5s3_head/all/3#22:addebba8:::10000014f8e.00000145:head#32ce3/data
/mnt/target/22.5s3_head/all/3#22:addebba8:::10000014f8e.00000145:head#32ce3/omap
/mnt/target/22.5s3_head/all/3#22:addebba8:::10000014f8e.00000145:head#32ce3/attr
/mnt/target/22.5s3_head/all/3#22:addebba8:::10000014f8e.00000145:head#32ce3/omap_header
I read all documentation, and a loot of therads on this list
( this list is really great for info about tools like ceph-objectstore-tool ),
but i can not find any way to remove/modify this broken object from bluestore.
My data are anacessable, i do not know when and if this bug will be fixed in
ceph code.
Should i focus on removing this strange/broken object (like hack
ceph-objectstore-tool or ceph-bluestore-tool),
or is there something else i can/should do to get my osd.1 up and running again
?
--
Kind Regards
Marek Grzybowski
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
