Hi again,
I managed to change the file with some other bucket's index.
--check-objects --fix worked but my hopes have failed as it didn't actually
read through the files or fixed anything. Any suggestions?
On Thu, Apr 4, 2013 at 5:56 PM, Erdem Agaoglu <erdem.agao...@gmail.com>wrote:
> Hi all,
>
> After a major failure, and getting our cluster health back OK (with some
> help from inktank folks, thanks), we found out that we have managed to
> corrupt one of our bucket indices. As far as i can track it, we are missing
> the omapheader on that specific index, so we're unable to use radosgw-admin
> tools to fix it.
>
> While a healthy (smaller) bucket answers
> # radosgw-admin bucket check -b imgdoviz
> { "existing_header": { "usage": { "rgw.main": { "size_kb": 4140,
> "size_kb_actual": 4484,
> "num_objects": 157}}},
> "calculated_header": { "usage": { "rgw.main": { "size_kb": 4140,
> "size_kb_actual": 4484,
> "num_objects": 157}}}}
>
> The faulty one fails with
> # radosgw-admin bucket check -b imgiz
> failed to list objects in bucket=imgiz(@.rgw.buckets[4470.1]) err=(22)
> Invalid argument
> failed to check index err=(22) Invalid argument
>
> When i push further
> # radosgw-admin bucket check -b imgiz --check-objects --fix
> failed to list objects in bucket=imgiz(@.rgw.buckets[4470.1]) err=(22)
> Invalid argument
> Checking objects, decreasing bucket 2-phase commit timeout.
> ** Note that timeout will reset only when operation completes successfully
> **
> ERROR: failed operation r=-22
> ERROR: failed operation r=-22
> ..
> last line keeps repeating without any progress.
>
> I checked the file omapheaders and while the healty bucket has:
> # rados -p .rgw.buckets getomapheader .dir.6912.3
> header (49 bytes) :
> 0000 : 03 02 2b 00 00 00 01 00 00 00 01 02 02 18 00 00 : ..+.............
> 0010 : 00 a8 af 40 00 00 00 00 00 00 10 46 00 00 00 00 : ...@.......F....
> 0020 : 00 9d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : ................
> 0030 : 00 : .
>
> the faulty one is missing it
> # rados -p .rgw.buckets getomapheader .dir.4470.1
> header (0 bytes) :
>
>
> I'm currently in the process of understanding how to create a readable
> header. My hopes are even while its wrong, radosgw-admin will be able to
> read through objects and fix the necessary parts. But i'm not sure how to
> set the new-header. It seems there is a setomapheader counterpart for
> getomapheader but it only accepts values from commandline so i don't know
> how to push a rgw-readable binary with it.
>
> Is this somewhat possible?
>
> Thanks in advance.
>
> --
> erdem agaoglu
>
--
erdem agaoglu
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
