Can you try copying one of these objects to itself? Would that work and/or
change the index entry? Another option would be to try copying all the
objects to a different bucket.
On Mon, Apr 8, 2013 at 9:48 AM, Erdem Agaoglu <erdem.agao...@gmail.com>wrote:
> omap header and all other omap attributes was destroyed. I copied another
> index over the destroyed one to get a somewhat valid header and it seems
> intact. After a 'check --fix':
>
> # rados -p .rgw.buckets getomapheader .dir.4470.1
> header (49 bytes) :
> 0000 : 03 02 2b 00 00 00 01 00 00 00 01 02 02 18 00 00 : ..+.............
> 0010 : 00 7d 7a 3f 6e 01 00 00 00 00 d0 00 7e 01 00 00 : .}z?n.......~...
> 0020 : 00 bb f5 01 00 00 00 00 00 00 00 00 00 00 00 00 : ................
> 0030 : 00 : .
>
>
> Rados shows objects are there:
>
> # rados ls -p .rgw.buckets |grep 4470.1_data/avatars
> 4470.1_data/avatars/11047/11047823_20101211154308.jpg
> 4470.1_data/avatars/106/106976-orig
> 4470.1_data/avatars/492/492923.jpg
> 4470.1_data/avatars/275/275479.jpg
> ...
>
>
> And i am able to GET them
>
> $ s3cmd get s3://imgiz/data/avatars/492/492923.jpg
> s3://imgiz/data/avatars/492/492923.jpg -> ./492923.jpg [1 of 1]
> 3587 of 3587 100% in 0s 93.40 kB/s done
>
>
> But unable to list them
>
> $ s3cmd ls s3://imgiz/data/avatars
> <NOTHING>
>
>
> My initial expectation was that 'bucket check --fix --check-objects' will
> actually read the files like 'rados ls' does and would recreate the missing
> omapkeys but it doesn't seem to do that. Now a simple check says
>
> # radosgw-admin bucket check -b imgiz
> { "existing_header": { "usage": { "rgw.main": { "size_kb": 6000607,
> "size_kb_actual": 6258740,
> "num_objects": 128443}}},
> "calculated_header": { "usage": { "rgw.main": { "size_kb": 6000607,
> "size_kb_actual": 6258740,
> "num_objects": 128443}}}}
>
> But i know we have more than 128k objects.
>
>
>
> On Mon, Apr 8, 2013 at 7:17 PM, Yehuda Sadeh <yeh...@inktank.com> wrote:
>
>> We'll need to have more info about the current state. Was just the
>> omap header destroyed, or does it still exist? What does the header
>> contain now? Are you able to actually access objects in that bucket,
>> but just fail to list them?
>>
>> On Mon, Apr 8, 2013 at 8:34 AM, Erdem Agaoglu <erdem.agao...@gmail.com>
>> wrote:
>> > Hi again,
>> >
>> > I managed to change the file with some other bucket's index.
>> --check-objects
>> > --fix worked but my hopes have failed as it didn't actually read
>> through the
>> > files or fixed anything. Any suggestions?
>> >
>> >
>> > On Thu, Apr 4, 2013 at 5:56 PM, Erdem Agaoglu <erdem.agao...@gmail.com>
>> > wrote:
>> >>
>> >> Hi all,
>> >>
>> >> After a major failure, and getting our cluster health back OK (with
>> some
>> >> help from inktank folks, thanks), we found out that we have managed to
>> >> corrupt one of our bucket indices. As far as i can track it, we are
>> missing
>> >> the omapheader on that specific index, so we're unable to use
>> radosgw-admin
>> >> tools to fix it.
>> >>
>> >> While a healthy (smaller) bucket answers
>> >> # radosgw-admin bucket check -b imgdoviz
>> >> { "existing_header": { "usage": { "rgw.main": { "size_kb": 4140,
>> >> "size_kb_actual": 4484,
>> >> "num_objects": 157}}},
>> >> "calculated_header": { "usage": { "rgw.main": { "size_kb": 4140,
>> >> "size_kb_actual": 4484,
>> >> "num_objects": 157}}}}
>> >>
>> >> The faulty one fails with
>> >> # radosgw-admin bucket check -b imgiz
>> >> failed to list objects in bucket=imgiz(@.rgw.buckets[4470.1]) err=(22)
>> >> Invalid argument
>> >> failed to check index err=(22) Invalid argument
>> >>
>> >> When i push further
>> >> # radosgw-admin bucket check -b imgiz --check-objects --fix
>> >> failed to list objects in bucket=imgiz(@.rgw.buckets[4470.1]) err=(22)
>> >> Invalid argument
>> >> Checking objects, decreasing bucket 2-phase commit timeout.
>> >> ** Note that timeout will reset only when operation completes
>> successfully
>> >> **
>> >> ERROR: failed operation r=-22
>> >> ERROR: failed operation r=-22
>> >> ..
>> >> last line keeps repeating without any progress.
>> >>
>> >> I checked the file omapheaders and while the healty bucket has:
>> >> # rados -p .rgw.buckets getomapheader .dir.6912.3
>> >> header (49 bytes) :
>> >> 0000 : 03 02 2b 00 00 00 01 00 00 00 01 02 02 18 00 00 :
>> ..+.............
>> >> 0010 : 00 a8 af 40 00 00 00 00 00 00 10 46 00 00 00 00 :
>> ...@.......F....
>> >> 0020 : 00 9d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 :
>> ................
>> >> 0030 : 00 : .
>> >>
>> >> the faulty one is missing it
>> >> # rados -p .rgw.buckets getomapheader .dir.4470.1
>> >> header (0 bytes) :
>> >>
>> >>
>> >> I'm currently in the process of understanding how to create a readable
>> >> header. My hopes are even while its wrong, radosgw-admin will be able
>> to
>> >> read through objects and fix the necessary parts. But i'm not sure how
>> to
>> >> set the new-header. It seems there is a setomapheader counterpart for
>> >> getomapheader but it only accepts values from commandline so i don't
>> know
>> >> how to push a rgw-readable binary with it.
>> >>
>> >> Is this somewhat possible?
>> >>
>> >> Thanks in advance.
>> >>
>> >> --
>> >> erdem agaoglu
>> >
>> >
>> >
>> >
>> > --
>> > erdem agaoglu
>> >
>> > _______________________________________________
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
>>
>
>
>
> --
> erdem agaoglu
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
