Just tried that file:
$ s3cmd mv s3://imgiz/data/avatars/492/492923.jpg
s3://imgiz/data/avatars/492/492923.jpg
ERROR: S3 error: 400 (InvalidRequest)
a more verbose output shows that the sign-headers was
'PUT\n\n\n\nx-amz-copy-source:/imgiz/data/avatars/492/492923.jpg\nx-amz-date:Mon,
08 Apr 2013 16:59:30
+0000\nx-amz-metadata-directive:COPY\n/imgiz/data/avatars/492/492923.jpg'
But i guess it doesn't work even if the index is correct. I get the same
response on a clear bucket too.
We might try that but we don't have a file list. I guess its possible with
'rados ls | grep | sed' ?
On Mon, Apr 8, 2013 at 7:53 PM, Yehuda Sadeh <yeh...@inktank.com> wrote:
> Can you try copying one of these objects to itself? Would that work and/or
> change the index entry? Another option would be to try copying all the
> objects to a different bucket.
>
>
> On Mon, Apr 8, 2013 at 9:48 AM, Erdem Agaoglu <erdem.agao...@gmail.com>wrote:
>
>> omap header and all other omap attributes was destroyed. I copied another
>> index over the destroyed one to get a somewhat valid header and it seems
>> intact. After a 'check --fix':
>>
>> # rados -p .rgw.buckets getomapheader .dir.4470.1
>> header (49 bytes) :
>> 0000 : 03 02 2b 00 00 00 01 00 00 00 01 02 02 18 00 00 : ..+.............
>> 0010 : 00 7d 7a 3f 6e 01 00 00 00 00 d0 00 7e 01 00 00 : .}z?n.......~...
>> 0020 : 00 bb f5 01 00 00 00 00 00 00 00 00 00 00 00 00 : ................
>> 0030 : 00 : .
>>
>>
>> Rados shows objects are there:
>>
>> # rados ls -p .rgw.buckets |grep 4470.1_data/avatars
>> 4470.1_data/avatars/11047/11047823_20101211154308.jpg
>> 4470.1_data/avatars/106/106976-orig
>> 4470.1_data/avatars/492/492923.jpg
>> 4470.1_data/avatars/275/275479.jpg
>> ...
>>
>>
>> And i am able to GET them
>>
>> $ s3cmd get s3://imgiz/data/avatars/492/492923.jpg
>> s3://imgiz/data/avatars/492/492923.jpg -> ./492923.jpg [1 of 1]
>> 3587 of 3587 100% in 0s 93.40 kB/s done
>>
>>
>> But unable to list them
>>
>> $ s3cmd ls s3://imgiz/data/avatars
>> <NOTHING>
>>
>>
>> My initial expectation was that 'bucket check --fix --check-objects' will
>> actually read the files like 'rados ls' does and would recreate the missing
>> omapkeys but it doesn't seem to do that. Now a simple check says
>>
>> # radosgw-admin bucket check -b imgiz
>> { "existing_header": { "usage": { "rgw.main": { "size_kb": 6000607,
>> "size_kb_actual": 6258740,
>> "num_objects": 128443}}},
>> "calculated_header": { "usage": { "rgw.main": { "size_kb": 6000607,
>> "size_kb_actual": 6258740,
>> "num_objects": 128443}}}}
>>
>> But i know we have more than 128k objects.
>>
>>
>>
>> On Mon, Apr 8, 2013 at 7:17 PM, Yehuda Sadeh <yeh...@inktank.com> wrote:
>>
>>> We'll need to have more info about the current state. Was just the
>>> omap header destroyed, or does it still exist? What does the header
>>> contain now? Are you able to actually access objects in that bucket,
>>> but just fail to list them?
>>>
>>> On Mon, Apr 8, 2013 at 8:34 AM, Erdem Agaoglu <erdem.agao...@gmail.com>
>>> wrote:
>>> > Hi again,
>>> >
>>> > I managed to change the file with some other bucket's index.
>>> --check-objects
>>> > --fix worked but my hopes have failed as it didn't actually read
>>> through the
>>> > files or fixed anything. Any suggestions?
>>> >
>>> >
>>> > On Thu, Apr 4, 2013 at 5:56 PM, Erdem Agaoglu <erdem.agao...@gmail.com
>>> >
>>> > wrote:
>>> >>
>>> >> Hi all,
>>> >>
>>> >> After a major failure, and getting our cluster health back OK (with
>>> some
>>> >> help from inktank folks, thanks), we found out that we have managed to
>>> >> corrupt one of our bucket indices. As far as i can track it, we are
>>> missing
>>> >> the omapheader on that specific index, so we're unable to use
>>> radosgw-admin
>>> >> tools to fix it.
>>> >>
>>> >> While a healthy (smaller) bucket answers
>>> >> # radosgw-admin bucket check -b imgdoviz
>>> >> { "existing_header": { "usage": { "rgw.main": { "size_kb": 4140,
>>> >> "size_kb_actual": 4484,
>>> >> "num_objects": 157}}},
>>> >> "calculated_header": { "usage": { "rgw.main": { "size_kb": 4140,
>>> >> "size_kb_actual": 4484,
>>> >> "num_objects": 157}}}}
>>> >>
>>> >> The faulty one fails with
>>> >> # radosgw-admin bucket check -b imgiz
>>> >> failed to list objects in bucket=imgiz(@.rgw.buckets[4470.1]) err=(22)
>>> >> Invalid argument
>>> >> failed to check index err=(22) Invalid argument
>>> >>
>>> >> When i push further
>>> >> # radosgw-admin bucket check -b imgiz --check-objects --fix
>>> >> failed to list objects in bucket=imgiz(@.rgw.buckets[4470.1]) err=(22)
>>> >> Invalid argument
>>> >> Checking objects, decreasing bucket 2-phase commit timeout.
>>> >> ** Note that timeout will reset only when operation completes
>>> successfully
>>> >> **
>>> >> ERROR: failed operation r=-22
>>> >> ERROR: failed operation r=-22
>>> >> ..
>>> >> last line keeps repeating without any progress.
>>> >>
>>> >> I checked the file omapheaders and while the healty bucket has:
>>> >> # rados -p .rgw.buckets getomapheader .dir.6912.3
>>> >> header (49 bytes) :
>>> >> 0000 : 03 02 2b 00 00 00 01 00 00 00 01 02 02 18 00 00 :
>>> ..+.............
>>> >> 0010 : 00 a8 af 40 00 00 00 00 00 00 10 46 00 00 00 00 :
>>> ...@.......F....
>>> >> 0020 : 00 9d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 :
>>> ................
>>> >> 0030 : 00 : .
>>> >>
>>> >> the faulty one is missing it
>>> >> # rados -p .rgw.buckets getomapheader .dir.4470.1
>>> >> header (0 bytes) :
>>> >>
>>> >>
>>> >> I'm currently in the process of understanding how to create a readable
>>> >> header. My hopes are even while its wrong, radosgw-admin will be able
>>> to
>>> >> read through objects and fix the necessary parts. But i'm not sure
>>> how to
>>> >> set the new-header. It seems there is a setomapheader counterpart for
>>> >> getomapheader but it only accepts values from commandline so i don't
>>> know
>>> >> how to push a rgw-readable binary with it.
>>> >>
>>> >> Is this somewhat possible?
>>> >>
>>> >> Thanks in advance.
>>> >>
>>> >> --
>>> >> erdem agaoglu
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > erdem agaoglu
>>> >
>>> > _______________________________________________
>>> > ceph-users mailing list
>>> > ceph-users@lists.ceph.com
>>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> >
>>>
>>
>>
>>
>> --
>> erdem agaoglu
>>
>
>
--
erdem agaoglu
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
