This is the log grepped with the relevant threadid. It shows 400 in the
last lines but nothing seems odd besides that.
http://pastebin.com/xWCYmnXV
Thanks for your interest.
On Mon, Apr 8, 2013 at 8:21 PM, Yehuda Sadeh <yeh...@inktank.com> wrote:
> Each bucket has a unique prefix which you can get by doing radosgw-admin
> bucket stats on that bucket. You can grep that prefix in 'rados ls -p
> .rgw.buckets'.
>
> Do you have any rgw log showing why you get the Invalid Request response?
> Can you also add 'debug ms = 1' for the log?
>
> Thanks
>
>
> On Mon, Apr 8, 2013 at 10:12 AM, Erdem Agaoglu <erdem.agao...@gmail.com>wrote:
>
>> Just tried that file:
>>
>> $ s3cmd mv s3://imgiz/data/avatars/492/492923.jpg
>> s3://imgiz/data/avatars/492/492923.jpg
>> ERROR: S3 error: 400 (InvalidRequest)
>>
>> a more verbose output shows that the sign-headers was
>> 'PUT\n\n\n\nx-amz-copy-source:/imgiz/data/avatars/492/492923.jpg\nx-amz-date:Mon,
>> 08 Apr 2013 16:59:30
>> +0000\nx-amz-metadata-directive:COPY\n/imgiz/data/avatars/492/492923.jpg'
>>
>> But i guess it doesn't work even if the index is correct. I get the same
>> response on a clear bucket too.
>>
>> We might try that but we don't have a file list. I guess its possible
>> with 'rados ls | grep | sed' ?
>>
>>
>> On Mon, Apr 8, 2013 at 7:53 PM, Yehuda Sadeh <yeh...@inktank.com> wrote:
>>
>>> Can you try copying one of these objects to itself? Would that work
>>> and/or change the index entry? Another option would be to try copying all
>>> the objects to a different bucket.
>>>
>>>
>>> On Mon, Apr 8, 2013 at 9:48 AM, Erdem Agaoglu
>>> <erdem.agao...@gmail.com>wrote:
>>>
>>>> omap header and all other omap attributes was destroyed. I copied
>>>> another index over the destroyed one to get a somewhat valid header and it
>>>> seems intact. After a 'check --fix':
>>>>
>>>> # rados -p .rgw.buckets getomapheader .dir.4470.1
>>>> header (49 bytes) :
>>>> 0000 : 03 02 2b 00 00 00 01 00 00 00 01 02 02 18 00 00 :
>>>> ..+.............
>>>> 0010 : 00 7d 7a 3f 6e 01 00 00 00 00 d0 00 7e 01 00 00 :
>>>> .}z?n.......~...
>>>> 0020 : 00 bb f5 01 00 00 00 00 00 00 00 00 00 00 00 00 :
>>>> ................
>>>> 0030 : 00 : .
>>>>
>>>>
>>>> Rados shows objects are there:
>>>>
>>>> # rados ls -p .rgw.buckets |grep 4470.1_data/avatars
>>>> 4470.1_data/avatars/11047/11047823_20101211154308.jpg
>>>> 4470.1_data/avatars/106/106976-orig
>>>> 4470.1_data/avatars/492/492923.jpg
>>>> 4470.1_data/avatars/275/275479.jpg
>>>> ...
>>>>
>>>>
>>>> And i am able to GET them
>>>>
>>>> $ s3cmd get s3://imgiz/data/avatars/492/492923.jpg
>>>> s3://imgiz/data/avatars/492/492923.jpg -> ./492923.jpg [1 of 1]
>>>> 3587 of 3587 100% in 0s 93.40 kB/s done
>>>>
>>>>
>>>> But unable to list them
>>>>
>>>> $ s3cmd ls s3://imgiz/data/avatars
>>>> <NOTHING>
>>>>
>>>>
>>>> My initial expectation was that 'bucket check --fix --check-objects'
>>>> will actually read the files like 'rados ls' does and would recreate the
>>>> missing omapkeys but it doesn't seem to do that. Now a simple check says
>>>>
>>>> # radosgw-admin bucket check -b imgiz
>>>> { "existing_header": { "usage": { "rgw.main": { "size_kb": 6000607,
>>>> "size_kb_actual": 6258740,
>>>> "num_objects": 128443}}},
>>>> "calculated_header": { "usage": { "rgw.main": { "size_kb": 6000607,
>>>> "size_kb_actual": 6258740,
>>>> "num_objects": 128443}}}}
>>>>
>>>> But i know we have more than 128k objects.
>>>>
>>>>
>>>>
>>>> On Mon, Apr 8, 2013 at 7:17 PM, Yehuda Sadeh <yeh...@inktank.com>wrote:
>>>>
>>>>> We'll need to have more info about the current state. Was just the
>>>>> omap header destroyed, or does it still exist? What does the header
>>>>> contain now? Are you able to actually access objects in that bucket,
>>>>> but just fail to list them?
>>>>>
>>>>> On Mon, Apr 8, 2013 at 8:34 AM, Erdem Agaoglu <erdem.agao...@gmail.com>
>>>>> wrote:
>>>>> > Hi again,
>>>>> >
>>>>> > I managed to change the file with some other bucket's index.
>>>>> --check-objects
>>>>> > --fix worked but my hopes have failed as it didn't actually read
>>>>> through the
>>>>> > files or fixed anything. Any suggestions?
>>>>> >
>>>>> >
>>>>> > On Thu, Apr 4, 2013 at 5:56 PM, Erdem Agaoglu <
>>>>> erdem.agao...@gmail.com>
>>>>> > wrote:
>>>>> >>
>>>>> >> Hi all,
>>>>> >>
>>>>> >> After a major failure, and getting our cluster health back OK (with
>>>>> some
>>>>> >> help from inktank folks, thanks), we found out that we have managed
>>>>> to
>>>>> >> corrupt one of our bucket indices. As far as i can track it, we are
>>>>> missing
>>>>> >> the omapheader on that specific index, so we're unable to use
>>>>> radosgw-admin
>>>>> >> tools to fix it.
>>>>> >>
>>>>> >> While a healthy (smaller) bucket answers
>>>>> >> # radosgw-admin bucket check -b imgdoviz
>>>>> >> { "existing_header": { "usage": { "rgw.main": { "size_kb": 4140,
>>>>> >> "size_kb_actual": 4484,
>>>>> >> "num_objects": 157}}},
>>>>> >> "calculated_header": { "usage": { "rgw.main": { "size_kb": 4140,
>>>>> >> "size_kb_actual": 4484,
>>>>> >> "num_objects": 157}}}}
>>>>> >>
>>>>> >> The faulty one fails with
>>>>> >> # radosgw-admin bucket check -b imgiz
>>>>> >> failed to list objects in bucket=imgiz(@.rgw.buckets[4470.1])
>>>>> err=(22)
>>>>> >> Invalid argument
>>>>> >> failed to check index err=(22) Invalid argument
>>>>> >>
>>>>> >> When i push further
>>>>> >> # radosgw-admin bucket check -b imgiz --check-objects --fix
>>>>> >> failed to list objects in bucket=imgiz(@.rgw.buckets[4470.1])
>>>>> err=(22)
>>>>> >> Invalid argument
>>>>> >> Checking objects, decreasing bucket 2-phase commit timeout.
>>>>> >> ** Note that timeout will reset only when operation completes
>>>>> successfully
>>>>> >> **
>>>>> >> ERROR: failed operation r=-22
>>>>> >> ERROR: failed operation r=-22
>>>>> >> ..
>>>>> >> last line keeps repeating without any progress.
>>>>> >>
>>>>> >> I checked the file omapheaders and while the healty bucket has:
>>>>> >> # rados -p .rgw.buckets getomapheader .dir.6912.3
>>>>> >> header (49 bytes) :
>>>>> >> 0000 : 03 02 2b 00 00 00 01 00 00 00 01 02 02 18 00 00 :
>>>>> ..+.............
>>>>> >> 0010 : 00 a8 af 40 00 00 00 00 00 00 10 46 00 00 00 00 :
>>>>> ...@.......F....
>>>>> >> 0020 : 00 9d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 :
>>>>> ................
>>>>> >> 0030 : 00 : .
>>>>> >>
>>>>> >> the faulty one is missing it
>>>>> >> # rados -p .rgw.buckets getomapheader .dir.4470.1
>>>>> >> header (0 bytes) :
>>>>> >>
>>>>> >>
>>>>> >> I'm currently in the process of understanding how to create a
>>>>> readable
>>>>> >> header. My hopes are even while its wrong, radosgw-admin will be
>>>>> able to
>>>>> >> read through objects and fix the necessary parts. But i'm not sure
>>>>> how to
>>>>> >> set the new-header. It seems there is a setomapheader counterpart
>>>>> for
>>>>> >> getomapheader but it only accepts values from commandline so i
>>>>> don't know
>>>>> >> how to push a rgw-readable binary with it.
>>>>> >>
>>>>> >> Is this somewhat possible?
>>>>> >>
>>>>> >> Thanks in advance.
>>>>> >>
>>>>> >> --
>>>>> >> erdem agaoglu
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > erdem agaoglu
>>>>> >
>>>>> > _______________________________________________
>>>>> > ceph-users mailing list
>>>>> > ceph-users@lists.ceph.com
>>>>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>> >
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> erdem agaoglu
>>>>
>>>
>>>
>>
>>
>> --
>> erdem agaoglu
>>
>
>
--
erdem agaoglu
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
