There seems to be an open issue at s3cmd
https://github.com/s3tools/s3cmd/issues/37. I'll try with other tools


On Mon, Apr 8, 2013 at 9:26 PM, Yehuda Sadeh <yeh...@inktank.com> wrote:

> This one fails because copy object into itself would only work if
> replacing it's attrs (X_AMZ_METADATA_DIRECTIVE=REPLACE).
>
> On Mon, Apr 8, 2013 at 10:35 AM, Erdem Agaoglu <erdem.agao...@gmail.com>
> wrote:
> > This is the log grepped with the relevant threadid. It shows 400 in the
> last
> > lines but nothing seems odd besides that.
> > http://pastebin.com/xWCYmnXV
> >
> > Thanks for your interest.
> >
> >
> > On Mon, Apr 8, 2013 at 8:21 PM, Yehuda Sadeh <yeh...@inktank.com> wrote:
> >>
> >> Each bucket has a unique prefix which you can get by doing radosgw-admin
> >> bucket stats on that bucket. You can grep that prefix in 'rados ls -p
> >> .rgw.buckets'.
> >>
> >> Do you have any rgw log showing why you get the Invalid Request
> response?
> >> Can you also add 'debug ms = 1' for the log?
> >>
> >> Thanks
> >>
> >>
> >> On Mon, Apr 8, 2013 at 10:12 AM, Erdem Agaoglu <erdem.agao...@gmail.com
> >
> >> wrote:
> >>>
> >>> Just tried that file:
> >>>
> >>> $ s3cmd mv s3://imgiz/data/avatars/492/492923.jpg
> >>> s3://imgiz/data/avatars/492/492923.jpg
> >>> ERROR: S3 error: 400 (InvalidRequest)
> >>>
> >>> a more verbose output shows that the sign-headers was
> >>>
> >>>
> 'PUT\n\n\n\nx-amz-copy-source:/imgiz/data/avatars/492/492923.jpg\nx-amz-date:Mon,
> >>> 08 Apr 2013 16:59:30
> >>>
> +0000\nx-amz-metadata-directive:COPY\n/imgiz/data/avatars/492/492923.jpg'
> >>>
> >>> But i guess it doesn't work even if the index is correct. I get the
> same
> >>> response on a clear bucket too.
> >>>
> >>> We might try that but we don't have a file list. I guess its possible
> >>> with 'rados ls | grep | sed' ?
> >>>
> >>>
> >>> On Mon, Apr 8, 2013 at 7:53 PM, Yehuda Sadeh <yeh...@inktank.com>
> wrote:
> >>>>
> >>>> Can you try copying one of these objects to itself? Would that work
> >>>> and/or change the index entry? Another option would be to try copying
> all
> >>>> the objects to a different bucket.
> >>>>
> >>>>
> >>>> On Mon, Apr 8, 2013 at 9:48 AM, Erdem Agaoglu <
> erdem.agao...@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>> omap header and all other omap attributes was destroyed. I copied
> >>>>> another index over the destroyed one to get a somewhat valid header
> and it
> >>>>> seems intact. After a 'check --fix':
> >>>>>
> >>>>>  # rados -p .rgw.buckets getomapheader .dir.4470.1
> >>>>> header (49 bytes) :
> >>>>> 0000 : 03 02 2b 00 00 00 01 00 00 00 01 02 02 18 00 00 :
> >>>>> ..+.............
> >>>>> 0010 : 00 7d 7a 3f 6e 01 00 00 00 00 d0 00 7e 01 00 00 :
> >>>>> .}z?n.......~...
> >>>>> 0020 : 00 bb f5 01 00 00 00 00 00 00 00 00 00 00 00 00 :
> >>>>> ................
> >>>>> 0030 : 00                                              : .
> >>>>>
> >>>>>
> >>>>> Rados shows objects are there:
> >>>>>
> >>>>> # rados ls -p .rgw.buckets |grep 4470.1_data/avatars
> >>>>> 4470.1_data/avatars/11047/11047823_20101211154308.jpg
> >>>>> 4470.1_data/avatars/106/106976-orig
> >>>>> 4470.1_data/avatars/492/492923.jpg
> >>>>> 4470.1_data/avatars/275/275479.jpg
> >>>>> ...
> >>>>>
> >>>>>
> >>>>> And i am able to GET them
> >>>>>
> >>>>> $ s3cmd get s3://imgiz/data/avatars/492/492923.jpg
> >>>>> s3://imgiz/data/avatars/492/492923.jpg -> ./492923.jpg  [1 of 1]
> >>>>>  3587 of 3587   100% in    0s    93.40 kB/s  done
> >>>>>
> >>>>>
> >>>>> But unable to list them
> >>>>>
> >>>>> $ s3cmd ls s3://imgiz/data/avatars
> >>>>> <NOTHING>
> >>>>>
> >>>>>
> >>>>> My initial expectation was that 'bucket check --fix --check-objects'
> >>>>> will actually read the files like 'rados ls' does and would recreate
> the
> >>>>> missing omapkeys but it doesn't seem to do that. Now a simple check
> says
> >>>>>
> >>>>> # radosgw-admin bucket check -b imgiz
> >>>>> { "existing_header": { "usage": { "rgw.main": { "size_kb": 6000607,
> >>>>>               "size_kb_actual": 6258740,
> >>>>>               "num_objects": 128443}}},
> >>>>>   "calculated_header": { "usage": { "rgw.main": { "size_kb": 6000607,
> >>>>>               "size_kb_actual": 6258740,
> >>>>>               "num_objects": 128443}}}}
> >>>>>
> >>>>> But i know we have more than 128k objects.
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Mon, Apr 8, 2013 at 7:17 PM, Yehuda Sadeh <yeh...@inktank.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> We'll need to have more info about the current state. Was just the
> >>>>>> omap header destroyed, or does it still exist? What does the header
> >>>>>> contain now? Are you able to actually access objects in that bucket,
> >>>>>> but just fail to list them?
> >>>>>>
> >>>>>> On Mon, Apr 8, 2013 at 8:34 AM, Erdem Agaoglu
> >>>>>> <erdem.agao...@gmail.com> wrote:
> >>>>>> > Hi again,
> >>>>>> >
> >>>>>> > I managed to change the file with some other bucket's index.
> >>>>>> > --check-objects
> >>>>>> > --fix worked but my hopes have failed as it didn't actually read
> >>>>>> > through the
> >>>>>> > files or fixed anything. Any suggestions?
> >>>>>> >
> >>>>>> >
> >>>>>> > On Thu, Apr 4, 2013 at 5:56 PM, Erdem Agaoglu
> >>>>>> > <erdem.agao...@gmail.com>
> >>>>>> > wrote:
> >>>>>> >>
> >>>>>> >> Hi all,
> >>>>>> >>
> >>>>>> >> After a major failure, and getting our cluster health back OK
> (with
> >>>>>> >> some
> >>>>>> >> help from inktank folks, thanks), we found out that we have
> managed
> >>>>>> >> to
> >>>>>> >> corrupt one of our bucket indices. As far as i can track it, we
> are
> >>>>>> >> missing
> >>>>>> >> the omapheader on that specific index, so we're unable to use
> >>>>>> >> radosgw-admin
> >>>>>> >> tools to fix it.
> >>>>>> >>
> >>>>>> >> While a healthy (smaller) bucket answers
> >>>>>> >> # radosgw-admin bucket check -b imgdoviz
> >>>>>> >> { "existing_header": { "usage": { "rgw.main": { "size_kb": 4140,
> >>>>>> >>               "size_kb_actual": 4484,
> >>>>>> >>               "num_objects": 157}}},
> >>>>>> >>   "calculated_header": { "usage": { "rgw.main": { "size_kb":
> 4140,
> >>>>>> >>               "size_kb_actual": 4484,
> >>>>>> >>               "num_objects": 157}}}}
> >>>>>> >>
> >>>>>> >> The faulty one fails with
> >>>>>> >> # radosgw-admin bucket check -b imgiz
> >>>>>> >> failed to list objects in bucket=imgiz(@.rgw.buckets[4470.1])
> >>>>>> >> err=(22)
> >>>>>> >> Invalid argument
> >>>>>> >> failed to check index err=(22) Invalid argument
> >>>>>> >>
> >>>>>> >> When i push further
> >>>>>> >> # radosgw-admin bucket check -b imgiz --check-objects --fix
> >>>>>> >> failed to list objects in bucket=imgiz(@.rgw.buckets[4470.1])
> >>>>>> >> err=(22)
> >>>>>> >> Invalid argument
> >>>>>> >> Checking objects, decreasing bucket 2-phase commit timeout.
> >>>>>> >> ** Note that timeout will reset only when operation completes
> >>>>>> >> successfully
> >>>>>> >> **
> >>>>>> >> ERROR: failed operation r=-22
> >>>>>> >> ERROR: failed operation r=-22
> >>>>>> >> ..
> >>>>>> >> last line keeps repeating without any progress.
> >>>>>> >>
> >>>>>> >> I checked the file omapheaders and while the healty bucket has:
> >>>>>> >> # rados -p .rgw.buckets getomapheader .dir.6912.3
> >>>>>> >> header (49 bytes) :
> >>>>>> >> 0000 : 03 02 2b 00 00 00 01 00 00 00 01 02 02 18 00 00 :
> >>>>>> >> ..+.............
> >>>>>> >> 0010 : 00 a8 af 40 00 00 00 00 00 00 10 46 00 00 00 00 :
> >>>>>> >> ...@.......F....
> >>>>>> >> 0020 : 00 9d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 :
> >>>>>> >> ................
> >>>>>> >> 0030 : 00                                              : .
> >>>>>> >>
> >>>>>> >> the faulty one is missing it
> >>>>>> >> # rados -p .rgw.buckets getomapheader .dir.4470.1
> >>>>>> >> header (0 bytes) :
> >>>>>> >>
> >>>>>> >>
> >>>>>> >> I'm currently in the process of understanding how to create a
> >>>>>> >> readable
> >>>>>> >> header. My hopes are even while its wrong, radosgw-admin will be
> >>>>>> >> able to
> >>>>>> >> read through objects and fix the necessary parts. But i'm not
> sure
> >>>>>> >> how to
> >>>>>> >> set the new-header. It seems there is a setomapheader counterpart
> >>>>>> >> for
> >>>>>> >> getomapheader but it only accepts values from commandline so i
> >>>>>> >> don't know
> >>>>>> >> how to push a rgw-readable binary with it.
> >>>>>> >>
> >>>>>> >> Is this somewhat possible?
> >>>>>> >>
> >>>>>> >> Thanks in advance.
> >>>>>> >>
> >>>>>> >> --
> >>>>>> >> erdem agaoglu
> >>>>>> >
> >>>>>> >
> >>>>>> >
> >>>>>> >
> >>>>>> > --
> >>>>>> > erdem agaoglu
> >>>>>> >
> >>>>>> > _______________________________________________
> >>>>>> > ceph-users mailing list
> >>>>>> > ceph-users@lists.ceph.com
> >>>>>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>>>>> >
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> erdem agaoglu
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> erdem agaoglu
> >>
> >>
> >
> >
> >
> > --
> > erdem agaoglu
>



-- 
erdem agaoglu
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Reply via email to