There seems to be an open issue at s3cmd https://github.com/s3tools/s3cmd/issues/37. I'll try with other tools
On Mon, Apr 8, 2013 at 9:26 PM, Yehuda Sadeh <yeh...@inktank.com> wrote: > This one fails because copy object into itself would only work if > replacing it's attrs (X_AMZ_METADATA_DIRECTIVE=REPLACE). > > On Mon, Apr 8, 2013 at 10:35 AM, Erdem Agaoglu <erdem.agao...@gmail.com> > wrote: > > This is the log grepped with the relevant threadid. It shows 400 in the > last > > lines but nothing seems odd besides that. > > http://pastebin.com/xWCYmnXV > > > > Thanks for your interest. > > > > > > On Mon, Apr 8, 2013 at 8:21 PM, Yehuda Sadeh <yeh...@inktank.com> wrote: > >> > >> Each bucket has a unique prefix which you can get by doing radosgw-admin > >> bucket stats on that bucket. You can grep that prefix in 'rados ls -p > >> .rgw.buckets'. > >> > >> Do you have any rgw log showing why you get the Invalid Request > response? > >> Can you also add 'debug ms = 1' for the log? > >> > >> Thanks > >> > >> > >> On Mon, Apr 8, 2013 at 10:12 AM, Erdem Agaoglu <erdem.agao...@gmail.com > > > >> wrote: > >>> > >>> Just tried that file: > >>> > >>> $ s3cmd mv s3://imgiz/data/avatars/492/492923.jpg > >>> s3://imgiz/data/avatars/492/492923.jpg > >>> ERROR: S3 error: 400 (InvalidRequest) > >>> > >>> a more verbose output shows that the sign-headers was > >>> > >>> > 'PUT\n\n\n\nx-amz-copy-source:/imgiz/data/avatars/492/492923.jpg\nx-amz-date:Mon, > >>> 08 Apr 2013 16:59:30 > >>> > +0000\nx-amz-metadata-directive:COPY\n/imgiz/data/avatars/492/492923.jpg' > >>> > >>> But i guess it doesn't work even if the index is correct. I get the > same > >>> response on a clear bucket too. > >>> > >>> We might try that but we don't have a file list. I guess its possible > >>> with 'rados ls | grep | sed' ? > >>> > >>> > >>> On Mon, Apr 8, 2013 at 7:53 PM, Yehuda Sadeh <yeh...@inktank.com> > wrote: > >>>> > >>>> Can you try copying one of these objects to itself? Would that work > >>>> and/or change the index entry? Another option would be to try copying > all > >>>> the objects to a different bucket. > >>>> > >>>> > >>>> On Mon, Apr 8, 2013 at 9:48 AM, Erdem Agaoglu < > erdem.agao...@gmail.com> > >>>> wrote: > >>>>> > >>>>> omap header and all other omap attributes was destroyed. I copied > >>>>> another index over the destroyed one to get a somewhat valid header > and it > >>>>> seems intact. After a 'check --fix': > >>>>> > >>>>> # rados -p .rgw.buckets getomapheader .dir.4470.1 > >>>>> header (49 bytes) : > >>>>> 0000 : 03 02 2b 00 00 00 01 00 00 00 01 02 02 18 00 00 : > >>>>> ..+............. > >>>>> 0010 : 00 7d 7a 3f 6e 01 00 00 00 00 d0 00 7e 01 00 00 : > >>>>> .}z?n.......~... > >>>>> 0020 : 00 bb f5 01 00 00 00 00 00 00 00 00 00 00 00 00 : > >>>>> ................ > >>>>> 0030 : 00 : . > >>>>> > >>>>> > >>>>> Rados shows objects are there: > >>>>> > >>>>> # rados ls -p .rgw.buckets |grep 4470.1_data/avatars > >>>>> 4470.1_data/avatars/11047/11047823_20101211154308.jpg > >>>>> 4470.1_data/avatars/106/106976-orig > >>>>> 4470.1_data/avatars/492/492923.jpg > >>>>> 4470.1_data/avatars/275/275479.jpg > >>>>> ... > >>>>> > >>>>> > >>>>> And i am able to GET them > >>>>> > >>>>> $ s3cmd get s3://imgiz/data/avatars/492/492923.jpg > >>>>> s3://imgiz/data/avatars/492/492923.jpg -> ./492923.jpg [1 of 1] > >>>>> 3587 of 3587 100% in 0s 93.40 kB/s done > >>>>> > >>>>> > >>>>> But unable to list them > >>>>> > >>>>> $ s3cmd ls s3://imgiz/data/avatars > >>>>> <NOTHING> > >>>>> > >>>>> > >>>>> My initial expectation was that 'bucket check --fix --check-objects' > >>>>> will actually read the files like 'rados ls' does and would recreate > the > >>>>> missing omapkeys but it doesn't seem to do that. Now a simple check > says > >>>>> > >>>>> # radosgw-admin bucket check -b imgiz > >>>>> { "existing_header": { "usage": { "rgw.main": { "size_kb": 6000607, > >>>>> "size_kb_actual": 6258740, > >>>>> "num_objects": 128443}}}, > >>>>> "calculated_header": { "usage": { "rgw.main": { "size_kb": 6000607, > >>>>> "size_kb_actual": 6258740, > >>>>> "num_objects": 128443}}}} > >>>>> > >>>>> But i know we have more than 128k objects. > >>>>> > >>>>> > >>>>> > >>>>> On Mon, Apr 8, 2013 at 7:17 PM, Yehuda Sadeh <yeh...@inktank.com> > >>>>> wrote: > >>>>>> > >>>>>> We'll need to have more info about the current state. Was just the > >>>>>> omap header destroyed, or does it still exist? What does the header > >>>>>> contain now? Are you able to actually access objects in that bucket, > >>>>>> but just fail to list them? > >>>>>> > >>>>>> On Mon, Apr 8, 2013 at 8:34 AM, Erdem Agaoglu > >>>>>> <erdem.agao...@gmail.com> wrote: > >>>>>> > Hi again, > >>>>>> > > >>>>>> > I managed to change the file with some other bucket's index. > >>>>>> > --check-objects > >>>>>> > --fix worked but my hopes have failed as it didn't actually read > >>>>>> > through the > >>>>>> > files or fixed anything. Any suggestions? > >>>>>> > > >>>>>> > > >>>>>> > On Thu, Apr 4, 2013 at 5:56 PM, Erdem Agaoglu > >>>>>> > <erdem.agao...@gmail.com> > >>>>>> > wrote: > >>>>>> >> > >>>>>> >> Hi all, > >>>>>> >> > >>>>>> >> After a major failure, and getting our cluster health back OK > (with > >>>>>> >> some > >>>>>> >> help from inktank folks, thanks), we found out that we have > managed > >>>>>> >> to > >>>>>> >> corrupt one of our bucket indices. As far as i can track it, we > are > >>>>>> >> missing > >>>>>> >> the omapheader on that specific index, so we're unable to use > >>>>>> >> radosgw-admin > >>>>>> >> tools to fix it. > >>>>>> >> > >>>>>> >> While a healthy (smaller) bucket answers > >>>>>> >> # radosgw-admin bucket check -b imgdoviz > >>>>>> >> { "existing_header": { "usage": { "rgw.main": { "size_kb": 4140, > >>>>>> >> "size_kb_actual": 4484, > >>>>>> >> "num_objects": 157}}}, > >>>>>> >> "calculated_header": { "usage": { "rgw.main": { "size_kb": > 4140, > >>>>>> >> "size_kb_actual": 4484, > >>>>>> >> "num_objects": 157}}}} > >>>>>> >> > >>>>>> >> The faulty one fails with > >>>>>> >> # radosgw-admin bucket check -b imgiz > >>>>>> >> failed to list objects in bucket=imgiz(@.rgw.buckets[4470.1]) > >>>>>> >> err=(22) > >>>>>> >> Invalid argument > >>>>>> >> failed to check index err=(22) Invalid argument > >>>>>> >> > >>>>>> >> When i push further > >>>>>> >> # radosgw-admin bucket check -b imgiz --check-objects --fix > >>>>>> >> failed to list objects in bucket=imgiz(@.rgw.buckets[4470.1]) > >>>>>> >> err=(22) > >>>>>> >> Invalid argument > >>>>>> >> Checking objects, decreasing bucket 2-phase commit timeout. > >>>>>> >> ** Note that timeout will reset only when operation completes > >>>>>> >> successfully > >>>>>> >> ** > >>>>>> >> ERROR: failed operation r=-22 > >>>>>> >> ERROR: failed operation r=-22 > >>>>>> >> .. > >>>>>> >> last line keeps repeating without any progress. > >>>>>> >> > >>>>>> >> I checked the file omapheaders and while the healty bucket has: > >>>>>> >> # rados -p .rgw.buckets getomapheader .dir.6912.3 > >>>>>> >> header (49 bytes) : > >>>>>> >> 0000 : 03 02 2b 00 00 00 01 00 00 00 01 02 02 18 00 00 : > >>>>>> >> ..+............. > >>>>>> >> 0010 : 00 a8 af 40 00 00 00 00 00 00 10 46 00 00 00 00 : > >>>>>> >> ...@.......F.... > >>>>>> >> 0020 : 00 9d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : > >>>>>> >> ................ > >>>>>> >> 0030 : 00 : . > >>>>>> >> > >>>>>> >> the faulty one is missing it > >>>>>> >> # rados -p .rgw.buckets getomapheader .dir.4470.1 > >>>>>> >> header (0 bytes) : > >>>>>> >> > >>>>>> >> > >>>>>> >> I'm currently in the process of understanding how to create a > >>>>>> >> readable > >>>>>> >> header. My hopes are even while its wrong, radosgw-admin will be > >>>>>> >> able to > >>>>>> >> read through objects and fix the necessary parts. But i'm not > sure > >>>>>> >> how to > >>>>>> >> set the new-header. It seems there is a setomapheader counterpart > >>>>>> >> for > >>>>>> >> getomapheader but it only accepts values from commandline so i > >>>>>> >> don't know > >>>>>> >> how to push a rgw-readable binary with it. > >>>>>> >> > >>>>>> >> Is this somewhat possible? > >>>>>> >> > >>>>>> >> Thanks in advance. > >>>>>> >> > >>>>>> >> -- > >>>>>> >> erdem agaoglu > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > -- > >>>>>> > erdem agaoglu > >>>>>> > > >>>>>> > _______________________________________________ > >>>>>> > ceph-users mailing list > >>>>>> > ceph-users@lists.ceph.com > >>>>>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >>>>>> > > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> erdem agaoglu > >>>> > >>>> > >>> > >>> > >>> > >>> -- > >>> erdem agaoglu > >> > >> > > > > > > > > -- > > erdem agaoglu > -- erdem agaoglu
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
