Re: [ceph-users] Fixing a rgw bucket index

2013-04-09 Thread Erdem Agaoglu
We ended up directly importing our original files to another bucket. Now we're cleaning the files in the broken bucket. Thanks for all the help. On Mon, Apr 8, 2013 at 10:27 PM, Erdem Agaoglu wrote: > There seems to be an open issue at s3cmd > https://github.com/s3tools/s3cmd/issues/37. I'll tr

Re: [ceph-users] Fixing a rgw bucket index

2013-04-08 Thread Erdem Agaoglu
There seems to be an open issue at s3cmd https://github.com/s3tools/s3cmd/issues/37. I'll try with other tools On Mon, Apr 8, 2013 at 9:26 PM, Yehuda Sadeh wrote: > This one fails because copy object into itself would only work if > replacing it's attrs (X_AMZ_METADATA_DIRECTIVE=REPLACE). > > O

Re: [ceph-users] Fixing a rgw bucket index

2013-04-08 Thread Yehuda Sadeh
This one fails because copy object into itself would only work if replacing it's attrs (X_AMZ_METADATA_DIRECTIVE=REPLACE). On Mon, Apr 8, 2013 at 10:35 AM, Erdem Agaoglu wrote: > This is the log grepped with the relevant threadid. It shows 400 in the last > lines but nothing seems odd besides tha

Re: [ceph-users] Fixing a rgw bucket index

2013-04-08 Thread Erdem Agaoglu
This is the log grepped with the relevant threadid. It shows 400 in the last lines but nothing seems odd besides that. http://pastebin.com/xWCYmnXV Thanks for your interest. On Mon, Apr 8, 2013 at 8:21 PM, Yehuda Sadeh wrote: > Each bucket has a unique prefix which you can get by doing radosgw

Re: [ceph-users] Fixing a rgw bucket index

2013-04-08 Thread Erdem Agaoglu
Just tried that file: $ s3cmd mv s3://imgiz/data/avatars/492/492923.jpg s3://imgiz/data/avatars/492/492923.jpg ERROR: S3 error: 400 (InvalidRequest) a more verbose output shows that the sign-headers was 'PUT\n\n\n\nx-amz-copy-source:/imgiz/data/avatars/492/492923.jpg\nx-amz-date:Mon, 08 Apr 2013

Re: [ceph-users] Fixing a rgw bucket index

2013-04-08 Thread Yehuda Sadeh
Can you try copying one of these objects to itself? Would that work and/or change the index entry? Another option would be to try copying all the objects to a different bucket. On Mon, Apr 8, 2013 at 9:48 AM, Erdem Agaoglu wrote: > omap header and all other omap attributes was destroyed. I copie

Re: [ceph-users] Fixing a rgw bucket index

2013-04-08 Thread Erdem Agaoglu
omap header and all other omap attributes was destroyed. I copied another index over the destroyed one to get a somewhat valid header and it seems intact. After a 'check --fix': # rados -p .rgw.buckets getomapheader .dir.4470.1 header (49 bytes) : : 03 02 2b 00 00 00 01 00 00 00 01 02 02 18

Re: [ceph-users] Fixing a rgw bucket index

2013-04-08 Thread Yehuda Sadeh
We'll need to have more info about the current state. Was just the omap header destroyed, or does it still exist? What does the header contain now? Are you able to actually access objects in that bucket, but just fail to list them? On Mon, Apr 8, 2013 at 8:34 AM, Erdem Agaoglu wrote: > Hi again,

Re: [ceph-users] Fixing a rgw bucket index

2013-04-08 Thread Erdem Agaoglu
Hi again, I managed to change the file with some other bucket's index. --check-objects --fix worked but my hopes have failed as it didn't actually read through the files or fixed anything. Any suggestions? On Thu, Apr 4, 2013 at 5:56 PM, Erdem Agaoglu wrote: > Hi all, > > After a major failure,

[ceph-users] Fixing a rgw bucket index

2013-04-04 Thread Erdem Agaoglu
Hi all, After a major failure, and getting our cluster health back OK (with some help from inktank folks, thanks), we found out that we have managed to corrupt one of our bucket indices. As far as i can track it, we are missing the omapheader on that specific index, so we're unable to use radosgw-