Hello Alfredo / Craig First of all Thank You So much for replying and giving your precious time to this problem.
@Alfredo : I tried version radosgw-agent version 1.2.2 and the case has progressed a lot. ( below are some the logs ) I am now getting *2015-04-28 00:35:14,781 5132 [radosgw_agent][INFO ] http://us-east-1.crosslogic.com:7480 <http://us-east-1.crosslogic.com:7480> endpoint does not support versioning* *2015-04-28 00:35:14,781 5132 [radosgw_agent][WARNIN] encountered issues reaching to endpoint http://us-east-1.crosslogic.com:7480 <http://us-east-1.crosslogic.com:7480>* *2015-04-28 00:35:14,782 5132 [radosgw_agent][WARNIN] HTTP Error 403: Forbidden* I am using CIVETWEB , any further help in this would be really helpful. [root@us-east-1 ceph]# [root@us-east-1 ceph]# radosgw-agent -c cluster-data-sync.conf 2015-04-28 00:35:14,750 5132 [radosgw_agent][INFO ] __ __ __ ___ ___ 2015-04-28 00:35:14,750 5132 [radosgw_agent][INFO ] /__` \ / |\ | / ` /\ / _` |__ |\ | | 2015-04-28 00:35:14,751 5132 [radosgw_agent][INFO ] .__/ | | \| \__, /~~\ \__> |___ | \| | 2015-04-28 00:35:14,751 5132 [radosgw_agent][INFO ] v1.2.2 2015-04-28 00:35:14,751 5132 [radosgw_agent][INFO ] agent options: 2015-04-28 00:35:14,752 5132 [radosgw_agent][INFO ] args: 2015-04-28 00:35:14,753 5132 [radosgw_agent][INFO ] conf : None 2015-04-28 00:35:14,753 5132 [radosgw_agent][INFO ] dest_access_key : **************** 2015-04-28 00:35:14,753 5132 [radosgw_agent][INFO ] dest_secret_key : **************** 2015-04-28 00:35:14,753 5132 [radosgw_agent][INFO ] destination : http://us-west-1.crosslogic.com:7480 2015-04-28 00:35:14,753 5132 [radosgw_agent][INFO ] incremental_sync_delay : 30 2015-04-28 00:35:14,754 5132 [radosgw_agent][INFO ] lock_timeout : 60 2015-04-28 00:35:14,754 5132 [radosgw_agent][INFO ] log_file : /var/log/radosgw/radosgw-sync-us-east-west.log 2015-04-28 00:35:14,756 5132 [radosgw_agent][INFO ] log_lock_time : 20 2015-04-28 00:35:14,756 5132 [radosgw_agent][INFO ] max_entries : 1000 2015-04-28 00:35:14,757 5132 [radosgw_agent][INFO ] metadata_only : False 2015-04-28 00:35:14,757 5132 [radosgw_agent][INFO ] num_workers : 1 2015-04-28 00:35:14,758 5132 [radosgw_agent][INFO ] object_sync_timeout : 216000 2015-04-28 00:35:14,758 5132 [radosgw_agent][INFO ] prepare_error_delay : 10 2015-04-28 00:35:14,758 5132 [radosgw_agent][INFO ] quiet : False 2015-04-28 00:35:14,758 5132 [radosgw_agent][INFO ] rgw_data_log_window : 30 2015-04-28 00:35:14,759 5132 [radosgw_agent][INFO ] source : None 2015-04-28 00:35:14,759 5132 [radosgw_agent][INFO ] src_access_key : **************** 2015-04-28 00:35:14,759 5132 [radosgw_agent][INFO ] src_secret_key : **************** 2015-04-28 00:35:14,759 5132 [radosgw_agent][INFO ] src_zone : None 2015-04-28 00:35:14,759 5132 [radosgw_agent][INFO ] sync_scope : incremental 2015-04-28 00:35:14,760 5132 [radosgw_agent][INFO ] test_server_host : None 2015-04-28 00:35:14,760 5132 [radosgw_agent][INFO ] test_server_port : 8080 2015-04-28 00:35:14,761 5132 [radosgw_agent][INFO ] verbose : False 2015-04-28 00:35:14,761 5132 [radosgw_agent][INFO ] versioned : False 2015-04-28 00:35:14,761 5132 [radosgw_agent.client][INFO ] creating connection to endpoint: http://us-west-1.crosslogic.com:7480 region map is: {u'us': [u'us-west', u'us-east']} *2015-04-28 00:35:14,781 5132 [radosgw_agent][INFO ] http://us-east-1.crosslogic.com:7480 <http://us-east-1.crosslogic.com:7480> endpoint does not support versioning* *2015-04-28 00:35:14,781 5132 [radosgw_agent][WARNIN] encountered issues reaching to endpoint http://us-east-1.crosslogic.com:7480 <http://us-east-1.crosslogic.com:7480>* *2015-04-28 00:35:14,782 5132 [radosgw_agent][WARNIN] HTTP Error 403: Forbidden* 2015-04-28 00:35:14,782 5132 [radosgw_agent.client][INFO ] creating connection to endpoint: http://us-east-1.crosslogic.com:7480 2015-04-28 00:35:14,784 5132 [radosgw_agent.client][INFO ] creating connection to endpoint: http://us-west-1.crosslogic.com:7480 2015-04-28 00:35:14,785 5132 [radosgw_agent.client][INFO ] creating connection to endpoint: http://us-east-1.crosslogic.com:7480 2015-04-28 00:35:14,787 5132 [radosgw_agent.client][INFO ] creating connection to endpoint: http://us-west-1.crosslogic.com:7480 *2015-04-28 00:35:14,807 5132 [radosgw_agent.sync][ERROR ] finding number of shards failed* 2015-04-28 00:35:14,807 5132 [radosgw_agent.sync][WARNIN] error preparing for sync, will retry. Traceback: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/radosgw_agent/sync.py", line 30, in prepare_sync syncer.prepare() File "/usr/lib/python2.7/site-packages/radosgw_agent/sync.py", line 208, in prepare self.init_num_shards() File "/usr/lib/python2.7/site-packages/radosgw_agent/sync.py", line 77, in init_num_shards self.num_shards = client.num_log_shards(self.src_conn, self.type) File "/usr/lib/python2.7/site-packages/radosgw_agent/client.py", line 403, in num_log_shards out = request(connection, 'get', 'admin/log', dict(type=shard_type)) File "/usr/lib/python2.7/site-packages/radosgw_agent/client.py", line 147, in request check_result_status(result) File "/usr/lib/python2.7/site-packages/radosgw_agent/client.py", line 89, in check_result_status exc.HttpError)(result.status, result.reason) *HttpError: Http error code 403 content Forbidden* *2015-04-28 00:35:24,826 5132 [radosgw_agent.sync][ERROR ] finding number of shards failed* 2015-04-28 00:35:24,826 5132 [radosgw_agent.sync][WARNIN] error preparing for sync, will retry. Traceback: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/radosgw_agent/sync.py", line 30, in prepare_sync syncer.prepare() File "/usr/lib/python2.7/site-packages/radosgw_agent/sync.py", line 208, in prepare self.init_num_shards() File "/usr/lib/python2.7/site-packages/radosgw_agent/sync.py", line 77, in init_num_shards self.num_shards = client.num_log_shards(self.src_conn, self.type) File "/usr/lib/python2.7/site-packages/radosgw_agent/client.py", line 403, in num_log_shards out = request(connection, 'get', 'admin/log', dict(type=shard_type)) File "/usr/lib/python2.7/site-packages/radosgw_agent/client.py", line 147, in request check_result_status(result) File "/usr/lib/python2.7/site-packages/radosgw_agent/client.py", line 89, in check_result_status exc.HttpError)(result.status, result.reason) HttpError: Http error code 403 content Forbidden ^C2015-04-28 00:35:25,609 5132 [radosgw_agent][ERROR ] KeyboardInterrupt [root@us-east-1 ceph]# [root@us-east-1 ceph]# curl http://us-east-1.crosslogic.com:7480 <?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns=" http://s3.amazonaws.com/doc/2006-03-01/ "><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult> [root@us-east-1 ceph]# [root@us-east-1 ceph]# [root@us-east-1 ceph]# curl http://us-west-1.crosslogic.com:7480 <?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns=" http://s3.amazonaws.com/doc/2006-03-01/ "><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult> [root@us-east-1 ceph]# [root@us-east-1 ceph]# [root@us-east-1 ceph]# radosgw-admin region-map get --name client.radosgw.us-east-1 { "regions": [ { "key": "us", "val": { "name": "us", "api_name": "us", "is_master": "true", "endpoints": [ "http:\/\/us-east-1.crosslogic.com:7480\/"], "master_zone": "us-east", "zones": [ { "name": "us-east", "endpoints": [ "http:\/\/us-east-1.crosslogic.com:7480\/"], "log_meta": "true", "log_data": "true"}, { "name": "us-west", "endpoints": [ "http:\/\/us-west-1.crosslogic.com:7480\/"], "log_meta": "true", "log_data": "true"}], "placement_targets": [ { "name": "default-placement", "tags": []}], "default_placement": "default-placement"}}], "master_region": "us", "bucket_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1}, "user_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1}}[root@us-east-1 ceph]# [root@us-east-1 ceph]# [root@us-east-1 ceph]# radosgw-admin region-map get --name client.radosgw.us-west-1 { "regions": [ { "key": "us", "val": { "name": "us", "api_name": "us", "is_master": "true", "endpoints": [ "http:\/\/us-east-1.crosslogic.com:7480\/"], "master_zone": "us-east", "zones": [ { "name": "us-east", "endpoints": [ "http:\/\/us-east-1.crosslogic.com:7480\/"], "log_meta": "true", "log_data": "true"}, { "name": "us-west", "endpoints": [ "http:\/\/us-west-1.crosslogic.com:7480\/"], "log_meta": "true", "log_data": "true"}], "placement_targets": [ { "name": "default-placement", "tags": []}], "default_placement": "default-placement"}}], "master_region": "us", "bucket_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1}, "user_quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1}}[root@us-east-1 ceph]# [root@us-east-1 ceph]# - VS - On Mon, Apr 27, 2015 at 11:59 PM, Alfredo Deza <ad...@redhat.com> wrote: > Hi Vickey (and all) > > It looks like this issue was introduced as part of the 1.2.1 release. > > I just finished getting 1.2.2 out (try upgrading please). You should no > longer see that > error. > > Hope that helps! > > -Alfredo > > ----- Original Message ----- > From: "Craig Lewis" <cle...@centraldesktop.com> > To: "Vickey Singh" <vickey.singh22...@gmail.com> > Cc: ceph-users@lists.ceph.com > Sent: Monday, April 27, 2015 4:23:52 PM > Subject: Re: [ceph-users] Ceph Radosgw multi zone data replication failure > > > [root@us-east-1 ceph] # ceph -s --name client.radosgw.us-east-1 > > > > [root@us-east-1 ceph]# ceph -s --name client.radosgw.us-west-1 > > Are you trying to setup two zones on one cluster? That's possible, but > you'll also want to spend some time on your CRUSH map making sure that the > two zones are as independent as possible (no shared disks, etc). > > Are you using Civetweb or Apache + FastCGI? > > Can you include the output (from both clusters): > radosgw-admin --name=client.radosgw.us-east-1 region get > radosgw-admin --name=client.radosgw.us-east-1 zone get > > Double check that both system users exist in both clusters, with the same > secret. > > > > > On Sun, Apr 26, 2015 at 8:01 AM, Vickey Singh < > vickey.singh22...@gmail.com > wrote: > > > > > > Hello Geeks > > > > > I am trying to setup Ceph Radosgw multi site data replication using > official documentation > http://ceph.com/docs/master/radosgw/federated-config/#multi-site-data-replication > > > > > Everything seems to work except radosgw-agent sync , Request you to please > check the below outputs and help me in any possible way. > > > > > Environment : > > > > > CentOS 7.0.1406 > > Ceph Versino 0.87.1 > > Rados Gateway configured using Civetweb > > > > > > > > Radosgw zone list : Works nicely > > > > > > [root@us-east-1 ceph]# radosgw-admin zone list --name > client.radosgw.us-east-1 > > { "zones": [ > > "us-west", > > "us-east"]} > > [root@us-east-1 ceph]# > > > > > Curl request to master zone : Works nicely > > > > > > [root@us-east-1 ceph]# curl http://us-east-1.crosslogic.com:7480 > > <?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns=" > http://s3.amazonaws.com/doc/2006-03-01/ > "><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult> > > [root@us-east-1 ceph]# > > > > > Curl request to secondary zone : Works nicely > > > > > [root@us-east-1 ceph]# curl http://us-west-1.crosslogic.com:7480 > > <?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns=" > http://s3.amazonaws.com/doc/2006-03-01/ > "><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult> > > [root@us-east-1 ceph]# > > > > > Rados Gateway agent configuration file : Seems correct, no TYPO errors > > > > > > [root@us-east-1 ceph] # cat cluster-data-sync.conf > > src_access_key: M7QAKDH8CYGTK86CG93U > > src_secret_key: 0xQR6PINk23W\/GYrWJ14aF+1stG56M6xMkqkdloO > > destination: http://us-west-1.crosslogic.com:7480 > > dest_access_key: ZQ32ES1WAWPG05YMZ7T7 > > dest_secret_key: INvk8AkrZRsejLEL34yRpMLmOqydt8ncOXy4RHCM > > log_file: /var/log/radosgw/radosgw-sync-us-east-west.log > > [root@us-east-1 ceph]# > > > > > Rados Gateway agent SYNC : Fails , however it can fetch region map so i > think src and dest KEYS are correct. But don't know why it fails on > AttributeError > > > > > > [root@us-east-1 ceph]# radosgw-agent -c cluster-data-sync.conf > > > region map is: {u'us': [u'us-west', u'us-east']} > > Traceback (most recent call last): > > File "/usr/bin/radosgw-agent", line 21, in <module> > > sys.exit(main()) > > File "/usr/lib/python2.7/site-packages/radosgw_agent/cli.py", line 275, in > main > > except client.ClientException as e: > > AttributeError: 'module' object has no attribute 'ClientException' > > [root@us-east-1 ceph]# > > > > > Can query to Ceph cluster using us-east-1 ID > > > > > [root@us-east-1 ceph] # ceph -s --name client.radosgw.us-east-1 > > cluster 9609b429-eee2-4e23-af31-28a24fcf5cbc > > health HEALTH_OK > > monmap e3: 3 mons at {ceph-node1= > 192.168.1.101:6789/0,ceph-node2=192.168.1.102:6789/0,ceph-node3=192.168.1.103:6789/0 > }, election epoch 448, quorum 0,1,2 ceph-node1,ceph-node2,ceph-node3 > > osdmap e1063: 9 osds: 9 up, 9 in > > pgmap v8473: 1500 pgs, 43 pools, 374 MB data, 2852 objects > > 1193 MB used, 133 GB / 134 GB avail > > 1500 active+clean > > [root@us-east-1 ceph]# > > > > > Can query to Ceph cluster using us-west-1 ID > > > > > > [root@us-east-1 ceph]# ceph -s --name client.radosgw.us-west-1 > > cluster 9609b429-eee2-4e23-af31-28a24fcf5cbc > > health HEALTH_OK > > monmap e3: 3 mons at {ceph-node1= > 192.168.1.101:6789/0,ceph-node2=192.168.1.102:6789/0,ceph-node3=192.168.1.103:6789/0 > }, election epoch 448, quorum 0,1,2 ceph-node1,ceph-node2,ceph-node3 > > osdmap e1063: 9 osds: 9 up, 9 in > > pgmap v8473: 1500 pgs, 43 pools, 374 MB data, 2852 objects > > 1193 MB used, 133 GB / 134 GB avail > > 1500 active+clean > > [root@us-east-1 ceph]# > > > > > Hope these packages are correct > > > > > > [root@us-east-1 ceph]# rpm -qa | egrep -i "ceph|radosgw" > > libcephfs1-0.87.1-0.el7.centos.x86_64 > > ceph-common-0.87.1-0.el7.centos.x86_64 > > python-ceph-0.87.1-0.el7.centos.x86_64 > > ceph-radosgw-0.87.1-0.el7.centos.x86_64 > > ceph-release-1-0.el7.noarch > > ceph-0.87.1-0.el7.centos.x86_64 > > radosgw-agent-1.2.1-0.el7.centos.noarch > > [root@us-east-1 ceph]# > > > > > > > > Regards > > VS > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
