Hello,
Is there an answer to why this is happening, I am facing the same issue, I
have the non-system user replicated to the slave zone, but still getting
403, the same thing happening when I am replicating from the master zone of
master region to master zone of secondary region.
I am using swift, and have created a non-system user for the same
-Hemant
On Tue, Nov 25, 2014 at 12:37 AM, Craig Lewis <cle...@centraldesktop.com>
wrote:
> I'm really not sure. I'm using the S3 interface rather than the Swift
> interface. Once my non-systm user replicated, I was able to access
> everything in the secondary cluster just fine.
>
> Hopefully somebody else with Swift experience will chime in.
>
>
>
> On Sat, Nov 22, 2014 at 12:47 AM, Vinod H I <vinvi...@gmail.com> wrote:
>
>> Thanks for the clarification.
>> Now I have done exactly as you suggested.
>> "us-east" is the master zone and "us-west" is the secondary zone.
>> Each zone has two system users "us-east" and "us-west".
>> These system users have same access/secret keys in both zones.
>> I have checked the pools to confirm that the non-system swift user which
>> i created("east-user:swift") in the primary has been replicated to the
>> secondary zone.
>> The buckets which are created in primary by the swift user are also there
>> in the pools of the secondary zone.
>> But when i try to authenticate this swift user in secondary zone, it says
>> access denied.
>>
>> Here are the relevant logs from the secondary zone, when i try to
>> authenticate the swift user.
>>
>> 2014-11-22 14:19:14.239976 7f73ecff9700 2
>> RGWDataChangesLog::ChangesRenewThread: start
>> 2014-11-22 14:19:14.243454 7f73fe236780 20 get_obj_state: rctx=0x2316ce0
>> obj=.us.rgw.root:region_info.us state=0x2319048 s->prefetch_data=0
>> 2014-11-22 14:19:14.243454 7f73fe236780 10 cache get: name=.us.rgw.root+
>> region_info.us : miss
>> 2014-11-22 14:19:14.252263 7f73fe236780 10 cache put: name=.us.rgw.root+
>> region_info.us
>> 2014-11-22 14:19:14.252283 7f73fe236780 10 adding .us.rgw.root+
>> region_info.us to cache LRU end
>> 2014-11-22 14:19:14.252310 7f73fe236780 20 get_obj_state: s->obj_tag was
>> set empty
>> 2014-11-22 14:19:14.252336 7f73fe236780 10 cache get: name=.us.rgw.root+
>> region_info.us : type miss (requested=1, cached=6)
>> 2014-11-22 14:19:14.252376 7f73fe236780 20 get_obj_state: rctx=0x2316ce0
>> obj=.us.rgw.root:region_info.us state=0x2319958 s->prefetch_data=0
>> 2014-11-22 14:19:14.252386 7f73fe236780 10 cache get: name=.us.rgw.root+
>> region_info.us : hit
>> 2014-11-22 14:19:14.252391 7f73fe236780 20 get_obj_state: s->obj_tag was
>> set empty
>> 2014-11-22 14:19:14.252404 7f73fe236780 20 get_obj_state: rctx=0x2316ce0
>> obj=.us.rgw.root:region_info.us state=0x2319958 s->prefetch_data=0
>> 2014-11-22 14:19:14.252409 7f73fe236780 20 state for obj=.us.rgw.root:
>> region_info.us is not atomic, not appending atomic test
>> 2014-11-22 14:19:14.252412 7f73fe236780 20 rados->read obj-ofs=0
>> read_ofs=0 read_len=524288
>> 2014-11-22 14:19:14.264611 7f73fe236780 20 rados->read r=0 bl.length=266
>> 2014-11-22 14:19:14.264650 7f73fe236780 10 cache put: name=.us.rgw.root+
>> region_info.us
>> 2014-11-22 14:19:14.264653 7f73fe236780 10 moving .us.rgw.root+
>> region_info.us to cache LRU end
>> 2014-11-22 14:19:14.264766 7f73fe236780 20 get_obj_state: rctx=0x2319860
>> obj=.us-west.rgw.root:zone_info.us-west state=0x2313b98 s->prefetch_data=0
>> 2014-11-22 14:19:14.264779 7f73fe236780 10 cache get:
>> name=.us-west.rgw.root+zone_info.us-west : miss
>> 2014-11-22 14:19:14.276114 7f73fe236780 10 cache put:
>> name=.us-west.rgw.root+zone_info.us-west
>> 2014-11-22 14:19:14.276131 7f73fe236780 10 adding
>> .us-west.rgw.root+zone_info.us-west to cache LRU end
>> 2014-11-22 14:19:14.276142 7f73fe236780 20 get_obj_state: s->obj_tag was
>> set empty
>> 2014-11-22 14:19:14.276161 7f73fe236780 10 cache get:
>> name=.us-west.rgw.root+zone_info.us-west : type miss (requested=1, cached=6)
>> 2014-11-22 14:19:14.276203 7f73fe236780 20 get_obj_state: rctx=0x2314660
>> obj=.us-west.rgw.root:zone_info.us-west state=0x2313b98 s->prefetch_data=0
>> 2014-11-22 14:19:14.276212 7f73fe236780 10 cache get:
>> name=.us-west.rgw.root+zone_info.us-west : hit
>> 2014-11-22 14:19:14.276218 7f73fe236780 20 get_obj_state: s->obj_tag was
>> set empty
>> 2014-11-22 14:19:14.276229 7f73fe236780 20 get_obj_state: rctx=0x2314660
>> obj=.us-west.rgw.root:zone_info.us-west state=0x2313b98 s->prefetch_data=0
>> 2014-11-22 14:19:14.276235 7f73fe236780 20 state for
>> obj=.us-west.rgw.root:zone_info.us-west is not atomic, not appending atomic
>> test
>> 2014-11-22 14:19:14.276238 7f73fe236780 20 rados->read obj-ofs=0
>> read_ofs=0 read_len=524288
>> 2014-11-22 14:19:14.290757 7f73fe236780 20 rados->read r=0 bl.length=997
>> 2014-11-22 14:19:14.290797 7f73fe236780 10 cache put:
>> name=.us-west.rgw.root+zone_info.us-west
>> 2014-11-22 14:19:14.290803 7f73fe236780 10 moving
>> .us-west.rgw.root+zone_info.us-west to cache LRU end
>> 2014-11-22 14:19:14.290857 7f73fe236780 2 zone us-west is NOT master
>> 2014-11-22 14:19:14.290931 7f73fe236780 20 get_obj_state: rctx=0x2313cc0
>> obj=.us-west.rgw.root:region_map state=0x2311e08 s->prefetch_data=0
>> 2014-11-22 14:19:14.290949 7f73fe236780 10 cache get:
>> name=.us-west.rgw.root+region_map : miss
>> 2014-11-22 14:19:14.298169 7f73fe236780 10 cache put:
>> name=.us-west.rgw.root+region_map
>> 2014-11-22 14:19:14.298184 7f73fe236780 10 adding
>> .us-west.rgw.root+region_map to cache LRU end
>> 2014-11-22 14:19:14.298195 7f73fe236780 20 get_obj_state: s->obj_tag was
>> set empty
>> 2014-11-22 14:19:14.298212 7f73fe236780 10 cache get:
>> name=.us-west.rgw.root+region_map : type miss (requested=1, cached=6)
>> 2014-11-22 14:19:14.298255 7f73fe236780 20 get_obj_state: rctx=0x2313cc0
>> obj=.us-west.rgw.root:region_map state=0x2311e08 s->prefetch_data=0
>> 2014-11-22 14:19:14.298267 7f73fe236780 10 cache get:
>> name=.us-west.rgw.root+region_map : hit
>> 2014-11-22 14:19:14.298272 7f73fe236780 20 get_obj_state: s->obj_tag was
>> set empty
>> 2014-11-22 14:19:14.298282 7f73fe236780 20 get_obj_state: rctx=0x2313cc0
>> obj=.us-west.rgw.root:region_map state=0x2311e08 s->prefetch_data=0
>> 2014-11-22 14:19:14.298286 7f73fe236780 20 state for
>> obj=.us-west.rgw.root:region_map is not atomic, not appending atomic test
>> 2014-11-22 14:19:14.298288 7f73fe236780 20 rados->read obj-ofs=0
>> read_ofs=0 read_len=524288
>> 2014-11-22 14:19:14.300462 7f73fe236780 20 rados->read r=0 bl.length=334
>> 2014-11-22 14:19:14.300486 7f73fe236780 10 cache put:
>> name=.us-west.rgw.root+region_map
>> 2014-11-22 14:19:14.300490 7f73fe236780 10 moving
>> .us-west.rgw.root+region_map to cache LRU end
>> 2014-11-22 14:19:14.545543 7f73fe236780 20 generating connection object
>> for zone us-east
>> 2014-11-22 14:19:14.548178 7f73fe236780 0 framework: fastcgi
>> 2014-11-22 14:19:14.548767 7f73fe236780 0 starting handler: fastcgi
>> 2014-11-22 14:19:14.549768 7f73ceffd700 20 UserSyncThread: start
>> 2014-11-22 14:19:14.550665 7f73cf7fe700 20 BucketsSyncThread: start
>> 2014-11-22 14:19:14.553127 7f73cdffb700 10 allocated request
>> req=0x7f73e000d010
>> 2014-11-22 14:19:14.553458 7f73cffff700 2 garbage collection: start
>> 2014-11-22 14:19:14.576569 7f73ceffd700 20 RGWRados::pool_iterate: got
>> east-user.buckets
>> 2014-11-22 14:19:14.731146 7f73ceffd700 20 RGWRados::pool_iterate: got
>> us-west
>> 2014-11-22 14:19:14.771842 7f73ceffd700 20 RGWRados::pool_iterate: got
>> us-east
>> 2014-11-22 14:19:14.803904 7f73ceffd700 20 RGWRados::pool_iterate: got
>> east-user
>> 2014-11-22 14:19:14.834887 7f73ceffd700 20 RGWUserStatsCache: sync
>> user=us-west
>> 2014-11-22 14:19:14.839166 7f73ceffd700 0 ERROR: can't read user header:
>> ret=-2
>> 2014-11-22 14:19:14.839182 7f73ceffd700 0 ERROR: sync_user() failed,
>> user=us-west ret=-2
>> 2014-11-22 14:19:14.839186 7f73ceffd700 20 RGWUserStatsCache: sync
>> user=us-east
>> 2014-11-22 14:19:14.846530 7f73ceffd700 0 ERROR: can't read user header:
>> ret=-2
>> 2014-11-22 14:19:14.846540 7f73ceffd700 0 ERROR: sync_user() failed,
>> user=us-east ret=-2
>> 2014-11-22 14:19:14.846543 7f73ceffd700 20 RGWUserStatsCache: sync
>> user=east-user
>> 2014-11-22 14:19:14.852550 7f73ceffd700 20 user is idle, not doing a full
>> sync (user=east-user)
>> 2014-11-22 14:19:15.994741 7f73cffff700 2 garbage collection: stop
>> 2014-11-22 14:19:21.917609 7f73cdffb700 20 enqueued request
>> req=0x7f73e000d010
>> 2014-11-22 14:19:21.917657 7f73cdffb700 20 RGWWQ:
>> 2014-11-22 14:19:21.917661 7f73cdffb700 20 req: 0x7f73e000d010
>> 2014-11-22 14:19:21.917679 7f73cdffb700 10 allocated request
>> req=0x7f73e0013dc0
>> 2014-11-22 14:19:21.918043 7f73b07c0700 20 dequeued request
>> req=0x7f73e000d010
>> 2014-11-22 14:19:21.918067 7f73b07c0700 20 RGWWQ: empty
>> 2014-11-22 14:19:21.918220 7f73b07c0700 20 DOCUMENT_ROOT=/var/www
>> 2014-11-22 14:19:21.918228 7f73b07c0700 20 FCGI_ROLE=RESPONDER
>> 2014-11-22 14:19:21.918228 7f73b07c0700 20 GATEWAY_INTERFACE=CGI/1.1
>> 2014-11-22 14:19:21.918228 7f73b07c0700 20 HTTP_ACCEPT_ENCODING=identity
>> 2014-11-22 14:19:21.918228 7f73b07c0700 20 HTTP_AUTHORIZATION=
>> 2014-11-22 14:19:21.918228 7f73b07c0700 20 HTTP_HOST=us-west-1.lt.com
>> 2014-11-22 14:19:21.918228 7f73b07c0700 20
>> HTTP_X_AUTH_KEY=MHA4vFaDy5XsJqpF5NuZLcBMCoJcuot44ASDuReY
>> 2014-11-22 14:19:21.918228 7f73b07c0700 20
>> HTTP_X_AUTH_USER=east-user:swift
>> 2014-11-22 14:19:21.918228 7f73b07c0700 20
>> PATH=/usr/local/bin:/usr/bin:/bin
>> 2014-11-22 14:19:21.918228 7f73b07c0700 20 QUERY_STRING=
>> 2014-11-22 14:19:21.918228 7f73b07c0700 20 REMOTE_ADDR=192.168.7.141
>> 2014-11-22 14:19:21.918228 7f73b07c0700 20 REMOTE_PORT=50857
>> 2014-11-22 14:19:21.918228 7f73b07c0700 20 REQUEST_METHOD=GET
>> 2014-11-22 14:19:21.918228 7f73b07c0700 20 REQUEST_URI=/auth/
>> 2014-11-22 14:19:21.918228 7f73b07c0700 20
>> SCRIPT_FILENAME=/var/www/s3gw.fcgi
>> 2014-11-22 14:19:21.918228 7f73b07c0700 20 SCRIPT_NAME=/auth/
>> 2014-11-22 14:19:21.918228 7f73b07c0700 20 SCRIPT_URI=
>> http://us-west-1.lt.com/auth/
>> 2014-11-22 14:19:21.918229 7f73b07c0700 20 SCRIPT_URL=/auth/
>> 2014-11-22 14:19:21.918229 7f73b07c0700 20 SERVER_ADDR=192.168.7.117
>> 2014-11-22 14:19:21.918229 7f73b07c0700 20 SERVER_ADMIN=
>> vinvi...@gmail.com
>> 2014-11-22 14:19:21.918229 7f73b07c0700 20 SERVER_NAME=us-west-1.lt.com
>> 2014-11-22 14:19:21.918229 7f73b07c0700 20 SERVER_PORT=80
>> 2014-11-22 14:19:21.918229 7f73b07c0700 20 SERVER_PROTOCOL=HTTP/1.1
>> 2014-11-22 14:19:21.918229 7f73b07c0700 20 SERVER_SIGNATURE=
>> 2014-11-22 14:19:21.918229 7f73b07c0700 20 SERVER_SOFTWARE=Apache/2.2.22
>> (Ubuntu)
>> 2014-11-22 14:19:21.918229 7f73b07c0700 1 ====== starting new request
>> req=0x7f73e000d010 =====
>> 2014-11-22 14:19:21.918229 7f73b07c0700 2 req 1:0.000000::GET
>> /auth/::initializing
>> 2014-11-22 14:19:21.918229 7f73b07c0700 10 host=us-west-1.lt.com
>> rgw_dns_name=us-west-1.lt.com
>> 2014-11-22 14:19:21.918288 7f73b07c0700 2 req 1:0.000053:swift-auth:GET
>> /auth/::getting op
>> 2014-11-22 14:19:21.918300 7f73b07c0700 2 req 1:0.000071:swift-auth:GET
>> /auth/:swift_auth_get:authorizing
>> 2014-11-22 14:19:21.918307 7f73b07c0700 2 req 1:0.000078:swift-auth:GET
>> /auth/:swift_auth_get:reading permissions
>> 2014-11-22 14:19:21.918313 7f73b07c0700 2 req 1:0.000084:swift-auth:GET
>> /auth/:swift_auth_get:init op
>> 2014-11-22 14:19:21.918319 7f73b07c0700 2 req 1:0.000090:swift-auth:GET
>> /auth/:swift_auth_get:verifying op mask
>> 2014-11-22 14:19:21.918325 7f73b07c0700 20 required_mask= 0 user.op_mask=7
>> 2014-11-22 14:19:21.918330 7f73b07c0700 2 req 1:0.000100:swift-auth:GET
>> /auth/:swift_auth_get:verifying op permissions
>> 2014-11-22 14:19:21.918336 7f73b07c0700 2 req 1:0.000107:swift-auth:GET
>> /auth/:swift_auth_get:verifying op params
>> 2014-11-22 14:19:21.918341 7f73b07c0700 2 req 1:0.000112:swift-auth:GET
>> /auth/:swift_auth_get:executing
>> 2014-11-22 14:19:21.918470 7f73b07c0700 20 get_obj_state:
>> rctx=0x7f73dc002030 obj=.us-west.users.swift:east-user:swift
>> state=0x7f73dc0066d8 s->prefetch_data=0
>> 2014-11-22 14:19:21.918494 7f73b07c0700 10 cache get:
>> name=.us-west.users.swift+east-user:swift : miss
>> 2014-11-22 14:19:21.931892 7f73b07c0700 10 cache put:
>> name=.us-west.users.swift+east-user:swift
>> 2014-11-22 14:19:21.931892 7f73b07c0700 10 adding
>> .us-west.users.swift+east-user:swift to cache LRU end
>> 2014-11-22 14:19:21.931892 7f73b07c0700 20 get_obj_state: s->obj_tag was
>> set empty
>> 2014-11-22 14:19:21.931892 7f73b07c0700 10 cache get:
>> name=.us-west.users.swift+east-user:swift : type miss (requested=1,
>> cached=6)
>> 2014-11-22 14:19:21.931893 7f73b07c0700 20 get_obj_state:
>> rctx=0x7f73dc007300 obj=.us-west.users.swift:east-user:swift
>> state=0x7f73dc006558 s->prefetch_data=0
>> 2014-11-22 14:19:21.931893 7f73b07c0700 10 cache get:
>> name=.us-west.users.swift+east-user:swift : hit
>> 2014-11-22 14:19:21.931893 7f73b07c0700 20 get_obj_state: s->obj_tag was
>> set empty
>> 2014-11-22 14:19:21.931893 7f73b07c0700 20 get_obj_state:
>> rctx=0x7f73dc007300 obj=.us-west.users.swift:east-user:swift
>> state=0x7f73dc006558 s->prefetch_data=0
>> 2014-11-22 14:19:21.931893 7f73b07c0700 20 state for
>> obj=.us-west.users.swift:east-user:swift is not atomic, not appending
>> atomic test
>> 2014-11-22 14:19:21.931893 7f73b07c0700 20 rados->read obj-ofs=0
>> read_ofs=0 read_len=524288
>> 2014-11-22 14:19:21.932003 7f73b07c0700 20 rados->read r=0 bl.length=13
>> 2014-11-22 14:19:21.932021 7f73b07c0700 10 cache put:
>> name=.us-west.users.swift+east-user:swift
>> 2014-11-22 14:19:21.932023 7f73b07c0700 10 moving
>> .us-west.users.swift+east-user:swift to cache LRU end
>> 2014-11-22 14:19:21.932054 7f73b07c0700 20 get_obj_state:
>> rctx=0x7f73dc006b30 obj=.us-west.users.uid:east-user state=0x7f73dc006498
>> s->prefetch_data=0
>> 2014-11-22 14:19:21.932062 7f73b07c0700 10 cache get:
>> name=.us-west.users.uid+east-user : miss
>> 2014-11-22 14:19:21.933559 7f73b07c0700 10 cache put:
>> name=.us-west.users.uid+east-user
>> 2014-11-22 14:19:21.933567 7f73b07c0700 10 adding
>> .us-west.users.uid+east-user to cache LRU end
>> 2014-11-22 14:19:21.933572 7f73b07c0700 20 get_obj_state: s->obj_tag was
>> set empty
>> 2014-11-22 14:19:21.933580 7f73b07c0700 10 cache get:
>> name=.us-west.users.uid+east-user : type miss (requested=1, cached=6)
>> 2014-11-22 14:19:21.933601 7f73b07c0700 20 get_obj_state:
>> rctx=0x7f73dc006b30 obj=.us-west.users.uid:east-user state=0x7f73dc006498
>> s->prefetch_data=0
>> 2014-11-22 14:19:21.933607 7f73b07c0700 10 cache get:
>> name=.us-west.users.uid+east-user : hit
>> 2014-11-22 14:19:21.933611 7f73b07c0700 20 get_obj_state: s->obj_tag was
>> set empty
>> 2014-11-22 14:19:21.933617 7f73b07c0700 20 get_obj_state:
>> rctx=0x7f73dc006b30 obj=.us-west.users.uid:east-user state=0x7f73dc006498
>> s->prefetch_data=0
>> 2014-11-22 14:19:21.933620 7f73b07c0700 20 state for
>> obj=.us-west.users.uid:east-user is not atomic, not appending atomic test
>> 2014-11-22 14:19:21.933622 7f73b07c0700 20 rados->read obj-ofs=0
>> read_ofs=0 read_len=524288
>> 2014-11-22 14:19:21.934709 7f73b07c0700 20 rados->read r=0 bl.length=310
>> 2014-11-22 14:19:21.934725 7f73b07c0700 10 cache put:
>> name=.us-west.users.uid+east-user
>> 2014-11-22 14:19:21.934727 7f73b07c0700 10 moving
>> .us-west.users.uid+east-user to cache LRU end
>> 2014-11-22 14:19:21.934790 7f73b07c0700 2 req 1:0.016560:swift-auth:GET
>> /auth/:swift_auth_get:http status=403
>> 2014-11-22 14:19:21.934794 7f73b07c0700 1 ====== req done
>> req=0x7f73e000d010 http_status=403 ======
>> 2014-11-22 14:19:21.934800 7f73b07c0700 20 process_request() returned -1
>>
>> Why am I not able to authenticate?
>>
>> On Fri, Nov 21, 2014 at 1:04 AM, Craig Lewis <cle...@centraldesktop.com>
>> wrote:
>>
>>> You need to create two system users, in both zones. They should have
>>> the same name, access key, and secret in both zones. By convention, these
>>> system users are named the same as the zones.
>>>
>>> You shouldn't use those system users for anything other than
>>> replication. You should create a non-system user to interact with the
>>> cluster. Just like you don't run as root all the time, you don't want to
>>> be a radosgw system user all the time. You only need to create this user
>>> in the primary zone.
>>>
>>> Once replication is working, it should copy the non-system user to the
>>> secondary cluster, as well as any buckets and objects this user creates.
>>>
>>>
>>> On Wed, Nov 19, 2014 at 1:16 AM, Vinod H I <vinvi...@gmail.com> wrote:
>>>
>>>> Hi,
>>>> I am using firefly version 0.80.7.
>>>> I am testing disaster recovery mechanism for rados gateways.
>>>> I have followed the federated gateway setup as mentioned in the docs.
>>>> There is one region with two zones on the same cluster.
>>>> After sync(using radosgw-agent, with "--sync-scope=full"), container
>>>> created by the swift user(with "--system" flag) on the master zone gateway
>>>> is not visible for the swift user(with "--system" flag) on the slave zone.
>>>> There are no error during the syncing process.
>>>> I tried by creating a new slave zone user with same uid and access and
>>>> secret keys as that of master. It did not work!
>>>> Any idea on how to be able to read the synced containers from the slave
>>>> zone?
>>>> Is there any requirement that the two zones must be on separate
>>>> clusters?
>>>> --
>>>> Vinod H I
>>>>
>>>>
>>>> _______________________________________________
>>>> ceph-users mailing list
>>>> ceph-users@lists.ceph.com
>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>
>>>>
>>>
>>
>>
>> --
>> Vinod H I
>>
>>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
