Re: [ceph-users] swiftclient call radosgw, it always response 401 Unauthorized
Hello, We definitely need more info. Does only HEAD-on-account fail or other requests are affected as well? Can we get more verbose RadosGW's log? In such situation debug_rgw=20 might be necessary. Regards, Radoslaw Zarzynski On Mon, Sep 19, 2016 at 12:33 PM, Orit Wasserman wrote: > Any ideas? > I would guess the swift user is not created correctly but as it with > keystone ... > > > -- Forwarded message -- > From: Brian Chang-Chien > Date: Mon, Sep 19, 2016 at 5:22 AM > Subject: Re: [ceph-users] swiftclient call radosgw, it always response > 401 Unauthorized > To: ceph-users@lists.ceph.com > > > no body meet this situation? Can somebody help me slove the issue, > please !!! THX > > 2016-09-16 13:02 GMT+08:00 Brian Chang-Chien : >> >> Can anyone know this problem,please help me to watch this >> >> >> 2016年9月13日 下午5:58，"Brian Chang-Chien" 寫道： >>> >>> Hi ,naga.b >>> >>> I use Ceph jewel 10.2.2 >>> my ceph.conf as follow >>> [global] >>> fsid = d056c174-2e3a-4c36-a067-cb774d176ce2 >>> mon_initial_members = brianceph >>> mon_host = 10.62.9.140 >>> auth_cluster_required = cephx >>> auth_service_required = cephx >>> auth_client_required = cephx >>> osd_crush_chooseleaf_type = 0 >>> osd_pool_default_size = 1 >>> osd_journal_size = 100 >>> [client.radosgw.gateway] >>> host = brianceph >>> keyring = /etc/ceph/ceph.client.radosgw.keyring >>> log_file = /var/log/ceph/radosgw.log >>> rgw_dns_name = brianceph >>> rgw_keystone_url = http://10.62.13.253:35357 >>> rgw_keystone_admin_token = 7bb8e26cbc714c47a26ffec3d96f246f >>> rgw_keystone_accepted_roles = admin, swiftuser >>> rgw_ketstone_token_cache_size = 200 >>> rgw_keystone_revocation_interval = 30 >>> rgw_s3_auth_use_keystone = true >>> nss_db_path = /var/ceph/nss >>> >>> and my radosgw.log >>> >>> 2016-09-13 17:42:38.638462 7efd964619c0 0 starting handler: fastcgi >>> 2016-09-13 17:42:38.638523 7efcadf9b700 0 ERROR: no socket server point >>> defined, cannot start fcgi frontend >>> 2016-09-13 17:47:33.597070 7efcdeffd700 1 == starting new request >>> req=0x7efcdeff7710 = >>> 2016-09-13 17:47:33.597329 7efcdeffd700 1 == req done >>> req=0x7efcdeff7710 op status=0 http_status=401 == >>> 2016-09-13 17:47:33.597379 7efcdeffd700 1 civetweb: 0x7efd2bb0: >>> 10.62.9.34 - - [13/Sep/2016:17:47:33 +0800] "HEAD /swift/v1 HTTP/1.1" 401 0 >>> - python-swiftclient-2.6.0 >>> 2016-09-13 17:47:34.755291 7efcd700 1 == starting new request >>> req=0x7efcdfff9710 = >>> 2016-09-13 17:47:34.755443 7efcd700 1 == req done >>> req=0x7efcdfff9710 op status=0 http_status=401 == >>> 2016-09-13 17:47:34.755481 7efcd700 1 civetweb: 0x7efd48004020: >>> 10.62.9.34 - - [13/Sep/2016:17:47:34 +0800] "HEAD /swift/v1 HTTP/1.1" 401 0 >>> - python-swiftclient-2.6.0 >>> 2016-09-13 17:49:04.718249 7efcdf7fe700 1 == starting new request >>> req=0x7efcdf7f8710 = >>> 2016-09-13 17:49:04.718438 7efcdf7fe700 1 == req done >>> req=0x7efcdf7f8710 op status=0 http_status=401 == >>> 2016-09-13 17:49:04.718483 7efcdf7fe700 1 civetweb: 0x7efd68001f60: >>> 10.62.9.34 - - [13/Sep/2016:17:49:04 +0800] "HEAD /swift/v1 HTTP/1.1" 401 0 >>> - python-swiftclient-2.6.0 >>> 2016-09-13 17:49:05.870115 7efcde7fc700 1 == starting new request >>> req=0x7efcde7f6710 = >>> 2016-09-13 17:49:05.870280 7efcde7fc700 1 == req done >>> req=0x7efcde7f6710 op status=0 http_status=401 == >>> 2016-09-13 17:49:05.870324 7efcde7fc700 1 civetweb: 0x7efd28000bb0: >>> 10.62.9.34 - - [13/Sep/2016:17:49:05 +0800] "HEAD /swift/v1 HTTP/1.1" 401 0 >>> - python-swiftclient-2.6.0 >>> 2016-09-13 17:51:32.036065 7efd157fa700 1 handle_sigterm >>> 2016-09-13 17:51:32.036099 7efd157fa700 1 handle_sigterm set alarm for 120 >>> 2016-09-13 17:51:32.036153 7efd964619c0 -1 shutting down >>> 2016-09-13 17:51:32.037977 7efd78df9700 0 monclient: hunting for new mon >>> 2016-09-13 17:51:32.038172 7efd783f6700 0 -- 10.62.9.140:0/1002906388 >> >>> 10.62.9.140:6789/0 pipe(0x7efd60016670 sd=7 :0 s=1 pgs=0 cs=0 l=1 >>> c=0x7efd60014d70).fault >>> 2016-09-13 17:51:32.906553 7efd964619c0 1 final shutdown >>> 2016-09-13 17:51:39.294948 7ff5175f29c0 0 deferred set uid:gid
Re: [ceph-users] swiftclient call radosgw, it always response 401 Unauthorized
Hi Brian, Responded inline. On Tue, Sep 20, 2016 at 5:45 AM, Brian Chang-Chien wrote: > > > 2016-09-20 10:14:38.761635 7f2049ffb700 20 > HTTP_X_AUTH_TOKEN=b243614d27244d00b12b2f366b58d709 > 2016-09-20 10:14:38.761636 7f2049ffb700 20 QUERY_STRING= > ... > 2016-09-20 10:14:38.761720 7f2049ffb700 2 req 3:0.78:swift:HEAD > /swift/v1:stat_account:authorizing > 2016-09-20 10:14:38.761725 7f2049ffb700 10 failed to authorize request > 2016-09-20 10:14:38.761726 7f2049ffb700 20 handler->ERRORHANDLER: err_no=-1 > new_err_no=-1 Those logs show there was no jump to the Keystone code at all. This is because the "token_id=..." debug message [1] is absent. The sole reason I see for such behavior is that the RadosGW instance internally sees rgw_keystone_url as empty [2][3]. Are you absolutely sure that the instance that got debug_rgw to its configuration file has rgw_keystone_url properly set? I mean whether the setting is in the same section, is written in pure ASCII (without some crazy UTF characters) and so on? I saw you posted the config earlier but we really need to double check. Could you also provide output from following curl command and corresponding RadosGW's log? 401 is fully expected as we'll intensionally send an invalid token. curl -i "http://:/swift/v1" -X HEAD -H "X-Auth-Token: random_string" > > > I also have some problems > > Q1 : if use keystone, radosgw need create user and subuser? > in the case , i create admin user and admin:admin subuser , but i think it > don't need , and i rght? Yup, this is unnecessary when using the Keystone auth. > > > Q2: > And i found a phenomenon, > Once I connect keystone and ceph radosgw before, and i use " rados --pool > default.rgw.users.uid ls " > It will detail a like token uid > > but if swift response 401 > i can't find the token uid > Do you know keystone how to add token user to default.rgw.users.uid > finally , hope bellow msgs can help me to slove > anyway, thx your support greate You don't need to add anything. RadosGW will create RGWUserInfo if necessary on the first, successfully authenticated request [4]. The RADOS object will be named after the tenant ID in Keystone. Best regards, Radoslaw Zarzynski [1] https://github.com/ceph/ceph/blob/v10.2.2/src/rgw/rgw_swift.cc#L472 [2] https://github.com/ceph/ceph/blob/v10.2.2/src/rgw/rgw_swift.cc#L766-L769 [3] https://github.com/ceph/ceph/blob/v10.2.2/src/rgw/rgw_swift.h#L59-L61 [4] https://github.com/ceph/ceph/blob/v10.2.2/src/rgw/rgw_swift.cc#L413 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] swiftclient call radosgw, it always response 401 Unauthorized
Hi, Responded inline. On Wed, Sep 21, 2016 at 4:54 AM, Brian Chang-Chien wrote: > > > [global] > ... > debug rgw = 20 > [client.radosgw.gateway] > host = brianceph > rgw keystone url = http://10.62.13.253:35357 > rgw keystone admin token = 7bb8e26cbc714c47a26ffec3d96f246f > rgw keystone accepted roles = admin, swiftuser, user, _member_, Member > rgw keystone token cache size = 500 > rgw keystone revocation interval = 60 > rgw keystone make new tenants = true > rgw s3 auth use keystone = true > nss db path = /var/ceph/nss The debug_rgw=20 has been put into the global section. I bet that's the sole reason why this particular RadosGW instance sees it. > and I still some config problem > > Q3 : when i edit /etc/ceph/ceph.conf , if my hostname is brianceph > the radosgw term in ceph.conf should be [client.radosgw.gateway] or > [client.radosgw.brianceph] > which one is correct? > > PS: when i create radosgw, i call th cmd " ceph-deploy rgw create brianceph" Most likely your current section naming is wrong. I haven't poked with ceph-deloy too much but I would say it should be [client.rgw.brianceph] or [client.radosgw.brianceph]. I don't have now any cluster alive to disambiguate, sorry. > Q4: when i finish to edit ceph.conf, i need restart radosgw service or > restart ceph service > in this case, i use ceph jewel, so which cmd need to call " systemctl > restart ceph-radosgw.target " or "systemctl ceph.target" Take a look on that: http://docs.ceph.com/docs/jewel/install/install-ceph-gateway/ > Q5: when i use ceph-deploy new brianceph, ceph will generate a ceph.cof, what > kind edit ceph.conf to create rgw is prefer > > Method1: i direct edit ceph.conf from ceph geerated, and use ceph-deploy > --overwrite-conf rgw create brianceph > > Method2(i used in the case) : first i call ceph-deploy rgw create brianceph, > and then edit ceph.conf in /etc/ceph/ folder , then call systemctl restart > ceph-radosgw.target > > > two methods i find some different issues, > in Method1, the radosgw item of the ceph.conf in /etc/ceph/ like "rgw > keystone url" will convert rgw_keystone_url , Spaces and underscores in configurables' names are treated in the same way. No difference. Regards, Radoslaw Zarzynski ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?
Hello Ben,
Could you provide full RadosGW's log for the failed request?
I mean the lines starting from header listing, through the start
marker ("== starting new request...") till the end marker?
At the moment we can't see any details related to the signature
calculation.
Regards,
Radek
On Thu, Apr 20, 2017 at 5:08 PM, Ben Morrice wrote:
> Hi all,
>
> I have tried upgrading one of our RGW servers from 10.2.5 to 10.2.7 (RHEL7)
> and authentication is in a very bad state. This installation is part of a
> multigw configuration, and I have just updated one host in the secondary
> zone (all other hosts/zones are running 10.2.5).
>
> On the 10.2.7 server I cannot authenticate as a user (normally backed by
> OpenStack Keystone), but even worse I can also not authenticate with an
> admin user.
>
> Please see [1] for the results of performing a list bucket operation with
> python boto (script works against rgw 10.2.5)
>
> Also, if I try to authenticate from the 'master' rgw zone with a
> "radosgw-admin sync status --rgw-zone=bbp-gva-master" I get:
>
> "ERROR: failed to fetch datalog info"
>
> "failed to retrieve sync info: (13) Permission denied"
>
> The above errors correlates to the errors in the log on the server running
> 10.2.7 (debug level 20) at [2]
>
> I'm not sure what I have done wrong or can try next?
>
> By the way, downgrading the packages from 10.2.7 to 10.2.5 returns
> authentication functionality
>
> [1]
> boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
> encoding="UTF-8"?>SignatureDoesNotMatchtx4-0058f8c86a-3fa2959-bbp-gva-secondary3fa2959-bbp-gva-secondary-bbp-gva
>
> [2]
> /bbpsrvc15.cscs.ch/admin/log
> 2017-04-20 16:43:04.916253 7ff87c6c0700 15 calculated
> digest=Ofg/f/NI0L4eEG1MsGk4PsVscTM=
> 2017-04-20 16:43:04.916255 7ff87c6c0700 15
> auth_sign=qZ3qsy7AuNCOoPMhr8yNoy5qMKU=
> 2017-04-20 16:43:04.916255 7ff87c6c0700 15 compare=34
> 2017-04-20 16:43:04.916266 7ff87c6c0700 10 failed to authorize request
> 2017-04-20 16:43:04.916268 7ff87c6c0700 20 handler->ERRORHANDLER:
> err_no=-2027 new_err_no=-2027
> 2017-04-20 16:43:04.916329 7ff87c6c0700 2 req 354:0.052585:s3:GET
> /admin/log:get_obj:op status=0
> 2017-04-20 16:43:04.916339 7ff87c6c0700 2 req 354:0.052595:s3:GET
> /admin/log:get_obj:http status=403
> 2017-04-20 16:43:04.916343 7ff87c6c0700 1 == req done
> req=0x7ff87c6ba710 op status=0 http_status=403 ==
> 2017-04-20 16:43:04.916350 7ff87c6c0700 20 process_request() returned -2027
> 2017-04-20 16:43:04.916390 7ff87c6c0700 1 civetweb: 0x7ff990015610:
> 10.80.6.26 - - [20/Apr/2017:16:43:04 +0200] "GET /admin/log HTTP/1.1" 403 0
> - -
> 2017-04-20 16:43:04.917212 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff9703a5440:18RGWMetaSyncShardCR: operate()
> 2017-04-20 16:43:04.917223 7ff9777e6700 20 rgw meta sync:
> incremental_sync:1544: shard_id=20
> mdlog_marker=1_1492686039.901886_5551978.1
> sync_marker.marker=1_1492686039.901886_5551978.1 period_marker=
> 2017-04-20 16:43:04.917227 7ff9777e6700 20 rgw meta sync:
> incremental_sync:1551: shard_id=20 syncing mdlog for shard_id=20
> 2017-04-20 16:43:04.917236 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
> 2017-04-20 16:43:04.917238 7ff9777e6700 20 rgw meta sync: operate:
> shard_id=20: init request
> 2017-04-20 16:43:04.917240 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
> 2017-04-20 16:43:04.917241 7ff9777e6700 20 rgw meta sync: operate:
> shard_id=20: reading shard status
> 2017-04-20 16:43:04.917303 7ff9777e6700 20 run: stack=0x7ff97000d420 is io
> blocked
> 2017-04-20 16:43:04.918285 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
> 2017-04-20 16:43:04.918295 7ff9777e6700 20 rgw meta sync: operate:
> shard_id=20: reading shard status complete
> 2017-04-20 16:43:04.918307 7ff9777e6700 20 rgw meta sync: shard_id=20
> marker=1_1492686039.901886_5551978.1 last_update=2017-04-20
> 13:00:39.0.901886s
> 2017-04-20 16:43:04.918316 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
> 2017-04-20 16:43:04.918317 7ff9777e6700 20 rgw meta sync: operate:
> shard_id=20: sending rest request
> 2017-04-20 16:43:04.918381 7ff9777e6700 20 RGWEnv::set(): HTTP_DATE: Thu Apr
> 20 14:43:04 2017
> 2017-04-20 16:43:04.918390 7ff9777e6700 20 > HTTP_DATE -> Thu Apr 20
> 14:43:04 2017
> 2017-04-20 16:43:04.918404 7ff9777e6700 10 get_canon_resource():
> dest=/admin/log
> 2017-04-20 16:43:04.918406 7ff9777e6700 10 generated canonical header: GET
>
> --
> Kind regards,
>
> Ben Morrice
>
> __
> Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
> EPFL / BBP
> Biotech Campus
> Chemin des Mines 9
> 1202 Geneva
> Switzerland
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cg
Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?
Thanks for the logs, Ben.
It looks that two completely different authenticators have failed:
the local, RADOS-backed auth (admin.txt) and Keystone-based
one as well. In the second case I'm pretty sure that Keystone has
rejected [1][2] to authenticate provided signature/StringToSign.
RGW tried to fallback to the local auth which obviously didn't have
any chance as the credentials were stored remotely. This explains
the presence of "error reading user info" in the user-keystone.txt.
What is common for both scenarios are the low-level things related
to StringToSign crafting/signature generation at RadosGW's side.
Following one has been composed for the request from admin.txt:
GET
Wed, 26 Apr 2017 09:18:42 GMT
/bbpsrvc15.cscs.ch/
If you could provide a similar log from v10.2.5, I would be really grateful.
Regards,
Radek
[1] https://github.com/ceph/ceph/blob/v10.2.7/src/rgw/rgw_rest_s3.cc#L3269-L3272
[2] https://github.com/ceph/ceph/blob/v10.2.7/src/rgw/rgw_common.h#L170
On Wed, Apr 26, 2017 at 11:29 AM, Morrice Ben wrote:
> Hello Radek,
>
> Please find attached the failed request for both the admin user and a
> standard user (backed by keystone).
>
> Kind regards,
>
> Ben Morrice
>
> __
> Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
> EPFL BBP
> Biotech Campus
> Chemin des Mines 9
> 1202 Geneva
> Switzerland
>
>
> From: Radoslaw Zarzynski
> Sent: Tuesday, April 25, 2017 7:38 PM
> To: Morrice Ben
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?
>
> Hello Ben,
>
> Could you provide full RadosGW's log for the failed request?
> I mean the lines starting from header listing, through the start
> marker ("== starting new request...") till the end marker?
>
> At the moment we can't see any details related to the signature
> calculation.
>
> Regards,
> Radek
>
> On Thu, Apr 20, 2017 at 5:08 PM, Ben Morrice wrote:
>> Hi all,
>>
>> I have tried upgrading one of our RGW servers from 10.2.5 to 10.2.7 (RHEL7)
>> and authentication is in a very bad state. This installation is part of a
>> multigw configuration, and I have just updated one host in the secondary
>> zone (all other hosts/zones are running 10.2.5).
>>
>> On the 10.2.7 server I cannot authenticate as a user (normally backed by
>> OpenStack Keystone), but even worse I can also not authenticate with an
>> admin user.
>>
>> Please see [1] for the results of performing a list bucket operation with
>> python boto (script works against rgw 10.2.5)
>>
>> Also, if I try to authenticate from the 'master' rgw zone with a
>> "radosgw-admin sync status --rgw-zone=bbp-gva-master" I get:
>>
>> "ERROR: failed to fetch datalog info"
>>
>> "failed to retrieve sync info: (13) Permission denied"
>>
>> The above errors correlates to the errors in the log on the server running
>> 10.2.7 (debug level 20) at [2]
>>
>> I'm not sure what I have done wrong or can try next?
>>
>> By the way, downgrading the packages from 10.2.7 to 10.2.5 returns
>> authentication functionality
>>
>> [1]
>> boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
>> > encoding="UTF-8"?>SignatureDoesNotMatchtx4-0058f8c86a-3fa2959-bbp-gva-secondary3fa2959-bbp-gva-secondary-bbp-gva
>>
>> [2]
>> /bbpsrvc15.cscs.ch/admin/log
>> 2017-04-20 16:43:04.916253 7ff87c6c0700 15 calculated
>> digest=Ofg/f/NI0L4eEG1MsGk4PsVscTM=
>> 2017-04-20 16:43:04.916255 7ff87c6c0700 15
>> auth_sign=qZ3qsy7AuNCOoPMhr8yNoy5qMKU=
>> 2017-04-20 16:43:04.916255 7ff87c6c0700 15 compare=34
>> 2017-04-20 16:43:04.916266 7ff87c6c0700 10 failed to authorize request
>> 2017-04-20 16:43:04.916268 7ff87c6c0700 20 handler->ERRORHANDLER:
>> err_no=-2027 new_err_no=-2027
>> 2017-04-20 16:43:04.916329 7ff87c6c0700 2 req 354:0.052585:s3:GET
>> /admin/log:get_obj:op status=0
>> 2017-04-20 16:43:04.916339 7ff87c6c0700 2 req 354:0.052595:s3:GET
>> /admin/log:get_obj:http status=403
>> 2017-04-20 16:43:04.916343 7ff87c6c0700 1 == req done
>> req=0x7ff87c6ba710 op status=0 http_status=403 ==
>> 2017-04-20 16:43:04.916350 7ff87c6c0700 20 process_request() returned -2027
>> 2017-04-20 16:43:04.916390 7ff87c6c0700 1 civetweb: 0x7ff990015610:
>> 10.80.6.26 - - [20/Apr/2017:16:43:04 +0200] "GET /admin/log HTTP/1.1" 403 0
>> - -
>> 2017-04-20 16:43:04.917212 7ff9777e6700 20
>> cr
Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?
Bingo! From the 10.2.5-admin:
GET
Thu, 27 Apr 2017 07:49:59 GMT
/
And also:
2017-04-27 09:49:59.117447 7f4a90ff9700 20 subdomain= domain=
in_hosted_domain=0 in_hosted_domain_s3website=0
2017-04-27 09:49:59.117449 7f4a90ff9700 20 final domain/bucket
subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
s->info.domain= s->info.request_uri=/
The most interesting part is the "final ... in_hosted_domain=0".
It looks we need to dig around RGWREST::preprocess(),
rgw_find_host_in_domains() & company.
There is a commit introduced in v10.2.6 that touches this area [1].
I'm definitely not saying it's the root cause. It might be that a change
in the code just unhidden a configuration issue [2].
I will talk about the problem on the today's sync-up.
Thanks for the logs!
Regards,
Radek
[1] https://github.com/ceph/ceph/commit/c9445faf7fac2ccb8a05b53152c0ca16d7f4c6d0
[2] http://tracker.ceph.com/issues/17440
On Thu, Apr 27, 2017 at 10:11 AM, Ben Morrice wrote:
> Hello Radek,
>
> Thank-you for your analysis so far! Please find attached logs for both the
> admin user and a keystone backed user from 10.2.5 (same host as before, I
> have simply downgraded the packages). Both users can authenticate and list
> buckets on 10.2.5.
>
> Also - I tried version 10.2.6 and see the same behavior as 10.2.7, so the
> bug i'm hitting looks like it was introduced in 10.2.6
>
> Kind regards,
>
> Ben Morrice
>
> __
> Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
> EPFL / BBP
> Biotech Campus
> Chemin des Mines 9
> 1202 Geneva
> Switzerland
>
> On 27/04/17 04:45, Radoslaw Zarzynski wrote:
>>
>> Thanks for the logs, Ben.
>>
>> It looks that two completely different authenticators have failed:
>> the local, RADOS-backed auth (admin.txt) and Keystone-based
>> one as well. In the second case I'm pretty sure that Keystone has
>> rejected [1][2] to authenticate provided signature/StringToSign.
>> RGW tried to fallback to the local auth which obviously didn't have
>> any chance as the credentials were stored remotely. This explains
>> the presence of "error reading user info" in the user-keystone.txt.
>>
>> What is common for both scenarios are the low-level things related
>> to StringToSign crafting/signature generation at RadosGW's side.
>> Following one has been composed for the request from admin.txt:
>>
>>GET
>>
>>
>>Wed, 26 Apr 2017 09:18:42 GMT
>>/bbpsrvc15.cscs.ch/
>>
>> If you could provide a similar log from v10.2.5, I would be really
>> grateful.
>>
>> Regards,
>> Radek
>>
>> [1]
>> https://github.com/ceph/ceph/blob/v10.2.7/src/rgw/rgw_rest_s3.cc#L3269-L3272
>> [2] https://github.com/ceph/ceph/blob/v10.2.7/src/rgw/rgw_common.h#L170
>>
>> On Wed, Apr 26, 2017 at 11:29 AM, Morrice Ben wrote:
>>>
>>> Hello Radek,
>>>
>>> Please find attached the failed request for both the admin user and a
>>> standard user (backed by keystone).
>>>
>>> Kind regards,
>>>
>>> Ben Morrice
>>>
>>> __
>>> Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
>>> EPFL BBP
>>> Biotech Campus
>>> Chemin des Mines 9
>>> 1202 Geneva
>>> Switzerland
>>>
>>>
>>> From: Radoslaw Zarzynski
>>> Sent: Tuesday, April 25, 2017 7:38 PM
>>> To: Morrice Ben
>>> Cc: ceph-users@lists.ceph.com
>>> Subject: Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?
>>>
>>> Hello Ben,
>>>
>>> Could you provide full RadosGW's log for the failed request?
>>> I mean the lines starting from header listing, through the start
>>> marker ("== starting new request...") till the end marker?
>>>
>>> At the moment we can't see any details related to the signature
>>> calculation.
>>>
>>> Regards,
>>> Radek
>>>
>>> On Thu, Apr 20, 2017 at 5:08 PM, Ben Morrice wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I have tried upgrading one of our RGW servers from 10.2.5 to 10.2.7
>>>> (RHEL7)
>>>> and authentication is in a very bad state. This installation is part of
>>>> a
>>>> multigw configuration, and I have just updated one host in the secondary
>>>> zone (all other hosts/zones ar
Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail?
Hello Łukasz, Thanks for your testing and sorry for my mistake. It looks that two commits need to be reverted to get the previous behaviour: The already mentioned one: https://github.com/ceph/ceph/commit/c9445faf7fac2ccb8a05b53152c0ca16d7f4c6d0 Its dependency: https://github.com/ceph/ceph/commit/b72fc1b820ede3cd186d887d9d30f7f91fe3764b They have been merged in the same pull request: https://github.com/ceph/ceph/pull/11760 and form the difference visible between v10.2.5 and v10.2.6 in the matter of "in_hosted_domain" handling: https://github.com/ceph/ceph/blame/v10.2.5/src/rgw/rgw_rest.cc#L1773 https://github.com/ceph/ceph/blame/v10.2.6/src/rgw/rgw_rest.cc#L1781-L1782 I'm really not sure we want to revert them. Still, it can be that they just unhide a misconfiguration issue while fixing the problems we had with handling of virtual hosted buckets. Regards, Radek On Wed, May 3, 2017 at 3:12 AM, Łukasz Jagiełło wrote: > Hi, > > I tried today revert [1] from 10.2.7 but the problem is still there even > without the change. Revert to 10.2.5 fix the issue instantly. > > https://github.com/ceph/ceph/commit/c9445faf7fac2ccb8a05b53152c0ca16d7f4c6d0 > > On Thu, Apr 27, 2017 at 4:53 AM, Radoslaw Zarzynski > wrote: >> >> Bingo! From the 10.2.5-admin: >> >> GET >> >> Thu, 27 Apr 2017 07:49:59 GMT >> / >> >> And also: >> >> 2017-04-27 09:49:59.117447 7f4a90ff9700 20 subdomain= domain= >> in_hosted_domain=0 in_hosted_domain_s3website=0 >> 2017-04-27 09:49:59.117449 7f4a90ff9700 20 final domain/bucket >> subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 >> s->info.domain= s->info.request_uri=/ >> >> The most interesting part is the "final ... in_hosted_domain=0". >> It looks we need to dig around RGWREST::preprocess(), >> rgw_find_host_in_domains() & company. >> >> There is a commit introduced in v10.2.6 that touches this area [1]. >> I'm definitely not saying it's the root cause. It might be that a change >> in the code just unhidden a configuration issue [2]. >> >> I will talk about the problem on the today's sync-up. >> >> Thanks for the logs! >> Regards, >> Radek >> >> [1] >> https://github.com/ceph/ceph/commit/c9445faf7fac2ccb8a05b53152c0ca16d7f4c6d0 >> [2] http://tracker.ceph.com/issues/17440 >> >> On Thu, Apr 27, 2017 at 10:11 AM, Ben Morrice wrote: >> > Hello Radek, >> > >> > Thank-you for your analysis so far! Please find attached logs for both >> > the >> > admin user and a keystone backed user from 10.2.5 (same host as before, >> > I >> > have simply downgraded the packages). Both users can authenticate and >> > list >> > buckets on 10.2.5. >> > >> > Also - I tried version 10.2.6 and see the same behavior as 10.2.7, so >> > the >> > bug i'm hitting looks like it was introduced in 10.2.6 >> > >> > Kind regards, >> > >> > Ben Morrice >> > >> > __ >> > Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670 >> > EPFL / BBP >> > Biotech Campus >> > Chemin des Mines 9 >> > 1202 Geneva >> > Switzerland >> > >> > On 27/04/17 04:45, Radoslaw Zarzynski wrote: >> >> >> >> Thanks for the logs, Ben. >> >> >> >> It looks that two completely different authenticators have failed: >> >> the local, RADOS-backed auth (admin.txt) and Keystone-based >> >> one as well. In the second case I'm pretty sure that Keystone has >> >> rejected [1][2] to authenticate provided signature/StringToSign. >> >> RGW tried to fallback to the local auth which obviously didn't have >> >> any chance as the credentials were stored remotely. This explains >> >> the presence of "error reading user info" in the user-keystone.txt. >> >> >> >> What is common for both scenarios are the low-level things related >> >> to StringToSign crafting/signature generation at RadosGW's side. >> >> Following one has been composed for the request from admin.txt: >> >> >> >>GET >> >> >> >> >> >>Wed, 26 Apr 2017 09:18:42 GMT >> >>/bbpsrvc15.cscs.ch/ >> >> >> >> If you could provide a similar log from v10.2.5, I would be really >> >> grateful. >> >> >> >> Regards, >> >> Radek >> >&
