Hi Radek, I can confirm, v10.2.7 without 2 commits you mentioned earlier works as expected.
Best, On Wed, May 3, 2017 at 2:59 AM, Radoslaw Zarzynski <rzarzyn...@mirantis.com> wrote: > Hello Łukasz, > > Thanks for your testing and sorry for my mistake. It looks that two commits > need to be reverted to get the previous behaviour: > > The already mentioned one: > https://github.com/ceph/ceph/commit/c9445faf7fac2ccb8a05b53152c0ca > 16d7f4c6d0 > Its dependency: > https://github.com/ceph/ceph/commit/b72fc1b820ede3cd186d887d9d30f7 > f91fe3764b > > They have been merged in the same pull request: > https://github.com/ceph/ceph/pull/11760 > and form the difference visible between v10.2.5 and v10.2.6 in the matter > of "in_hosted_domain" handling: > https://github.com/ceph/ceph/blame/v10.2.5/src/rgw/rgw_rest.cc#L1773 > https://github.com/ceph/ceph/blame/v10.2.6/src/rgw/rgw_ > rest.cc#L1781-L1782 > > I'm really not sure we want to revert them. Still, it can be that they just > unhide a misconfiguration issue while fixing the problems we had with > handling of virtual hosted buckets. > > Regards, > Radek > > On Wed, May 3, 2017 at 3:12 AM, Łukasz Jagiełło > <jagiello.luk...@gmail.com> wrote: > > Hi, > > > > I tried today revert [1] from 10.2.7 but the problem is still there even > > without the change. Revert to 10.2.5 fix the issue instantly. > > > > https://github.com/ceph/ceph/commit/c9445faf7fac2ccb8a05b53152c0ca > 16d7f4c6d0 > > > > On Thu, Apr 27, 2017 at 4:53 AM, Radoslaw Zarzynski > > <rzarzyn...@mirantis.com> wrote: > >> > >> Bingo! From the 10.2.5-admin: > >> > >> GET > >> > >> Thu, 27 Apr 2017 07:49:59 GMT > >> / > >> > >> And also: > >> > >> 2017-04-27 09:49:59.117447 7f4a90ff9700 20 subdomain= domain= > >> in_hosted_domain=0 in_hosted_domain_s3website=0 > >> 2017-04-27 09:49:59.117449 7f4a90ff9700 20 final domain/bucket > >> subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 > >> s->info.domain= s->info.request_uri=/ > >> > >> The most interesting part is the "final ... in_hosted_domain=0". > >> It looks we need to dig around RGWREST::preprocess(), > >> rgw_find_host_in_domains() & company. > >> > >> There is a commit introduced in v10.2.6 that touches this area [1]. > >> I'm definitely not saying it's the root cause. It might be that a change > >> in the code just unhidden a configuration issue [2]. > >> > >> I will talk about the problem on the today's sync-up. > >> > >> Thanks for the logs! > >> Regards, > >> Radek > >> > >> [1] > >> https://github.com/ceph/ceph/commit/c9445faf7fac2ccb8a05b53152c0ca > 16d7f4c6d0 > >> [2] http://tracker.ceph.com/issues/17440 > >> > >> On Thu, Apr 27, 2017 at 10:11 AM, Ben Morrice <ben.morr...@epfl.ch> > wrote: > >> > Hello Radek, > >> > > >> > Thank-you for your analysis so far! Please find attached logs for both > >> > the > >> > admin user and a keystone backed user from 10.2.5 (same host as > before, > >> > I > >> > have simply downgraded the packages). Both users can authenticate and > >> > list > >> > buckets on 10.2.5. > >> > > >> > Also - I tried version 10.2.6 and see the same behavior as 10.2.7, so > >> > the > >> > bug i'm hitting looks like it was introduced in 10.2.6 > >> > > >> > Kind regards, > >> > > >> > Ben Morrice > >> > > >> > ____________________________________________________________ > __________ > >> > Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670 > >> > EPFL / BBP > >> > Biotech Campus > >> > Chemin des Mines 9 > >> > 1202 Geneva > >> > Switzerland > >> > > >> > On 27/04/17 04:45, Radoslaw Zarzynski wrote: > >> >> > >> >> Thanks for the logs, Ben. > >> >> > >> >> It looks that two completely different authenticators have failed: > >> >> the local, RADOS-backed auth (admin.txt) and Keystone-based > >> >> one as well. In the second case I'm pretty sure that Keystone has > >> >> rejected [1][2] to authenticate provided signature/StringToSign. > >> >> RGW tried to fallback to the local auth which obviously didn't have > >> >> any chance as the credentials were stored remotely. This explains > >> >> the presence of "error reading user info" in the user-keystone.txt. > >> >> > >> >> What is common for both scenarios are the low-level things related > >> >> to StringToSign crafting/signature generation at RadosGW's side. > >> >> Following one has been composed for the request from admin.txt: > >> >> > >> >> GET > >> >> > >> >> > >> >> Wed, 26 Apr 2017 09:18:42 GMT > >> >> /bbpsrvc15.cscs.ch/ > >> >> > >> >> If you could provide a similar log from v10.2.5, I would be really > >> >> grateful. > >> >> > >> >> Regards, > >> >> Radek > >> >> > >> >> [1] > >> >> > >> >> https://github.com/ceph/ceph/blob/v10.2.7/src/rgw/rgw_rest_ > s3.cc#L3269-L3272 > >> >> [2] https://github.com/ceph/ceph/blob/v10.2.7/src/rgw/rgw_ > common.h#L170 > >> >> > >> >> On Wed, Apr 26, 2017 at 11:29 AM, Morrice Ben <ben.morr...@epfl.ch> > >> >> wrote: > >> >>> > >> >>> Hello Radek, > >> >>> > >> >>> Please find attached the failed request for both the admin user and > a > >> >>> standard user (backed by keystone). > >> >>> > >> >>> Kind regards, > >> >>> > >> >>> Ben Morrice > >> >>> > >> >>> ____________________________________________________________ > __________ > >> >>> Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670 > >> >>> EPFL BBP > >> >>> Biotech Campus > >> >>> Chemin des Mines 9 > >> >>> 1202 Geneva > >> >>> Switzerland > >> >>> > >> >>> ________________________________________ > >> >>> From: Radoslaw Zarzynski <rzarzyn...@mirantis.com> > >> >>> Sent: Tuesday, April 25, 2017 7:38 PM > >> >>> To: Morrice Ben > >> >>> Cc: ceph-users@lists.ceph.com > >> >>> Subject: Re: [ceph-users] RGW 10.2.5->10.2.7 authentication fail? > >> >>> > >> >>> Hello Ben, > >> >>> > >> >>> Could you provide full RadosGW's log for the failed request? > >> >>> I mean the lines starting from header listing, through the start > >> >>> marker ("====== starting new request...") till the end marker? > >> >>> > >> >>> At the moment we can't see any details related to the signature > >> >>> calculation. > >> >>> > >> >>> Regards, > >> >>> Radek > >> >>> > >> >>> On Thu, Apr 20, 2017 at 5:08 PM, Ben Morrice <ben.morr...@epfl.ch> > >> >>> wrote: > >> >>>> > >> >>>> Hi all, > >> >>>> > >> >>>> I have tried upgrading one of our RGW servers from 10.2.5 to 10.2.7 > >> >>>> (RHEL7) > >> >>>> and authentication is in a very bad state. This installation is > part > >> >>>> of > >> >>>> a > >> >>>> multigw configuration, and I have just updated one host in the > >> >>>> secondary > >> >>>> zone (all other hosts/zones are running 10.2.5). > >> >>>> > >> >>>> On the 10.2.7 server I cannot authenticate as a user (normally > backed > >> >>>> by > >> >>>> OpenStack Keystone), but even worse I can also not authenticate > with > >> >>>> an > >> >>>> admin user. > >> >>>> > >> >>>> Please see [1] for the results of performing a list bucket > operation > >> >>>> with > >> >>>> python boto (script works against rgw 10.2.5) > >> >>>> > >> >>>> Also, if I try to authenticate from the 'master' rgw zone with a > >> >>>> "radosgw-admin sync status --rgw-zone=bbp-gva-master" I get: > >> >>>> > >> >>>> "ERROR: failed to fetch datalog info" > >> >>>> > >> >>>> "failed to retrieve sync info: (13) Permission denied" > >> >>>> > >> >>>> The above errors correlates to the errors in the log on the server > >> >>>> running > >> >>>> 10.2.7 (debug level 20) at [2] > >> >>>> > >> >>>> I'm not sure what I have done wrong or can try next? > >> >>>> > >> >>>> By the way, downgrading the packages from 10.2.7 to 10.2.5 returns > >> >>>> authentication functionality > >> >>>> > >> >>>> [1] > >> >>>> boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden > >> >>>> <?xml version="1.0" > >> >>>> > >> >>>> > >> >>>> encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</ > Code><RequestId>tx000000000000000000004-0058f8c86a-3fa2959-bbp-gva- > secondary</RequestId><HostId>3fa2959-bbp-gva-secondary-bbp- > gva</HostId></Error> > >> >>>> > >> >>>> [2] > >> >>>> /bbpsrvc15.cscs.ch/admin/log > >> >>>> 2017-04-20 16:43:04.916253 7ff87c6c0700 15 calculated > >> >>>> digest=Ofg/f/NI0L4eEG1MsGk4PsVscTM= > >> >>>> 2017-04-20 16:43:04.916255 7ff87c6c0700 15 > >> >>>> auth_sign=qZ3qsy7AuNCOoPMhr8yNoy5qMKU= > >> >>>> 2017-04-20 16:43:04.916255 7ff87c6c0700 15 compare=34 > >> >>>> 2017-04-20 16:43:04.916266 7ff87c6c0700 10 failed to authorize > >> >>>> request > >> >>>> 2017-04-20 16:43:04.916268 7ff87c6c0700 20 handler->ERRORHANDLER: > >> >>>> err_no=-2027 new_err_no=-2027 > >> >>>> 2017-04-20 16:43:04.916329 7ff87c6c0700 2 req 354:0.052585:s3:GET > >> >>>> /admin/log:get_obj:op status=0 > >> >>>> 2017-04-20 16:43:04.916339 7ff87c6c0700 2 req 354:0.052595:s3:GET > >> >>>> /admin/log:get_obj:http status=403 > >> >>>> 2017-04-20 16:43:04.916343 7ff87c6c0700 1 ====== req done > >> >>>> req=0x7ff87c6ba710 op status=0 http_status=403 ====== > >> >>>> 2017-04-20 16:43:04.916350 7ff87c6c0700 20 process_request() > returned > >> >>>> -2027 > >> >>>> 2017-04-20 16:43:04.916390 7ff87c6c0700 1 civetweb: > 0x7ff990015610: > >> >>>> 10.80.6.26 - - [20/Apr/2017:16:43:04 +0200] "GET /admin/log > HTTP/1.1" > >> >>>> 403 0 > >> >>>> - - > >> >>>> 2017-04-20 16:43:04.917212 7ff9777e6700 20 > >> >>>> cr:s=0x7ff97000d420:op=0x7ff9703a5440:18RGWMetaSyncShardCR: > operate() > >> >>>> 2017-04-20 16:43:04.917223 7ff9777e6700 20 rgw meta sync: > >> >>>> incremental_sync:1544: shard_id=20 > >> >>>> mdlog_marker=1_1492686039.901886_5551978.1 > >> >>>> sync_marker.marker=1_1492686039.901886_5551978.1 period_marker= > >> >>>> 2017-04-20 16:43:04.917227 7ff9777e6700 20 rgw meta sync: > >> >>>> incremental_sync:1551: shard_id=20 syncing mdlog for shard_id=20 > >> >>>> 2017-04-20 16:43:04.917236 7ff9777e6700 20 > >> >>>> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: > >> >>>> operate() > >> >>>> 2017-04-20 16:43:04.917238 7ff9777e6700 20 rgw meta sync: operate: > >> >>>> shard_id=20: init request > >> >>>> 2017-04-20 16:43:04.917240 7ff9777e6700 20 > >> >>>> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: > >> >>>> operate() > >> >>>> 2017-04-20 16:43:04.917241 7ff9777e6700 20 rgw meta sync: operate: > >> >>>> shard_id=20: reading shard status > >> >>>> 2017-04-20 16:43:04.917303 7ff9777e6700 20 run: > stack=0x7ff97000d420 > >> >>>> is > >> >>>> io > >> >>>> blocked > >> >>>> 2017-04-20 16:43:04.918285 7ff9777e6700 20 > >> >>>> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: > >> >>>> operate() > >> >>>> 2017-04-20 16:43:04.918295 7ff9777e6700 20 rgw meta sync: operate: > >> >>>> shard_id=20: reading shard status complete > >> >>>> 2017-04-20 16:43:04.918307 7ff9777e6700 20 rgw meta sync: > shard_id=20 > >> >>>> marker=1_1492686039.901886_5551978.1 last_update=2017-04-20 > >> >>>> 13:00:39.0.901886s > >> >>>> 2017-04-20 16:43:04.918316 7ff9777e6700 20 > >> >>>> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: > >> >>>> operate() > >> >>>> 2017-04-20 16:43:04.918317 7ff9777e6700 20 rgw meta sync: operate: > >> >>>> shard_id=20: sending rest request > >> >>>> 2017-04-20 16:43:04.918381 7ff9777e6700 20 RGWEnv::set(): > HTTP_DATE: > >> >>>> Thu > >> >>>> Apr > >> >>>> 20 14:43:04 2017 > >> >>>> 2017-04-20 16:43:04.918390 7ff9777e6700 20 > HTTP_DATE -> Thu Apr > 20 > >> >>>> 14:43:04 2017 > >> >>>> 2017-04-20 16:43:04.918404 7ff9777e6700 10 get_canon_resource(): > >> >>>> dest=/admin/log > >> >>>> 2017-04-20 16:43:04.918406 7ff9777e6700 10 generated canonical > >> >>>> header: > >> >>>> GET > >> >>>> > >> >>>> -- > >> >>>> Kind regards, > >> >>>> > >> >>>> Ben Morrice > >> >>>> > >> >>>> > >> >>>> ____________________________________________________________ > __________ > >> >>>> Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670 > >> >>>> EPFL / BBP > >> >>>> Biotech Campus > >> >>>> Chemin des Mines 9 > >> >>>> 1202 Geneva > >> >>>> Switzerland > >> >>>> > >> >>>> _______________________________________________ > >> >>>> ceph-users mailing list > >> >>>> ceph-users@lists.ceph.com > >> >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > > >> > > >> _______________________________________________ > >> ceph-users mailing list > >> ceph-users@lists.ceph.com > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > > -- > > Łukasz Jagiełło > > lukasz<at>jagiello<dot>org > -- Łukasz Jagiełło lukasz<at>jagiello<dot>org
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
