On Thu, Jan 15, 2015 at 04:49:37PM -0600, Matt Riedemann wrote: > > > On 1/15/2015 11:40 AM, Matt Riedemann wrote: > > > > > >On 1/13/2015 9:27 PM, Matt Riedemann wrote: > >> > >> > >>On 1/13/2015 12:11 PM, Steven Hardy wrote: > >>>On Tue, Jan 13, 2015 at 10:00:04AM -0600, Matt Riedemann wrote: > >>>>Looks like the fix we merged didn't actually fix the problem. I have > >>>>a patch > >>>>[1] to uncap the boto requirement on master and it's failing the ec2 > >>>>tests > >>>>in tempest the same as before. > >>> > >>>FWIW, I just re-tested and boto 2.35.1 works fine for me locally, if you > >>>revert my patch it breaks again with "Signature not provided" errors > >>>(for > >>>all ec2 API requests). > >>> > >>>If you look at the failures in the log, it actually looks like a > >>>different > >>>problem: > >>> > >>>EC2ResponseError: EC2ResponseError: 401 Unauthorized > >>> > >>>This is not the same as the original error which rejected any request > >>>inside the nova API before even calling keystone with a message like > >>>this: > >>> > >>>AuthFailure: Signature not provided > >>> > >>>AFAICT this means my patch is working, and there's a different problem > >>>affecting only a subset of the ec2 boto tests. > >>> > >>>Steve > >>> > >>>__________________________________________________________________________ > >>> > >>> > >>>OpenStack Development Mailing List (not for usage questions) > >>>Unsubscribe: > >>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >>> > >> > >>Yeah, new bug reported, looks like we're hitting 401 Unauthorized errors > >>when trying to create security groups in the test: > >> > >>https://bugs.launchpad.net/nova/+bug/1410622 > >> > > > >I have a debug patch up here to try and recreate the tempest failures > >with latest boto but using a nova debug change also to get more > >information when we fail. > > > >https://review.openstack.org/#/c/147601/ > > > > I finally narrowed this down to some code in keystone where it generates a > signature and compares that to what nova is passing in on the request for > ec2 credentials and they are different so keystone is rejecting the request > with a 401. > > http://logs.openstack.org/01/147601/3/check/check-tempest-dsvm-full/96bb05e/logs/apache/keystone.txt.gz#_2015-01-15_22_00_27_046 > > I'm assuming something needs to change in keystone to support the version 4 > format?
Keystone already supports the hmac v4 format, the code which creates the signature keystone compares with that in the request actually lives in keystoneclient: https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/contrib/ec2/utils.py#L156 We have been bitten by a couple of bugs in the past where boto changed the way it calculated the signature (presumably in a way compatible with AWS), but generally that breaks all requests not just a specific one as in this case (CreateSecurityGroup). Interestingly, this seems to work OK for me locally, e.g using euca-add-group, which AFAICS is the same API action the test is failing on: -bash-4.2$ source /opt/stack/devstack/accrc/demo/demo -bash-4.2$ euca-add-group --debug --description foo shtest2 2015-01-15 23:33:22,890 euca2ools [DEBUG]:Using access key provided by client. 2015-01-15 23:33:22,890 euca2ools [DEBUG]:Using secret key provided by client. 2015-01-15 23:33:22,890 euca2ools [DEBUG]:Method: POST 2015-01-15 23:33:22,890 euca2ools [DEBUG]:Path: /services/Cloud/ 2015-01-15 23:33:22,890 euca2ools [DEBUG]:Data: 2015-01-15 23:33:22,890 euca2ools [DEBUG]:Headers: {} 2015-01-15 23:33:22,891 euca2ools [DEBUG]:Host: 192.168.0.4 2015-01-15 23:33:22,891 euca2ools [DEBUG]:Port: 8773 2015-01-15 23:33:22,891 euca2ools [DEBUG]:Params: {'Action': 'CreateSecurityGroup', 'GroupName': 'shtest2', 'Version': '2009-11-30', 'GroupDescription': 'foo'} 2015-01-15 23:33:22,891 euca2ools [DEBUG]:establishing HTTP connection: kwargs={'port': 8773, 'timeout': 70} 2015-01-15 23:33:22,891 euca2ools [DEBUG]:Token: None 2015-01-15 23:33:22,891 euca2ools [DEBUG]:CanonicalRequest: POST /services/Cloud/ host:192.168.0.4:8773 x-amz-date:20150115T233322Z host;x-amz-date a364b884b3e72160b8850f80c1b5b559011b38313da026d04dd60b745c0fd135 2015-01-15 23:33:22,892 euca2ools [DEBUG]:StringToSign: AWS4-HMAC-SHA256 20150115T233322Z 20150115/168/192/aws4_request 6b751f8ad4ca1935c68a35f196928c1626d8fdb0ae4bf901050ab757649c9a9a 2015-01-15 23:33:22,892 euca2ools [DEBUG]:Signature: 3a0c741e4a8abf22d83ea8d909bb90e11627905338e42fdfaa51296fa5f2dd31 2015-01-15 23:33:22,892 euca2ools [DEBUG]:Final headers: {'Content-Length': '84', 'User-Agent': 'Boto/2.35.1 Python/2.7.5 Linux/3.17.7-200.fc20.x86_64', 'Host': '192.168.0.4:8773', 'X-Amz-Date': '20150115T233322Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Authorization': 'AWS4-HMAC-SHA256 Credential=8943303abb374011be595587aa1fc986/20150115/168/192/aws4_request,SignedHeaders=host;x-amz-date,Signature=3a0c741e4a8abf22d83ea8d909bb90e11627905338e42fdfaa51296fa5f2dd31'} 2015-01-15 23:33:22,994 euca2ools [DEBUG]:Response headers: [('date', 'Thu, 15 Jan 2015 23:33:22 GMT'), ('content-length', '407'), ('content-type', 'text/xml')] 2015-01-15 23:33:22,995 euca2ools [DEBUG]:<CreateSecurityGroupResponse xmlns="http://ec2.amazonaws.com/doc/2009-11-30/";> <requestId>req-cf3700c4-51fa-4d1f-bc6b-5ee45321a215</requestId> <securityGroupSet> <item> <ipPermissions/> <groupName>shtest2</groupName> <groupDescription>foo</groupDescription> <ownerId>e463cc1c9e814680bd96f1f87ca61e37</ownerId> </item> </securityGroupSet> </CreateSecurityGroupResponse> -bash-4.2$ pip freeze | grep boto boto==2.35.1 So I guess the next step is to figure out what's different between the request made by euca-add-group and the failing test (assuming others can reproduce the successful call via euca-add-group that is..). Steve __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
