Reviewed: https://review.openstack.org/416810
Committed:
https://git.openstack.org/cgit/openstack/nova/commit/?id=180e6340a595ec047c59365465f36fed7a669ec3
Submitter: Jenkins
Branch: master
commit 180e6340a595ec047c59365465f36fed7a669ec3
Author: Matt Riedemann <mrie...@us.ibm.com>
Date: Wed Jan 4 21:11:10 2017 -0500
Handle Unauthorized exception in report client's safe_connect()
If nova is misconfigured for placement auth, as seen in:
I6351c1b2ca7ea4448e13eb87455bff4058df4fa7
Keystone returns an Unauthorized 401 exception which the
scheduler report client's safe_connect() method wasn't handling
and disabling future attempts to talk to the placement service.
This patch adds that check to the report client. Note also that
we already check for this in the "nova-status upgrade check"
command.
Change-Id: Id52de64528383de70fc5a8951dde764ba34e369d
Closes-Bug: #1654107
** Changed in: nova
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1654107
Title:
resource tracking fails with Unauthorized from placement API (keystone
v3)
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Compute (nova) newton series:
Confirmed
Bug description:
Seen here:
http://logs.openstack.org/65/416765/1/check/gate-tempest-dsvm-neutron-
identity-v3-only-full-ubuntu-xenial-
nv/faf1363/logs/screen-n-cpu.txt.gz#_2017-01-04_23_15_06_674
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager
[req-832f822c-02b3-453b-b80b-61ad7f06f401 - -] Error updating resources for
node ubuntu-xenial-osic-cloud1-disk-6483179.
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager Traceback (most
recent call last):
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File
"/opt/stack/new/nova/nova/compute/manager.py", line 6537, in
update_available_resource_for_node
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager
rt.update_available_resource(context, nodename)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File
"/opt/stack/new/nova/nova/compute/resource_tracker.py", line 540, in
update_available_resource
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager
self._update_available_resource(context, resources)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File
"/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line
271, in inner
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager return f(*args,
**kwargs)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File
"/opt/stack/new/nova/nova/compute/resource_tracker.py", line 564, in
_update_available_resource
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager
self._init_compute_node(context, resources)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File
"/opt/stack/new/nova/nova/compute/resource_tracker.py", line 451, in
_init_compute_node
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager
self.scheduler_client.update_resource_stats(self.compute_node)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File
"/opt/stack/new/nova/nova/scheduler/client/__init__.py", line 60, in
update_resource_stats
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager
self.reportclient.update_resource_stats(compute_node)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File
"/opt/stack/new/nova/nova/scheduler/client/__init__.py", line 37, in
__run_method
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager return
getattr(self.instance, __name)(*args, **kwargs)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File
"/opt/stack/new/nova/nova/scheduler/client/report.py", line 462, in
update_resource_stats
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager
compute_node.hypervisor_hostname)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File
"/opt/stack/new/nova/nova/scheduler/client/report.py", line 282, in
_ensure_resource_provider
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager rp =
self._get_resource_provider(uuid)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File
"/opt/stack/new/nova/nova/scheduler/client/report.py", line 47, in wrapper
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager return f(self,
*a, **k)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File
"/opt/stack/new/nova/nova/scheduler/client/report.py", line 195, in
_get_resource_provider
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager resp =
self.get("/resource_providers/%s" % uuid)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File
"/opt/stack/new/nova/nova/scheduler/client/report.py", line 160, in get
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager
endpoint_filter=self.ks_filter, raise_exc=False)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 710, in
get
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager return
self.request(url, 'GET', **kwargs)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File
"/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 101, in
inner
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager return
wrapped(*args, **kwargs)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 467, in
request
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager auth_headers =
self.get_auth_headers(auth)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 770, in
get_auth_headers
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager return
auth.get_headers(self, **kwargs)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 90, in
get_headers
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager token =
self.get_token(session)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line
90, in get_token
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager return
self.get_access(session).auth_token
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line
136, in get_access
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager self.auth_ref =
self.get_auth_ref(session)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py",
line 198, in get_auth_ref
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager return
self._plugin.get_auth_ref(session, **kwargs)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py",
line 167, in get_auth_ref
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager
authenticated=False, log=False, **rkwargs)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 718, in
post
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager return
self.request(url, 'POST', **kwargs)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File
"/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 101, in
inner
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager return
wrapped(*args, **kwargs)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager File
"/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 607, in
request
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager raise
exceptions.from_response(resp, method, url)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager Unauthorized: The
request you have made requires authentication. (HTTP 401) (Request-ID:
req-38ce6df3-1dca-4b66-a680-d2691bb4955f)
2017-01-04 23:15:06.674 7096 ERROR nova.compute.manager
The scheduler report client's safe_connect method isn't handle
Unauthorized exceptions, which are happening in the keystone v3 jobs
now with the placement API, at least until this devstack patch is
merged:
https://review.openstack.org/#/c/416599/
We handle Unauthorized in the new nova-status upgrade check command
but not the safe_connect() decorator, which we need to do.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1654107/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
