Reviewed: https://review.opendev.org/c/openstack/nova/+/901815
Committed:
https://opendev.org/openstack/nova/commit/1f8ce3cc70ceaf8edd25b1ae2646e525cf2b3973
Submitter: "Zuul (22348)"
Branch: master
commit 1f8ce3cc70ceaf8edd25b1ae2646e525cf2b3973
Author: Rajesh Tailor <ratai...@redhat.com>
Date: Fri Nov 24 13:25:25 2023 +0530
Add retry on attachment_update
In nova-live-migration job, evacuation failures are causing
POST_FAILURE.
As per discussion on bug, it looks like 500 is coming from cinder
which is cause of this failure.
Similar to attachment_delete method, this change adds a retry
mechanism in cinder API calls attachment_update method.
Closes-Bug: #1970642
Change-Id: I1da3c8481f7e7a1e8776cf03f5c4cf117b7fabaa
** Changed in: nova
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1970642
Title:
evacuation failure causes POST_FAILURE in the nova-live-migration job
Status in OpenStack Compute (nova):
Fix Released
Bug description:
The nova-live-migration job has a post task to test evacuation. This
test fails intermittently causing the job to fail with POST_FAILURE
status.
Evacuation fails due to libvirt connection error.
job log:
2022-04-27 14:24:41.092806 | controller | + /opt/stack/nova/roles/run-
evacuate-hook/files/test_evacuate.sh:evacuate_and_wait_for_active:34 :
echo 'Timed out waiting for server evacuate-bfv-test to go to ACTIVE
status'
2022-04-27 14:24:41.092962 | controller | Timed out waiting for server
evacuate-bfv-test to go to ACTIVE status
nova-compute log:
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server [None
req-a012194e-8201-43fa-ae56-a9194a5fd5ad demo admin] Exception during
message handling: nova.exception.HypervisorUnavailable: Connection to
the hypervisor is broken on host
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server Traceback (most recent
call last):
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/virt/libvirt/host.py", line 586, in
get_connection
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server conn =
self._get_connection()
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/virt/libvirt/host.py", line 566, in
_get_connection
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server
self._queue_conn_event_handler(
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
227, in __exit__
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server
self.force_reraise()
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
200, in force_reraise
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server raise self.value
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/virt/libvirt/host.py", line 558, in
_get_connection
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server self._wrapped_conn
= self._get_new_connection()
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/virt/libvirt/host.py", line 502, in
_get_new_connection
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server wrapped_conn =
self._connect(self._uri, self._read_only)
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/virt/libvirt/host.py", line 353, in _connect
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server return
self._libvirt_proxy.openAuth(uri, auth, flags)
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 193,
in doit
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server result =
proxy_call(self._autowrap, f, *args, **kwargs)
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 151,
in proxy_call
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server rv = execute(f,
*args, **kwargs)
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 132,
in execute
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server six.reraise(c, e,
tb)
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/six.py", line 719, in reraise
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server raise value
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 86,
in tworker
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server rv = meth(*args,
**kwargs)
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/libvirt.py", line 104, in openAuth
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server if ret is
None:raise libvirtError('virConnectOpenAuth() failed')
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server libvirt.libvirtError:
Failed to connect socket to '/var/run/libvirt/libvirt-sock':
Connection refused
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server During handling of the
above exception, another exception occurred:
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server Traceback (most recent
call last):
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py",
line 165, in _process_incoming
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-
packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-
packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py",
line 241, in inner
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server return func(*args,
**kwargs)
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/exception_wrapper.py", line 71, in wrapped
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server
_emit_versioned_exception_notification(
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
227, in __exit__
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server
self.force_reraise()
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
200, in force_reraise
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server raise self.value
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/exception_wrapper.py", line 63, in wrapped
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server return f(self,
context, *args, **kw)
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 184, in
decorated_function
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server
LOG.warning("Failed to revert task state for instance. "
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
227, in __exit__
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server
self.force_reraise()
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
200, in force_reraise
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server raise self.value
Apr 27 14:13:41.050639 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 155, in
decorated_function
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/utils.py", line 1439, in
decorated_function
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 212, in
decorated_function
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server
compute_utils.add_instance_fault_from_exc(context,
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
227, in __exit__
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server
self.force_reraise()
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line
200, in force_reraise
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server raise self.value
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 201, in
decorated_function
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 3548, in
rebuild_instance
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server
self._do_rebuild_instance_with_claim(
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/manager.py", line 3625, in
_do_rebuild_instance_with_claim
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server claim_context =
rebuild_claim(
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/usr/local/lib/python3.8/dist-
packages/oslo_concurrency/lockutils.py", line 391, in inner
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server return f(*args,
**kwargs)
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/resource_tracker.py", line 204, in
rebuild_claim
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server return
self._move_claim(
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/resource_tracker.py", line 280, in
_move_claim
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server if
self.disabled(nodename):
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/compute/resource_tracker.py", line 666, in
disabled
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server not
self.driver.node_is_available(nodename))
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/virt/driver.py", line 1564, in node_is_available
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server if nodename in
self.get_available_nodes():
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 11037, in
get_available_nodes
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server return
[self._host.get_hostname()]
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/virt/libvirt/host.py", line 1051, in
get_hostname
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server hostname =
self.get_connection().getHostname()
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server File
"/opt/stack/nova/nova/virt/libvirt/host.py", line 596, in
get_connection
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server raise
exception.HypervisorUnavailable()
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server
nova.exception.HypervisorUnavailable: Connection to the hypervisor is
broken on host
Apr 27 14:13:41.061728 ubuntu-focal-iweb-mtl01-0029462367 nova-
compute[97475]: ERROR oslo_messaging.rpc.server
Example failure:
https://zuul.opendev.org/t/openstack/build/6c792e4ee72542cbaf386710e57a4b01/log/controller/logs/screen-
n-cpu.txt#12329
I see multiple matching failures from the last two weeks:
https://paste.opendev.org/show/bbB6cuBQBGSUPyAxPXva/
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1970642/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
