Public bug reported:
I am running wallaby release in production and today when my rabbitMQ
went down so i have rebuild whole rabbitMQ cluster but after that all my
compute nodes nova-compute service started throwing errors like
following. Not sure what is wrong here even i re-build rabbitMQ. any
idea how to fix?
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager
[req-092f8eb6-1bcc-4545-979a-a8e0eb28da65 - - - - -] Error updating resources
for node ostack-phx-comp-sriov-1-1.v1v0x.net.:
oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to
message ID c7bba4980b9b49a0a74e946d35cf06a1
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager Traceback (most
recent call last):
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 433, in get
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager return
self._queues[msg_id].get(block=True, timeout=timeout)
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/queue.py",
line 322, in get
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager return
waiter.wait()
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/queue.py",
line 141, in wait
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager return
get_hub().switch()
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/hub.py",
line 313, in switch
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager return
self.greenlet.switch()
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager _queue.Empty
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager During handling of
the above exception, another exception occurred:
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager Traceback (most
recent call last):
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/compute/manager.py",
line 9934, in _update_available_resource_for_node
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager
self.rt.update_available_resource(context, nodename,
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/compute/resource_tracker.py",
line 895, in update_available_resource
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager
self._update_available_resource(context, resources, startup=startup)
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_concurrency/lockutils.py",
line 360, in inner
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager return f(*args,
**kwargs)
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/compute/resource_tracker.py",
line 975, in _update_available_resource
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager
self._update(context, cn, startup=startup)
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/compute/resource_tracker.py",
line 1230, in _update
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager
self.pci_tracker.save(context)
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/pci/manager.py",
line 95, in save
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager dev.save()
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_versionedobjects/base.py",
line 209, in wrapper
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager updates, result =
self.indirection_api.object_action(
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/conductor/rpcapi.py",
line 247, in object_action
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager return
cctxt.call(context, 'object_action', objinst=objinst,
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/rpc/client.py",
line 175, in call
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager
self.transport._send(self.target, msg_ctxt, msg,
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/transport.py",
line 123, in _send
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager return
self._driver.send(target, ctxt, message,
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 680, in send
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager return
self._send(target, ctxt, message, wait_for_reply, timeout,
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 669, in _send
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager result =
self._waiter.wait(msg_id, timeout,
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 559, in wait
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager message =
self.waiters.get(msg_id, timeout=timeout)
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 435, in get
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager raise
oslo_messaging.MessagingTimeout(
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager
oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to
message ID c7bba4980b9b49a0a74e946d35cf06a1
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager
Feb 10 02:44:47 ostack-phx-comp-sriov-1-1.v1v0x.net nova-compute[827533]:
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task
[req-b111abf5-19d8-4424-9934-f368ad55216b - - - - -] Error during
ComputeManager._run_image_cache_manager_pass:
oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to
message ID 64a5bc6b8a4a4fd49458f742a6ed7a30
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task Traceback (most
recent call last):
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 433, in get
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task return
self._queues[msg_id].get(block=True, timeout=timeout)
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/queue.py",
line 322, in get
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task return
waiter.wait()
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/queue.py",
line 141, in wait
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task return
get_hub().switch()
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/hub.py",
line 313, in switch
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task return
self.greenlet.switch()
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task _queue.Empty
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task During handling
of the above exception, another exception occurred:
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task Traceback (most
recent call last):
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_service/periodic_task.py",
line 216, in run_periodic_tasks
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task task(self,
context)
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/compute/manager.py",
line 10481, in _run_image_cache_manager_pass
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task
self.driver.manage_image_cache(context, filtered_instances)
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py",
line 10649, in manage_image_cache
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task
self.image_cache_manager.update(context, all_instances)
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/virt/libvirt/imagecache.py",
line 381, in update
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task running =
self._list_running_instances(context, all_instances)
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/virt/imagecache.py",
line 56, in _list_running_instances
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task
instance_bdms = objects.BlockDeviceMappingList.bdms_by_instance_uuid(
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/objects/block_device.py",
line 370, in bdms_by_instance_uuid
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task bdms =
cls.get_by_instance_uuids(context, instance_uuids)
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_versionedobjects/base.py",
line 175, in wrapper
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task result =
cls.indirection_api.object_class_action_versions(
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/conductor/rpcapi.py",
line 240, in object_class_action_versions
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task return
cctxt.call(context, 'object_class_action_versions',
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/rpc/client.py",
line 175, in call
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task
self.transport._send(self.target, msg_ctxt, msg,
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/transport.py",
line 123, in _send
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task return
self._driver.send(target, ctxt, message,
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 680, in send
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task return
self._send(target, ctxt, message, wait_for_reply, timeout,
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 669, in _send
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task result =
self._waiter.wait(msg_id, timeout,
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 559, in wait
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task message =
self.waiters.get(msg_id, timeout=timeout)
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 435, in get
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task raise
oslo_messaging.MessagingTimeout(
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task
oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to
message ID 64a5bc6b8a4a4fd49458f742a6ed7a30
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1960465
Title:
nova-compute Timed out waiting for a reply to message ID
Status in OpenStack Compute (nova):
New
Bug description:
I am running wallaby release in production and today when my rabbitMQ
went down so i have rebuild whole rabbitMQ cluster but after that all
my compute nodes nova-compute service started throwing errors like
following. Not sure what is wrong here even i re-build rabbitMQ. any
idea how to fix?
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager
[req-092f8eb6-1bcc-4545-979a-a8e0eb28da65 - - - - -] Error updating resources
for node ostack-phx-comp-sriov-1-1.v1v0x.net.:
oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to
message ID c7bba4980b9b49a0a74e946d35cf06a1
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager Traceback (most
recent call last):
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 433, in get
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager return
self._queues[msg_id].get(block=True, timeout=timeout)
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/queue.py",
line 322, in get
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager return
waiter.wait()
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/queue.py",
line 141, in wait
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager return
get_hub().switch()
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/hub.py",
line 313, in switch
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager return
self.greenlet.switch()
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager _queue.Empty
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager During handling of
the above exception, another exception occurred:
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager Traceback (most
recent call last):
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/compute/manager.py",
line 9934, in _update_available_resource_for_node
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager
self.rt.update_available_resource(context, nodename,
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/compute/resource_tracker.py",
line 895, in update_available_resource
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager
self._update_available_resource(context, resources, startup=startup)
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_concurrency/lockutils.py",
line 360, in inner
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager return f(*args,
**kwargs)
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/compute/resource_tracker.py",
line 975, in _update_available_resource
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager
self._update(context, cn, startup=startup)
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/compute/resource_tracker.py",
line 1230, in _update
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager
self.pci_tracker.save(context)
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/pci/manager.py",
line 95, in save
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager dev.save()
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_versionedobjects/base.py",
line 209, in wrapper
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager updates, result =
self.indirection_api.object_action(
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/conductor/rpcapi.py",
line 247, in object_action
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager return
cctxt.call(context, 'object_action', objinst=objinst,
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/rpc/client.py",
line 175, in call
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager
self.transport._send(self.target, msg_ctxt, msg,
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/transport.py",
line 123, in _send
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager return
self._driver.send(target, ctxt, message,
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 680, in send
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager return
self._send(target, ctxt, message, wait_for_reply, timeout,
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 669, in _send
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager result =
self._waiter.wait(msg_id, timeout,
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 559, in wait
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager message =
self.waiters.get(msg_id, timeout=timeout)
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 435, in get
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager raise
oslo_messaging.MessagingTimeout(
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager
oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to
message ID c7bba4980b9b49a0a74e946d35cf06a1
2022-02-10 02:42:36.067 827533 ERROR nova.compute.manager
Feb 10 02:44:47 ostack-phx-comp-sriov-1-1.v1v0x.net nova-compute[827533]:
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task
[req-b111abf5-19d8-4424-9934-f368ad55216b - - - - -] Error during
ComputeManager._run_image_cache_manager_pass:
oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to
message ID 64a5bc6b8a4a4fd49458f742a6ed7a30
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task Traceback (most
recent call last):
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 433, in get
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task return
self._queues[msg_id].get(block=True, timeout=timeout)
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/queue.py",
line 322, in get
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task return
waiter.wait()
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/queue.py",
line 141, in wait
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task return
get_hub().switch()
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/hub.py",
line 313, in switch
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task return
self.greenlet.switch()
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task _queue.Empty
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task During handling
of the above exception, another exception occurred:
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task Traceback (most
recent call last):
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_service/periodic_task.py",
line 216, in run_periodic_tasks
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task task(self,
context)
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/compute/manager.py",
line 10481, in _run_image_cache_manager_pass
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task
self.driver.manage_image_cache(context, filtered_instances)
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py",
line 10649, in manage_image_cache
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task
self.image_cache_manager.update(context, all_instances)
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/virt/libvirt/imagecache.py",
line 381, in update
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task running =
self._list_running_instances(context, all_instances)
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/virt/imagecache.py",
line 56, in _list_running_instances
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task
instance_bdms = objects.BlockDeviceMappingList.bdms_by_instance_uuid(
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/objects/block_device.py",
line 370, in bdms_by_instance_uuid
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task bdms =
cls.get_by_instance_uuids(context, instance_uuids)
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_versionedobjects/base.py",
line 175, in wrapper
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task result =
cls.indirection_api.object_class_action_versions(
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/conductor/rpcapi.py",
line 240, in object_class_action_versions
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task return
cctxt.call(context, 'object_class_action_versions',
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/rpc/client.py",
line 175, in call
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task
self.transport._send(self.target, msg_ctxt, msg,
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/transport.py",
line 123, in _send
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task return
self._driver.send(target, ctxt, message,
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 680, in send
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task return
self._send(target, ctxt, message, wait_for_reply, timeout,
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 669, in _send
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task result =
self._waiter.wait(msg_id, timeout,
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 559, in wait
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task message =
self.waiters.get(msg_id, timeout=timeout)
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task File
"/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 435, in get
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task raise
oslo_messaging.MessagingTimeout(
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task
oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to
message ID 64a5bc6b8a4a4fd49458f742a6ed7a30
2022-02-10 02:44:47.651 827533 ERROR oslo_service.periodic_task
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1960465/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
