Public bug reported: Description =========== Nova-conductor put instance in error if the exception is not know in _build_live_migrate_task during the live-migration. [1] The exception come from _call_livem_checks_on_host and we can see raise exception.MigrationPreCheckError if we facing to messaging.MessagingTimeout. [2] The function check_can_live_migrate_destination do a check also on souce host with check_can_live_migrate_source [3] and this check can also return MessagingTimeout and this one is not catch properly because it's a remote "Remote error: MessagingTimeout" due to dest host try to contact source host and this source host not reply.
[1] https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L523 [2] https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L363 [3] https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L8546 Steps to reproduce ================== # Deploy devstack multinode # Create an instance openstack server create --image a3cf22ec-3e24-404c-83cd-47a95874e164 --flavor m1.small --network dd824883-17b8-4ecd-881d-6b3cbd758bb6 test-check_can_live_migrate_source-on-dest-node # In the dest node add in check_can_live_migrate_source (nova/compute/rpcapi.py) a sleep to have time to stop nova-compute on the source node % git diff nova/compute/rpcapi.py diff --git a/nova/compute/rpcapi.py b/nova/compute/rpcapi.py index b58004c6e6..00ca0bd109 100644 --- a/nova/compute/rpcapi.py +++ b/nova/compute/rpcapi.py @@ -608,6 +608,8 @@ class ComputeAPI(object): client = self.router.client(ctxt) source = _compute_host(None, instance) cctxt = client.prepare(server=source, version=version) + import time + time.sleep(600) return cctxt.call(ctxt, 'check_can_live_migrate_source', instance=instance, dest_check_data=dest_check_data) # Stop nova-compute and waiting # After few minutes instance go to error state # We can found in nova super conductor this log error: Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR nova.conductor.manager Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: WARNING nova.scheduler.utils [None req-8795982e-8a37-4d87-9695-806039a3d89b admin admin] [instance: 4969fe65-11ec-495f-a036-386f83d404b0] Setting instance to ERROR state.: oslo_messaging.rpc.client.RemoteError: Remote error: MessagingTimeout Timed out waiting for a reply to message ID c685b202642c469eac1dc06ac187a49c Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server [None req-8795982e-8a37-4d87-9695-806039a3d89b admin admin] Exception during message handling: nova.exception.MigrationError: Migration error: Remote error: MessagingTimeout Timed out waiting for a reply to message ID c685b202642c469eac1dc06ac187a49c Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/manager.py", line 505, in _live_migrate Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc. server task.execute() Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/base.py", line 25, in wrap Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server self.force_reraise() Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server raise self.value Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/base.py", line 23, in wrap Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server return original(self) Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/base.py", line 40, in execute Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server return self._execute() Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/live_migrate.py", line 100, in _execute Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server self.destination, dest_node, self.limits = self._find_destination() Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/live_migrate.py", line 550, in _find_destination Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server self._call_livem_checks_on_host(host, provider_mapping) Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/live_migrate.py", line 360, in _call_livem_checks_on_host Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server check_can_live_migrate_destination(self.context, self.instance, Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/rpcapi.py", line 604, in check_can_live_migrate_destination Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server return cctxt.call(ctxt, 'check_can_live_migrate_destination', **kwargs) Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in call Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server result = self.transport._send( Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server return self._driver.send(target, ctxt, message, Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server return self._send(target, ctxt, message, wait_for_reply, timeout, Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server raise result Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server oslo_messaging.rpc.client.RemoteError: Remote error: MessagingTimeout Timed out waiting for a reply to message ID c685b202642c469eac1dc06ac187a49c Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server ['Traceback (most recent call last):\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 441, in get\n return self. _queues[msg_id].get(block=True, timeout=timeout)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/queue.py", line 322, in get\n return waiter.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/queue.py", line 141, in w ait\n return get_hub().switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch\n return self.greenlet.switch()\n', '_queue.Empty\n', '\nDuring handling of the above exception, another exception occurred:\n\n ', 'Traceback (most recent call last):\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/ oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_ args)\n', ' File "/opt/stack/nova/nova/exception_wrapper.py", line 65, in wrapped\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/exception_wrapper.py", line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/opt/st ack/nova/nova/compute/utils.py", line 1439, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 213, in decorated_function\n with excutils.save_and_reraise_exception():\n', ' File "/opt /stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/op t/stack/nova/nova/compute/manager.py", line 203, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 8546, in check_can_live_migrate_destination\n migrate_data = self.compute_rpcapi.chec k_can_live_migrate_source(\n', ' File "/opt/stack/nova/nova/compute/rpcapi.py", line 613, in check_can_live_migrate_source\n return cctxt.call(ctxt, \'check_can_live_migrate_source\',\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/cl ient.py", line 190, in call\n result = self.transport._send(\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send\n return self._driver.send(target, ctxt, message,\n', ' File "/opt/stack/data/venv/lib/pytho n3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send\n return self._send(target, ctxt, message, wait_for_reply, timeout,\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 678, in _send\n result = self._waiter.wait(msg_id, timeout,\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 567, in wait\n message = self.waiters.get(msg_id, timeout=timeout)\n', ' File "/opt/stack/data/venv/lib/python3. 10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 443, in get\n raise oslo_messaging.MessagingTimeout(\n', 'oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID c685b202642c469eac1dc06ac187a49c\n']. Expected result =============== The live-migration task are in pre check so we need to have instance in Active state with migration task in error Actual result ============= The instance in ERROR state and the migration is in error Environment =========== I have reproduce the issue on devstack multinode with default config ** Affects: nova Importance: Undecided Assignee: Pierre Libeau (pierre-libeau) Status: New ** Changed in: nova Assignee: (unassigned) => Pierre Libeau (pierre-libeau) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2044235 Title: nova-condutor put instance in error during live-migration due to remote error MessagingTimeout Status in OpenStack Compute (nova): New Bug description: Description =========== Nova-conductor put instance in error if the exception is not know in _build_live_migrate_task during the live-migration. [1] The exception come from _call_livem_checks_on_host and we can see raise exception.MigrationPreCheckError if we facing to messaging.MessagingTimeout. [2] The function check_can_live_migrate_destination do a check also on souce host with check_can_live_migrate_source [3] and this check can also return MessagingTimeout and this one is not catch properly because it's a remote "Remote error: MessagingTimeout" due to dest host try to contact source host and this source host not reply. [1] https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L523 [2] https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L363 [3] https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L8546 Steps to reproduce ================== # Deploy devstack multinode # Create an instance openstack server create --image a3cf22ec-3e24-404c-83cd-47a95874e164 --flavor m1.small --network dd824883-17b8-4ecd-881d-6b3cbd758bb6 test-check_can_live_migrate_source-on-dest-node # In the dest node add in check_can_live_migrate_source (nova/compute/rpcapi.py) a sleep to have time to stop nova-compute on the source node % git diff nova/compute/rpcapi.py diff --git a/nova/compute/rpcapi.py b/nova/compute/rpcapi.py index b58004c6e6..00ca0bd109 100644 --- a/nova/compute/rpcapi.py +++ b/nova/compute/rpcapi.py @@ -608,6 +608,8 @@ class ComputeAPI(object): client = self.router.client(ctxt) source = _compute_host(None, instance) cctxt = client.prepare(server=source, version=version) + import time + time.sleep(600) return cctxt.call(ctxt, 'check_can_live_migrate_source', instance=instance, dest_check_data=dest_check_data) # Stop nova-compute and waiting # After few minutes instance go to error state # We can found in nova super conductor this log error: Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR nova.conductor.manager Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: WARNING nova.scheduler.utils [None req-8795982e-8a37-4d87-9695-806039a3d89b admin admin] [instance: 4969fe65-11ec-495f-a036-386f83d404b0] Setting instance to ERROR state.: oslo_messaging.rpc.client.RemoteError: Remote error: MessagingTimeout Timed out waiting for a reply to message ID c685b202642c469eac1dc06ac187a49c Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server [None req-8795982e-8a37-4d87-9695-806039a3d89b admin admin] Exception during messa ge handling: nova.exception.MigrationError: Migration error: Remote error: MessagingTimeout Timed out waiting for a reply to message ID c685b202642c469eac1dc06ac187a49c Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/manager.py", line 505, in _live_migrate Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rp c.server task.execute() Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/base.py", line 25, in wrap Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server self.force_reraise() Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server raise self.value Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/base.py", line 23, in wrap Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server return original(self) Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/base.py", line 40, in execute Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server return self._execute() Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/live_migrate.py", line 100, in _execute Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server self.destination, dest_node, self.limits = self._find_destination() Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/live_migrate.py", line 550, in _find_destination Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server self._call_livem_checks_on_host(host, provider_mapping) Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/conductor/tasks/live_migrate.py", line 360, in _call_livem_checks_on_host Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server check_can_live_migrate_destination(self.context, self.instance, Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/rpcapi.py", line 604, in check_can_live_migrate_destination Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server return cctxt.call(ctxt, 'check_can_live_migrate_destination', **kwargs) Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in call Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server result = self.transport._send( Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server return self._driver.send(target, ctxt, message, Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server return self._send(target, ctxt, message, wait_for_reply, timeout, Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server raise result Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server oslo_messaging.rpc.client.RemoteError: Remote error: MessagingTimeout Timed out waiting for a reply to message ID c685b202642c469eac1dc06ac187a49c Nov 21 16:40:58 devstack2-multi-node-1-cp nova-conductor[143072]: ERROR oslo_messaging.rpc.server ['Traceback (most recent call last):\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 441, in get\n return self. _queues[msg_id].get(block=True, timeout=timeout)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/queue.py", line 322, in get\n return waiter.wait()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/queue.py", line 141, in w ait\n return get_hub().switch()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch\n return self.greenlet.switch()\n', '_queue.Empty\n', '\nDuring handling of the above exception, another exception occurred:\n\n ', 'Traceback (most recent call last):\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/ oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_ args)\n', ' File "/opt/stack/nova/nova/exception_wrapper.py", line 65, in wrapped\n with excutils.save_and_reraise_exception():\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/opt/stack/nova/nova/exception_wrapper.py", line 63, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/opt/st ack/nova/nova/compute/utils.py", line 1439, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 213, in decorated_function\n with excutils.save_and_reraise_exception():\n', ' File "/opt /stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/op t/stack/nova/nova/compute/manager.py", line 203, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 8546, in check_can_live_migrate_destination\n migrate_data = self.compute_rpcapi.chec k_can_live_migrate_source(\n', ' File "/opt/stack/nova/nova/compute/rpcapi.py", line 613, in check_can_live_migrate_source\n return cctxt.call(ctxt, \'check_can_live_migrate_source\',\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/cl ient.py", line 190, in call\n result = self.transport._send(\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send\n return self._driver.send(target, ctxt, message,\n', ' File "/opt/stack/data/venv/lib/pytho n3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send\n return self._send(target, ctxt, message, wait_for_reply, timeout,\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 678, in _send\n result = self._waiter.wait(msg_id, timeout,\n', ' File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 567, in wait\n message = self.waiters.get(msg_id, timeout=timeout)\n', ' File "/opt/stack/data/venv/lib/python3. 10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 443, in get\n raise oslo_messaging.MessagingTimeout(\n', 'oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID c685b202642c469eac1dc06ac187a49c\n']. Expected result =============== The live-migration task are in pre check so we need to have instance in Active state with migration task in error Actual result ============= The instance in ERROR state and the migration is in error Environment =========== I have reproduce the issue on devstack multinode with default config To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2044235/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
