Public bug reported:
Test failure during vm destroy as:-
2025-01-31 11:57:58,135 120175 INFO [tempest.common.waiters] State
transition "RESIZE/resize_prep" ==> "RESIZE/resize_migrating" after 8 second
wait
2025-01-31 11:58:01,827 120175 INFO [tempest.common.waiters] State
transition "RESIZE/resize_migrating" ==> "RESIZE/resize_migrated" after
11 second wait
2025-01-31 11:58:05,489 120175 INFO [tempest.common.waiters] State
transition "RESIZE/resize_migrated" ==> "RESIZE/resize_finish" after 15
second wait
2025-01-31 11:58:15,869 120175 INFO [tempest.common.waiters] State
transition "RESIZE/resize_finish" ==> "VERIFY_RESIZE/None" after 25
second wait
Response - Headers: {'date': 'Fri, 31 Jan 2025 11:59:10 GMT', 'server':
'Apache/2.4.58 (Ubuntu)', 'content-length': '1718', 'content-type':
'application/json', 'openstack-api-version': 'compute 2.72',
'x-openstack-nova-api-version': '2.72', 'vary':
'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id':
'req-d15679f5-e820-42ac-b93e-76024fa41940', 'x-compute-request-id':
'req-d15679f5-e820-42ac-b93e-76024fa41940', 'connection': 'close', 'status':
'200', 'content-location':
'https://158.69.67.245/compute/v2.1/servers/007a11e4-32d4-4b1c-9199-56a664fe86ed'}
Body: b'{"server": {"id": "007a11e4-32d4-4b1c-9199-56a664fe86ed",
"name": "tempest-QoSBandwidthAndPacketRateTests-server-1599599327", "status":
"ERROR", "tenant_id": "fab8f5903cec400ab103cf64a8530b57", "user_id":
"35b1e65e2c354a48b77f2f7247070fde", "metadata": {}, "hostId":
"8ac31065c7b89285e06e31bd99c60023f629010753c70c6f324d93ea", "image": {"id":
"acb24912-db2b-4ee1-8a71-9ce02d0abf0f", "links": [{"rel": "bookmark", "href":
"https://158.69.67.245/compute/images/acb24912-db2b-4ee1-8a71-9ce02d0abf0f"}]},
"flavor": {"vcpus": 1, "ram": 192, "disk": 2, "ephemeral": 0, "swap": 0,
"original_name": "m1.nanoextra-391323443", "extra_specs": {}}, "created":
"2025-01-31T11:57:38Z", "updated": "2025-01-31T11:59:09Z", "addresses":
{"tempest-network-smoke--673944778": [{"version": 4, "addr": "10.1.0.22",
"OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:60:17:c9"}]},
"accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href":
"https://158.69.67.245/compute/v2.1/ser
vers/007a11e4-32d4-4b1c-9199-56a664fe86ed"}, {"rel": "bookmark", "href":
"https://158.69.67.245/compute/servers/007a11e4-32d4-4b1c-9199-56a664fe86ed"}],
"OS-DCF:diskConfig": "MANUAL", "fault": {"code": 500, "created":
"2025-01-31T11:59:09Z", "message": "libvirtError"},
"OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": null,
"OS-SRV-USG:launched_at": "2025-01-31T11:58:13.000000",
"OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "default"}],
"OS-EXT-STS:task_state": "resize_reverting", "OS-EXT-STS:vm_state": "error",
"OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": [],
"locked": false, "description": null, "tags": [], "trusted_image_certificates":
null, "server_groups": []}}'
2025-01-31 11:59:10,617 120175 INFO [tempest.common.waiters] State
transition "REVERT_RESIZE/resize_reverting" ==> "ERROR/resize_reverting" after
54 second wait
Traceback (most recent call last):
File "/opt/stack/tempest/tempest/common/waiters.py", line 136, in
wait_for_server_termination
raise lib_exc.DeleteErrorException(details, server_id=server_id)
tempest.lib.exceptions.DeleteErrorException: Resource %(resource_id)s failed to
delete and is in ERROR status
Details: Server 007a11e4-32d4-4b1c-9199-56a664fe86ed failed to delete and is in
ERROR status. Fault: {'code': 500, 'created': '2025-01-31T11:59:09Z',
'message': 'libvirtError'}.
}}}
Traceback (most recent call last):
File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 65, in
wrapper
return f(*func_args, **func_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/stack/tempest/tempest/scenario/test_network_qos_placement.py",
line 851, in test_server_resize_revert
waiters.wait_for_server_status(
File "/opt/stack/tempest/tempest/common/waiters.py", line 80, in
wait_for_server_status
raise exceptions.BuildErrorException(details, server_id=server_id)
tempest.exceptions.BuildErrorException: Server
007a11e4-32d4-4b1c-9199-56a664fe86ed failed to build and is in ERROR status
Details: Fault: {'code': 500, 'created': '2025-01-31T11:59:09Z', 'message':
'libvirtError'}.
Nova compute logs:-
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: WARNING
nova.virt.libvirt.driver [None req-5e44a4a9-7f73-4019-aec0-86bec9372149
tempest-QoSBandwidthAndPacketRateTests-1787381104
tempest-QoSBandwidthAndPacketRateTests-1787381104-project-member] [instance:
007a11e4-32d4-4b1c-9199-56a664fe86ed] Cannot destroy instance, general system
call failure: libvirt.libvirtError: Failed to terminate process 53286 with
SIGKILL: Device or resource busy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [None req-5e44a4a9-7f73-4019-aec0-86bec9372149
tempest-QoSBandwidthAndPacketRateTests-1787381104
tempest-QoSBandwidthAndPacketRateTests-1787381104-project-member] [instance:
007a11e4-32d4-4b1c-9199-56a664fe86ed] Setting instance vm_state to ERROR:
libvirt.libvirtError: Failed to terminate process 53286 with SIGKILL: Device or
resource busy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] Traceback
(most recent call last):
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/opt/stack/nova/nova/compute/manager.py", line 11382, in
_error_out_instance_on_exception
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] yield
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/opt/stack/nova/nova/compute/manager.py", line 5914, in revert_resize
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]
self.driver.destroy(context, instance, network_info,
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 1622, in destroy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]
self._destroy(instance)
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 1567, in _destroy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] with
excutils.save_and_reraise_exception():
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py",
line 227, in __exit__
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]
self.force_reraise()
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py",
line 200, in force_reraise
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] raise
self.value
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 1535, in _destroy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]
guest.poweroff()
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/opt/stack/nova/nova/virt/libvirt/guest.py", line 173, in poweroff
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]
self._domain.destroy()
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line
186, in doit
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]
result = proxy_call(self._autowrap, f, *args, **kwargs)
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line
144, in proxy_call
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] rv =
execute(f, *args, **kwargs)
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line
125, in execute
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] raise
e.with_traceback(tb)
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line 82,
in tworker
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] rv =
meth(*args, **kwargs)
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]
^^^^^^^^^^^^^^^^^^^^^
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/usr/lib/python3/dist-packages/libvirt.py", line 1461, in destroy
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] raise
libvirtError('virDomainDestroy() failed')
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]
libvirt.libvirtError: Failed to terminate process 53286 with SIGKILL: Device or
resource busy
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]
Seen twice, ex builds:-
https://0c63ab9652170854bf26-a09d1a3317eb4b9b558e42ad19c25861.ssl.cf2.rackcdn.com/940474/1/gate/neutron-ovs-tempest-multinode-full/5184c7f/testr_results.html
https://97ad4d1320a89f3380ef-01f5fc3a5734547a13a0f54725d40b32.ssl.cf5.rackcdn.com/936364/4/gate/nova-multi-cell/3003ee6/testr_results.html
Opensearch(creds: openstack/openstack)
https://opensearch.logs.openstack.org/_dashboards/app/data-
explorer/discover?security_tenant=global#?_a=(discover:(columns:!(_source),isDirty:!f,sort:!()),metadata:(indexPattern:'94869730-aea8-11ec-9e6a-83741af3fdcd',view:discover))&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_q=(filters:!(),query:(language:kuery,query:'message:%22libvirt.libvirtError:%20Failed%20to%20terminate%20process%22'))
Found an old bug https://bugs.launchpad.net/nova/+bug/1353939 under which a
workaround was applied to retry 6 times before giving up but that was
targetting libvirt < 4.7.0 with
https://review.opendev.org/c/openstack/nova/+/639091. And then it was removed
with support removal of old libvirt versions
https://review.opendev.org/c/openstack/nova/+/746984.
This bug is to track what exactly triggering this behavior now and if
some workaround flag can be added in nova to have some attempts on
temporary failure during destroy as before.
Also some discussion on nova channel
https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-
nova.2025-02-06.log.html#t2025-02-06T15:04:13
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2097586
Title:
Random test failures with libvirt.libvirtError: Failed to terminate
process 53286 with SIGKILL: Device or resource busy
Status in OpenStack Compute (nova):
New
Bug description:
Test failure during vm destroy as:-
2025-01-31 11:57:58,135 120175 INFO [tempest.common.waiters] State
transition "RESIZE/resize_prep" ==> "RESIZE/resize_migrating" after 8 second
wait
2025-01-31 11:58:01,827 120175 INFO [tempest.common.waiters] State
transition "RESIZE/resize_migrating" ==> "RESIZE/resize_migrated"
after 11 second wait
2025-01-31 11:58:05,489 120175 INFO [tempest.common.waiters] State
transition "RESIZE/resize_migrated" ==> "RESIZE/resize_finish" after
15 second wait
2025-01-31 11:58:15,869 120175 INFO [tempest.common.waiters] State
transition "RESIZE/resize_finish" ==> "VERIFY_RESIZE/None" after 25
second wait
Response - Headers: {'date': 'Fri, 31 Jan 2025 11:59:10 GMT', 'server':
'Apache/2.4.58 (Ubuntu)', 'content-length': '1718', 'content-type':
'application/json', 'openstack-api-version': 'compute 2.72',
'x-openstack-nova-api-version': '2.72', 'vary':
'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id':
'req-d15679f5-e820-42ac-b93e-76024fa41940', 'x-compute-request-id':
'req-d15679f5-e820-42ac-b93e-76024fa41940', 'connection': 'close', 'status':
'200', 'content-location':
'https://158.69.67.245/compute/v2.1/servers/007a11e4-32d4-4b1c-9199-56a664fe86ed'}
Body: b'{"server": {"id": "007a11e4-32d4-4b1c-9199-56a664fe86ed",
"name": "tempest-QoSBandwidthAndPacketRateTests-server-1599599327", "status":
"ERROR", "tenant_id": "fab8f5903cec400ab103cf64a8530b57", "user_id":
"35b1e65e2c354a48b77f2f7247070fde", "metadata": {}, "hostId":
"8ac31065c7b89285e06e31bd99c60023f629010753c70c6f324d93ea", "image": {"id":
"acb24912-db2b-4ee1-8a71-9ce02d0abf0f", "links": [{"rel": "bookmark", "href":
"https://158.69.67.245/compute/images/acb24912-db2b-4ee1-8a71-9ce02d0abf0f"}]},
"flavor": {"vcpus": 1, "ram": 192, "disk": 2, "ephemeral": 0, "swap": 0,
"original_name": "m1.nanoextra-391323443", "extra_specs": {}}, "created":
"2025-01-31T11:57:38Z", "updated": "2025-01-31T11:59:09Z", "addresses":
{"tempest-network-smoke--673944778": [{"version": 4, "addr": "10.1.0.22",
"OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:60:17:c9"}]},
"accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href":
"https://158.69.67.245/compute/v2.1/s
ervers/007a11e4-32d4-4b1c-9199-56a664fe86ed"}, {"rel": "bookmark", "href":
"https://158.69.67.245/compute/servers/007a11e4-32d4-4b1c-9199-56a664fe86ed"}],
"OS-DCF:diskConfig": "MANUAL", "fault": {"code": 500, "created":
"2025-01-31T11:59:09Z", "message": "libvirtError"},
"OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": null,
"OS-SRV-USG:launched_at": "2025-01-31T11:58:13.000000",
"OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "default"}],
"OS-EXT-STS:task_state": "resize_reverting", "OS-EXT-STS:vm_state": "error",
"OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": [],
"locked": false, "description": null, "tags": [], "trusted_image_certificates":
null, "server_groups": []}}'
2025-01-31 11:59:10,617 120175 INFO [tempest.common.waiters] State
transition "REVERT_RESIZE/resize_reverting" ==> "ERROR/resize_reverting" after
54 second wait
Traceback (most recent call last):
File "/opt/stack/tempest/tempest/common/waiters.py", line 136, in
wait_for_server_termination
raise lib_exc.DeleteErrorException(details, server_id=server_id)
tempest.lib.exceptions.DeleteErrorException: Resource %(resource_id)s failed
to delete and is in ERROR status
Details: Server 007a11e4-32d4-4b1c-9199-56a664fe86ed failed to delete and is
in ERROR status. Fault: {'code': 500, 'created': '2025-01-31T11:59:09Z',
'message': 'libvirtError'}.
}}}
Traceback (most recent call last):
File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 65, in
wrapper
return f(*func_args, **func_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/stack/tempest/tempest/scenario/test_network_qos_placement.py",
line 851, in test_server_resize_revert
waiters.wait_for_server_status(
File "/opt/stack/tempest/tempest/common/waiters.py", line 80, in
wait_for_server_status
raise exceptions.BuildErrorException(details, server_id=server_id)
tempest.exceptions.BuildErrorException: Server
007a11e4-32d4-4b1c-9199-56a664fe86ed failed to build and is in ERROR status
Details: Fault: {'code': 500, 'created': '2025-01-31T11:59:09Z', 'message':
'libvirtError'}.
Nova compute logs:-
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: WARNING
nova.virt.libvirt.driver [None req-5e44a4a9-7f73-4019-aec0-86bec9372149
tempest-QoSBandwidthAndPacketRateTests-1787381104
tempest-QoSBandwidthAndPacketRateTests-1787381104-project-member] [instance:
007a11e4-32d4-4b1c-9199-56a664fe86ed] Cannot destroy instance, general system
call failure: libvirt.libvirtError: Failed to terminate process 53286 with
SIGKILL: Device or resource busy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [None req-5e44a4a9-7f73-4019-aec0-86bec9372149
tempest-QoSBandwidthAndPacketRateTests-1787381104
tempest-QoSBandwidthAndPacketRateTests-1787381104-project-member] [instance:
007a11e4-32d4-4b1c-9199-56a664fe86ed] Setting instance vm_state to ERROR:
libvirt.libvirtError: Failed to terminate process 53286 with SIGKILL: Device or
resource busy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] Traceback
(most recent call last):
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/opt/stack/nova/nova/compute/manager.py", line 11382, in
_error_out_instance_on_exception
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] yield
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/opt/stack/nova/nova/compute/manager.py", line 5914, in revert_resize
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]
self.driver.destroy(context, instance, network_info,
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 1622, in destroy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]
self._destroy(instance)
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 1567, in _destroy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] with
excutils.save_and_reraise_exception():
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py",
line 227, in __exit__
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]
self.force_reraise()
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py",
line 200, in force_reraise
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] raise
self.value
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 1535, in _destroy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]
guest.poweroff()
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/opt/stack/nova/nova/virt/libvirt/guest.py", line 173, in poweroff
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]
self._domain.destroy()
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line
186, in doit
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]
result = proxy_call(self._autowrap, f, *args, **kwargs)
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line
144, in proxy_call
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] rv =
execute(f, *args, **kwargs)
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line
125, in execute
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] raise
e.with_traceback(tb)
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line 82,
in tworker
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] rv =
meth(*args, **kwargs)
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]
^^^^^^^^^^^^^^^^^^^^^
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File
"/usr/lib/python3/dist-packages/libvirt.py", line 1461, in destroy
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] raise
libvirtError('virDomainDestroy() failed')
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]
libvirt.libvirtError: Failed to terminate process 53286 with SIGKILL: Device or
resource busy
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR
nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]
Seen twice, ex builds:-
https://0c63ab9652170854bf26-a09d1a3317eb4b9b558e42ad19c25861.ssl.cf2.rackcdn.com/940474/1/gate/neutron-ovs-tempest-multinode-full/5184c7f/testr_results.html
https://97ad4d1320a89f3380ef-01f5fc3a5734547a13a0f54725d40b32.ssl.cf5.rackcdn.com/936364/4/gate/nova-multi-cell/3003ee6/testr_results.html
Opensearch(creds: openstack/openstack)
https://opensearch.logs.openstack.org/_dashboards/app/data-
explorer/discover?security_tenant=global#?_a=(discover:(columns:!(_source),isDirty:!f,sort:!()),metadata:(indexPattern:'94869730-aea8-11ec-9e6a-83741af3fdcd',view:discover))&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_q=(filters:!(),query:(language:kuery,query:'message:%22libvirt.libvirtError:%20Failed%20to%20terminate%20process%22'))
Found an old bug https://bugs.launchpad.net/nova/+bug/1353939 under which a
workaround was applied to retry 6 times before giving up but that was
targetting libvirt < 4.7.0 with
https://review.opendev.org/c/openstack/nova/+/639091. And then it was removed
with support removal of old libvirt versions
https://review.opendev.org/c/openstack/nova/+/746984.
This bug is to track what exactly triggering this behavior now and if
some workaround flag can be added in nova to have some attempts on
temporary failure during destroy as before.
Also some discussion on nova channel
https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-
nova.2025-02-06.log.html#t2025-02-06T15:04:13
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2097586/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
